<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I have four file servers all of which have at least two (some three) aggregates made of two 10GB lines. Three of the four servers are getting pretty good throughput as reported by iperf:</div><div><br></div><div>(hvfs1 has iperf -s, other hosts iperf -c)</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 5] 0.00-10.00 sec 8.04 GBytes 6.91 Gbits/sec sender</div><div>[ 5] 0.00-10.00 sec 8.04 GBytes 6.91 Gbits/sec receiver</div><div><br></div><div>but one, fs2, is getting roughly half that speed when it is running iperf -c:</div><div><br></div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 4] 0.00-10.00 sec 3.81 GBytes 3.27 Gbits/sec sender</div><div>[ 4] 0.00-10.00 sec 3.81 GBytes 3.27 Gbits/sec receiver</div><div>(with occasional bumps to 4.05 Gb/s)</div><div><br></div><div>Even stranger, when I run iperf -c on fs2 and iperf -s on hvfs1 I get a better (though still not as good) result bouncing between 4.6 Gb/s and 5.2 Gb/s.</div><div><br></div><div>It doesn't matter if I test via aggr0 or aggr_nas0, I get roughly the same numbers. It also doesn't make any difference if aggr0 is dual-i40e or dual-ixgbe devices.</div><div><br></div><div>Tests were done with hvfs1 acting as the iperf -s server via aggr_front0 (the equivalent to aggr0 on the other hosts) or aggr_nas0 as appropriate. All hosts are directly connected to the same Juniper switch. (aggr info is postpended at the end of this email.)</div><div><br></div><div>One theory I had was that fs2 wasn't actually using the full throughput of the bond so I physically removed one of the two cables. Sure enough, the bandwidth reported by iperf remained around 3.2Gb/s. I tested this with both aggr0 and aggr_nas0 with the same result. There are times where it gets closer to the 6Gb of the other hosts so it is clearly getting more than just one link's worth but not often.</div><div><br></div><div>Unlike the other servers, fs2 is running 151026. The rest are all on 151022. I didn't see any references to aggr in the release notes for 151024 or 151026 but I'm not going to claim exhaustive search.</div><div><br></div><div>Aggregates were created with the device equivalent of</div><div><br></div><div> dladm create-aggr -L active -l i40e0 -l i40e1 aggr0</div><div><br></div><div>My question is, what blatantly obvious thing did I miss in creating the aggregates on fs2?</div><div><br></div><div>thanks,</div><div>nomad</div><div><br></div><div><br></div><div>aggr info:</div><div><br></div><div>fs1 (omnios-r151022-5e982daae6):</div><div>LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS</div><div>aggr0 L4 auto passive short ----- (not used for this test)</div><div>aggr_nas0 L4 auto active short -----</div><div>aggr_front0 L4 auto active short -----</div><div><br></div><div>LINK CLASS MTU STATE BRIDGE OVER</div><div>e1000g1 phys 1500 up -- --</div><div>e1000g0 phys 1500 up -- --</div><div>aggr0 aggr 1500 up -- e1000g0,e1000g1</div><div>ixgbe0 phys 1500 up -- --</div><div>ixgbe1 phys 1500 up -- --</div><div>aggr_nas0 aggr 1500 up -- ixgbe0,ixgbe1</div><div>ixgbe2 phys 1500 up -- --</div><div>ixgbe3 phys 1500 up -- --</div><div>aggr_front0 aggr 1500 up -- ixgbe2,ixgbe3</div><div><br></div><div><br></div><div>fs2 (omnios-r151026-51c7d6fd75):</div><div>LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS</div><div>aggr_nas0 L4 auto active short -----</div><div>aggr_net10 L4 auto active short ----- (not used for this test)</div><div>aggr0 L4 auto active short -----</div><div><br></div><div><div>LINK CLASS MTU STATE BRIDGE OVER</div><div>i40e2 phys 1500 up -- --</div><div>igb0 phys 1500 up -- --</div><div>i40e0 phys 1500 up -- --</div><div>i40e3 phys 1500 up -- --</div><div>igb1 phys 1500 up -- --</div><div>i40e1 phys 1500 up -- --</div><div>ixgbe0 phys 1500 unknown -- --</div><div>ixgbe1 phys 1500 unknown -- --</div><div>aggr_nas0 aggr 1500 up -- i40e2,i40e3</div><div>aggr_net10 aggr 1500 up -- igb0,igb1</div><div>aggr0 aggr 1500 up -- i40e0,i40e1</div><div><br></div><div><br></div><div>hvfs1 (omnios-r151022-5e982daae6):</div><div>LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS</div><div>aggr0 L4 auto active short ----- (not used for this test)</div><div>aggr_front0 L4 auto active short -----</div><div>aggr_nas0 L4 auto active short -----</div><div><br></div><div>LINK CLASS MTU STATE BRIDGE OVER</div><div>ixgbe0 phys 1500 up -- --</div><div>igb0 phys 1500 up -- --</div><div>igb1 phys 1500 up -- --</div><div>igb2 phys 1500 unknown -- --</div><div>oce0 phys 1500 up -- --</div><div>igb3 phys 1500 unknown -- --</div><div>ixgbe1 phys 1500 up -- --</div><div>oce1 phys 1500 up -- --</div><div>aggr0 aggr 1500 up -- igb0,igb1</div><div>aggr_front0 aggr 1500 up -- ixgbe0,ixgbe1</div><div>aggr_nas0 aggr 1500 up -- oce0,oce1</div><div><br></div><div><br></div><div>hvfs2 (omnios-r151022-89f6242508):</div><div>LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS</div><div>aggr_nas0 L4 auto active short -----</div><div>aggr0 L4 auto active short -----</div><div><br></div><div>LINK CLASS MTU STATE BRIDGE OVER</div><div>ixgbe2 phys 1500 up -- --</div><div>ixgbe0 phys 1500 up -- --</div><div>ixgbe3 phys 1500 up -- --</div><div>ixgbe1 phys 1500 up -- --</div><div>aggr_nas0 aggr 1500 up -- ixgbe0,ixgbe1</div><div>aggr0 aggr 1500 up -- ixgbe3,ixgbe2</div></div><div><br></div></div></div></div>