<div dir="ltr"><div><div><div><div><div><div>Hello Garrett,<br><br></div>No, no 802.3ad going on in this config.<br><br></div>Here is a basic schematic:<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQVkVqcE5OQUJyUUU/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQVkVqcE5OQUJyUUU/view?usp=sharing</a><br><br></div>Here is the Nexenta MPIO iSCSI Setup Document that I used as a guide:<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQbjEyUTBjN2tTNWM/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQbjEyUTBjN2tTNWM/view?usp=sharing</a><br><br></div><div>Note that I am using an MTU of 3000 on both the 10G and 1G NICs. The switch is set to allow 9148-byte frames, and I'm not seeing any errors/buffer overruns on the switch.<br></div><div><br></div>Here is a screenshot of a packet capture from a read operation on the guest OS (from it's local drive, which is actually a VMDK file on the storage server). In this example, only a single 1G ESXi kernel interface (vmk1) is bound to the software iSCSI initiator.<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQa2NYdXhpZkpkbU0/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQa2NYdXhpZkpkbU0/view?usp=sharing</a><br><br></div>Note that there's a nice, well-behaved window sizing process taking place. The ESXi decreases the scaled window by 11 or 12 for each ACK, then bumps it back up to 512.<br><br></div>Here is a similar screenshot of a single-interface write operation:<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQbU1RZHRnakxDSFU/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQbU1RZHRnakxDSFU/view?usp=sharing</a><br><div><div><div><div><br></div><div>There are no pauses or gaps in the transmission rate in the single-interface transfers.<br><br><br></div><div>In the next screenshots, I have enabled an additional 1G interface on the ESXi host, and bound it to the iSCSI initiator. The new interface is bound to a separate physical port, uses a different VLAN on the switch, and talks to a different 10G port on the storage server. <br><br>First, let's look at a write operation on the guest OS, which happily pumps data at near-line-rate to the storage server. <br><br>Here is a sequence number trace diagram. Note how the transfer has a nice, smooth increment rate over the entire transfer.<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQWHNIa0drWnNxMmM/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQWHNIa0drWnNxMmM/view?usp=sharing</a><br><br></div><div>Here are screenshots from packet captures on both 1G interfaces:<br><br></div><div><a href="https://drive.google.com/file/d/0BwyUMjibonYQRWhyVVQ4djNaU3c/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQRWhyVVQ4djNaU3c/view?usp=sharing</a><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQaTVjTEtTRloyR2c/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQaTVjTEtTRloyR2c/view?usp=sharing</a><br><br></div><div>Note how we again see nice, smooth window adjustment, and no gaps in transmission.<br><br><br></div><div>But now, let's look at the problematic two-interface Read operation.<br></div><div>First, the sequence graph:<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQTzdFVWdQMWZ6LUU/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQTzdFVWdQMWZ6LUU/view?usp=sharing</a><br><br></div><div>As you can see, there are gaps and jumps in the transmission throughout the transfer.<br></div><div>It is very illustrative to look at captures of the gaps, which are occurring on both interfaces:<br><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQc0VISXN6eVFwQzg/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQc0VISXN6eVFwQzg/view?usp=sharing</a><br><a href="https://drive.google.com/file/d/0BwyUMjibonYQVFREUHp3TGFiUU0/view?usp=sharing">https://drive.google.com/file/d/0BwyUMjibonYQVFREUHp3TGFiUU0/view?usp=sharing</a><br><br></div><div>As you can see, there are ~.4 second pauses in transmission from the storage server, which kills the transfer rate.<br></div><div>It's clear that the ESXi box ACKs the prior iSCSI operation to completion, then makes a new LUN request, which the storage server immediately replies to. The ESXi ACKs the response packet from the storage server, then waits...and waits....and waits... until eventually the storage server starts transmitting again.<br><br></div><div>Because the pause happens while the ESXi client is waiting for a packet from the storage server, that tells me that the gaps are not an artifact of traffic being switched between both active interfaces, but are actually indicative of short hangs occurring on the server.<br></div><div><br></div><div>Having a pause or two in transmission is no big deal, but in my case, it is happening constantly, and dropping my overall read transfer rate down to 20-60MB/s, which is slower than the single interface transfer rate (~90-100MB/s).<br><br></div><div>Decreasing the MTU makes the pauses shorter, increasing them makes the pauses longer.<br><br></div><div>Another interesting thing is that if I set the multipath io interval to 3 operations instead of 1, I get better throughput. In other words, the less frequently I swap IP addresses on my iSCSI requests from the ESXi unit, the fewer pauses I see.<br><br></div><div>Basically, COMSTAR seems to choke each time an iSCSI request from a new IP arrives.<br></div><div><br></div><div>Because the single interface transfer is near line rate, that tells me that the storage system (mpt_sas, zfs, etc) is working fine. It's only when multiple paths are attempted that iSCSI falls on its face during reads.<br><br></div><div>All of these captures were taken without a cache device being attached to the storage zpool, so this isn't looking like some kind of ZFS ARC problem. As mentioned previously, local transfers to/from the zpool are showing ~300-500 MB/s rates over long transfers (10G+).<br><br></div><div>-Warren V<br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 1, 2015 at 9:11 PM, Garrett D'Amore <span dir="ltr"><<a href="mailto:garrett@damore.org" target="_blank">garrett@damore.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">I’m not sure I’ve followed properly. You have *two* interfaces. You are not trying to provision these in an aggr are you? As far as I’m aware, VMware does not support 802.3ad link aggregations. (Its possible that you can make it work with ESXi if you give the entire NIC to the guest — but I’m skeptical.) The problem is that if you try to use link aggregation, some packets (up to half!) will be lost. TCP and other protocols fare poorly in this situation.<div><br></div><div>Its possible I’ve totally misunderstood what you’re trying to do, in which case I apologize.</div><div><br></div><div>The idle thing is a red-herring — the cpu is waiting for work to do, probably because packets haven’t arrived (or where dropped by the hypervisor!) I wouldn’t read too much into that except that your network stack is in trouble. I’d look a bit more closely at the kstats for tcp — I suspect you’ll see retransmits or out of order values that are unusually high — if so this may help validate my theory above.</div><div><br></div><div><span style="white-space:pre-wrap"> </span>- Garrett<br><div><div><div><br><div><blockquote type="cite"><div><div class="h5"><div>On Mar 1, 2015, at 9:03 PM, W Verb via illumos-developer <<a href="mailto:developer@lists.illumos.org" target="_blank">developer@lists.illumos.org</a>> wrote:</div><br></div></div><div><div><div class="h5"><div dir="ltr"><p>Hello all,</p><div> <br></div><p>Well, I no longer blame the ixgbe driver for the problems I'm seeing.</p><p><br></p><p>I tried Joerg's updated driver, which didn't improve the issue. So I went back to the drawing board and rebuilt the
server from scratch.</p><p>What I noted is that if I have only a single 1-gig
physical interface active on the ESXi host, everything works as expected. As
soon as I enable two interfaces, I start seeing the performance problems I've
described.</p><p>Response pauses from the server that I see in TCPdumps are still leading me to believe the problem is
delay on the server side, so I ran a series of kernel dtraces and produced some
flamegraphs.</p><p><br></p><p><span></span>This was taken
during a read operation with two active 10G interfaces on the server, with a single target being shared by two tpgs- one tpg for each 10G physical port. The host device has two 1G ports enabled, with VLANs separating the active ports into 10G/1G pairs. ESXi is set to multipath using both VLANS with a round-robin IO interval of 1.<br></p><p><a href="https://drive.google.com/file/d/0BwyUMjibonYQd3ZYOGh4d2pteGs/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQd3ZYOGh4d2pteGs/view?usp=sharing</a></p><p><br><span></span></p><p><span></span>This was taken
during a write operation:</p><p><a href="https://drive.google.com/file/d/0BwyUMjibonYQMnBtU1Q2SXM2ams/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQMnBtU1Q2SXM2ams/view?usp=sharing</a></p><p><br></p><p>I then rebooted the server and disabled C-State, ACPI
T-State, and general EIST (Turbo boost) functionality in the CPU.</p><p>I when I attempted to boot my guest VM, the iSCSI
transfer gradually ground to a halt during the boot loading process, and the guest OS never did complete its boot process.<br></p><p>Here is a flamegraph taken while iSCSI is slowly dying:</p><p><a href="https://drive.google.com/file/d/0BwyUMjibonYQM21JeFZPX3dZWTg/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQM21JeFZPX3dZWTg/view?usp=sharing</a></p><div> <br></div><p>I edited out cpu_idle_adaptive from the dtrace output and regenerated the
slowdown graph:</p><p><a href="https://drive.google.com/file/d/0BwyUMjibonYQbTVwV3NvXzlPS1E/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQbTVwV3NvXzlPS1E/view?usp=sharing</a></p><div> <br></div><p>I then edited cpu_idle_adaptive out
of the speedy write operation and regenerated that graph:<br></p><p><a href="https://drive.google.com/file/d/0BwyUMjibonYQeWFYM0pCMDZ1X2s/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQeWFYM0pCMDZ1X2s/view?usp=sharing</a></p><div> <br></div><p>I have zero experience with interpreting flamegraphs, but
the most significant difference I see between the slow
read example and the fast write example is in unix`thread_start --> unix`idle. There's a good chunk of "unix`i86_mwait" in the read example that is not present in the write example at all.<br></p><p>Disabling the l2arc cache device didn't make a
difference, and I had to reenable EIST support on the CPU to get my VMs to boot.<br></p><p>I am seeing a variety of bug reports going back to 2010 regarding excessive mwait operations, with the suggested solutions usually being to set "cpupm enable poll-mode" in power.conf. That change also had no effect on speed.</p><p>-Warren V</p><div> <br></div><div> <br></div><div> <br></div><p>-----Original Message-----</p><p>From: Chris Siebenmann [<a href="mailto:cks@cs.toronto.edu" target="_blank">mailto:cks@cs.toronto.edu</a>]</p><p>Sent: Monday, February 23, 2015 8:30 AM</p><p>To: W Verb</p><p>Cc: <a href="mailto:omnios-discuss@lists.omniti.com" target="_blank">omnios-discuss@lists.omniti.com</a>;
<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a></p><p>Subject: Re: [OmniOS-discuss] The ixgbe driver, Lindsay
Lohan, and the Greek economy</p><div> <br></div><p>> Chris, thanks for your specific details. I'd
appreciate it if you </p><p>> could tell me which copper NIC you tried, as well as
to pass on the </p><p>> iSCSI tuning parameters.</p><div> <br></div><p><span> </span>Our copper NIC
experience is with onboard X540-AT2 ports on SuperMicro hardware (which have
the guaranteed 10-20 msec lock hold) and dual-port 82599EB TN cards (which have
some sort of driver/hardware failure under load that eventually leads to
2-second lock holds). I can't recommend either with the current driver; we had
to revert to 1G networking in order to get stable servers.</p><div> <br></div><p><span> </span>The iSCSI
parameter modifications we do, across both initiators and targets, are:</p><div> <br></div><p><span> </span>initialr2t<span> </span>no</p><p><span> </span>firstburstlength<span> </span>128k</p><p><span> </span>maxrecvdataseglen<span> </span>128k<span> </span>[only
on Linux backends]</p><p><span> </span>maxxmitdataseglen<span> </span>128k<span> </span>[only
on Linux backends]</p><div> <br></div><p>The OmniOS initiator doesn't need tuning for more than
the first two parameters; on the Linux backends we tune up all four. My
extended thoughts on these tuning parameters and why we touch them can be found</p><p>here:</p><div> <br></div><p><span> </span><a href="http://utcc.utoronto.ca/~cks/space/blog/tech/UnderstandingiSCSIProtocol" target="_blank">http://utcc.utoronto.ca/~cks/space/blog/tech/UnderstandingiSCSIProtocol</a></p><p><span> </span><a href="http://utcc.utoronto.ca/~cks/space/blog/tech/LikelyISCSITuning" target="_blank">http://utcc.utoronto.ca/~cks/space/blog/tech/LikelyISCSITuning</a></p><div> <br></div><p>The short version is that these parameters probably only
make a small difference but their overall goal is to do 128KB ZFS reads and
writes in single iSCSI operations (although they will be fragmented at the TCP</p><p>layer) and to do iSCSI writes without a back-and-forth
delay between initiator and target (that's 'initialr2t no').</p><div> <br></div><p><span> </span>I think basically
everyone should use InitialR2T set to no and in fact that it should be the
software default. These days only unusually limited iSCSI targets should need
it to be otherwise and they can change their setting for it (initiator and
target must both agree to it being 'yes', so either can veto it).</p><div> <br></div><p><span> </span>- cks</p><div> <br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 23, 2015 at 8:21 AM, Joerg Goltermann <span dir="ltr"><<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br><br>
I think your problem is caused by your link properties or your<br>
switch settings. In general the standard ixgbe seems to perform<br>
well.<br><br>
I had trouble after changing the default flow control settings to "bi"<br>
and this was my motivation to update the ixgbe driver a long time ago.<br>
After I have updated our systems to ixgbe 2.5.8 I never had any<br>
problems ....<br><br>
Make sure your switch has support for jumbo frames and you use<br>
the same mtu on all ports, otherwise the smallest will be used.<br><br>
What switch do you use? I can tell you nice horror stories about<br>
different vendors....<br><br>
- Joerg<span><br><br>
On 23.02.2015 10:31, W Verb wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
Thank you Joerg,<br><br>
I've downloaded the package and will try it tomorrow.<br><br>
The only thing I can add at this point is that upon review of my<br>
testing, I may have performed my "pkg -u" between the initial quad-gig<br>
performance test and installing the 10G NIC. So this may be a new<br>
problem introduced in the latest updates.<br><br>
Those of you who are running 10G and have not upgraded to the latest<br>
kernel, etc, might want to do some additional testing before running the<br>
update.<br><br>
-Warren V<br><br>
On Mon, Feb 23, 2015 at 1:15 AM, Joerg Goltermann <<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a><br></span><span>
<mailto:<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>>> wrote:<br><br>
Hi,<br><br>
I remember there was a problem with the flow control settings in the<br>
ixgbe<br>
driver, so I updated it a long time ago for our internal servers to<br>
2.5.8.<br>
Last weekend I integrated the latest changes from the FreeBSD driver<br>
to bring<br>
the illumos ixgbe to 2.5.25 but I had no time to test it, so it's<br>
completely<br>
untested!<br><br><br>
If you would like to give the latest driver a try you can fetch the<br>
kernel modules from<br></span>
<a href="https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.__<u></u>php/s/Fb4so9RsNnXA7r9</a><span><br>
<<a href="https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.<u></u>php/s/Fb4so9RsNnXA7r9</a>><br><br>
Clone your boot environment, place the modules in the new environment<br>
and update the boot-archive of the new BE.<br><br>
- Joerg<br><br><br><br><br><br>
On 23.02.2015 02:54, W Verb wrote:<br><br>
By the way, to those of you who have working setups: please send me<br>
your pool/volume settings, interface linkprops, and any kernel<br>
tuning<br>
parameters you may have set.<br><br>
Thanks,<br>
Warren V<br><br>
On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip<br></span><div><div>
<<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a> <mailto:<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a>>> wrote:<br><br>
I can't say I totally agree with your performance<br>
assessment. I run Intel<br>
X520 in all my OmniOS boxes.<br><br>
Here is a capture of nfssvrtop I made while running many<br>
storage vMotions<br>
between two OmniOS boxes hosting NFS datastores. This is a<br>
10 host VMware<br>
cluster. Both OmniOS boxes are dual 10G connected with<br>
copper twin-ax to<br>
the in rack Nexus 5010.<br><br>
VMware does 100% sync writes, I use ZeusRAM SSDs for log<br>
devices.<br><br>
-Chip<br><br>
2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB,<br>
swrite: 15985 KB,<br>
awrite: 1875455 KB<br><br>
Ver Client NFSOPS Reads SWrites AWrites<br>
Commits Rd_bw<br>
SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align%<br><br>
4 10.28.17.105 0 0 0 0<br>
0 0<br>
0 0 0 0 0 0 0<br><br>
4 10.28.17.215 0 0 0 0<br>
0 0<br>
0 0 0 0 0 0 0<br><br>
4 10.28.17.213 0 0 0 0<br>
0 0<br>
0 0 0 0 0 0 0<br><br>
4 10.28.16.151 0 0 0 0<br>
0 0<br>
0 0 0 0 0 0 0<br><br>
4 all 1 0 0 0<br>
0 0<br>
0 0 0 0 0 0 0<br><br>
3 10.28.16.175 3 0 3 0<br>
0 1<br>
11 0 4806 48 0 0 85<br><br>
3 10.28.16.183 6 0 6 0<br>
0 3<br>
162 0 549 124 0 0 73<br><br>
3 10.28.16.180 11 0 10 0<br>
0 3<br>
27 0 776 89 0 0 67<br><br>
3 10.28.16.176 28 2 26 0<br>
0 10<br>
405 0 2572 198 0 0 100<br><br>
3 10.28.16.178 4606 4602 4 0<br>
0 294534<br>
3 0 723 49 0 0 99<br><br>
3 10.28.16.179 4905 4879 26 0<br>
0 312208<br>
311 0 735 271 0 0 99<br><br>
3 10.28.16.181 5515 5502 13 0<br>
0 352107<br>
77 0 89 87 0 0 99<br><br>
3 10.28.16.184 12095 12059 10 0<br>
0 763014<br>
39 0 249 147 0 0 99<br><br>
3 10.28.58.1 15401 6040 116 6354<br>
53 191605<br>
474 202346 192 96 144 83 99<br><br></div></div>
3 all <a href="tel:42574%2033086" value="+14257433086" target="_blank">42574 33086</a> <tel:42574%20%20%2033086> 217<span><br>
6354 53 1913488<br>
1582 202300 348 138 153 105 99<br><br><br><br><br><br>
On Fri, Feb 20, 2015 at 11:46 PM, W Verb <<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a><br></span><div><div>
<mailto:<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>>> wrote:<br><br><br>
Hello All,<br><br>
Thank you for your replies.<br>
I tried a few things, and found the following:<br><br>
1: Disabling hyperthreading support in the BIOS drops<br>
performance overall<br>
by a factor of 4.<br>
2: Disabling VT support also seems to have some effect,<br>
although it<br>
appears to be minor. But this has the amusing side<br>
effect of fixing the<br>
hangs I've been experiencing with fast reboot. Probably<br>
by disabling kvm.<br>
3: The performance tests are a bit tricky to quantify<br>
because of caching<br>
effects. In fact, I'm not entirely sure what is<br>
happening here. It's just<br>
best to describe what I'm seeing:<br><br>
The commands I'm using to test are<br>
dd if=/dev/zero of=./test.dd bs=2M count=5000<br>
dd of=/dev/null if=./test.dd bs=2M count=5000<br>
The host vm is running Centos 6.6, and has the latest<br>
vmtools installed.<br>
There is a host cache on an SSD local to the host that<br>
is also in place.<br>
Disabling the host cache didn't immediately have an<br>
effect as far as I could<br>
see.<br><br>
The host MTU set to 3000 on all iSCSI interfaces for all<br>
tests.<br><br>
Test 1: Right after reboot, with an ixgbe MTU of 9000,<br>
the write test<br>
yields an average speed over three tests of 137MB/s. The<br>
read test yields an<br>
average over three tests of 5MB/s.<br><br>
Test 2: After setting "ifconfig ixgbe0 mtu 3000", the<br>
write tests yield<br>
140MB/s, and the read tests yield 53MB/s. It's important<br>
to note here that<br>
if I cut the read test short at only 2-3GB, I get<br>
results upwards of<br>
350MB/s, which I assume is local cache-related distortion.<br><br>
Test 3: MTU of 1500. Read tests are up to 156 MB/s.<br>
Write tests yield<br>
about 142MB/s.<br>
Test 4: MTU of 1000: Read test at 182MB/s.<br>
Test 5: MTU of 900: Read test at 130 MB/s.<br>
Test 6: MTU of 1000: Read test at 160MB/s. Write tests<br>
are now<br>
consistently at about 300MB/s.<br>
Test 7: MTU of 1200: Read test at 124MB/s.<br>
Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s.<br><br>
A few final notes:<br>
L1ARC grabs about 10GB of RAM during the tests, so<br>
there's definitely some<br>
read caching going on.<br>
The write operations are easier to observe with iostat,<br>
and I'm seeing io<br>
rates that closely correlate with the network write speeds.<br><br><br>
Chris, thanks for your specific details. I'd appreciate<br>
it if you could<br>
tell me which copper NIC you tried, as well as to pass<br>
on the iSCSI tuning<br>
parameters.<br><br>
I've ordered an Intel EXPX9502AFXSR, which uses the<br>
82598 chip instead of<br>
the 82599 in the X520. If I get similar results with my<br>
fiber transcievers,<br>
I'll see if I can get a hold of copper ones.<br><br>
But I should mention that I did indeed look at PHY/MAC<br>
error rates, and<br>
they are nil.<br><br>
-Warren V<br><br>
On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann<br></div></div>
<<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a> <mailto:<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a>>><div><div><br>
wrote:<br><br><br>
After installation and configuration, I observed<br>
all kinds of bad<br>
behavior<br>
in the network traffic between the hosts and the<br>
server. All of this<br>
bad<br>
behavior is traced to the ixgbe driver on the<br>
storage server. Without<br>
going<br>
into the full troubleshooting process, here are<br>
my takeaways:<br><br>
[...]<br><br>
For what it's worth, we managed to achieve much<br>
better line rates on<br>
copper 10G ixgbe hardware of various descriptions<br>
between OmniOS<br>
and CentOS 7 (I don't think we ever tested OmniOS to<br>
OmniOS). I don't<br>
believe OmniOS could do TCP at full line rate but I<br>
think we managed 700+<br>
Mbytes/sec on both transmit and receive and we got<br>
basically disk-limited<br>
speeds with iSCSI (across multiple disks on<br>
multi-disk mirrored pools,<br>
OmniOS iSCSI initiator, Linux iSCSI targets).<br><br>
I don't believe we did any specific kernel tuning<br>
(and in fact some of<br>
our attempts to fiddle ixgbe driver parameters blew<br>
up in our face).<br>
We did tune iSCSI connection parameters to increase<br>
various buffer<br>
sizes so that ZFS could do even large single<br>
operations in single iSCSI<br>
transactions. (More details available if people are<br>
interested.)<br><br>
10: At the wire level, the speed problems are<br>
clearly due to pauses in<br>
response time by omnios. At 9000 byte frame<br>
sizes, I see a good number<br>
of duplicate ACKs and fast retransmits during<br>
read operations (when<br>
omnios is transmitting). But below about a<br>
4100-byte MTU on omnios<br>
(which seems to correlate to 4096-byte iSCSI<br>
block transfers), the<br>
transmission errors fade away and we only see<br>
the transmission pause<br>
problem.<br><br><br>
This is what really attracted my attention. In<br>
our OmniOS setup, our<br>
specific Intel hardware had ixgbe driver issues that<br>
could cause<br>
activity stalls during once-a-second link heartbeat<br>
checks. This<br>
obviously had an effect at the TCP and iSCSI layers.<br>
My initial message<br>
to illumos-developer sparked a potentially<br>
interesting discussion:<br><br><br></div></div>
<a href="http://www.listbox.com/member/__archive/182179/2014/10/sort/__time_rev/page/16/entry/6:405/__20141003125035:6357079A-4B1D-__11E4-A39C-D534381BA44D/" target="_blank">http://www.listbox.com/member/<u></u>__archive/182179/2014/10/sort/<u></u>__time_rev/page/16/entry/6:<u></u>405/__20141003125035:6357079A-<u></u>4B1D-__11E4-A39C-D534381BA44D/</a><span><br>
<<a href="http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/" target="_blank">http://www.listbox.com/<u></u>member/archive/182179/2014/10/<u></u>sort/time_rev/page/16/entry/6:<u></u>405/20141003125035:6357079A-<u></u>4B1D-11E4-A39C-D534381BA44D/</a>><br><br>
If you think this is a possibility in your setup,<br>
I've put the DTrace<br>
script I used to hunt for this up on the web:<br><br></span>
<a href="http://www.cs.toronto.edu/~__cks/src/omnios-ixgbe/ixgbe___delay.d" target="_blank">http://www.cs.toronto.edu/~__<u></u>cks/src/omnios-ixgbe/ixgbe___<u></u>delay.d</a><span><br>
<<a href="http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d" target="_blank">http://www.cs.toronto.edu/~<u></u>cks/src/omnios-ixgbe/ixgbe_<u></u>delay.d</a>><br><br>
This isn't the only potential source of driver<br>
stalls by any means, it's<br>
just the one I found. You may also want to look at<br>
lockstat in general,<br>
as information it reported is what led us to look<br>
specifically at the<br>
ixgbe code here.<br><br>
(If you suspect kernel/driver issues, lockstat<br>
combined with kernel<br>
source is a really excellent resource.)<br><br>
- cks<br><br><br><br><br></span>
______________________________<u></u>___________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti" target="_blank">OmniOS-discuss@lists.omniti</a>.__<u></u>com<br>
<mailto:<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.<u></u>omniti.com</a>><br>
<a href="http://lists.omniti.com/__mailman/listinfo/omnios-__discuss" target="_blank">http://lists.omniti.com/__<u></u>mailman/listinfo/omnios-__<u></u>discuss</a><br>
<<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/<u></u>mailman/listinfo/omnios-<u></u>discuss</a>><br><br><br>
______________________________<u></u>___________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti" target="_blank">OmniOS-discuss@lists.omniti</a>.__<u></u>com<br>
<mailto:<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.<u></u>omniti.com</a>><br>
<a href="http://lists.omniti.com/__mailman/listinfo/omnios-__discuss" target="_blank">http://lists.omniti.com/__<u></u>mailman/listinfo/omnios-__<u></u>discuss</a><span><br>
<<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/<u></u>mailman/listinfo/omnios-<u></u>discuss</a>><br><br><br>
--<br>
OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg<br></span>
Tel: <a href="tel:%2B49%20911%2039905-0" value="+49911399050" target="_blank">+49 911 39905-0</a> <tel:%2B49%20911%2039905-0> - Fax: +49 911<br>
39905-55 <tel:%2B49%20911%2039905-55> - <a href="http://www.osn.de/" target="_blank">http://www.osn.de</a><span><br>
HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann<br><br><br></span></blockquote><div><div><br>
-- <br>
OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg<br>
Tel: <a href="tel:%2B49%20911%2039905-0" value="+49911399050" target="_blank">+49 911 39905-0</a> - Fax: <a href="tel:%2B49%20911%2039905-55" value="+499113990555" target="_blank">+49 911 39905-55</a> - <a href="http://www.osn.de/" target="_blank">http://www.osn.de</a><br>
HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann<br></div></div></blockquote></div><br></div></div></div><div bgcolor="#ffffff" style="width:auto;margin:0;padding:5px;background-color:#fff;clear:both;border-top:1px solid #ccc"><table style="background-color:#fff" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td><font style="font-family:helvetica,sans-serif" color="#333333" size="1">
<strong>illumos-developer</strong> | <a href="https://www.listbox.com/member/archive/182179/=now" style="text-decoration:none;color:#669933;border-bottom:1px solid #444444" title="Go to archives for illumos-developer" target="_blank">Archives</a>
<a border="0" href="https://www.listbox.com/member/archive/rss/182179/21239177-3604570e" style="text-decoration:none;color:#669933" title="RSS feed for illumos-developer" target="_blank"><img src="http://postlink.www.listbox.com/1851406/833487e62783d55fe81f119fb93ef644/21239177/8c349c6f.jpg?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2ZlZWQtaWNvbi0xMHgxMC5qcGc" border="0"></a>
| <a href="https://www.listbox.com/member/?member_id=21239177&id_secret=21239177-2d0c9337" style="text-decoration:none;color:#669933;border-bottom:1px solid #444444" title="" target="_blank">Modify</a>
Your Subscription</font></td><td align="right" valign="top"><a href="http://www.listbox.com/" style="border-bottom:none" target="_blank">
<img src="http://postlink.www.listbox.com/1851407/3379085af0f1cf7fc3708f04b4471ae2/21239177/8c349c6f.png?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2xpc3Rib3gtbG9nby1zbWFsbC5wbmc" title="Powered by Listbox" border="0"></a></td></tr></tbody></table></div></div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div>