<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Chris,<div>Check all cable ends for errors. In an extreme case, we sent an Engineer to Germany</div><div>to solve a "performance problem" that turned out to be a bad SFP+ transceiver. The bill</div><div>was, shall we say, higher than the average service call :-)</div><div> -- richard</div><div><br><div><div>On Aug 4, 2012, at 4:41 PM, Chris Nagele wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>I've been testing an environment before we send it over to the data<br>center. The setup is an OmniOS install (Supermicro X9, 24 drives,<br>DDRdrive for ZIL) used as an NFS file server and multiple Debian<br>machines connected via directly attached 10GB SFP+ twin-axial cables.<br>We went with the 10Gb in order to get lower latency to compensate for<br>the high TCP overhead of NFS.<br><br>For some reason, I can't get ping times below 0.2ms when using OmniOS.<br>If I install Debian on the same machines I get consistent 0.08ms<br>between the machines. In addition, iperf shows a maximum of 6.6Gb/s on<br>OmniOS, and about 9.89Gb/s on Debian.<br><br>We are using the x520-DA2, which is well supported in Illumos. I also<br>tried on Nexenta and OpenIndiana with the same result. I was thinking<br>of trying different cards (Myricom maybe), different cabling (CX4?),<br>or even Infiniband since it is known for low latency.<br><br>Any recommendations? I am wondering if I just need to adjust some<br>tunables or maybe update a driver.<br><br>Thanks,<br>Chris<br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>http://lists.omniti.com/mailman/listinfo/omnios-discuss<br></div></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">ZFS Performance and Training</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><a href="mailto:Richard.Elling@RichardElling.com">Richard.Elling@RichardElling.com</a><br>+1-760-896-4422<br><br><br></div></span></div></span></div></span></span> </div> <br></div></body></html>