[OmniOS-discuss] Low latency networking

[Chris Nagele]

I've been testing an environment before we send it over to the data
center. The setup is an OmniOS install (Supermicro X9, 24 drives,
DDRdrive for ZIL) used as an NFS file server and multiple Debian
machines connected via directly attached 10GB SFP+ twin-axial cables.
We went with the 10Gb in order to get lower latency to compensate for
the high TCP overhead of NFS.

For some reason, I can't get ping times below 0.2ms when using OmniOS.
If I install Debian on the same machines I get consistent 0.08ms
between the machines. In addition, iperf shows a maximum of 6.6Gb/s on
OmniOS, and about 9.89Gb/s on Debian.

We are using the x520-DA2, which is well supported in Illumos. I also
tried on Nexenta and OpenIndiana with the same result. I was thinking
of trying different cards (Myricom maybe), different cabling (CX4?),
or even Infiniband since it is known for low latency.

Any recommendations? I am wondering if I just need to adjust some
tunables or maybe update a driver.

Thanks,
Chris