[OmniOS-discuss] Slow NFS speeds at rsize > 128k

[Dan McDonald]

> On Jan 7, 2015, at 5:28 AM, Stephan Budach <stephan.budach at JVM.DE> wrote:
> 
> Hello everyone,
> 
> I am sharing my zfs via NFS to a couple of OVM nodes. I noticed really bad NFS read performance, when rsize goes beyond 128k, whereas the performance is just fine at 32k. The issue is, that the ovs-agent, which is performing the actual mount, doesn't accept or pass any NFS mount options to the NFS server. To give some numbers, a rsize of 1mb results in a read throughput of approx. 2Mb/s, whereas a rsize of 32k gives me 110Mb/s. Mounting a NFS export from a OEL 6u4 box has no issues with this, as the read speeds from this export are 108+MB/s regardles of the rsize of the NFS mount.
> 
> The OmniOS box is currently connected to a 10GbE port at our core 6509, but the NFS client is connected through a 1GbE port only. MTU is at 1500 and can currently not be upped.
> Anyone having a tip, why a rsize of 64k+ will result in such a performance drop?

Assuming you're running over TCP, perhaps you need to increase the receive window?

	ndd -set /dev/tcp tcp_xmit_hiwat 1048576

Dan