<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>On Sep 12, 2013, at 9:38 AM, Ian Kaufman <<a href="mailto:ikaufman@eng.ucsd.edu">ikaufman@eng.ucsd.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hi Muhammad,<br><br>What is the output of "sharectl get nfs"? Have you done any NFS tuning?<br><br>Even though the system says you have 7GB of RAM free, you are running<br>a very slim system for ZFS. Do you have compression enabled?<br>Deduplication? How big is your filesystem? Here are some RAM rules of<br>thumb for ZFS (anfd other advice):<br><br>For decent ZFS performance, you need at least 1GB of RAM for every TB<br>of data, plus at least 1 extra GB for OS<br></blockquote><div><br></div><div>Disagree. We've done many, many studies of this and the RAM size is a </div><div>function of the working set size (WSS), not total space. For example, in the </div><div>copying case here, there is no expectation of reuse, so most of the data will </div><div>efficiently flow through the MRU side of the ARC. The rules of thumb you're </div><div>reaching for are for frequently used data on the MFU side of the ARC. For </div><div>OSes like OmniOS, we see the MFU WSS peaking around 200 MB, as an </div><div>example.</div><br><blockquote type="cite"><br>If deduplication is turned on, you need about 4GB of RAM per TB of<br>data in addition to normal system and ZFS RAM if using 64K blocks on<br>average, if using smaller 8K blocks on average, then you need about<br>32GB of RAM per TB of data.<br><br>RAIDZ2 vdevs should have at least 5 disks and no more than 10. Some<br>believe 6 or 8 disks per vdev for RAIDZ2 is optimal.<br></blockquote><div><br></div><div>It depends entirely on your workload. For itty-bitty files, fewer disks is faster</div><div>but less space efficient. For big files, more disks is faster and more space</div><div>efficient. If you don't know, then the rule is to protect your data.</div><div><br></div><div>NB, a 4-disk raidz2 offers better slightly data dependability, the same available</div><div>space, but slightly lower streaming write performance than 4 disks in a 2-way</div><div>mirror configuration.</div><br><blockquote type="cite"><br>Improve read performance with L2ARC and SSDs - improve write<br>performance with either RAMDISK SSDs or SLC Flash Memory SSDs, and<br>mirror the devices.<br></blockquote><div><br></div><div>This is more likely the issue. NFS for many clients is a synchronous workload</div><div>where performance is improved by using a separate log device (SSD usually).</div><div>The quick test is to "zfs set sync=disabled file/system" and see if the perf</div><div>improves. If it does, then this well-known behavior is no mystery and the </div><div>usual methods to solve it apply. I'll bet a steak dinner that this is the case, since</div><div>the ssh/scp case is an asynchronous write workload and performs much better :-)</div><div> -- richard</div><br><blockquote type="cite"><br>Ian<br><br><br><br><br>On Wed, Sep 11, 2013 at 11:42 PM, Muhammad Yousuf Khan <<a href="mailto:sirtcp@gmail.com">sirtcp@gmail.com</a>> wrote:<br><blockquote type="cite">Just notice and would like to share my new finding.<br><br>as i mentioned earlier i am using RAIDz2 and hosted a VM on NFS. so for<br>testing i "SCP" the whole HD image of VM to some other Linux system so that<br>i can learn if the issue is with Drive speed or the issue is with NFS<br>protocol, (of course not a bug with NFS but i have done something wrong in<br>configuring)<br><br>here is the SCP result. the virtual harddisk image size is 32 GB.<br>i am running this command from OmniOS SSH.<br><br><br>vm-100001-disk-1.qco 50% |************** | 16461 MB 09:42<br>ETA<br><br><br>you can see it copy 16 GB and i confirm that it took 3 to 4 minutes to copy<br>that huge data.<br><br>it seems fine to me. same RAIDz2 but protocol change (SCP on SSH) so i<br>think the issue is with NFS. not with the Hardware speed/specs limitation.<br><br>any idea please.<br><br>Thanks,<br>Myk<br><br><br><br>On Thu, Sep 12, 2013 at 2:40 AM, Muhammad Yousuf Khan <<a href="mailto:sirtcp@gmail.com">sirtcp@gmail.com</a>><br>wrote:<br><blockquote type="cite"><br>first of sorry for starting a new thread for same issue explained earlier<br>because. i<br>mistakenly deleted the email. so my apologizes.<br><br>my omniOS server hardware is as under.<br>dual 3.2GHz Processors, 12 GB RAM. dell 490 workstation<br>4 sata drives set with RAIDz2<br>1 40GB very old IDE drive for OmniOS root system<br><br>i have 3 servers for testing.<br>1. KVM virtualization server (Proxmox)<br>2. OmniOS (NFS)<br>3. Desktop machine<br><br>all three are connected via 1GB LAN broadcast domain.<br><br>i have hosted a test VM (windows 2003 server) on NFS and when i am trying<br>to copy the data (2gb File) from VM via Microsoft sharing to a desktop<br>machine. i doing something very abnormal which i can not understand why<br><br>just FYI : all the virtual machines hosted on KVM (proxmox server) local<br>harddrives are on mdadm linux raid. and working great. and i see no issue in<br>copy and pasting data from or to the local VM machines.<br><br>In attached graphics you will see 2 test of sending 2GB file.<br>in "snap a.1" for few seconds file transfer stuck at 2 percent and then<br>due to unknown reasons bumps to 15%. and it continues for few seconds and<br>when file reaches the end it again goes down to 3 Percent as you can see in<br>"snap a.2"<br><br>in "snap b.1" you can see it starts at 3% max and for 2 minutes it<br>continues and speed only changes btw 1% to 3%. see in "snap b.2"<br><br>some says disable sync for testing and see if performance increases, sorry<br>it didnt help me. same thing happens, whether i disable or enable the sync<br>it makes no difference for me<br><br>and Some says to upgrade the RAM which i am already looking but just a<br>part of my confusion when i see vmstat it shows me i have 7GB free RAM<br><br>is there anyone can please explain me why and what is happening behind all<br>this, the workstation hosting OmniOS is a good machine. why it is doing some<br>thing very unexpected.<br><br>Any advice, suggestion, tip etc. will be highly appreciated.<br><br>Thanks<br>Myk<br><br></blockquote><br><br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>http://lists.omniti.com/mailman/listinfo/omnios-discuss<br><br></blockquote><br><br><br>-- <br>Ian Kaufman<br>Research Systems Administrator<br>UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu<br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>http://lists.omniti.com/mailman/listinfo/omnios-discuss<br></blockquote></div><br><div apple-content-edited="true"> <div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br></div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><a href="mailto:Richard.Elling@RichardElling.com">Richard.Elling@RichardElling.com</a><br>+1-760-896-4422<br><br><br></div></span></div></span></div></span></div></span></div> </div> <br></body></html>