<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks Rune, more below...</div><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 19, 2015, at 5:23 AM, Rune Tipsmark <<a href="mailto:rt@steait.net" class="">rt@steait.net</a>> wrote:</div><div class=""><div dir="ltr" style="font-size: 12pt; background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div class=""><br class=""></div> <div style="-ms-word-wrap: break-word;" class=""> <hr tabindex="-1" style="width: 98%; display: inline-block;" class=""> <div id="divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b> Richard Elling <<a href="mailto:richard.elling@richardelling.com" class="">richard.elling@richardelling.com</a>><br class=""> <b class="">Sent:</b> Monday, January 19, 2015 1:57 PM<br class=""> <b class="">To:</b> Rune Tipsmark<br class=""> <b class="">Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com" class="">omnios-discuss@lists.omniti.com</a><br class=""> <b class="">Subject:</b> Re: [OmniOS-discuss] ZFS Volumes and vSphere Disks - Storage vMotion Speed</font> <div class=""> </div> </div> <div class=""><br class=""> <div class=""> <blockquote type="cite" class=""> <div class="">On Jan 19, 2015, at 3:55 AM, Rune Tipsmark <<a href="mailto:rt@steait.net" class="">rt@steait.net</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""><style type="text/css" class="">  </style> <div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" dir="ltr" class=""><p class="">hi all,</p> <div class=""> <br class="webkit-block-placeholder"> </div><p class="">just in case there are other people out there using their ZFS box against vSphere 5.1 or later... I found my storage vmotion were slow... really slow... not much info available and so after a while of trial and error I found a nice combo that works very well in terms of performance, latency as well as throughput and storage vMotion.</p> <div class=""> <br class="webkit-block-placeholder"> </div><p class="">- <strong class="">Use</strong><strong class=""> ZFS volumes</strong> instead of thin provisioned LU's - Volumes support two of the VAAI features</p> <div class=""><br class=""> </div> </div> </div> </blockquote> <div class=""><br class=""> </div> AFAIK, ZFS is not available in VMware. Do you mean run iSCSI to connect the ESX box to</div> <div class="">the server running ZFS? If so...<br class=""> </div> <div class="">>> I run 8G Fibre Channel<br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>ok, still it is COMSTAR, so the backend is the same</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-size: 12pt; background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div style="-ms-word-wrap: break-word;" class=""><div class=""><div class=""> </div> <blockquote type="cite" class=""> <div class=""> <div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" dir="ltr" class=""><p class="">- <strong class="">Use thick provisioning disks</strong>, lazy zeroed disks in my case <strong class=""> reduced storage vMotion by 90%</strong> or so - machine 1 dropped from 8½ minutes to 23 seconds and machine 2 dropped from ~7 minutes to 54 seconds... a rather nice improvement simply by changing from thin to thick provisioning.</p> <div class=""><br class=""> </div> </div> </div> </blockquote> <div class=""></div> <div class=""><br class=""> </div> <div class="">This makes no difference in ZFS. The "thick provisioned" volume is simply a volume with a reservation.</div> <div class="">All allocations are copy-on-write. So the only difference between a "thick" and "thin" volume occurs when</div> <div class="">you run out of space in the pool.<br class=""> </div> <div class="">>> I am talking thick provisioning in VMware, that's where it makes a huge difference<br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>yes, you should always let VMware think it is thick provisioned, even if it isn't. VMware is too</div><div>ignorant of copy-on-write file systems to be able to make good decisions.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-size: 12pt; background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div style="-ms-word-wrap: break-word;" class=""><div class=""><div class=""> </div> <blockquote type="cite" class=""> <div class=""> <div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" dir="ltr" class=""><p class="">- I dropped my Qlogic HBA <strong class="">max queue depth from default 64 to 16</strong> on all ESXi hosts and now I see an average latency of less than 1ms per data store (on 8G fibre channel). Of course there are spikes when doing storage vMotion at these speeds but its well worth it.</p> <div class=""><br class=""> </div> </div> </div> </blockquote> <div class=""></div> <div class=""><br class=""> </div> <div class=""></div> <div class="">I usually see storage vmotion running at wire speed for well configured systems. When you get </div> <div class=""></div> <div class="">into the 2GByte/sec range this can get tricky, because maintaining that flow through the RAM</div> <div class=""></div> <div class="">and disks requires nontrivial amounts of hardware.</div> <div class=""></div> <div class="">>> I don't even get close to wire speed unfortunately my SLOGs can only do around 5-600 MBbyte/sec with sync=always.<br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>Indeed, the systems we make fast have enough hardware to be fast.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-size: 12pt; background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div style="-ms-word-wrap: break-word;" class=""><div class=""><div class=""> </div> <div class=""></div> <div class="">More likely, you're seeing the effects of caching, which is very useful for storage vmotion and</div> <div class=""></div> <div class="">allows you to hit line rate.</div> <div class=""><br class=""> </div> <div class=""><span class="">>> Not sure this is the case with using sync=always? </span></div></div></div></div></div></blockquote><div><br class=""></div><div>Caching will make a big difference. You should also see effective use of the ZFS prefetcher.</div><div><br class=""></div><div>Thanks for sharing your experience.</div><div> -- richard</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-size: 12pt; background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div style="-ms-word-wrap: break-word;" class=""><div class=""> <blockquote type="cite" class=""> <div class=""> <div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" dir="ltr" class=""> <div class=""> <br class="webkit-block-placeholder"> </div><p class="">I am getting to the point where I am almost happy with my ZFS backend for vSphere.</p> <div class=""><br class=""> </div> </div> </div> </blockquote> <div class=""></div> <div class=""><br class=""> </div> <div class=""></div> <div class="">excellent!</div> <div class=""></div> <div class=""> -- richard</div> <div class=""></div> <div class=""><br class=""> </div> <div class=""></div> <br class=""> </div> </div> </div> </div></blockquote></div><br class=""><style type="text/css" style="display:none" class=""></style></body></html>