<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""> <div><blockquote type="cite" class=""><div class="">On Dec 23, 2015, at 12:58 AM, Dan Vatca <<a href="mailto:dan@syneto.eu" class="">dan@syneto.eu</a>> wrote:</div> <div class=""><div dir="ltr" class="">If you need latency, you will most likely need a latency distribution histogram, and not an average latency.<div class="">With averages you will lose latency outliers that are very important. Here's a good read with lots of references on this topic: <a href="https://www.vividcortex.com/blog/why-percentiles-dont-work-the-way-you-think" class="">https://www.vividcortex.com/blog/why-percentiles-dont-work-the-way-you-think</a></div><div class="">To currently do this on OmniOS, you need to use dtrace to aggregate (quantize) time differences between nfsv3:::op-read-start and nfsv3:::op-read-done (same for write).</div></div></div></blockquote><div> </div><div>Indeed, distributions are much more enlightening than averages.</div><div> </div><div>Unfortunately, the new kstats added for NFS server operations on a per-mountpoint basis</div><div>are implemented using the Riemann sums (KSTAT_TYPE_IO) and it is not possible to obtain</div><div>per-operation information needed for min/max or distribution. These are the same type of </div><div>kstat used for the iostat command.</div><div> </div><div>Shameless plug, nfssvrtop has proven to be useful in watching NFS traffic and uses the </div><div>op-read-start/op-read-done method.</div><div><a href="https://github.com/richardelling/tools" class="">https://github.com/richardelling/tools</a></div><div> </div><div> — richard</div> <blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""> </div></div><div class="gmail_extra"> <div class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div style="font-family: Monaco; font-size: 12px;" class=""><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">Dan Vâtca</div><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">CTO at Syneto</div><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">Tel: +40723604357, Skype: dan_vatca</div><a href="http://www.syneto.net/" target="_blank" class=""><img src="https://syneto.eu/wp-content/uploads/2014/07/storage_logo_icons8.png" class=""></a></div></div></div></div></div></div> <div class="gmail_quote">On Wed, Dec 23, 2015 at 2:44 AM, 張峻宇 <<a href="mailto:jerry1209@cht.com.tw" target="_blank" class="">jerry1209@cht.com.tw</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="ZH-TW" link="#0563C1" vlink="#954F72" class=""> <div class="">Hi all, According to the release note of OmniOS r151016, we could get “IOPS, bandwidth, and latency kstats for NFS server” there is lots of information showing when I use enter command #kstat, I want to get the “nfs read & write latency for NFS server” Q1 : Is the ‘nfs:0:rfsprocio_v4_write:wtime’ & ‘nfs:0:rfsprocio_v4_read:wtime’ meant write & read latency ? Q2 : I mounted the nfs share directory, and write lots file to it, the number of ‘nfs:0:rfsprocio_v4_write:wtime’ & ‘nfs:0:rfsprocio_v4_read:wtime’ still zero. Why ? #kstat –p –m nfs –n rfsprocio_v4_write nfs:0:rfsprocio_v4_write:class rfsprocio_v4nfs:0:rfsprocio_v4_write:crtime 50.833043074nfs:0:rfsprocio_v4_write:nread 3932160nfs:0:rfsprocio_v4_write:nwritten 5374607360nfs:0:rfsprocio_v4_write:rcnt 0nfs:0:rfsprocio_v4_write:reads 163840nfs:0:rfsprocio_v4_write:rlastupdate 12048225488385nfs:0:rfsprocio_v4_write:rlentime 33429565743nfs:0:rfsprocio_v4_write:rtime 23992279289nfs:0:rfsprocio_v4_write:snaptime 269635.483575440nfs:0:rfsprocio_v4_write:wcnt 0nfs:0:rfsprocio_v4_write:wlastupdate 0nfs:0:rfsprocio_v4_write:wlentime 0nfs:0:rfsprocio_v4_write:writes 163840 / number of writes /nfs:0:rfsprocio_v4_write:wtime 0 / wait queue - time spent waiting / #kstat –p –m nfs –n rfsprocio_v4_read nfs:0:rfsprocio_v4_read:class rfsprocio_v4nfs:0:rfsprocio_v4_read:crtime 50.833003263nfs:0:rfsprocio_v4_read:nread 0nfs:0:rfsprocio_v4_read:nwritten 0nfs:0:rfsprocio_v4_read:rcnt 0nfs:0:rfsprocio_v4_read:reads 0nfs:0:rfsprocio_v4_read:rlastupdate 0nfs:0:rfsprocio_v4_read:rlentime 0nfs:0:rfsprocio_v4_read:rtime 0nfs:0:rfsprocio_v4_read:snaptime 269635.483080962nfs:0:rfsprocio_v4_read:wcnt 0nfs:0:rfsprocio_v4_read:wlastupdate 0nfs:0:rfsprocio_v4_read:wlentime 0nfs:0:rfsprocio_v4_read:writes 0nfs:0:rfsprocio_v4_read:wtime 0 Best regards,---------------------------------------------張峻宇中華電信研究院雲端運算研究所TEL: 03-4245663 </div> 本信件可能包含中華電信股份有限公司機密資訊,非指定之收件者,請勿蒐集、處理或利用本信件內容,並請銷毀此信件. 如為指定收件者,應確實保護郵件中本公司之營業機密及個人資料,不得任意傳佈或揭露,並應自行確認本郵件之附檔與超連結之安全性,以共同善盡資訊安全與個資保護責任. Please be advised that this email message (including any attachments) contains confidential information and may be legally privileged. If you are not the intended recipient, please destroy this message and all attachments from your system and do not further collect, process, or use them. Chunghwa Telecom and all its subsidiaries and associated companies shall not be liable for the improper or incomplete transmission of the information contained in this email nor for any delay in its receipt or damage to your system. If you are the intended recipient, please protect the confidential and/or personal information contained in this email with due care. Any unauthorized use, disclosure or distribution of this message in whole or in part is strictly prohibited. Also, please self-inspect attachments and hyperlinks contained in this email to ensure the information security and to protect personal information. </div> _______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" rel="noreferrer" target="_blank" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> </blockquote></div> </div> _______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> http://lists.omniti.com/mailman/listinfo/omnios-discuss </div></blockquote></div> <div class=""> <div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""> </div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><a href="mailto:Richard.Elling@richardelling.com" class="">Richard.Elling@RichardElling.com</a> +1-760-896-4422 </div></div></div></div></div> </div> </body></html>