<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)">  <style>  </style> </head> <body lang=CS link=blue vlink=blue style='word-wrap: break-word;-webkit-nbsp-mode: space; -webkit-line-break: after-white-space'> <div class=Section1> Hi Richard,<o:p></o:p> <o:p> </o:p> thank you for your reply.<o:p></o:p> <o:p> </o:p> 1. Workload is still the same or very similar. Zvols, which we deleted from our pool were disconnected from KVM server a few days before, so the only change was, that we deleted that zvols with all snapshots.<o:p></o:p> 2. As you wrote, our customers are fine for now :) We have monitoring of all our virtual servers running from that storage server, and there is no noticeable change in workload or latencies. <o:p></o:p> 3. That could be the reason, of course. But in the graph are only data from arcstat.pl script. We can see, that arcstat is reporting heavy read accesses every 5 seconds (propably some update of ARC after ZFS writes data to disks from ZIL? All of them are marked as "cache hits" by arcstat script) and with only few ARC accesses between that 5 seconds periody. Before we deleted that zvols (about 0.7 TB data from 10 TB pool, which have 5 TB of free space) there were about 40k accesses every 5 seconds, now there are no more than 2k accesses every 5 seconds.<o:p></o:p> <o:p> </o:p> Most of our zvols have 8K volblocksize (including deleted zvols), only few have 64K. Unfortunately I have no data about size of the read before that change. But we have two more storage servers, with similary high ARC read accesses every 5 seconds as on the first pool before deletion. Maybe I should try to delete some data on that pools and see what happen with more detailed monitoring.<o:p></o:p> <o:p> </o:p> Thank you,<o:p></o:p> Filip<o:p></o:p> <o:p></o:p> <o:p> </o:p> <div> <div class=MsoNormal align=center style='text-align:center'> <hr size=2 width="100%" align=center tabindex=-1> </div> From: Richard Elling [mailto:richard.elling@richardelling.com] Sent: Wednesday, May 07, 2014 3:56 AM To: Filip Marvan Cc: omnios-discuss@lists.omniti.com Subject: Re: [OmniOS-discuss] Strange ARC reads numbers<o:p></o:p> </div> <o:p> </o:p> Hi Filip,<o:p></o:p> <div> <div> <div> <o:p> </o:p> </div> <div> There are two primary reasons for reduction in the number of ARC reads.<o:p></o:p> </div> <div> 1. the workload isn't reading as much as it used to<o:p></o:p> </div> <div> 2. the latency of reads has increased<o:p></o:p> </div> <div> 3. your measurement is b0rken<o:p></o:p> </div> <div> there are three reasons...<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> The data you shared clearly shows reduction in reads, but doesn't contain the answers<o:p></o:p> </div> <div> to the cause. Usually, if #2 is the case, then the phone will be ringing with angry customers<o:p></o:p> </div> <div> on the other end.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> If the above 3 are not the case, then perhaps it is something more subtle. The arcstat reads<o:p></o:p> </div> <div> does not record the size of the read. To get the read size for zvols is a little tricky, you can<o:p></o:p> </div> <div> infer it from the pool statistics in iostat. The subtleness here is that if the volblocksize is <o:p></o:p> </div> <div> different between the old and new zvols, then the number of (block) reads will be different<o:p></o:p> </div> <div> for the same workload.<o:p></o:p> </div> <div> -- richard<o:p></o:p> </div> </div> <o:p> </o:p> </div> </div> </body> </html>