<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Feb 12, 2014 at 10:55 AM, Saso Kiselkov <span dir="ltr"><<a href="mailto:skiselkov.ml@gmail.com" target="_blank">skiselkov.ml@gmail.com</a>></span> wrote:<br>

<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can you also try running this piece of dtrace on the machine? I have a<br>
hypothesis that I'd like to test:<br>
<br>
dtrace -n 'fbt::l2arc_evict:entry{printf("dev=%p; l2ad_first=%d",<br>
args[0], args[0]->l2ad_first)}'<br>
<br>
(let it run for about 5-10s and post the output)<br></blockquote><div> </div><div><br></div><div>Output from the above dtrace:<br><br>dtrace: description 'fbt::l2arc_evict:entry' matched 1 probe<br>CPU     ID                    FUNCTION:NAME<br>

 12  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br> 12  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br> 12  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>

 14  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  0  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>  5  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>

  5  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br> 19  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>

 21  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br> 23  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  0  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>

 11  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br> 13  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br> 14  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>

 15  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br> 19  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>

  3  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  3  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  3  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>

  3  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br> 16  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  3  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>

 13  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>  3  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>  0  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>

 21  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  2  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br> 18  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>

  1  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>

  2  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>  2  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  2  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>

 20  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>  0  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>  0  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>

  0  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>  2  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br>

  0  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br>  8  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>  1  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br>

  4  40955                l2arc_evict:entry dev=ffffff4377e25058; l2ad_first=1<br> 14  40955                l2arc_evict:entry dev=ffffff4377e25148; l2ad_first=1<br> 12  40955                l2arc_evict:entry dev=ffffff432153be68; l2ad_first=1<br>

 21  40955                l2arc_evict:entry dev=ffffff43782a8240; l2ad_first=1<br> <br></div><div>And on your other question:<br><br>> What's the block size of your SSDs and the average recordsize of the<br>> data on them?<br>

<br></div><div>I'm actually not sure what you're asking about.. the recordsize on the pool itself? It's 128K. I have different volblocksize on various volumes. Or information about the SSD itself?<br><br>* /dev/rdsk/c2t500117310015D579d0 partition map<br>
*<br>* Dimensions:<br>*     512 bytes/sector<br>* 781422768 sectors<br>* 781422701 accessible sectors<br>*<br>* Flags:<br>*   1: unmountable<br>*  10: read-only<br>*<br>* Unallocated space:<br>*       First     Sector    Last<br>
*       Sector     Count    Sector<br>*          34       222       255<br>*<br>*                          First     Sector    Last<br>* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory<br>       0      4    00        256 781406095 781406350<br>
       8     11    00  781406351     16384 781422734<br></div></div></div></div>