<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Am 19.04.16 um 23:31 schrieb wuffers:<br>
</div>
<blockquote
cite="mid:CA+tR_Kwce_=qierL8oBBQVmHp5Rx6dLJZTqiN+dHndR-2-uPGA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">You might want to check this old thread:
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://lists.omniti.com/pipermail/omnios-discuss/2014-July/002927.html">http://lists.omniti.com/pipermail/omnios-discuss/2014-July/002927.html</a><br>
</div>
<div><br>
</div>
<div>Richard Elling had some interesting insights on how the
scrub works:</div>
<div><br>
</div>
<div>"So I think the pool is not scheduling scrub I/Os very
well. You can increase the number of</div>
<div>scrub I/Os in the scheduler by adjusting the
zfs_vdev_scrub_max_active tunable. The</div>
<div>default is 2, but you'll have to consider that a share (in
the stock market sense) where</div>
<div>the active sync reads and writes are getting 10 each. You
can try bumping up the value</div>
<div>and see what happens over some time, perhaps 10 minutes or
so -- too short of a time</div>
<div>and you won't get a good feeling for the impact (try this
in off-peak time).</div>
<div><span class="" style="white-space:pre"> </span>echo
zfs_vdev_scrub_max_active/W0t5 | mdb -kw</div>
<div>will change the value from 2 to 5, increasing its share of
the total I/O workload.</div>
<div><br>
</div>
<div>You can see the progress of scan (scrubs do scan) workload
by looking at the ZFS</div>
<div>debug messages.</div>
<div><span class="" style="white-space:pre"> </span>echo
::zfs_dbgmsg | mdb -k</div>
<div>These will look mysterious... they are. But the interesting
bits are about how many blocks</div>
<div>are visited in some amount of time (txg sync interval).
Ideally, this will change as you </div>
<div>adjust zfs_vdev_scrub_max_active."</div>
<div><br>
</div>
<div>I had to increase my zfs_vdev_scrub_max_active parameter
higher than 5, but it sounds like the default setting for that
tunable is no longer satisfactory for today's high performance
systems.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Apr 17, 2016 at 4:07 PM,
Stephan Budach <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:stephan.budach@jvm.de" target="_blank">stephan.budach@jvm.de</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Am
17.04.16 um 20:42 schrieb Dale Ghent:
<div>
<div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Apr 17, 2016, at 9:07 AM, Stephan Budach <<a
moz-do-not-send="true"
href="mailto:stephan.budach@JVM.DE"
target="_blank">stephan.budach@JVM.DE</a>>
wrote:<br>
<br>
Well… searching the net somewhat more thoroughfully,
I came across an archived discussion which deals
also with a similar issue. Somewhere down the
conversation, this parameter got suggested:<br>
<br>
echo "zfs_scrub_delay/W0" | mdb -kw<br>
<br>
I just tried that as well and although the caculated
speed climbs rathet slowly up, iostat now shows
approx. 380 MB/s read from the devices, which rates
at 24 MB/s per single device * 8 *2.<br>
<br>
Being curious, I issued a echo "zfs_scrub_delay/W1"
| mdb -kw to see what would happen and that command
immediately drowned the rate on each device down to
1.4 MB/s…<br>
<br>
What is the rational behind that? Who wants to wait
for weeks for a scrub to finish? Usually, I am
having znapzend run as well, creating snapshots on a
regular basis. Wouldn't that hurt scrub performance
even more?<br>
</blockquote>
zfs_scrub_delay is described here:<br>
<br>
<a moz-do-not-send="true"
href="http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#63"
rel="noreferrer" target="_blank">http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#63</a><br>
<br>
How busy are your disks if you subtract the IO caused
by a scrub? Are you doing these scrubs with your VMs
causing normal IO as well?<br>
<br>
Scrubbing, overall, is treated as a background
maintenance process. As such, it is designed to not
interfere with "production IO" requests. It used to be
that scrubs ran as fast as disk IO and bus bandwidth
would allow, which in turn severely impacted the IO
performance of running applications, and in some cases
this would cause problems for production or user
services. The scrub delay setting which you've
discovered is the main governor of this scrub throttle
code[1], and by setting it to 0, you are effectively
removing the delay it imposes on itself to allow
non-scrub/resilvering IO requests to finish.<br>
<br>
The solution in your case is specific to yourself and
how you operate your servers and services. Can you
accept degraded application IO while a scrub or
resilver is running? Can you not? Maybe only during
certain times?<br>
<br>
/dale<br>
<br>
[1] <a moz-do-not-send="true"
href="http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#1841"
rel="noreferrer" target="_blank">http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#1841</a><br>
</blockquote>
</div>
</div>
I do get the notion if this, but if the increase from 0 to 1
reduces the throughput from 24Mb/s to 1MB/s, this seems way
overboard to me. Having to wait for a couple of hours when
running with 0 as opposed to days (up to 10) when running at
1 - on a 1.3 TB zpool - doesn't seem to be the right
choice. If this tunable offered some more room for choice,
that would be great, but it obviously doesn't.<br>
<br>
It's the weekend and my VMs aren't excatly hogging their
disks, so there was plenty of I/O available… I'd wish for a
more granular setting regarding this setting.<br>
<br>
Anyway, the scrub finished a couple of hours later and of
course, I can always set this tunable to 0, should I need
it,<br>
<br>
Thanks,
<div class="HOEnZb">
<div class="h5"><br>
Stephan<br>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
Interesting read - and it surely works. If you set the tunable
before you start the scrub you can immediately see the thoughput
being much higher than with the standard setting. Now, what would be
the tunable to use zfs_vdev_scrub_max_active or zfs_scrub_delay?<br>
<br>
Cheers,<br>
Stephan<br>
<br>
</body>
</html>