<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">A couple things that I've discovered over time that might help:<div class=""><br class=""></div><div class="">Don't ever use the root user for zpool queries such as "zpool status". If you have a really bad failing disk a zpool status command can take forever to complete when ran as root. A "su nobody -c 'zpool status'" will return results almost instantly. So if your device discovery script(s) use zpool commands, that might be a choking point.</div><div class=""><br class=""></div><div class=""># make sure to prevent scsi bus resets (in /kernel/drv/sd.conf) especially in an HA environment</div><div class="">allow-bus-device-reset=0;<br class=""> <br class="">Also, depending on the disk model, I've found that some of them wreak havoc on the SAS topology itself when they start to fail. Some just handle errors really badly and can flood the SAS channel. If you have a SAS switch in between, you might be able to get an idea of which device is causing the grief from there based on the counts.</div><div class=""><br class=""></div><div class="">In my case I have had horrible experiences with the WD WD4001FYYG. That model of drive has caused me an insane amount of headache. The disk scan on boot literally takes 13 seconds per-disk (when the disks are perfectly good and much much longer when one is dying). If I replace them with another make/model drive, the disk scan is done in a fraction of a second. Also, booting the same machine into any linux os the scan completes in a fraction of a second. Must be something about that model's firmware that doesn't play nicely with Illumos's driver. Anyway, that's a story for another time ;)</div><div class=""><br class=""></div><div class="">I've reduced the drive scan time at boot down to 5 seconds per disk instead of the 13 seconds per disk for that horrible accursed drive by adding this to /kernel/drv/sd.conf</div><div class=""><br class=""></div><div class="">sd-config-list= "WD WD4001FYYG","power-condition:false";<br class=""><br class=""></div><div class="">Followed by this command to commit it:</div><div class="">update_drv -vf sd<br class=""><br class=""></div><div class="">Hope this helps.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Michael</div><div class=""><br class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 22, 2017, at 1:41 PM, Schweiss, Chip <<a href="mailto:chip@innovates.com" class="">chip@innovates.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I'm talking about an offline pool. I started this thread after rebooting a server that is part of an HA pair. The other server has the pools online. It's been over 4 hours now and it still hasn't completed its disk scan. <div class=""><br class=""></div><div class="">Every tool I have that helps me locate disks, suffers from the same insane command timeout to happen many times before moving on. Operations that typically take seconds blow up to hours really fast because of a few dead disks. </div><div class=""><br class=""></div><div class="">-Chip</div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Jun 22, 2017 at 3:12 PM, Dale Ghent <span dir="ltr" class=""><<a href="mailto:daleg@omniti.com" target="_blank" class="">daleg@omniti.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class=""> Have you able to and have tried offlining it in the zpool?<br class=""> <br class=""> zpool offline thepool <disk><br class=""> <br class=""> I'm assuming the pool has some redundancy which would allow for this.<br class=""> <span class="HOEnZb"><font color="#888888" class=""><br class=""> /dale<br class=""> </font></span><div class="HOEnZb"><div class="h5"><br class=""> > On Jun 22, 2017, at 11:54 AM, Schweiss, Chip <<a href="mailto:chip@innovates.com" class="">chip@innovates.com</a>> wrote:<br class=""> ><br class=""> > When ever a disk goes south, several disk related takes become painfully slow. Boot up times can jump into the hours to complete the disk scans.<br class=""> ><br class=""> > The logs slowly get these type messages:<br class=""> ><br class=""> > genunix: WARNING /pci@0,0/pci8086,340c@5/<wbr class="">pci15d9,400@0 (mpt_sas0):<br class=""> > Timeout of 60 seconds expired with 1 commands on target 16 lun 0<br class=""> ><br class=""> > I thought this /etc/system setting would reduce the timeout to 5 seconds:<br class=""> > set sd:sd_io_time = 5<br class=""> ><br class=""> > But this doesn't seem to change anything.<br class=""> ><br class=""> > Is there anyway to make this a more reasonable timeout, besides pulling the disk that's causing it? Just locating the defective disk is also painfully slow because of this problem.<br class=""> ><br class=""> > -Chip<br class=""> </div></div><div class="HOEnZb"><div class="h5">> ______________________________<wbr class="">_________________<br class=""> > OmniOS-discuss mailing list<br class=""> > <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.<wbr class="">com</a><br class=""> > <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" rel="noreferrer" target="_blank" class="">http://lists.omniti.com/<wbr class="">mailman/listinfo/omnios-<wbr class="">discuss</a><br class=""> <br class=""> </div></div></blockquote></div><br class=""></div> _______________________________________________<br class="">OmniOS-discuss mailing list<br class=""><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss<br class=""></div></blockquote></div><br class=""></div></body></html>