<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 18, 2015, at 11:25 AM, Jeff Stockett <<a href="mailto:jstockett@molalla.com" class="">jstockett@molalla.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class="">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)" class="">
<style class=""><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SourceCodeProSemibold;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="EN-US" link="blue" vlink="purple" class="">
<div class="WordSection1"><p class="MsoNormal">A drive failed in one of our supermicro 5048R-E1CR36L servers running omnios r151012 last night, and somewhat unexpectedly, the whole system seems to have panicked.
<o:p class=""></o:p></p><p class="MsoNormal"><o:p class=""> </o:p></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:43:08 zfs01 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,2f02@1/pci15d9,808@0 (mpt_sas0):<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:43:08 zfs01 Log info 0x31140000 received for target 29 w50000c0f01f1bf06.<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:43:08 zfs01 scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc</span></p></div></div></div></blockquote><div><br class=""></div><div>[forward reference]</div><br class=""><blockquote type="cite" class=""><div class=""><div lang="EN-US" link="blue" vlink="purple" class=""><div class="WordSection1"><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class=""><o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 843051 kern.info] NOTICE: SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error,
VER: 1, SEVERITY: Major<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 unix: [ID 836849 kern.notice]
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 ^Mpanic[cpu0]/thread=ffffff00f3ecbc40:
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 918906 kern.notice] I/O to pool 'dpool' appears to be hung.<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 unix: [ID 100000 kern.notice]
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecba20 zfs:vdev_deadman+10b ()</span></p></div></div></div></blockquote><div><br class=""></div><div>Bugs notwithstanding, the ZFS deadman timer occurs when a ZFS I/O does not</div><div>complete in 10,000 seconds (by default). The problem likely lies below ZFS. For this</div><div>reason, the deadman timer was invented -- don't blame ZFS for a problem below ZFS.</div><br class=""><blockquote type="cite" class=""><div class=""><div lang="EN-US" link="blue" vlink="purple" class=""><div class="WordSection1"><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class=""><o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecba70 zfs:vdev_deadman+4a ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbac0 zfs:vdev_deadman+4a ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbaf0 zfs:spa_deadman+ad ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbb90 genunix:cyclic_softint+fd
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbba0 unix:cbe_low_level+14 ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbbf0 unix:av_dispatch_softvect+78
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3ecbc20 apix:apix_dispatch_softint+35
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05990 unix:switch_sp_and_call+13
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e059e0 apix:apix_do_softint+6c ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05a40 apix:apix_do_interrupt+34a
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05a50 unix:cmnint+ba ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05bc0 unix:acpi_cpu_cstate+11b ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05bf0 unix:cpu_acpi_idle+8d ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05c00 unix:cpu_idle_adaptive+13
()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05c20 unix:idle+a7 ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 655072 kern.notice] ffffff00f3e05c30 unix:thread_start+8 ()<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 unix: [ID 100000 kern.notice]
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:36 zfs01 genunix: [ID 672855 kern.notice] syncing file systems...<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:38 zfs01 genunix: [ID 904073 kern.notice] done<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:39 zfs01 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset
65536, content: kernel<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 04:44:39 zfs01 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 1 reset
port<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 05:17:56 zfs01 genunix: [ID 100000 kern.notice]
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 05:17:56 zfs01 genunix: [ID 665016 kern.notice] ^M100% done: 8607621 pages dumped,
<o:p class=""></o:p></span></p><p class="MsoNormal" style="line-height:11.25pt;vertical-align:baseline"><span style="font-size: 7.5pt; font-family: SourceCodeProSemibold, serif;" class="">May 18 05:17:56 zfs01 genunix: [ID 851671 kern.notice] dump succeeded<o:p class=""></o:p></span></p><p class="MsoNormal"><o:p class=""> </o:p></p><p class="MsoNormal">The disks are all 4TB WD40001FYYG enterprise SAS drives.</p><div class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div><div>I've had such bad luck with that model, IMNSHO I recommend replacing with anything else :-(</div><div><br class=""></div><div>That said, I don't think it is a root cause for this panic. To get the trail of tears, you'll need to</div><div>look at the FMA ereports for the 10,000 seconds prior to the panic. fmdump has a -t option you'll</div><div>find useful. The [foreward reference] is the result of a SCSI reset of the target, LUN, or HBA.</div><div>These occur when the sd driver has not had a reply and issues one of those types of resets *or*</div><div>the device or something in the data path resets.</div><div><br class=""></div><div>HTH,</div><div> -- richard</div><br class=""><blockquote type="cite" class=""><div class=""><div lang="EN-US" link="blue" vlink="purple" class=""><div class="WordSection1"><p class="MsoNormal"> Googling seems to indicate it is a known problem with the way the various subsystems sometimes interact. Is there any way to fix/workaround this issue?<o:p class=""></o:p></p>
</div>
</div>
_______________________________________________<br class="">OmniOS-discuss mailing list<br class=""><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss<br class=""></div></blockquote></div><br class=""></body></html>