<div dir="ltr"><div><div>Just to add to this, I had a 4th kernel panic, and this was a 3rd different type. I did a memtest on the unit after this last panic, and it ran successfully (24+ hours). I'm skeptical that it's memory, or something to do with the IOCLogInfo=0x31120303 error (last 2 panics didn't have that - I may start another thread on that), as I've been running this config with Hyper-V hosts just fine. Adding an ESXi host (just one for now) into the mix seems to make things unstable.<br>
<br></div><div>Should I be starting an issue in the Illumos issue report (<a href="https://www.illumos.org/projects/illumos-gate/issues/new">https://www.illumos.org/projects/illumos-gate/issues/new</a>), and if so, just one report or one for each panic type?<br>
</div><div><br>List of kernel panics so far:<br><br></div>Panic 1: anon_decref: slot count 0<br></div>Panic 2-3: kernel heap corruption detected<br>Panic 4: BAD TRAP: type=e (#pf Page fault) rp=ffffff01e97d7a70 addr=1500010 occurred in module "genunix" due to an illegal access to a user address<br>
<div><div><br></div><div>Latest crash file here: <br><a href="https://drive.google.com/file/d/0B7mCJnZUzJPKWW83TFBhVHpVajQ">https://drive.google.com/file/d/0B7mCJnZUzJPKWW83TFBhVHpVajQ</a><br></div><div><br>TIME UUID SUNW-MSG-ID<br>
Nov 17 2013 09:22:20.799446000 9d55f532-d39f-4dea-8f57-d3b24c8e9dff SUNOS-8000-KL<br><br> TIME CLASS ENA<br> Nov 17 09:22:20.7654 ireport.os.sunos.panic.dump_available 0x0000000000000000<br>
Nov 17 09:21:14.0267 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000<br><br>nvlist version: 0<br> version = 0x0<br> class = list.suspect<br> uuid = 9d55f532-d39f-4dea-8f57-d3b24c8e9dff<br>
code = SUNOS-8000-KL<br> diag-time = 1384698140 767808<br> de = fmd:///module/software-diagnosis<br> fault-list-sz = 0x1<br> fault-list = (array of embedded nvlists)<br> (start fault-list[0])<br>
nvlist version: 0<br> version = 0x0<br> class = defect.sunos.kernel.panic<br> certainty = 0x64<br> asru = sw:///:path=/var/crash/unknown/.9d55f532-d39f-4dea-8f57-d3b24c8e9dff<br>
resource = sw:///:path=/var/crash/unknown/.9d55f532-d39f-4dea-8f57-d3b24c8e9dff<br> savecore-succcess = 1<br> dump-dir = /var/crash/unknown<br> dump-files = vmdump.3<br>
os-instance-uuid = 9d55f532-d39f-4dea-8f57-d3b24c8e9dff<br> panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff01e97d7a70 addr=1500010 occurred in module "genunix" due to an illegal access to a user address<br>
panicstack = unix:die+df () | unix:trap+db3 () | unix:cmntrap+e6 () | genunix:anon_decref+35 () | genunix:anon_free+74 () | genunix:segvn_free+242 () | genunix:seg_free+30 () | genunix:segvn_unmap+cde () | genunix:as_free+e7 () | genunix:relvm+220 () | genunix:proc_exit+454 () | genunix:exit+15 () | genunix:rexit+18 () | unix:brand_sys_sysenter+1c9 () |<br>
crashtime = 1384592942<br> panic-time = Sat Nov 16 04:09:02 2013 EST<br> (end fault-list[0])<br><br> fault-status = 0x1<br> severity = Major<br> __ttl = 0x1<br> __tod = 0x5288d11c 0x2fa693f0<br>
</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Nov 16, 2013 at 2:48 AM, wuffers <span dir="ltr"><<a href="mailto:moo@wuffers.net" target="_blank">moo@wuffers.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>When it pours, it rains. With r151006y, I had two kernel panics in quick succession while trying to create some zero thick eager disks (4 at the same time) in ESXi. They are now "kernel heap corruption detected" instead of anon_decref.<br>
<br></div><div>Kernel panic 2 (dump info: <a href="https://drive.google.com/file/d/0B7mCJnZUzJPKMHhqZHJnaDEzYkk" target="_blank">https://drive.google.com/file/d/0B7mCJnZUzJPKMHhqZHJnaDEzYkk</a>)<br><a href="http://i.imgur.com/eIssxmc.png?1" target="_blank">http://i.imgur.com/eIssxmc.png?1</a><br>
<a href="http://i.imgur.com/MXJy4zP.png?1" target="_blank">http://i.imgur.com/MXJy4zP.png?1</a><br><br></div>TIME UUID SUNW-MSG-ID<br>Nov 16 2013 00:51:24.912170000 5998ba1e-3aa5-ccac-e885-be4897cfcfe8 SUNOS-8000-KL<br>
<br> TIME CLASS ENA<br> Nov 16 00:51:24.8638 ireport.os.sunos.panic.dump_available 0x0000000000000000<br> Nov 16 00:49:58.8671 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000<div class="im">
<br>
<br>nvlist version: 0<br> version = 0x0<br> class = list.suspect<br></div> uuid = 5998ba1e-3aa5-ccac-e885-be4897cfcfe8<br> code = SUNOS-8000-KL<br> diag-time = 1384581084 866703<div class="im">
<br> de = fmd:///module/software-diagnosis<br>
fault-list-sz = 0x1<br> fault-list = (array of embedded nvlists)<br> (start fault-list[0])<br> nvlist version: 0<br> version = 0x0<br> class = defect.sunos.kernel.panic<br>
certainty = 0x64<br></div> asru = sw:///:path=/var/crash/unknown/.5998ba1e-3aa5-ccac-e885-be4897cfcfe8<br> resource = sw:///:path=/var/crash/unknown/.5998ba1e-3aa5-ccac-e885-be4897cfcfe8<div class="im">
<br>
savecore-succcess = 1<br> dump-dir = /var/crash/unknown<br></div> dump-files = vmdump.1<br> os-instance-uuid = 5998ba1e-3aa5-ccac-e885-be4897cfcfe8<br> panicstr = kernel heap corruption detected<br>
panicstack = fffffffffba49c04 () | genunix:kmem_slab_free+c1 () | genunix:kmem_magazine_destroy+6e () | genunix:kmem_depot_ws_reap+5d () | genunix:kmem_cache_magazine_purge+118 () | genunix:kmem_cache_magazine_resize+40 () | genunix:taskq_thread+2d0 () | unix:thread_start+8 () |<br>
crashtime = 1384577735<br> panic-time = Fri Nov 15 23:55:35 2013 EST<div class="im"><br> (end fault-list[0])<br><br> fault-status = 0x1<br> severity = Major<br> __ttl = 0x1<br>
</div> __tod = 0x528707dc 0x365e9c10<br>
<br>kernel panic 3 (dump info: <a href="https://drive.google.com/file/d/0B7mCJnZUzJPKbnZIeWZzQjhUOTQ" target="_blank">https://drive.google.com/file/d/0B7mCJnZUzJPKbnZIeWZzQjhUOTQ</a>):<br></div>(looked the same, no screenshots)<br>
<br><div>
TIME UUID SUNW-MSG-ID<br>Nov 16 2013 01:44:43.327489000 a6592c60-199f-ead5-9586-ff013bf5ab2d SUNOS-8000-KL<br><br> TIME CLASS ENA<br>
Nov 16 01:44:43.2941 ireport.os.sunos.panic.dump_available 0x0000000000000000<br> Nov 16 01:44:03.5356 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000<div class="im"><br><br>nvlist version: 0<br> version = 0x0<br>
class = list.suspect<br></div> uuid = a6592c60-199f-ead5-9586-ff013bf5ab2d<br> code = SUNOS-8000-KL<br> diag-time = 1384584283 296816<div class="im"><br> de = fmd:///module/software-diagnosis<br>
fault-list-sz = 0x1<br>
fault-list = (array of embedded nvlists)<br> (start fault-list[0])<br> nvlist version: 0<br> version = 0x0<br> class = defect.sunos.kernel.panic<br> certainty = 0x64<br>
</div>
asru = sw:///:path=/var/crash/unknown/.a6592c60-199f-ead5-9586-ff013bf5ab2d<br> resource = sw:///:path=/var/crash/unknown/.a6592c60-199f-ead5-9586-ff013bf5ab2d<div class="im"><br> savecore-succcess = 1<br>
dump-dir = /var/crash/unknown<br></div> dump-files = vmdump.2<br> os-instance-uuid = a6592c60-199f-ead5-9586-ff013bf5ab2d<br> panicstr = kernel heap corruption detected<br>
panicstack = fffffffffba49c04 () | genunix:kmem_slab_free+c1 () | genunix:kmem_magazine_destroy+6e () | genunix:kmem_cache_magazine_purge+dc () | genunix:kmem_cache_magazine_resize+40 () | genunix:taskq_thread+2d0 () | unix:thread_start+8 () |<br>
crashtime = 1384582658<br> panic-time = Sat Nov 16 01:17:38 2013 EST<div class="im"><br> (end fault-list[0])<br><br> fault-status = 0x1<br> severity = Major<br> __ttl = 0x1<br>
</div> __tod = 0x5287145b 0x138515e8<br>
<br><br>---<br></div><div>Now, having looked through all 3, I can see in the first two there were some warnings:<br><pre>WARNING: /<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">pci at 0</a>,0/<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">pci8086,3c08 at 3</a>/<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">pci1000,3030 at 0</a> (mpt_sas1):
mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31120303<br><br></pre>The /var/adm/message also had a sprinkling of these:<br>Nov 15 23:36:43 san1 scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci8086,3c08@3/pci1000,3030@0 (mpt_sas1):<br>
Nov 15 23:36:43 san1 mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31120303<br>Nov 15 23:36:43 san1 scsi: [ID 365881 <a href="http://kern.info" target="_blank">kern.info</a>] /pci@0,0/pci8086,3c08@3/pci1000,3030@0 (mpt_sas1):<br>
Nov 15 23:36:43 san1 Log info 0x31120303 received for target 10.<br>Nov 15 23:36:43 san1 scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Following this <a href="http://lists.omniti.com/pipermail/omnios-discuss/2013-March/000544.html" target="_blank">http://lists.omniti.com/pipermail/omnios-discuss/2013-March/000544.html</a> to map the target disk, it's my Stec ZeusRAM ZIL drive that's configured as a mirror (if I've done it right). I didn't see these errors in the 3rd dump, so don't know if it's contributing. I may try to do a memtest tomorrow on the system just in case it's some hardware issues.<br>
<br>My zpool status shows all my drives okay with no known data errors.<br><br></div><div class="gmail_extra">Not sure how to proceed from here.. my Hyper-V hosts have been using the SAN with no issues for 2+ months since it's been up and configured, using SRP and IB. I'd expect the VM hosts to crash before my SAN does.<br>
<br></div><div class="gmail_extra">Of course, I can make the vmdump.x files available to anyone who wants to look at them (7GB, 8GB, 4GB).<br></div><div class="gmail_extra"><br></div></div>
</blockquote></div><br></div>