<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<div style="margin-top: 0px; margin-bottom: 0px;">hi guys,</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">my omnios zfs server crashed today and I got a complete core dump and I was wondering if I am on the right track...</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">here is what I did so far...</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;"><a href="mailto:root@zfs10:/root" target="_blank">root@zfs10:/root</a># fmdump -Vp -u 775e0fc1-dcd2-4cb2-b800-88a1b9910f94<br>
TIME UUID SUNW-MSG-ID<br>
Apr 17 2015 22:48:13.667749000 775e0fc1-dcd2-4cb2-b800-88a1b9910f94 SUNOS-8000-KL</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> TIME CLASS ENA<br>
Apr 17 22:48:13.6544 ireport.os.sunos.panic.dump_available 0x0000000000000000<br>
Apr 17 22:45:46.3335 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000</div>
<div style="margin-top: 0px; margin-bottom: 0px;">nvlist version: 0<br>
version = 0x0<br>
class = list.suspect<br>
uuid = 775e0fc1-dcd2-4cb2-b800-88a1b9910f94<br>
code = SUNOS-8000-KL<br>
diag-time = 1429303693 655062<br>
de = fmd:///module/software-diagnosis<br>
fault-list-sz = 0x1<br>
fault-list = (array of embedded nvlists)<br>
(start fault-list[0])<br>
nvlist version: 0<br>
version = 0x0<br>
class = defect.sunos.kernel.panic<br>
certainty = 0x64<br>
asru = sw:///:path=/var/crash/unknown/.775e0fc1-dcd2-4cb2-b800-88a1b9910f94<br>
resource = sw:///:path=/var/crash/unknown/.775e0fc1-dcd2-4cb2-b800-88a1b9910f94<br>
savecore-succcess = 1<br>
dump-dir = /var/crash/unknown<br>
dump-files = vmdump.1<br>
os-instance-uuid = 775e0fc1-dcd2-4cb2-b800-88a1b9910f94<br>
panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff01701bb960 addr=ec6093a0 occurred in module "unix" due to an illegal access to a user address<br>
panicstack = unix:die+df () | unix:trap+db3 () | unix:cmntrap+e6 () | unix:bzero+184 () | zfs:l2arc_write_buffers+1f8 () | zfs:l2arc_feed_thread+240 () | unix:thread_start+8 () |<br>
crashtime = 1429299093<br>
panic-time = Fri Apr 17 21:31:33 2015 CEST<br>
(end fault-list[0])</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> fault-status = 0x1<br>
severity = Major<br>
__ttl = 0x1<br>
__tod = 0x5531718d 0x27cd0a88</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">//then extract the dump file:</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">savecore: not enough space in /var/crash/unknown (14937 MB avail, 27154 MB needed)<br>
<a href="mailto:root@zfs10:/var/crash/unknown" target="_blank">root@zfs10:/var/crash/unknown</a># savecore -f /pool01/ISO/vmdump.1 /pool01/ISO/<br>
savecore: System dump time: Fri Apr 17 21:31:33 2015</div>
<div style="margin-top: 0px; margin-bottom: 0px;">savecore: saving system crash dump in /pool01/ISO//{unix,vmcore}.1<br>
Constructing namelist /pool01/ISO//unix.1<br>
Constructing corefile /pool01/ISO//vmcore.1<br>
3:33 100% done: 6897249 of 6897249 pages saved</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">// then mdb and $c to see last process before the crash...</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;"><a href="mailto:root@zfs10:/pool01/ISO" target="_blank">root@zfs10:/pool01/ISO</a># mdb unix.1 vmcore.1<br>
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sata sd ip hook neti sockfs arp usba stmf stmf_sbd fctl md lofs mpt_sas random ufs idm smbsrv nfs crypto ptm cpc kvm fcp fcip logindmux nsmb nsctl sdbc ii sv rdc ]<br>
> $c<br>
bzero+0x184()<br>
l2arc_write_buffers+0x1f8(ffffff328f860000, ffffff331782d8d8, 800000, ffffff01701bbbec)<br>
l2arc_feed_thread+0x240()<br>
thread_start+8()</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">// based on this I believe my m2 sata L2 cache Samsung ssd drives used for L2arc in the zpool are ready to be thrown into the bin ....
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">Is there some way I can gather more info and confirm I am on the right track?</div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;"> </div>
<div style="margin-top: 0px; margin-bottom: 0px;">br,</div>
<div style="margin-top: 0px; margin-bottom: 0px;">Rune</div>
</body>
</html>