<div dir="ltr"><div>Hi.</div><div><br></div><div>I recently got a kernel panic on my OmniOS ZFS storage server.</div><div>I have a 500GB iSCSI target from my ISP that I have created a zpool on, and one ZFS dataset.</div><div>
My OmniOS machines uses this zpool to archive some data not used in production.</div><div>I mainly use rsync to move files between.</div><div><br></div><div>This morning, which the iSCSI connection had been up since yesterday I was going to sync about 20GB of files</div>
<div>to the iSCSI target.</div><div><br></div><div>My server got into panic mode.</div><div><br></div><div>This is what was in my messages.log file :</div><div><br></div><div><br></div><div><font face="courier new, monospace">May 21 09:13:17 media savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=8 (#df Double fault) rp=ffffff04e3069f10 addr=0</font></div>
<div><font face="courier new, monospace">May 21 09:13:17 media savecore: [ID 365739 auth.error] Saving compressed system crash dump in /var/crash/unknown/vmdump.0</font></div><div><font face="courier new, monospace">May 21 09:14:24 media savecore: [ID 849871 auth.error] Decompress the crash dump with</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media 'savecore -vf /var/crash/unknown/vmdump.0'</font></div><div><font face="courier new, monospace">May 21 09:14:24 media fmd: [ID 377184 daemon.error] SUNW-MSG-ID: SUNOS-8000-KL, TYPE: Defect, VER: 1, SEVERITY: Major</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media EVENT-TIME: Wed May 21 09:14:24 GMT 2014</font></div><div><font face="courier new, monospace">May 21 09:14:24 media PLATFORM: X10SAE, CSN: 0123456789, HOSTNAME: media</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media SOURCE: software-diagnosis, REV: 0.1</font></div><div><font face="courier new, monospace">May 21 09:14:24 media EVENT-ID: f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media DESC: The system has rebooted after a kernel panic. Refer to <a href="http://illumos.org/msg/SUNOS-8000-KL">http://illumos.org/msg/SUNOS-8000-KL</a> for more information.</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media AUTO-RESPONSE: The failed system image was dumped to the dump device. If savecore is enabled (see dumpadm(1M)) a copy of the dump will be written to the savecore directory /var/crash/unknown.</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media IMPACT: There may be some performance impact while the panic is copied to the savecore directory. Disk space usage by panics can be substantial.</font></div>
<div><font face="courier new, monospace">May 21 09:14:24 media REC-ACTION: If savecore is not enabled then please take steps to preserve the crash image.</font></div><div><font face="courier new, monospace">May 21 09:14:24 media Use 'fmdump -Vp -u f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0' to view more panic detail. Please refer to the knowledge article for additional information.</font></div>
<div><br></div><div><br></div><div>By issuing fmdump -Vp -u f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</div><div>I have this data :</div><div><br></div><div><br></div><div><font face="courier new, monospace">TIME UUID SUNW-MSG-ID</font></div>
<div><font face="courier new, monospace">maí 21 2014 09:14:24.861678000 f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0 SUNOS-8000-KL</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> TIME CLASS ENA</font></div>
<div><font face="courier new, monospace"> maí 21 09:14:24.7657 ireport.os.sunos.panic.dump_available 0x0000000000000000</font></div><div><font face="courier new, monospace"> maí 21 09:13:17.7666 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">nvlist version: 0</font></div><div><font face="courier new, monospace"> version = 0x0</font></div><div><font face="courier new, monospace"> class = list.suspect</font></div>
<div><font face="courier new, monospace"> uuid = f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</font></div><div><font face="courier new, monospace"> code = SUNOS-8000-KL</font></div><div><font face="courier new, monospace"> diag-time = 1400663664 781451</font></div>
<div><font face="courier new, monospace"> de = fmd:///module/software-diagnosis</font></div><div><font face="courier new, monospace"> fault-list-sz = 0x1</font></div><div><font face="courier new, monospace"> fault-list = (array of embedded nvlists)</font></div>
<div><font face="courier new, monospace"> (start fault-list[0])</font></div><div><font face="courier new, monospace"> nvlist version: 0</font></div><div><font face="courier new, monospace"> version = 0x0</font></div>
<div><font face="courier new, monospace"> class = defect.sunos.kernel.panic</font></div><div><font face="courier new, monospace"> certainty = 0x64</font></div><div><font face="courier new, monospace"> asru = sw:///:path=/var/crash/unknown/.f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</font></div>
<div><font face="courier new, monospace"> resource = sw:///:path=/var/crash/unknown/.f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</font></div><div><font face="courier new, monospace"> savecore-succcess = 1</font></div>
<div><font face="courier new, monospace"> dump-dir = /var/crash/unknown</font></div><div><font face="courier new, monospace"> dump-files = vmdump.0</font></div><div><font face="courier new, monospace"> os-instance-uuid = f8b8e00b-a409-6f50-aaa6-bc6c3ebadad0</font></div>
<div><font face="courier new, monospace"> panicstr = BAD TRAP: type=8 (#df Double fault) rp=ffffff04e3069f10 addr=0</font></div><div><font face="courier new, monospace"> panicstack = unix:real_mode_stop_cpu_stage2_end+9de3 () | unix:trap+ca5 () | unix:_patch_xrstorq_rbx+196 () | zfs:zio_vdev_delegated_io+86 () | zfs:vdev_queue_aggregate+298 () | zfs:vdev_queue_io_to_issue+5e () | zfs:vdev_queue_io_done+88 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () | zfs:zio_execute+88 () | zfs:vdev_queue_io_done+78 () | zfs:zio_vdev_io_done+80 () |</font></div>
<div><font face="courier new, monospace"> crashtime = 1400662215</font></div><div><font face="courier new, monospace"> panic-time = Wed May 21 08:50:15 2014 GMT</font></div><div><font face="courier new, monospace"> (end fault-list[0])</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> fault-status = 0x1</font></div><div><font face="courier new, monospace"> severity = Major</font></div><div>
<font face="courier new, monospace"> __ttl = 0x1</font></div><div><font face="courier new, monospace"> __tod = 0x537c6e70 0x335c29b0</font></div><div> </div><div>Does anyone see anything ? I have no clue or knowledge/experience in debugging kernel based crashes.</div>
<div>The only feature that I have enabled on this zPool and or ZFS dataset is a Lz4 compression on the zfs dataset.</div><div><br></div><div>Is there any zfs, iSCSI improvements in the latest OmniOS release ?</div><div><br>
</div><div>Any help, and or information would be much appreciated.</div><div><br></div><div>Thanks allot people.</div><div><br></div><div>Best regards,</div><div><br></div><div>Svavar Orn</div><div><span class="" style="white-space:pre"> </span></div>
</div>