<html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi Dan, just wanted to know, if you would be interested in the core dump of this crash: Jan 11 07:28:52 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f090d09613000056b8a83b0007 (sd27): Jan 11 07:28:52 zfsha02gh79 incomplete write- retrying Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f0564d504f4f4c3035534c3133 (sd47): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,3">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,3</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f0564d504f4f4c3035534c3039 (sd44): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,0">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,0</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f0564d504f4f4c3035534c3131 (sd46): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,2">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,2</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f0564d504f4f4c3035534c3130 (sd45): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,1">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,1</a> Jan 12 17:30:22 zfsha02gh79 iscsi: [ID 431120 kern.warning] WARNING: iscsi connection(26/3f) closing connection - target requested reason:0x7 Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f090d09613000056b8a7f10003 (sd19): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,2">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,2</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f090d09613000056b8a7fc0004 (sd21): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,3">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,3</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 /scsi_vhci/disk@g600144f090d09613000056b8a84a0008 (sd29): Command Timeout on path <a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,7">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,7</a> Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Jan 12 17:30:22 zfsha02gh79 unix: [ID 836849 kern.notice] Jan 12 17:30:22 zfsha02gh79 ^Mpanic[cpu1]/thread=ffffff00f6539c40: Jan 12 17:30:22 zfsha02gh79 genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff00f6539610 addr=10 occurred in module "scsi_vhci" due to a NULL pointer dereference Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice] Jan 12 17:30:22 zfsha02gh79 unix: [ID 839527 kern.notice] sched: Jan 12 17:30:22 zfsha02gh79 unix: [ID 753105 kern.notice] #pf Page fault Jan 12 17:30:22 zfsha02gh79 unix: [ID 532287 kern.notice] Bad kernel fault at addr=0x10 Jan 12 17:30:22 zfsha02gh79 unix: [ID 243837 kern.notice] pid=0, pc=0xfffffffff7948e15, sp=0xffffff00f6539700, eflags=0x10246 Jan 12 17:30:22 zfsha02gh79 unix: [ID 211416 kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 1426f8<smep,osxsav,vmxe,xmme,fxsr,pge,mce,pae,pse,de> Jan 12 17:30:22 zfsha02gh79 unix: [ID 624947 kern.notice] cr2: 10 Jan 12 17:30:22 zfsha02gh79 unix: [ID 625075 kern.notice] cr3: c000000 Jan 12 17:30:22 zfsha02gh79 unix: [ID 625715 kern.notice] cr8: 0 Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice] Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] rdi: ffffff226adb90d8 rsi: 1 rdx: ffffff227063d400 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] rcx: 2 r8: 0 r9: fffffffff794bd10 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] rax: 0 rbx: 1 rbp: ffffff00f6539780 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] r10: 0 r11: ffffff00f65397b0 r12: 2 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] r13: 1 r14: 0 r15: 4 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] fsb: 0 gsb: ffffff21f0e81040 ds: 4b Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] es: 4b fs: 0 gs: 1c3 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] trp: e err: 0 rip: fffffffff7948e15 Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] cs: 30 rfl: 10246 rsp: ffffff00f6539700 Jan 12 17:30:22 zfsha02gh79 unix: [ID 266532 kern.notice] ss: 38 Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice] Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f65394f0 unix:die+df () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539600 unix:trap+dd8 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539610 unix:_cmntrap+e6 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539780 scsi_vhci:vhci_scsi_reset_target+75 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f65397d0 scsi_vhci:vhci_recovery_reset+7d () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539820 scsi_vhci:vhci_pathinfo_offline+e5 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f65398c0 scsi_vhci:vhci_pathinfo_state_change+d5 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539950 genunix:i_mdi_pi_state_change+16a () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539990 genunix:mdi_pi_offline+39 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539a20 iscsi:iscsi_lun_offline+b3 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539a60 iscsi:iscsi_sess_offline_luns+4d () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539ab0 iscsi:iscsi_sess_state_logged_in+11e () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539b00 iscsi:iscsi_sess_state_machine+13e () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539b60 iscsi:iscsi_client_notify_task+17e () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539c20 genunix:taskq_thread+2d0 () Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice] ffffff00f6539c30 unix:thread_start+8 () Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice] Jan 12 17:30:22 zfsha02gh79 genunix: [ID 672855 kern.notice] syncing file systems... Jan 12 17:30:24 zfsha02gh79 genunix: [ID 904073 kern.notice] done Jan 12 17:30:22 zfsha02gh79 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel Jan 12 17:30:22 zfsha02gh79 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port Jan 12 17:48:01 zfsha02gh79 genunix: [ID 100000 kern.notice] Jan 12 17:48:02 zfsha02gh79 genunix: [ID 665016 kern.notice] ^M100% done: 4721646 pages dumped, This happend on a rather higher load situation, when I was copying a 200G file from a snapshot back to it's original place on its zvol, when this happened. Luckily these are RSF-1 nodes and the other one took over very quickliy, such as that my VM cluster didn't even seem to notice this issue. However, at that time I was conencted to the crashing host via ssh and my heart skipped a beat. ;) As I have (unvoluntarily) freed this node of it's duties, I could jump to r020 on it, but I wonder if there has been any changes to the scsi_vhci layer at all in recent times… Cheers, Stephan <pre class="moz-signature" cols="72">-- Krebs's 3 Basic Rules for Online Safety 1st - "If you didn't go looking for it, don't install it!" 2nd - "If you installed it, update it." 3rd - "If you no longer need it, remove it." <a class="moz-txt-link-freetext" href="http://krebsonsecurity.com/2011/05/krebss-3-basic-rules-for-online-safety">http://krebsonsecurity.com/2011/05/krebss-3-basic-rules-for-online-safety</a> Stephan Budach Head of IT Jung von Matt AG Glashüttenstraße 79 20357 Hamburg Tel: +49 40-4321-1353 Fax: +49 40-4321-1114 E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:stephan.budach@jvm.de">stephan.budach@jvm.de</a> Internet: <a class="moz-txt-link-freetext" href="http://www.jvm.com">http://www.jvm.com</a> CiscoJabber Video: <a class="moz-txt-link-freetext" href="https://exp-e2.jvm.de/call/stephan.budach">https://exp-e2.jvm.de/call/stephan.budach</a> Vorstand: Dr. Peter Figge, Jean-Remy von Matt, Larissa Pohl, Thomas Strerath, Götz Ulmer Vorsitzender des Aufsichtsrates: Hans Hermann Münchmeyer AG HH HRB 72893 </pre> </body> </html>