<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Today the server crashed again. I’m not sure if it’s because I was running SMART short self-tests or not, but it looks like it started around that time. <div class=""><br class=""></div><div class="">I’m still running smart tests, but it looks like there are no errors on the drives, although some tests take up to 30min to finish… iostat -E also reports no errors.</div><div class=""><br class=""></div><div class="">When it froze, I started iostat and tried to write a file to ZFS pool. As usual, it froze, but I left iostat running, hoping it will give me some infos… After 30 or something minutes, system become responsible again and this is how my iostat output looks like:</div><div class=""><a href="http://pastebin.com/W4EWgnzq" class="">http://pastebin.com/W4EWgnzq</a></div><div class=""><br class=""></div><div class="">System got responsible at 'Fri May 29 11:38:45 CEST 2015'.</div><div class=""><br class=""></div><div class="">It’s weird to say the least. It looks like there is something in write buffer that hogs the ZFS for quite some time and gets released or times-out after a certain time. But I’m not sure that it is and what thing has such a long timeout. It looks like freeze lasted for 15 minutes.</div><div class=""><br class=""></div><div class="">Matej</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 28 May 2015, at 18:30, Anon <<a href="mailto:anon@omniti.com" class="">anon@omniti.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Have you verified that your disks are not having any issues with <span style="font-family:monospace,monospace" class="">smartctl</span> and <span style="font-family:monospace,monospace" class="">iostat -E</span> ?<br class=""><br class=""></div>I'd suggest running a short test on the disks: <span style="font-family:monospace,monospace" class="">smartctl -d sat,12 -t short /path/to/disk</span> (note: you may need to append s2 to the physical disk name).<br class=""><br class=""></div>I built a test target and iSCSI initiator and wrote 1G from /dev/zero and ended up crashing the sesssion; are your sessions under load?<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, May 27, 2015 at 2:58 AM, Matej Zerovnik <span dir="ltr" class=""><<a href="mailto:matej@zunaj.si" target="_blank" class="">matej@zunaj.si</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Hello Josten,<div class=""><br class=""></div><div class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On 26 May 2015, at 22:18, Anon <<a href="mailto:anon@omniti.com" target="_blank" class="">anon@omniti.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hi Matej,<br class=""><br class="">Do you have sar running on your system? I'd recommend maybe running it at a short interval so that you can get historical disk statistics. You can use this info to rule out if its the disks or not. You can also use iotop -P to get a real time view of %IO to see if it's the disks. You can also use zpool iostat -v 1.<br class=""></div></div></blockquote><div class=""><br class=""></div></span><div class="">I didn’t have sar or iotop running, but I had 'iostat -xn' and 'zpool iostat -v 1' running when things stopped working, but there is nothing unusual in there. Write ops suddenly fall to 0 and that’s it. Reads are still happening and according to network traffic, there is outgoing traffic when I’m unable to write to the ZFS FS (even locally on the server). I created a simple text file, so next time system hangs, I will be able to check if system is readable (currently, I only have iscsi volumes, so I’m unable to check that locally on server).</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class="">Also, do you have baseline benchmark of performance and know if you're meeting/exceeding it? The baseline should be for random and sequential IO; you can use bonnie++ to get this information.<br class=""></div></div></blockquote><div class=""><br class=""></div></span><div class="">I can, with 99,99% say, I’m exceeding performance of the pool itself. It’s a single raidz2 vdev with 50 hard drives and 70 connected clients. some are idling, but 10-20 clients are pushing data to server. I know zpool configuration is very bad, but that’s a legacy I can’t change easily. I’m already syncing data to another 7 vdev server, but since this server is so busy, transfers are happening VERY SLOW (read, zfs sync doing 10MB/s).</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class="">Are you able to share your ZFS configuration and iSCSI configuration?<br class=""></div></div></blockquote><div class=""><br class=""></div></span>Sure! Here are zfs settings:</div><div class=""><br class=""></div><div class="">zfs get all data:</div><div class=""><div class="">NAME PROPERTY VALUE SOURCE</div><div class="">data type filesystem -</div><div class="">data creation Fri Oct 25 20:26 2013 -</div><div class="">data used 104T -</div><div class="">data available 61.6T -</div><div class="">data referenced 1.09M -</div><div class="">data compressratio 1.08x -</div><div class="">data mounted yes -</div><div class="">data quota none default</div><div class="">data reservation none default</div><div class="">data recordsize 128K default</div><div class="">data mountpoint /volumes/data received</div><div class="">data sharenfs off default</div><div class="">data checksum on default</div><div class="">data compression off received</div><div class="">data atime off local</div><div class="">data devices on default</div><div class="">data exec on default</div><div class="">data setuid on default</div><div class="">data readonly off local</div><div class="">data zoned off default</div><div class="">data snapdir hidden default</div><div class="">data aclmode discard default</div><div class="">data aclinherit restricted default</div><div class="">data canmount on default</div><div class="">data xattr on default</div><div class="">data copies 1 default</div><div class="">data version 5 -</div><div class="">data utf8only off -</div><div class="">data normalization none -</div><div class="">data casesensitivity sensitive -</div><div class="">data vscan off default</div><div class="">data nbmand off default</div><div class="">data sharesmb off default</div><div class="">data refquota none default</div><div class="">data refreservation none default</div><div class="">data primarycache all default</div><div class="">data secondarycache all default</div><div class="">data usedbysnapshots 0 -</div><div class="">data usedbydataset 1.09M -</div><div class="">data usedbychildren 104T -</div><div class="">data usedbyrefreservation 0 -</div><div class="">data logbias latency default</div><div class="">data dedup off local</div><div class="">data mlslabel none default</div><div class="">data sync standard default</div><div class="">data refcompressratio 1.00x -</div><div class="">data written 1.09M -</div><div class="">data logicalused 98.1T -</div><div class="">data logicalreferenced 398K -</div><div class="">data filesystem_limit none default</div><div class="">data snapshot_limit none default</div><div class="">data filesystem_count none default</div><div class="">data snapshot_count none default</div><div class="">data redundant_metadata all default</div><div class="">data nms:dedup-dirty on received</div><div class="">data nms:description datauporabnikov received</div><div class=""><br class=""></div><div class="">I’m not sure what iSCSI configuration do you want/need? But as far as I figured out in the last 'freeze', iSCSI is not the problem, since I’m unable to write to ZFS volume even if I’m local on the server itself.</div><span class=""><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class="">For iSCSI, can you take a look at this: <a href="http://docs.oracle.com/cd/E23824_01/html/821-1459/fpjwy.html#fsume" target="_blank" class="">http://docs.oracle.com/cd/E23824_01/html/821-1459/fpjwy.html#fsume</a><br class=""></div></div></blockquote><div class=""><br class=""></div></span>Interesting. I tried running 'iscsiadm list target' but it doesn’t return anything. There is also nothing in /var/adm/messages as usual:) But target service is online (according to svcs), clients are connected and having traffic.</div><div class=""><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class="">Do you have detailed logs for the clients experiencing the issues? If not are you able to enable verbose logging (such as debug level logs)?<br class=""></div></div></blockquote><div class=""><br class=""></div></span><div class="">I have clients logs, but they mostly just report loosing connections and reconnecting:</div><div class=""><br class=""></div><div class="">Example 1:</div><div class="">Apr 29 10:33:53 eee kernel: connection1:0: detected conn error (1021)<br class="">Apr 29 10:33:54 eee iscsid: Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)<br class="">Apr 29 10:33:56 eee iscsid: connection1:0 is operational after recovery (1 attempts)<br class="">Apr 29 10:36:37 eee kernel: connection1:0: detected conn error (1021)<br class="">Apr 29 10:36:37 eee iscsid: Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)<br class="">Apr 29 10:36:40 eee iscsid: connection1:0 is operational after recovery (1 attempts)</div><div class="">Apr 29 10:36:50 eee kernel: sd 3:0:0:0: Device offlined - not ready after error recovery<br class="">Apr 29 10:36:51 eee kernel: sd 3:0:0:0: Device offlined - not ready after error recovery<br class="">Apr 29 10:36:51 eee kernel: sd 3:0:0:0: Device offlined - not ready after error recovery</div><div class=""><br class=""></div><div class="">Example 2:</div><div class="">Apr 16 08:41:40 vf kernel: connection1:0: pdu (op 0x5e itt 0x1) rejected. Reason code 0x7<br class="">Apr 16 08:43:11 vf kernel: connection1:0: pdu (op 0x5e itt 0x1) rejected. Reason code 0x7<br class="">Apr 16 08:44:13 vf kernel: connection1:0: pdu (op 0x5e itt 0x1) rejected. Reason code 0x7<br class="">Apr 16 08:45:51 vf kernel: connection1:0: detected conn error (1021) Apr 16 08:45:51 317 iscsid: Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)<br class="">Apr 16 08:45:53 vf iscsid: connection1:0 is operational after recovery (1 attempts)</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">I’m already in contact with OmniTI regarding our new build, but in the mean time, I would love for our clients to be able to use the storage so I’m trying to resolve the current issue somehow…</div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">Matej</div><div class=""><br class=""></div><div class=""><br class=""></div></font></span></div></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></body></html>