<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hey there,<div class=""> </div><div class="">we were having the same problems with an old storage with SATA drives and a new one with SAS drives and all HW from Nexenta HCL list, so everything should work. Yet iSCSI target died every 2-6 days nonetheless. With the help of memory dump and Dan, it looks like, we managed to solve the issue.</div><div class=""> </div><div class="">In my case, one of the threads was waiting for memory to free. I remembered I hadn’t set max ARC sizeon my system, so it was possible for ARC to eat all my free memory and could not release it quick enough. After settings max ARC size, everything works for the last month. I still can’t say if this really fixed the problem or not, but so far so good:)</div><div class=""> </div><div class="">Matej</div><div class=""> </div><div class=""> <div><blockquote type="cite" class=""><div class="">On 13 Jan 2016, at 05:52, John Barfield <<a href="mailto:john.barfield@bissinc.com" class="">john.barfield@bissinc.com</a>> wrote:</div> <div class=""> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252" class=""> <meta content="text/html; charset=Windows-1252" class=""> <div class="" style="word-wrap:break-word"> <div class="">Oh I didnt catch that detail. </div> <div class=""> </div> <div class="">Okay well nevermind :)</div> <div class=""> </div> <div class=""> <div class="acompli_signature">Sent from <a href="https://aka.ms/qtex0l" class="">Outlook Mobile</a></div> </div> <div class="gmail_quote">On Tue, Jan 12, 2016 at 8:21 PM -0800, "Brian Hechinger" <<a href="mailto:wonko@4amlunch.net" target="_blank" class="">wonko@4amlunch.net</a>> wrote: </div> <div class="">In my case the SATA disks aren’t on the 1068E. <div class=""> </div> <div class="">-brian</div> <div class=""> <div class=""> <blockquote type="cite" class=""> <div class="">On Jan 12, 2016, at 11:19 PM, John Barfield <<a href="mailto:john.barfield@bissinc.com" class="">john.barfield@bissinc.com</a>> wrote:</div> <div class=""> <div id="compose" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; padding-left:16px; padding-right:16px; padding-bottom:8px"> <div class="">BTW I left off that it has the same LSI controller chipset <div class="acompli_signature">Sent from <a href="https://aka.ms/qtex0l" class="">Outlook Mobile</a></div> </div> </div> <div class="gmail_quote" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px"> _____________________________ From: John Barfield <<a href="mailto:john.barfield@bissinc.com" class="">john.barfield@bissinc.com</a>> Sent: Tuesday, January 12, 2016 10:17 PM Subject: Re: [OmniOS-discuss] [discuss] COMSTAR hanging To: <<a href="mailto:discuss@lists.illumos.org" class="">discuss@lists.illumos.org</a>>, omnios-discuss <<a href="mailto:omnios-discuss@lists.omniti.com" class="">omnios-discuss@lists.omniti.com</a>> <div class=""> <div class="">My input may or may not be valid but Im going to throw it out there anyway :)</div> <div class=""> </div> <div class="">do you have any Mpt disconnect errors in /var/adm/messages? </div> <div class=""> </div> <div class="">Also do you have smartmontools installed? </div> <div class=""> </div> <div class="">I ran into similiar issues just booting a sunfire x4540 recently off of OmniOS live, i/o would just hang while probing device nodes.</div> <div class=""> </div> <div class="">I found the drive that was acting up and pulled it. </div> <div class=""> </div> <div class="">All of a sudden everything miraculously worked amazing. </div> <div class=""> </div> <div class="">I compiled smartmontools after I got it to boot and found 10 drives out of 48 with bad sectors in prefail state.</div> <div class=""> </div> <div class="">I dont know if this happens with SAS drives or not but Im using SATA and saw this was a common issue in old opensolaris threads. -barfield <div class="x_acompli_signature">Sent from <a href="https://aka.ms/qtex0l" class="">Outlook Mobile</a></div> </div> <div class="x_gmail_quote">On Tue, Jan 12, 2016 at 8:08 PM -0800, "Brian Hechinger" <<a href="mailto:wonko@4amlunch.net" class="">wonko@4amlunch.net</a>> wrote: </div> </div> <div class="PlainText">In the meantime I’ve removed the SLOG and L2ARC just in case. I don’t think that’s it though. At least will have some sort of data point to work with here. :) -brian > On Jan 12, 2016, at 10:55 PM, Brian Hechinger <<a href="mailto:wonko@4amlunch.net" class="">wonko@4amlunch.net</a>> wrote: > > Ok, it has happened. > > Checking this here, the pool seems to be fine. I can read and write files. > > except ‘zpool status’ is now currently hanging. I can still read/write from the pool, however. > > I can telnet to port 3260, but restarting target services has hung. > > root@basket1:/tank/Share# svcs -a | grep stmf > online Jan_05 svc:/system/stmf:default > root@basket1:/tank/Share# svcs -a | grep target > disabled Jan_05 svc:/system/fcoe_target:default > online Jan_05 svc:/network/iscsi/target:default > online Jan_05 svc:/system/ibsrp/target:default > root@basket1:/tank/Share# svcadm restart /system/ibsrp/target > root@basket1:/tank/Share# svcadm restart /network/iscsi/target > root@basket1:/tank/Share# svcadm restart /system/stmf > root@basket1:/tank/Share# svcs -a | grep target > disabled Jan_05 svc:/system/fcoe_target:default > online* 22:43:03 svc:/system/ibsrp/target:default > online* 22:43:13 svc:/network/iscsi/target:default > root@basket1:/tank/Share# svcs -a | grep stmf > online* 22:43:18 svc:/system/stmf:default > root@basket1:/tank/Share# > > I’m doing a crash dump reboot. I’ll post the output somewhere. > > The output of echo '$<threadlist' | mdb -k is attached. > > <threadlist.out> > >> On Jan 8, 2016, at 3:11 PM, Matej Zerovnik <<a href="mailto:matej@zunaj.si" class="">matej@zunaj.si</a>> wrote: >> >> Is the pool usable during comstar hang? >> Can you write and read from the pool (test both, in my case, when pool froze, I wasn’t able to write to the pool, but I could read). >> >> Again, this might not be connected with Comstar, but in my case, Comstar and pool hang were exchanging. >> >> Matej >> >>> On 08 Jan 2016, at 20:11, Brian Hechinger <<a href="mailto:wonko@4amlunch.net" class="">wonko@4amlunch.net</a>> wrote: >>> >>> Yeah, I’m using the 1068E to boot from (this has been supported since before Illumos) but that doesn’t have anything accessed by COMSTAR. >>> >>> It’s the ICH10R SATA that hosts the disks that COMSTAR shares out space from. >>> >>> -brian >>> >>>> On Jan 8, 2016, at 1:31 PM, Richard Jahnel <<a href="mailto:rjahnel@ellipseinc.com" class="">rjahnel@ellipseinc.com</a>> wrote: >>>> >>>> First off, love SuperMicro good choice IMHO. >>>> >>>> This board has two on board controllers. >>>> >>>> LSI SAS1068E (not 100% sure there are working illumos drivers for this one) >>>> >>>> And >>>> >>>> Intel ICH10R SATA (So I'm guessing your using this one.) >>>> >>>> -----Original Message----- >>>> From: OmniOS-discuss [ <a href="mailto:omnios-discuss-bounces@lists.omniti.com" class="">mailto:omnios-discuss-bounces@lists.omniti.com</a>] On Behalf Of Brian Hechinger >>>> Sent: Friday, January 08, 2016 12:16 PM >>>> To: Matej Zerovnik <<a href="mailto:matej@zunaj.si" class="">matej@zunaj.si</a>> >>>> Cc: omnios-discuss <<a href="mailto:omnios-discuss@lists.omniti.com" class="">omnios-discuss@lists.omniti.com</a>> >>>> Subject: Re: [OmniOS-discuss] [discuss] COMSTAR hanging >>>> >>>> >>>>> Which controller exactly do you have? >>>> >>>> Whatever ACHI stuff is built into the motherboard. Motherboard is X8DTL-3F. >>>> >>>>> Do you know firmware version? >>>> >>>> I’m assuming this is linked to the BIOS version? >>>> >>>>> Which hard drives? >>>> >>>> Hitachi-HUA723030ALA640-MKAOAA50-2.73TB >>>> >>>>> It might not tell much, but it’s good to have as much information as possible. >>>>> >>>>> When comstar hangs, can you telnet to the iSCSI port? >>>>> What does svcs says, is the service running? >>>>> What happens in you try to restart it? >>>>> How do you restart it? >>>> >>>> I’ll try all these things next time. >>>> >>>>> In my case, svcs reported service running, but when I tried to telnet, there was no connection as well as there was no listening port opened when checking with 'netstat -an'. If I tried to restart target and stmf service, but stmf service got stucked in online* state and would not start. Reboot was the only solution in my case, but as I said, latest 014 release is working OK (but then again, load got reduced). >>>> >>>> All good info. Thanks! >>>> >>>> -brian >>>> >>>>> >>>>> Matej >>>>> >>>>>> On 08 Jan 2016, at 17:50, Dave Pooser <<a href="mailto:dave-oo@pooserville.com" class="">dave-oo@pooserville.com</a>> wrote: >>>>>> >>>>>>>> On Jan 8, 2016, at 11:22 AM, Brian Hechinger <<a href="mailto:wonko@4amlunch.net" class="">wonko@4amlunch.net</a>> wrote: >>>>>>>> >>>>>>>> No, ZFS raid10 >>>>>>> >>>>>>> Saw the HW-RAID term, and got concerned. That's what, raidz2 in ZFS-ese? >>>>>> >>>>>> It's a zpool with multiple mirror vdevs. >>>>>> >>>>>> -- >>>>>> Dave Pooser >>>>>> Cat-Herder-in-Chief, <a href="http://pooserville.com/" class="">Pooserville.com</a> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OmniOS-discuss mailing list >>>>>> <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> >>>>>> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> >>>>> >>>>> _______________________________________________ >>>>> OmniOS-discuss mailing list >>>>> <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> >>>>> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> >>>> >>>> _______________________________________________ >>>> OmniOS-discuss mailing list >>>> <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> >>>> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> >>> >> > <a href="http://www.listbox.com/" class="">http://www.listbox.com</a> </div> <div bgcolor="#ffffff" id="listbox-footer" class="" style="width:auto; margin:0px; padding:5px; background-color:rgb(255,255,255); clear:both; border-top-width:1px; border-top-style:solid; border-top-color:rgb(204,204,204)"> <table bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" width="100%" class="" style="background-color:rgb(255,255,255)"> <tbody class=""> <tr class=""> <td class="">illumos-discuss | <a href="https://www.listbox.com/member/archive/182180/=now" title="Go to archives for illumos-discuss" class="" style="text-decoration:none; color:rgb(102,153,51); border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(68,68,68)">Archives</a> <a border="0" href="https://www.listbox.com/member/archive/rss/182180/26677440-40b316d8" title="RSS feed for illumos-discuss" class="" style="text-decoration:none; color:rgb(102,153,51)"><img border="0" src="http://postlink.www.listbox.com/2033704/833487e62783d55fe81f119fb93ef644/26677440/3044d385.jpg?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2ZlZWQtaWNvbi0xMHgxMC5qcGc" class=""></a> | <a href="https://www.listbox.com/member/?member_id=26677440&id_secret=26677440-8fd7f4fe" title="" class="" style="text-decoration:none; color:rgb(102,153,51); border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(68,68,68)">Modify</a> Your Subscription</td> <td align="right" valign="top" class=""><a href="http://www.listbox.com/" class="" style="border-bottom-style:none"><img border="0" src="http://postlink.www.listbox.com/2033705/3379085af0f1cf7fc3708f04b4471ae2/26677440/3044d385.png?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2xpc3Rib3gtbG9nby1zbWFsbC5wbmc" title="Powered by Listbox" class=""></a></td> </tr> </tbody> </table> </div> </div> <span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important">_______________________________________________<br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px"> <span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important">OmniOS-discuss mailing list<br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px"> <span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important"><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px"> <span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important"><a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px"> </div> </blockquote> </div> </div> </div> </div> _______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a> http://lists.omniti.com/mailman/listinfo/omnios-discuss </div></blockquote></div> </div></body></html>