<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Title" content=""> <meta name="Keywords" content=""> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body bgcolor="white" lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> I did compile it. It seems to have fixed the deadlocsk but we’re still having performance issues that we were not having before. IMHO this is some type of regression. <o:p></o:p> <o:p> </o:p> Here is the binary location:<o:p></o:p> <o:p> </o:p> <a href="http://download.txtelecom.net/zfs">http://download.txtelecom.net/zfs</a><o:p></o:p> <o:p> </o:p> Install procedure (As root):<o:p></o:p> (Props to Dan McDonald for helping me with this process)<o:p></o:p> <o:p> </o:p> <![if !supportLists]>· <![endif]>beadm list (Note your current BE)<o:p></o:p> <![if !supportLists]>· <![endif]>zfs snap rpool@WHATEVER<o:p></o:p> <![if !supportLists]>· <![endif]>beadm create OmniOS-r151020-cstm<o:p></o:p> <![if !supportLists]>· <![endif]>beadm mount OmniOS-r151020-cstm /mnt<o:p></o:p> <![if !supportLists]>· <![endif]>cd /mnt/kernel/fs/amd64/<o:p></o:p> <![if !supportLists]>· <![endif]>mv zfs zfs.old<o:p></o:p> <![if !supportLists]>· <![endif]>wget download.txtelecom.net/zfs<o:p></o:p> <![if !supportLists]>· <![endif]>cd ~<o:p></o:p> <![if !supportLists]>· <![endif]>beadm unmount /mnt<o:p></o:p> <![if !supportLists]>· <![endif]>beadm activate OmniOS-r151020<o:p></o:p> <![if !supportLists]>· <![endif]>shutdown –i6 –g0<o:p></o:p> <o:p> </o:p> <o:p> </o:p> The system will reboot twice…the first SAN I did this on rebooted both times flawlessly. On the Second system it froze while shutting down and we had to power cycle it. But it still came back online using the new build. Use this at your own risk. <o:p></o:p> <o:p> </o:p> If it does fail completely simply choose your old BE on next reboot within grub. (Step 1 you’ll have a list)<o:p></o:p> <o:p> </o:p> <div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in"> However, please see the following comments from my customer which are still occurring:<o:p></o:p> </div> <o:p> </o:p> John,<o:p></o:p> I’ve copied 6GB, 37GB, and 61GB files from the thor /mnt local disk to a location on the San.<o:p></o:p> Monitoring only shell activity on a few machines, nothing locked up (except thor). Although trying a ssh to thor took a minute to connect.<o:p></o:p> While the ‘cp’ command was working, I monitor thor using ‘top’ command.<o:p></o:p> * Of course, the load factor started to rise from 1 to over 5 on thor.<o:p></o:p> * As the file was copying, I see the destination file size increase.<o:p></o:p> * However, when the destination file size reached 6 or 37 or 61GB, thor’s load factor is still very high and remained high for a while.<o:p></o:p> * If logged into thor, no work can be done while the ‘cp’ is ongoing.<o:p></o:p> * Only after the ‘cp’ command was completed did thor’s load factor start to drop and work could be done if logged into thor.<o:p></o:p> * Other machines, e.g. flash, linux8, had no issues. . . as far as I could tell.<o:p></o:p> <div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in"> * I did not invoke any of the EDA tools.<o:p></o:p> </div> <o:p> </o:p> Additionally the VMware datastore on NFS(ZFS) bug that I was hoping to fix was only ½ resolved as well. <o:p></o:p> <o:p> </o:p> We can delete large files now but it brings the SAN to a crawl. Better than it used to be when it would come to a halt. <o:p></o:p> <o:p> </o:p> I’d love to see if I could get a kernel hacker to look at both of these issues with us. <o:p></o:p> <o:p> </o:p> <div> John Barfield<o:p></o:p> Engineering and Stuff <o:p> </o:p> <o:p> </o:p> M: +1 (214) 425-0783 O: +1 (214) 506-8354 <o:p></o:p> <a href="mailto:john.barfield@bissinc.com">john.barfield@bissinc.com</a><o:p></o:p> <o:p> </o:p> <img border="0" width="72" height="28" id="_x0000_i1026" src="cid:image001.png@01D2B829.5033AEE0"><o:p></o:p> 4925 Greenville Ave, Ste 900<o:p></o:p> Dallas, TX 75206<o:p></o:p> <o:p> </o:p> For Support Requests:<o:p></o:p> <a href="http://support.bissinc.com/">http://support.bissinc.com</a> or <a href="mailto:support@bissinc.com">support@bissinc.com</a><o:p></o:p> <o:p> </o:p> Follow us on Twitter for Network Status & Updates! <o:p></o:p> <o:p> </o:p> </div> <a href="https://twitter.com/johnbarfield"><img border="0" width="24" height="24" id="_x0000_i1025" src="cid:image002.gif@01D2B829.5033AEE0"></a> <o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: wuffers <moo@wuffers.net> Date: Tuesday, April 18, 2017 at 9:38 AM To: Dan McDonald <danmcd@omniti.com> Cc: John Barfield <john.barfield@bissinc.com>, "omnios-discuss@lists.omniti.com" <omnios-discuss@lists.omniti.com> Subject: Re: [OmniOS-discuss] ZPOOL bug after upgrade to r151020<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> I upgraded to r151020 in late Jan, and saw some strangeness with arcstat (l2size and l2asize were huge) before I did a reboot due to some instability a few weeks ago. I thought it was just a case of not using the latest arcstat, and things were running fine after a reboot so didn't pursue it. <o:p></o:p> <div> <o:p> </o:p> </div> <div> I saw this post last week, and confirmed it was within my environment, so did the remove/re-add of the cache devices, then a complete reboot as well. The cache devices reported back their actual proper size (400GB) via "zpool iostat -v". Today I checked it again and this is what I see:<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> # arcstat<o:p></o:p> </div> <div> <div> read hits miss hit% l2read l2hits l2miss l2hit% arcsz l2size l2asize<o:p></o:p> </div> <div> 465 412 53 88 53 50 3 94 230G 4.4T 3.2T<o:p></o:p> </div> </div> <div> <o:p> </o:p> </div> <div> # zpool iostat -v<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> (other info snipped for brevity)<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> <div> cache - - - - - -<o:p></o:p> </div> <div> c2t500117310015D579d0 816G 16.0E 54 23 2.32M 1.46M<o:p></o:p> </div> <div> c2t50011731001631FDd0 816G 16.0E 54 23 2.32M 1.46M<o:p></o:p> </div> <div> c12t500117310015D59Ed0 815G 16.0E 55 23 2.35M 1.46M<o:p></o:p> </div> <div> c12t500117310015D54Ed0 816G 16.0E 55 23 2.36M 1.46M<o:p></o:p> </div> </div> <div> <o:p> </o:p> </div> <div> <o:p> </o:p> </div> <div> I'm just waiting for the next lockup/crash..<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> John, were you able to compile the fix, and if so, be able send me a copy? <o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Thanks.<o:p></o:p> </div> <div> <o:p> </o:p> </div> </div> <div> <o:p> </o:p> <div> On Mon, Apr 10, 2017 at 10:00 AM, Dan McDonald <<a href="mailto:danmcd@omniti.com" target="_blank">danmcd@omniti.com</a>> wrote:<o:p></o:p> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> > On Apr 9, 2017, at 10:27 PM, John Barfield <<a href="mailto:john.barfield@bissinc.com">john.barfield@bissinc.com</a>> wrote: > > Thank you Dan. > > Do you happen to have the process or know the location of a process document for only building ZFS? > > Ive re-built only nfs from illumos-gate in the past to resolve a bug but im wondering how I would build and install only zfs. (if its even possible). > > There are 2 bugs that we're suffering with at two different customer sites that didnt get into r151020 and Im not sure that we can make it till r151022 is released. > > Thanks for any advice You can build zfs the way you likely built NFS. Build it, replace it on an alternate BE (in zfs's case: /kernel/fs/amd64/zfs), and reboot. The only gotcha might be if a bugfix covers more than just ZFS itself... but for 7504, that's NOT the case. :)<o:p></o:p> <div> <div> Dan _______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><o:p></o:p> </div> </div> </blockquote> </div> <o:p> </o:p> </div> </div> </body> </html>