From skeltonr at btconnect.com Fri Sep 1 23:04:07 2017 From: skeltonr at btconnect.com (Richard Skelton) Date: Sat, 02 Sep 2017 00:04:07 +0100 Subject: [OmniOS-discuss] ZFS cache device allocation a free seem strange on my OmniOS r151022 system. Message-ID: <59A9E767.1030608@btconnect.com> Hi, The cache allocation a free seem strange on my OmniOS r151022 system. The cache device is an Intel SSD DC P3700 Series 800GB, 1/2 Height PCIe 3.0, 20nm, MLC root at omnios:/scratch# zpool list -v scratch NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT scratch 2.66T 1.94T 738G - 17% 72% 1.00x ONLINE - mirror 544G 395G 149G - 16% 72% c7t5000CCA018309809d0 - - - - - - c18t5000CCA038718395d0 - - - - - - mirror 544G 396G 148G - 15% 72% c14t5000CCA038710745d0 - - - - - - c11t5000CCA038721F5Dd0 - - - - - - mirror 544G 396G 148G - 18% 72% c12t5000CCA03871048Dd0 - - - - - - c13t5000CCA0387102E1d0 - - - - - - mirror 544G 398G 146G - 19% 73% c15t5000CCA03871058Dd0 - - - - - - c16t5000CCA0387224EDd0 - - - - - - mirror 544G 396G 148G - 17% 72% c17t5000CCA038717F65d0 - - - - - - c10t5000CCA038717AC9d0 - - - - - - log - - - - - - c6t0d0 3.84G 0 3.84G - 0% 0% cache - - - - - - c3t1d0 745G *1.07T 16.0E - 0% 147%* spare - - - - - - c9t5000CCA038717B2Dd0 - - - - - - c8t5000CCA0387104B1d0 - - - - - - root at omnios:/scratch# format /pci at 0,0/pci8086,35c5 at 1f,2/disk at 4,0 1. c2t5d0 /pci at 0,0/pci8086,35c5 at 1f,2/disk at 5,0 2. c3t1d0 /pci at 75,0/pci8086,6f0a at 3,2/pci8086,3702 at 0/blkdev at 1,0 3. c6t0d0 /pci at 75,0/pci8086,6f06 at 2,2/pci19e3,81 at 0/sd at 0,0 Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From an3s.annema at gmail.com Mon Sep 4 18:17:36 2017 From: an3s.annema at gmail.com (Andries Annema) Date: Mon, 4 Sep 2017 20:17:36 +0200 Subject: [OmniOS-discuss] pkglint: Python 2.7 MemoryError In-Reply-To: <52dddb24-3ebf-4de1-36b8-c88715c8b0b0@gmail.com> References: <52dddb24-3ebf-4de1-36b8-c88715c8b0b0@gmail.com> Message-ID: <2835e1ba-390e-31c0-c6ec-f0530b9053d4@gmail.com> Finally found the time to solve this issue. And actually, it is a little embarrassing. Category "user error". Anyway, for the record, here is the cause and the solution. While I did give the VM 1, 2 and even 4 GB of RAM to play with, I tend to deploy my zones with a memory cap. In this case, I had this zone set to only use 256MB which was clearly too little. But, when I increased RAM to the VM, I forgot to also increase the memory cap on the zone.... D'oh! Increased it to 512MB and bam! No "MemoryError" anymore whatsoever. Andries On 2017-07-13 19:41, Andries Annema wrote: > Hi guys, > > I've got a freshly installed r151022 VM here (under VMWare > Workstation) to fiddle around a bit. Very clean, no extra packages > installed yet, not even NFS- or SMB-shares. Only two non-global zones: > one to host a repository and one as build environment for some > packages I use myself. > I have done all this on r151014 machines (VM/test as well as > physical/production) as well, but have never bumped into the error I'm > getting now. > > Not sure if it is package-specific or more of a general error - I > assume the latter - but for completeness sake, I'm trying to put > Znapzend 0.17.0 into a neat IPS packages. > .configure, gmake, gmake install, as well as creating the SMF method > and manifest files, and the IPS package manifest go just like I got > before on r151014, but now on r151022 the "pkglint" process fails. > This is what it throws at me: > > ==begin quote== > builder at vm09ngz02build:/tank/build/1_scratch$ sudo pkglint -c > ./lint-cache -r http:// znapzend.p5m.3.res > Lint engine setup... > > Ignoring -r option, existing image found. > Lint setup 3?????????????????????????????? 1485/1838Traceback (most > recent call last): > ? File "/usr/bin/pkglint", line 317, in > ??? __ret = main_func() > ? File "/usr/bin/pkglint", line 148, in main_func > ??? release=opts.release) > ? File "/usr/lib/python2.7/vendor-packages/pkg/lint/engine.py", line > 607, in setup > ??? checker.startup(self) > ? File > "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line > 174, in startup > ??? seed_dict(manifest, "path", self.ref_paths) > ? File > "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line > 126, in seed_dict > ??? for action in mf_gen(atype): > ? File > "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line > 123, in mf_gen > ??? for a in mf.gen_actions(): > ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line > 1898, in gen_actions > ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line > 1563, in __load > ??? self.set_content(excludes=self.excludes, pathname=self.pathname) > ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line > 1052, in set_content > ??? self.add_action(action, excludes) > ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line > 1094, in add_action > ??? self.actions.append(action) > MemoryError > Error: > > This is an internal error in pkg(5) version 1493165709.? Please log a > Service Request about this issue including the information above and this > message. > ==end quote== > > Any ideas? Am I doing something wrong or is there a bug in Python 2.7 > or even pkg(5)? > Assuming it simply had not enough memory to get the job done, I tried > increasing the VM's assigned memory from 1 to 2, even to 4GB. But that > didn't help. > > This proves I'm still a novice on compiling and maintaining > packages... *sigh* > Anyway, any hints appreciated! Tnx. > > Regards, > Andries > > > From johan.kragsterman at capvert.se Tue Sep 5 07:29:40 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Tue, 5 Sep 2017 09:29:40 +0200 Subject: [OmniOS-discuss] email notifications from SMF and FMA Message-ID: An HTML attachment was scrubbed... URL: From doug at will.to Tue Sep 5 12:28:19 2017 From: doug at will.to (Doug Hughes) Date: Tue, 5 Sep 2017 08:28:19 -0400 Subject: [OmniOS-discuss] email notifications from SMF and FMA In-Reply-To: References: Message-ID: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent. Sent from my android device. -----Original Message----- From: Johan Kragsterman To: omnios-discuss Sent: Tue, 05 Sep 2017 3:44 Subject: [OmniOS-discuss] email notifications from SMF and FMA Hi! I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials. I know how to use the SMF service to send the mail. I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it? But the rest...? I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail. Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Tue Sep 5 14:23:15 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Tue, 5 Sep 2017 16:23:15 +0200 Subject: [OmniOS-discuss] Ang: Re: Ang: Re: email notifications from SMF and FMA In-Reply-To: <05bdf145-bd95-e97c-fb3a-0d1f5ed7454b@will.to> References: <05bdf145-bd95-e97c-fb3a-0d1f5ed7454b@will.to>, <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> Message-ID: Hi! Of coarse, forgot to include list, haven't been using this in a while, therefore a bit rusty... -----Doug Hughes skrev: ----- Till: Johan Kragsterman Fr?n: Doug Hughes Datum: 2017-09-05 15:44 ?rende: Re: Ang: Re: [OmniOS-discuss] email notifications from SMF and FMA Perhaps you meant to reply-all? I don't know what SMF uses for sure, but I'd be surprised if it was not the local mailer. nullmailer seems like it would be a good choice, though it would require some porting. postfix in relay mode would be very low overhead. It is not really the overhead I am worrying about, it is an extra service that isn't needed, I want to remove every possible piece that can cause failure. I'm sure someone on this list knows what SMF uses. And as well how to send mail from SMF to remote hosts as well, including auth. BR Johan On 9/5/2017 9:07 AM, Johan Kragsterman wrote: > Hi! > > -----Doug Hughes skrev: ----- > Till: omnios-discuss , Johan Kragsterman > Fr?n: Doug Hughes > Datum: 2017-09-05 14:28 > ?rende: Re: [OmniOS-discuss] email notifications from SMF and FMA > > - wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an - > - outbound relay agent. > > > > > > Yes, of coarse, but this is a SAN array controller, and if it is something I want to avoid, it is to run unnecessary services on a machine like that. I want to keep it as clean as possible. > > By the way, do someone know what SMF is using for sending mail? Is it good ol' mailx, or...? > > If I knew where to look, I'm sure I could fix it, since I know how to send mail with mailx to remote hosts. > > > > BR Johan > > > > > > > > Sent from my android device. > > -----Original Message----- > From: Johan Kragsterman > To: omnios-discuss > Sent: Tue, 05 Sep 2017 3:44 > Subject: [OmniOS-discuss] email notifications from SMF and FMA > > Hi! > > > I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials. > > I know how to use the SMF service to send the mail. > > I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it? > > But the rest...? > > I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail. > > > Best regards from/Med v?nliga h?lsningar fr?n > > Johan Kragsterman > > Capvert > > > From rjahnel at ellipseinc.com Tue Sep 5 15:03:33 2017 From: rjahnel at ellipseinc.com (Richard Jahnel) Date: Tue, 5 Sep 2017 15:03:33 +0000 Subject: [OmniOS-discuss] ZFS cache device allocation a free seem strange on my OmniOS r151022 system. In-Reply-To: <59A9E767.1030608@btconnect.com> References: <59A9E767.1030608@btconnect.com> Message-ID: <65DC5816D4BEE043885A89FD54E273FC781396AF@MAIL101.Ellipseinc.com> Yes, this issue has been lurking around since they made the arcs compressible. I really wish they would revert that change. There have been at least 3 illumos issues filed for this and two fixes attempted. So far it still persist. For now I have my cache drives removed from my production pools. From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Richard Skelton Sent: Friday, September 1, 2017 6:04 PM To: omniOS-discuss Subject: [OmniOS-discuss] ZFS cache device allocation a free seem strange on my OmniOS r151022 system. Hi, The cache allocation a free seem strange on my OmniOS r151022 system. The cache device is an Intel SSD DC P3700 Series 800GB, 1/2 Height PCIe 3.0, 20nm, MLC root at omnios:/scratch# zpool list -v scratch NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT scratch 2.66T 1.94T 738G - 17% 72% 1.00x ONLINE - mirror 544G 395G 149G - 16% 72% c7t5000CCA018309809d0 - - - - - - c18t5000CCA038718395d0 - - - - - - mirror 544G 396G 148G - 15% 72% c14t5000CCA038710745d0 - - - - - - c11t5000CCA038721F5Dd0 - - - - - - mirror 544G 396G 148G - 18% 72% c12t5000CCA03871048Dd0 - - - - - - c13t5000CCA0387102E1d0 - - - - - - mirror 544G 398G 146G - 19% 73% c15t5000CCA03871058Dd0 - - - - - - c16t5000CCA0387224EDd0 - - - - - - mirror 544G 396G 148G - 17% 72% c17t5000CCA038717F65d0 - - - - - - c10t5000CCA038717AC9d0 - - - - - - log - - - - - - c6t0d0 3.84G 0 3.84G - 0% 0% cache - - - - - - c3t1d0 745G 1.07T 16.0E - 0% 147% spare - - - - - - c9t5000CCA038717B2Dd0 - - - - - - c8t5000CCA0387104B1d0 - - - - - - root at omnios:/scratch# format /pci at 0,0/pci8086,35c5 at 1f,2/disk at 4,0 1. c2t5d0 /pci at 0,0/pci8086,35c5 at 1f,2/disk at 5,0 2. c3t1d0 /pci at 75,0/pci8086,6f0a at 3,2/pci8086,3702 at 0/blkdev at 1,0 3. c6t0d0 /pci at 75,0/pci8086,6f06 at 2,2/pci19e3,81 at 0/sd at 0,0 Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From matjaz85 at gmail.com Tue Sep 5 17:21:10 2017 From: matjaz85 at gmail.com (=?utf-8?B?TWF0amHFviBN?=) Date: Tue, 5 Sep 2017 19:21:10 +0200 Subject: [OmniOS-discuss] email notifications from SMF and FMA In-Reply-To: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> References: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> Message-ID: The easiest way to send an email from CLI or crontab I found is Heirloom mailx. You can find it here: http://heirloom.sourceforge.net/mailx.html You will need to make/make install yourself. And then you can use info on how to set it up from here: https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/ BR, Matjaz > On 5. sep. 2017, at 14:28, Doug Hughes wrote: > > wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent. > > Sent from my android device. > > -----Original Message----- > From: Johan Kragsterman > To: omnios-discuss > Sent: Tue, 05 Sep 2017 3:44 > Subject: [OmniOS-discuss] email notifications from SMF and FMA > > Hi! > > > I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials. > > I know how to use the SMF service to send the mail. > > I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com :465, would it? > > But the rest...? > > I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail. > > > Best regards from/Med v?nliga h?lsningar fr?n > > Johan Kragsterman > > Capvert > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Tue Sep 5 17:30:02 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Tue, 5 Sep 2017 19:30:02 +0200 Subject: [OmniOS-discuss] Ang: Re: email notifications from SMF and FMA In-Reply-To: References: , <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> Message-ID: Hi! Thanks, Matjaz, but I wanted to use the SMF service: svc:/system/fm/notify-params:default. I just wondered how I would configure the mail address(in SMF) to be able to send to remote mail host with auth. BR Johan Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Matjaž M skrev: ----- Till: Doug Hughes Fr?n: Matjaž M Datum: 2017-09-05 19:21 Kopia: omnios-discuss , Johan Kragsterman ?rende: Re: [OmniOS-discuss] email notifications from SMF and FMA The easiest way to send an email from CLI or crontab I found is Heirloom mailx. You can find it here: http://heirloom.sourceforge.net/mailx.html You will need to make/make install yourself. And then you can use info on how to set it up from here: https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/ BR, Matjaz On 5. sep. 2017, at 14:28, Doug Hughes wrote: wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent. Sent from my android device. -----Original Message----- From: Johan Kragsterman To: omnios-discuss Sent: Tue, 05 Sep 2017 3:44 Subject: [OmniOS-discuss] email notifications from SMF and FMA Hi! I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials. I know how to use the SMF service to send the mail. I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it? But the rest...? I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail. Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From jimklimov at cos.ru Tue Sep 5 17:39:28 2017 From: jimklimov at cos.ru (Jim Klimov) Date: Tue, 05 Sep 2017 17:39:28 +0000 Subject: [OmniOS-discuss] email notifications from SMF and FMA In-Reply-To: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> References: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost> Message-ID: <7211FA9F-2A6A-4F10-A138-4DF774C5B5CF@cos.ru> On September 5, 2017 2:28:19 PM GMT+02:00, Doug Hughes wrote: >wouldn't it be easier to use your local MTA (postfix, sendmail) to do >this? Then SMF just has to execute /bin/mail or equivalent and your MTA >can act as an outbound relay agent. > >Sent from my android device. > >-----Original Message----- >From: Johan Kragsterman >To: omnios-discuss >Sent: Tue, 05 Sep 2017 3:44 >Subject: [OmniOS-discuss] email notifications from SMF and FMA > >Hi! > > >I need to send mails from SMF to a remote SMTP server, and want/need to >include auth credentials. > > >I know how to use the SMF service to send the mail. > > >I know how to get the user/passwd into encode_base64, but how do I pass >it on to the mail? As well as the port. I guess the port would look >like this: root at mailserver.com:465, would it? > > >But the rest...? > > >I've seen Peter Tribble been blogging about these features, but not >exactly how it would look like to pass on the credentials to your mail. > > >Best regards from/Med v?nliga h?lsningar fr?n > >Johan Kragsterman > > >Capvert Not sure: you're most interested in such notifications when something has failed. This state can preclude running an external submission program on the local system, or having it queue-dequeue your message. But generally (e.g. to forward user mails or logs/events summaries posted from app servers and other systems especially in highly volatile development setups where connectivity and remote services are not 100% granted) it is often convenient to have dumb programs post to localhost:25 and a real MTA/MSA repost that - often to a common org mail relay that is properly known in worldwide DNS and has secure setup to pass offsite recipients' antispams. Jim -- Typos courtesy of K-9 Mail on my Android From stephan.budach at jvm.de Thu Sep 7 08:33:54 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Thu, 7 Sep 2017 10:33:54 +0200 (CEST) Subject: [OmniOS-discuss] COMSTAR and blocksizes In-Reply-To: <1413021884.656.1504772876331.JavaMail.stephan.budach@stephanbudach.local> Message-ID: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> Hi, I am having trouble getting an issue sorted out, where omniOS 151020 complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, like this: Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! These COMSTAR LUNs are configured to export a blocksize 8k like this: LU Name: 600144F0564D504F4F4C3037534C3132 Operational Status: Online Provider Name : sbd Alias : nfsvmpool07Slot12 View Entry Count : 1 Data File : /dev/rdsk/c3t50015178F364A264d0p1 Meta File : not set Size : 200042414080 Block Size : 8192 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Enabled Access State : Active Now, the system seems to recognize the 8k, but for whatever reason, doesn't adjust the block size accordingly. It does that for the 4k LUNs, however, so I am unsure, on how to tackle this? Any tipp, anyone could share? Thanks, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From johan.kragsterman at capvert.se Thu Sep 7 10:08:45 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Thu, 7 Sep 2017 12:08:45 +0200 Subject: [OmniOS-discuss] Ang: COMSTAR and blocksizes In-Reply-To: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> Message-ID: Hi! -----"OmniOS-discuss" skrev: ----- Till: omnios-discuss at lists.omniti.com Fr?n: Stephan Budach S?nt av: "OmniOS-discuss" Datum: 2017-09-07 10:38 ?rende: [OmniOS-discuss] COMSTAR and blocksizes Hi, I am having trouble getting an issue sorted out, where omniOS 151020 complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, like this: Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! These COMSTAR LUNs are configured to export a blocksize 8k like this: LU Name: 600144F0564D504F4F4C3037534C3132 Operational Status: Online Provider Name : sbd Alias : nfsvmpool07Slot12 View Entry Count : 1 Data File : /dev/rdsk/c3t50015178F364A264d0p1 Meta File : not set Size : 200042414080 Block Size : 8192 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Enabled Access State : Active Now, the system seems to recognize the 8k, but for whatever reason, doesn't adjust the block size accordingly. It does that for the 4k LUNs, however, so I am unsure, on how to tackle this? Any tipp, anyone could share? Thanks, Stephan What's in the bottom here? Data File : /dev/rdsk/c3t50015178F364A264d0p1 It dosesn't look like a zvol? /Johan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From stephan.budach at jvm.de Thu Sep 7 10:22:06 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Thu, 7 Sep 2017 12:22:06 +0200 (CEST) Subject: [OmniOS-discuss] Ang: COMSTAR and blocksizes In-Reply-To: References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> Message-ID: <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local> ----- Urspr?ngliche Mail ----- > Von: "Johan Kragsterman" > An: "Stephan Budach" > CC: omnios-discuss at lists.omniti.com > Gesendet: Donnerstag, 7. September 2017 12:08:45 > Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes > > > Hi! > > -----"OmniOS-discuss" > skrev: ----- > Till: omnios-discuss at lists.omniti.com > Fr?n: Stephan Budach > S?nt av: "OmniOS-discuss" > Datum: 2017-09-07 10:38 > ?rende: [OmniOS-discuss] COMSTAR and blocksizes > > Hi, > > I am having trouble getting an issue sorted out, where omniOS 151020 > complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, > like this: > > Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned with > 8192 disk sector size in 10 seconds. They are handled through Read > Modify Write but the performance is very low! > Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned with > 8192 disk sector size in 10 seconds. They are handled through Read > Modify Write but the performance is very low! > Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): > Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned with > 8192 disk sector size in 10 seconds. They are handled through Read > Modify Write but the performance is very low! > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): > Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned with > 8192 disk sector size in 10 seconds. They are handled through Read > Modify Write but the performance is very low! > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned with > 8192 disk sector size in 10 seconds. They are handled through Read > Modify Write but the performance is very low! > > These COMSTAR LUNs are configured to export a blocksize 8k like this: > > LU Name: 600144F0564D504F4F4C3037534C3132 > Operational Status: Online > Provider Name : sbd > Alias : nfsvmpool07Slot12 > View Entry Count : 1 > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > Meta File : not set > Size : 200042414080 > Block Size : 8192 > Management URL : not set > Vendor ID : SUN > Product ID : COMSTAR > Serial Num : not set > Write Protect : Disabled > Writeback Cache : Enabled > Access State : Active > > Now, the system seems to recognize the 8k, but for whatever reason, > doesn't adjust the block size accordingly. It does that for the 4k > LUNs, however, so I am unsure, on how to tackle this? Any tipp, > anyone could share? > > Thanks, > Stephan > > > > > > > What's in the bottom here? > > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > > It dosesn't look like a zvol? > > /Johan It isn't one - it's a raw partition, an Intel SSD in this case. Cheers, Stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From sylvain at yesik.it Fri Sep 8 12:38:17 2017 From: sylvain at yesik.it (Sylvain Leroux) Date: Fri, 8 Sep 2017 14:38:17 +0200 Subject: [OmniOS-discuss] Status of the OmniOS project Message-ID: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> Hi everyone, After Oracle has laid off the core Solaris developers, I'm writing an article for https://itsfoss.com about the various OpenSolaris-based/Illumos-based distributions. I've already heard about OmniOS previously and wanted to mention it as the de-facto Illumos server distribution. However while doing some researches, I found that article: https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/ Here is the headline: """ Development of OmniOS ? an Oracle-free open-source variant of Solaris ? is being killed after five years of work. """ The author then reports several quotes from OmniTI chief executive Robert Treat to support that assertion. On the other hand, by what I saw on the mailing list the project _seems_ still to be active. Could you give more informations about the current state of the project? Thanks in advance for your time, - Sylvain -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- https://yesik.it -- -- Follow me on Twitter! -- https://twitter.com/Yes_I_Know_IT -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From sylvain at yesik.it Fri Sep 8 12:50:27 2017 From: sylvain at yesik.it (Sylvain Leroux) Date: Fri, 8 Sep 2017 14:50:27 +0200 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> Message-ID: Thank you Robert for the link. I wasn't aware at all of the "Community Edition". It is unfortunate it does not yet appear in the first results when searching on Google for "OmniOS". Hopefully this will change soon. Cheers :) On 09/08/2017 02:44 PM, DELROY A BLAKE wrote: > > > http://www.omniosce.org/about.html > > The community is alive and well . > > >> On Sep 8, 2017, at 8:38 AM, Sylvain Leroux > > wrote: >> >> Hi everyone, >> >> After Oracle has laid off the core Solaris developers, I'm writing an >> article for https://itsfoss.com about the various >> OpenSolaris-based/Illumos-based distributions. >> >> I've already heard about OmniOS previously and wanted to mention it as >> the de-facto Illumos server distribution. >> However while doing some researches, I found that article: >> >> https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/ >> >> Here is the headline: >> """ >> Development of OmniOS ? an Oracle-free open-source variant of Solaris ? >> is being killed after five years of work. >> """ >> >> The author then reports several quotes from OmniTI chief executive >> Robert Treat to support that assertion. >> On the other hand, by what I saw on the mailing list the project _seems_ >> still to be active. >> >> >> Could you give more informations about the current state of the project? >> >> >> Thanks in advance for your time, >> - Sylvain >> >> -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- >> https://yesik.it -- -- Follow me on Twitter! -- >> https://twitter.com/Yes_I_Know_IT >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- https://yesik.it -- -- Follow me on LinkedIn! -- https://fr.linkedin.com/in/sylvain-leroux-knows-it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From jesus at omniti.com Fri Sep 8 12:53:45 2017 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 8 Sep 2017 08:53:45 -0400 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> Message-ID: My two cents on this. Sensational title and misleading. If you read the archives you'll see that OmniTI forced themselves out of corporate stewardship with the desire for the OmniOS product to be community developed, managed and governed. There was never any intention (or outcome) that resembles killing. One of the advantages of this move is that it prevents OmniOS from meeting any of the Oracle-induced outcomes brutally met by Solaris. The community has taken over... and releases are even on schedule. http://www.omniosce.org/ As far as developer activity, the OmniOS project has wider participation and more activity now than it did prior to the transition. If you're looking for an "It's dead Jim" article, you might not get what you're looking for. Best regards, Theo On Fri, Sep 8, 2017 at 8:38 AM, Sylvain Leroux wrote: > Hi everyone, > > After Oracle has laid off the core Solaris developers, I'm writing an > article for https://itsfoss.com about the various > OpenSolaris-based/Illumos-based distributions. > > I've already heard about OmniOS previously and wanted to mention it as > the de-facto Illumos server distribution. > However while doing some researches, I found that article: > > https://www.theregister.co.uk/2017/04/25/oracle_free_ > solaris_project_stops/ > > Here is the headline: > """ > Development of OmniOS ? an Oracle-free open-source variant of Solaris ? > is being killed after five years of work. > """ > > The author then reports several quotes from OmniTI chief executive > Robert Treat to support that assertion. > On the other hand, by what I saw on the mailing list the project _seems_ > still to be active. > > > Could you give more informations about the current state of the project? > > > Thanks in advance for your time, > - Sylvain > > -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- > https://yesik.it -- -- Follow me on Twitter! -- > https://twitter.com/Yes_I_Know_IT > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From skiselkov.ml at gmail.com Fri Sep 8 12:54:25 2017 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Fri, 8 Sep 2017 14:54:25 +0200 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> Message-ID: <8decfd16-7b34-d227-d0aa-4ffc7dd3691d@gmail.com> On 9/8/17 2:38 PM, Sylvain Leroux wrote: > Hi everyone, > > After Oracle has laid off the core Solaris developers, I'm writing an > article for https://itsfoss.com about the various > OpenSolaris-based/Illumos-based distributions. > > I've already heard about OmniOS previously and wanted to mention it as > the de-facto Illumos server distribution. > However while doing some researches, I found that article: > > https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/ > > Here is the headline: > """ > Development of OmniOS ? an Oracle-free open-source variant of Solaris ? > is being killed after five years of work. > """ > > The author then reports several quotes from OmniTI chief executive > Robert Treat to support that assertion. > On the other hand, by what I saw on the mailing list the project _seems_ > still to be active. > > > Could you give more informations about the current state of the project? > > > Thanks in advance for your time, While "OmniOS" is no longer being developed by OmniTI, its development has been taken over by the community (at OmniTI's request, actually) and can be found under http://www.omniosce.org with new releases still ongoing. Cheers, -- Saso -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From sylvain at yesik.it Fri Sep 8 13:03:58 2017 From: sylvain at yesik.it (Sylvain Leroux) Date: Fri, 8 Sep 2017 15:03:58 +0200 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> Message-ID: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> Some comments inline: On 09/08/2017 02:53 PM, Theo Schlossnagle wrote: > The community has taken over... and releases are even on schedule. > > http://www.omniosce.org/ > > As far as developer activity, the OmniOS project has wider > participation and more activity now than it did prior to the transition. Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of OmniOS "Community Edition". And this project certainly deserves being advertised. So if I may add my own little stone to the wall for that, I will be glad to do it. > > If you're looking for an "It's dead Jim" article, you might not get > what you're looking for. As a matter of fact, I was looking to the exact opposite: I I used a lot Solaris in the 90s and early 2000s. And I was strongly disappointed to see what happened to that great OS years after years under the umbrella of Oracle in the 2010s. My goal was to say Solaris was still alive -- and there are actively maintained projects and distributions that have taken up the succession. Regards, - Sylvain -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- https://yesik.it -- -- Follow me on LinkedIn! -- https://fr.linkedin.com/in/sylvain-leroux-knows-it -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From chip at innovates.com Fri Sep 8 13:43:15 2017 From: chip at innovates.com (Schweiss, Chip) Date: Fri, 8 Sep 2017 08:43:15 -0500 Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos In-Reply-To: References: <4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org> <19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org> <78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com> <328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org> Message-ID: Now that I'm back to working on this. The only way I could get the firmware updated was booting into the UEFI shell. A bit of a pain but it worked. Unfortunately, it has not changed the behavior of the HBA. Where do I go from here? Any hope of getting this working on OmniOS? -Chip On Thu, Aug 31, 2017 at 9:53 AM, Schweiss, Chip wrote: > This server will be serving NFS for vSphere. It is running OmniOS CE, > nothing VMware. > > I'm working on flashing firmware now and will report back any changes. > > -Chip > > On Thu, Aug 31, 2017 at 9:42 AM, Dale Ghent wrote: > >> >> > On Aug 31, 2017, at 9:29 AM, Schweiss, Chip wrote: >> > >> > I've added mpt_sas "pciex1000,c9" to /etc/driver_aliases and rebooted. >> > >> > Looks like it's partially working, but it's not fully functional. >> Service are timing out: >> > >> > Here's what I see in /var/adm/messages: >> > >> > >> > Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING: >> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0): >> > Aug 31 08:15:49 vsphere-zfs01 MPT Firmware Fault, code: 2667 >> > Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING: >> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0): >> >> The driver is reporting that the MPT IOC (IO Controller) is reporting a >> fault. It's just reading this condition off the controller chip itself, and >> unfortunately there doesn't seem to be a handy reference published by >> LSI/Avago regarding what 2667h actually means. >> >> However I note from your machine's hostname that this is perhaps a ESI >> guest that is being given the HBA in passthrough mode? It would seem that >> someone else has encountered a similar issue as yourself in this case, with >> the same MPT fault code, but on Linux running Proxmox. According to this >> forum thread, they ended up flashing the firmware on the card to something >> newer and the problem went away: >> >> https://forum.proxmox.com/threads/pci-passthrough.16483/ >> >> I would suggest Tim's approach and flashing your card up to the newest IT >> (not IR) firmware. >> >> /dale >> >> >> ------------------------------------------ >> illumos-zfs >> Archives: https://illumos.topicbox.com/groups/zfs/discussions/T372d7dd >> d75316296-Mb0dd6c92e5393440a8b0c8fb >> Powered by Topicbox: https://topicbox.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From illumos at cucumber.demon.co.uk Fri Sep 8 13:52:24 2017 From: illumos at cucumber.demon.co.uk (Andrew Gabriel) Date: Fri, 8 Sep 2017 14:52:24 +0100 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> Message-ID: <4032b9e4-a9e1-7f20-553a-4fc7a700c8c2@cucumber.demon.co.uk> On 08/09/2017 14:03, Sylvain Leroux wrote: > My goal was to say Solaris was still alive -- and there are actively > maintained projects and distributions that have taken up the succession. Do check out Bryan Cantrill's blog on this, if you haven't seen it yet. http://dtrace.org/blogs/bmc/2017/09/04/the-sudden-death-and-eternal-life-of-solaris/ -- Andrew From jimklimov at cos.ru Fri Sep 8 14:04:11 2017 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 08 Sep 2017 14:04:11 +0000 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> Message-ID: On September 8, 2017 3:03:58 PM GMT+02:00, Sylvain Leroux wrote: >Some comments inline: > > >On 09/08/2017 02:53 PM, Theo Schlossnagle wrote: >> The community has taken over... and releases are even on schedule. >> >> http://www.omniosce.org/ >> >> As far as developer activity, the OmniOS project has wider >> participation and more activity now than it did prior to the >transition. >Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of >OmniOS "Community Edition". And this project certainly deserves being >advertised. So if I may add my own little stone to the wall for that, I >will be glad to do it. > > >> >> If you're looking for an "It's dead Jim" article, you might not get >> what you're looking for. >As a matter of fact, I was looking to the exact opposite: I I used a >lot >Solaris in the 90s and early 2000s. And I was strongly disappointed to >see what happened to that great OS years after years under the umbrella >of Oracle in the 2010s. > >My goal was to say Solaris was still alive -- and there are actively >maintained projects and distributions that have taken up the >succession. > >Regards, >- Sylvain >My goal was to say Solaris was still alive -- and there are actively >maintained projects and distributions that have taken up the >succession. To that effect, OpenIndiana Hipster is another major general-purpose illumos distribution that's being actively debeloped too, aimed at both desktop and server, with a lot of software and GUI desktop pre-packaged in its standard repository - in contrast with OmniOS that aims to be the minimal foundation of an OS. Also there are many smaller projects, outlined on wiki.illumos.org as well as unknown cores of storage, networking and other appliances. Hope this helps, Jim -- Typos courtesy of K-9 Mail on my Android From david.ledger at ivdcs.co.uk Fri Sep 8 14:21:30 2017 From: david.ledger at ivdcs.co.uk (David Ledger) Date: Fri, 08 Sep 2017 15:21:30 +0100 Subject: [OmniOS-discuss] networking between zones Message-ID: We have several zones running on an OmniOS server that use exclusive addressing but in the same subnet at the global zone. These work fine. We now need to set up a couple of zones that have their own subnet, but talk to the outside world through the global zone. These will need to be network isolated from the existing zones and with access controlled, presumably by ipf/ipnat filtering done in the global zone. I?m having difficulty setting this up. It is readily admitted on the ?net that Solaris network config is different to anything else, and that it has moved on in stages from the old hosts, hostname etc. files that were so easy back in the 80?s. Could someone please point me to some documentation that is relevant to OmniOS. We?re not up to r151022 even yet. I can?t tell from what?s out there whether I should be using ipfilter config within svcs, or ipf/ipnat type files or what. Some things say I need an etherstub, others say I don?t. The .cfg file gives the zone a vmic, but the zone can?t see it, but the zone can use it ??? David the confused. David Ledger - computers since ?69, Unix since ?83, Web since ?95 From jimklimov at cos.ru Fri Sep 8 14:18:07 2017 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 08 Sep 2017 14:18:07 +0000 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> Message-ID: <21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru> On September 8, 2017 4:04:11 PM GMT+02:00, Jim Klimov wrote: >On September 8, 2017 3:03:58 PM GMT+02:00, Sylvain Leroux > wrote: >>Some comments inline: >> >> >>On 09/08/2017 02:53 PM, Theo Schlossnagle wrote: >>> The community has taken over... and releases are even on schedule. >>> >>> http://www.omniosce.org/ >>> >>> As far as developer activity, the OmniOS project has wider >>> participation and more activity now than it did prior to the >>transition. >>Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of >>OmniOS "Community Edition". And this project certainly deserves being >>advertised. So if I may add my own little stone to the wall for that, >I >>will be glad to do it. >> >> >>> >>> If you're looking for an "It's dead Jim" article, you might not get >>> what you're looking for. >>As a matter of fact, I was looking to the exact opposite: I I used a >>lot >>Solaris in the 90s and early 2000s. And I was strongly disappointed to >>see what happened to that great OS years after years under the >umbrella >>of Oracle in the 2010s. >> >>My goal was to say Solaris was still alive -- and there are actively >>maintained projects and distributions that have taken up the >>succession. >> >>Regards, >>- Sylvain > >>My goal was to say Solaris was still alive -- and there are actively >>maintained projects and distributions that have taken up the >>succession. > >To that effect, OpenIndiana Hipster is another major general-purpose >illumos distribution that's being actively debeloped too, aimed at both >desktop and server, with a lot of software and GUI desktop pre-packaged >in its standard repository - in contrast with OmniOS that aims to be >the minimal foundation of an OS. > >Also there are many smaller projects, outlined on wiki.illumos.org as >well as unknown cores of storage, networking and other appliances. > >Hope this helps, >Jim >-- >Typos courtesy of K-9 Mail on my Android ...And for clarification - I spoke of general purpose distros but ended on a note of the hearts of appliances. In that area of purpose-optimized illumos products there are of course some well-known larger bi-lateral opensource commercial players, such as Joyent (now part of Samsung) aimed at hypervisors and related storage needs, Delphix for databases, Nexenta for storage... Also, the OpenZFS community codebase is a stripped-down illumos source tree that grows with BSD, Linux and other contributions. Jim -- Typos courtesy of K-9 Mail on my Android From jdg117 at elvis.arl.psu.edu Fri Sep 8 14:23:07 2017 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Fri, 08 Sep 2017 10:23:07 -0400 Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos In-Reply-To: Your message of "Fri, 08 Sep 2017 08:43:15 CDT." References: <4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org> <19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org> <78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com> <328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org> Message-ID: <201709081423.v88EN7Xe013815@elvis.arl.psu.edu> In message , "Schweiss, Chip" writes: >Now that I'm back to working on this. The only way I could get the >firmware updated was booting into the UEFI shell. A bit of a pain but it >worked. Did you go to P14 or P15? >Unfortunately, it has not changed the behavior of the HBA. > >Where do I go from here? Any hope of getting this working on OmniOS? Can you reproduce with FreeBSD? John groenveld at acm.org From danmcd at kebe.com Fri Sep 8 14:31:22 2017 From: danmcd at kebe.com (Dan McDonald) Date: Fri, 8 Sep 2017 10:31:22 -0400 Subject: [OmniOS-discuss] networking between zones In-Reply-To: References: Message-ID: <20170908143122.GA33110@everywhere.local> On Fri, Sep 08, 2017 at 03:21:30PM +0100, David Ledger wrote: > > We now need to set up a couple of zones that have their own subnet, but talk > to the outside world through the global zone. These will need to be network > isolated from the existing zones and with access controlled, presumably by > ipf/ipnat filtering done in the global zone. I?m having difficulty setting > this up. It is readily admitted on the ?net that Solaris network config is > different to anything else, and that it has moved on in stages from the old > hosts, hostname etc. files that were so easy back in the 80?s. I think you wish to create an etherstub (in-machine "LAN" as it were). From global: dladm create-etherstub internal0 And once created, you create vnics attached to that etherstub: dladm create-vnic -l internal0 stubnet0 dladm create-vnic -l internal0 stubnet1 ... And then you assign the vnics to your "have their own subnet" zones like you would any other nic. You will also need your global, or even a dedicated router zone, attach to both the etherstub and the external network (running ipf or whatever else). Does this help? Dan From david.ledger at ivdcs.co.uk Fri Sep 8 14:40:44 2017 From: david.ledger at ivdcs.co.uk (David Ledger) Date: Fri, 08 Sep 2017 15:40:44 +0100 Subject: [OmniOS-discuss] networking between zones In-Reply-To: <20170908143122.GA33110@everywhere.local> References: <20170908143122.GA33110@everywhere.local> Message-ID: <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> On 8 Sep 2017, at 15:31, Dan McDonald wrote: > On Fri, Sep 08, 2017 at 03:21:30PM +0100, David Ledger wrote: >> > > > >> We now need to set up a couple of zones that have their own subnet, >> but talk >> to the outside world through the global zone. These will need to be >> network >> isolated from the existing zones and with access controlled, >> presumably by >> ipf/ipnat filtering done in the global zone. I?m having difficulty >> setting >> this up. It is readily admitted on the ?net that Solaris network >> config is >> different to anything else, and that it has moved on in stages from >> the old >> hosts, hostname etc. files that were so easy back in the 80?s. > > I think you wish to create an etherstub (in-machine "LAN" as it were). > From > global: > > dladm create-etherstub internal0 > > And once created, you create vnics attached to that etherstub: > > dladm create-vnic -l internal0 stubnet0 > dladm create-vnic -l internal0 stubnet1 > ... > > And then you assign the vnics to your "have their own subnet" zones > like you > would any other nic. You will also need your global, or even a > dedicated > router zone, attach to both the etherstub and the external network > (running > ipf or whatever else). > > Does this help? > > Dan Knowing that certainly helps, thanks. Can I also ignore those online documents that say it?s all done by configurations within the svcs setup and that I need to disable one version of ipfilter and enable another? David From danmcd at kebe.com Fri Sep 8 14:54:33 2017 From: danmcd at kebe.com (Dan McDonald) Date: Fri, 8 Sep 2017 10:54:33 -0400 Subject: [OmniOS-discuss] networking between zones In-Reply-To: <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> References: <20170908143122.GA33110@everywhere.local> <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> Message-ID: <20170908145433.GC33110@everywhere.local> On Fri, Sep 08, 2017 at 03:40:44PM +0100, David Ledger wrote: > > Knowing that certainly helps, thanks. Can I also ignore those online > documents that say it?s all done by configurations within the svcs setup and > that I need to disable one version of ipfilter and enable another? I use stock ipfilter (ipnat only, to be honest) in its own zone. I didn't use SMF magic to configure it, save "svcadm enable ipfilter" once had /etc/ipf/ipnat.conf where I wanted it. Dan From jdg117 at elvis.arl.psu.edu Fri Sep 8 14:57:08 2017 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Fri, 08 Sep 2017 10:57:08 -0400 Subject: [OmniOS-discuss] networking between zones In-Reply-To: Your message of "Fri, 08 Sep 2017 15:40:44 BST." <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> References: <20170908143122.GA33110@everywhere.local> <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> Message-ID: <201709081457.v88Ev8I3014024@elvis.arl.psu.edu> In message <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9 at ivdcs.co.uk>, "David Ledger" w rites: >Knowing that certainly helps, thanks. Can I also ignore those online >documents that say it???s all done by configurations within the svcs You assign the VNIC's to your zones via zonecfg(1M). >setup and that I need to disable one version of ipfilter and enable >another? PF has not yet been ported to illumos. If you absolutely must move to PF, you'll need to run it under a VBox or KVM VM. I use native IPF for my Crossbow network in a box. John groenveld at acm.org From philipcristiano at philipcristiano.com Fri Sep 8 15:25:07 2017 From: philipcristiano at philipcristiano.com (Philip Cristiano) Date: Fri, 08 Sep 2017 11:25:07 -0400 Subject: [OmniOS-discuss] networking between zones In-Reply-To: <201709081457.v88Ev8I3014024@elvis.arl.psu.edu> References: <20170908143122.GA33110@everywhere.local> <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> <201709081457.v88Ev8I3014024@elvis.arl.psu.edu> Message-ID: <1504884307.139027.1099714504.7EB2D20B@webmail.messagingengine.com> This helped me quite a bit when trying to do the same thing http://www.scalingbits.com/book/export/html/479 -- Philip Cristiano philipcristiano at philipcristiano.com On Fri, Sep 8, 2017, at 10:57 AM, John D Groenveld wrote: > In message <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9 at ivdcs.co.uk>, "David > Ledger" w > rites: > >Knowing that certainly helps, thanks. Can I also ignore those online > >documents that say it?s all done by configurations within the svcs > > You assign the VNIC's to your zones via zonecfg(1M). > > >setup and that I need to disable one version of ipfilter and enable > >another? > > PF has not yet been ported to illumos. > If you absolutely must move to PF, you'll need to run it under > a VBox or KVM VM. > > I use native IPF for my Crossbow network in a box. > > John > groenveld at acm.org > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From chip at innovates.com Fri Sep 8 15:38:03 2017 From: chip at innovates.com (Schweiss, Chip) Date: Fri, 8 Sep 2017 10:38:03 -0500 Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos In-Reply-To: References: <4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org> <19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org> <78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com> <328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org> Message-ID: Robert, That is awesome. I'd definitely be interested in testing this. I'll get my feet wet with building OmniOS CE with it. Thanks! -Chip On Fri, Sep 8, 2017 at 10:07 AM, Robert Mustacchi wrote: > On 9/8/17 6:43 , Schweiss, Chip wrote: > > Now that I'm back to working on this. The only way I could get the > > firmware updated was booting into the UEFI shell. A bit of a pain but > it > > worked. > > > > Unfortunately, it has not changed the behavior of the HBA. > > > > Where do I go from here? Any hope of getting this working on OmniOS? > > Hi Chip, > > I'm just catching up on this. So, I do have some good news and bad news. > First, the bad news. This is based on the SAS3224 chipset, which it > appears the 16e is also describing itself as. Of note, this uses a > slightly newer version of the MPI specification and the driver as it is > written doesn't quite notice that it requires slightly different > behavior and a simple PCI ID update isn't sufficient. > > The good news is that I just finished doing this work for the LSI > 9305-24i and was going to send that up to illumos shortly. If you want, > I can send those changes your way if you're comfortable building illumos > and want to test that. > > Robert > > > On Thu, Aug 31, 2017 at 9:53 AM, Schweiss, Chip > wrote: > > > >> This server will be serving NFS for vSphere. It is running OmniOS CE, > >> nothing VMware. > >> > >> I'm working on flashing firmware now and will report back any changes. > >> > >> -Chip > >> > >> On Thu, Aug 31, 2017 at 9:42 AM, Dale Ghent > wrote: > >> > >>>> On Aug 31, 2017, at 9:29 AM, Schweiss, Chip > wrote: > >>>> > >>>> I've added mpt_sas "pciex1000,c9" to /etc/driver_aliases and rebooted. > >>>> > >>>> Looks like it's partially working, but it's not fully functional. > >>> Service are timing out: > >>>> > >>>> Here's what I see in /var/adm/messages: > >>>> > >>>> > >>>> Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING: > >>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0): > >>>> Aug 31 08:15:49 vsphere-zfs01 MPT Firmware Fault, code: 2667 > >>>> Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING: > >>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0): > >>> > >>> The driver is reporting that the MPT IOC (IO Controller) is reporting a > >>> fault. It's just reading this condition off the controller chip > itself, and > >>> unfortunately there doesn't seem to be a handy reference published by > >>> LSI/Avago regarding what 2667h actually means. > >>> > >>> However I note from your machine's hostname that this is perhaps a ESI > >>> guest that is being given the HBA in passthrough mode? It would seem > that > >>> someone else has encountered a similar issue as yourself in this case, > with > >>> the same MPT fault code, but on Linux running Proxmox. According to > this > >>> forum thread, they ended up flashing the firmware on the card to > something > >>> newer and the problem went away: > >>> > >>> https://forum.proxmox.com/threads/pci-passthrough.16483/ > >>> > >>> I would suggest Tim's approach and flashing your card up to the newest > IT > >>> (not IR) firmware. > >>> > >>> /dale > >>> > > > > ------------------------------------------ > > illumos-zfs > > Archives: https://illumos.topicbox.com/groups/zfs/discussions/ > T372d7ddd75316296-M4bd824d5e1881e2772ee518a > > Powered by Topicbox: https://topicbox.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at yesik.it Fri Sep 8 15:40:22 2017 From: sylvain at yesik.it (Sylvain Leroux) Date: Fri, 8 Sep 2017 17:40:22 +0200 Subject: [OmniOS-discuss] Status of the OmniOS project In-Reply-To: <21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru> References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it> <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it> <21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru> Message-ID: On 09/08/2017 04:18 PM, Jim Klimov wrote: > On September 8, 2017 4:04:11 PM GMT+02:00, Jim Klimov wrote: >> To that effect, OpenIndiana Hipster is another major general-purpose >> illumos distribution that's being actively debeloped too, aimed at both >> desktop and server, with a lot of software and GUI desktop pre-packaged >> in its standard repository - in contrast with OmniOS that aims to be >> the minimal foundation of an OS. >> >> Also there are many smaller projects, outlined on wiki.illumos.org as >> well as unknown cores of storage, networking and other appliances. >> >> Hope this helps, >> Jim >> -- >> Typos courtesy of K-9 Mail on my Android > ...And for clarification - I spoke of general purpose distros but ended on a note of the hearts of appliances. > > In that area of purpose-optimized illumos products there are of course some well-known larger bi-lateral opensource commercial players, such as Joyent (now part of Samsung) aimed at hypervisors and related storage needs, Delphix for databases, Nexenta for storage... Thank you Jim. Indeed OpenIndiana is clearly the Illumos flagship distribution. I couldn't have missed it. And in fact, I probably have some OpenIndiana 1 installation CD handing around somewhere! Concerning the distributions mentioned on wiki.illumos.org: Unfortunately, I can't provide an exhaustive coverage of all available distributions. My editor want I focus on 4 or 5 of them. So I focused on what I considered "enterprise grade" and/or "production ready" solutions. Namely: - OpenIndiana, - SmartOS - NexentaStor (now closed source) - OmniOSce I examined a couple of others I didn't know before like Tribblix (http://www.tribblix.org/) or Dyson OS (https://www.osdyson.org/projects/dyson/wiki). But they seems much less accomplished. Or not very active like Dilos (http://www.dilos.org/). Concerning Delphix -- I didn't know that one. I will definitely take a look at that... Once again, thank you Jim - Sylvain -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- https://yesik.it -- -- Follow me on Google+! -- https://plus.google.com/+YesikIT -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From omnios at citrus-it.net Fri Sep 8 16:50:35 2017 From: omnios at citrus-it.net (Andy Fiddaman) Date: Fri, 8 Sep 2017 16:50:35 +0000 (UTC) Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos In-Reply-To: References: <4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org> <19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org> <78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com> <328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org> Message-ID: On Fri, 8 Sep 2017, Schweiss, Chip wrote: ; Robert, ; ; That is awesome. I'd definitely be interested in testing this. ; ; I'll get my feet wet with building OmniOS CE with it. Hi Chip, If you don't already know about it you might want to take a look at https://github.com/omniosorg/omni/ which makes building OmniOS on OmniOS nice and easy, globally or in a zone. You won't need to build the userland bits so the whole process should take around an hour. If you run into trouble, come and chat to us at https://gitter.im/omniosorg/Lobby Cheers, Andy -- Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From stephan.budach at jvm.de Fri Sep 8 18:23:29 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Fri, 8 Sep 2017 20:23:29 +0200 (CEST) Subject: [OmniOS-discuss] Ang: COMSTAR and blocksizes In-Reply-To: <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local> References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local> Message-ID: <471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de> ----- Urspr?ngliche Mail ----- > Von: "Stephan Budach" > An: "Johan Kragsterman" > CC: omnios-discuss at lists.omniti.com > Gesendet: Donnerstag, 7. September 2017 12:22:06 > Betreff: Re: [OmniOS-discuss] Ang: COMSTAR and blocksizes > > > > ----- Urspr?ngliche Mail ----- > > Von: "Johan Kragsterman" > > An: "Stephan Budach" > > CC: omnios-discuss at lists.omniti.com > > Gesendet: Donnerstag, 7. September 2017 12:08:45 > > Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes > > > > > > Hi! > > > > -----"OmniOS-discuss" > > skrev: ----- > > Till: omnios-discuss at lists.omniti.com > > Fr?n: Stephan Budach > > S?nt av: "OmniOS-discuss" > > Datum: 2017-09-07 10:38 > > ?rende: [OmniOS-discuss] COMSTAR and blocksizes > > > > Hi, > > > > I am having trouble getting an issue sorted out, where omniOS > > 151020 > > complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, > > like this: > > > > Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned > > with > > 8192 disk sector size in 10 seconds. They are handled through Read > > Modify Write but the performance is very low! > > Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > > Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned > > with > > 8192 disk sector size in 10 seconds. They are handled through Read > > Modify Write but the performance is very low! > > Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): > > Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned > > with > > 8192 disk sector size in 10 seconds. They are handled through Read > > Modify Write but the performance is very low! > > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > > /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): > > Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned with > > 8192 disk sector size in 10 seconds. They are handled through Read > > Modify Write but the performance is very low! > > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > > Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned > > with > > 8192 disk sector size in 10 seconds. They are handled through Read > > Modify Write but the performance is very low! > > > > These COMSTAR LUNs are configured to export a blocksize 8k like > > this: > > > > LU Name: 600144F0564D504F4F4C3037534C3132 > > Operational Status: Online > > Provider Name : sbd > > Alias : nfsvmpool07Slot12 > > View Entry Count : 1 > > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > > Meta File : not set > > Size : 200042414080 > > Block Size : 8192 > > Management URL : not set > > Vendor ID : SUN > > Product ID : COMSTAR > > Serial Num : not set > > Write Protect : Disabled > > Writeback Cache : Enabled > > Access State : Active > > > > Now, the system seems to recognize the 8k, but for whatever reason, > > doesn't adjust the block size accordingly. It does that for the 4k > > LUNs, however, so I am unsure, on how to tackle this? Any tipp, > > anyone could share? > > > > Thanks, > > Stephan > > > > > > > > > > > > > > What's in the bottom here? > > > > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > > > > It dosesn't look like a zvol? > > > > /Johan > > It isn't one - it's a raw partition, an Intel SSD in this case. > > Cheers, > Stephan > Hi all, I just exported this zpool from my omniOS 020 and imported it on a recent oi box and the issue with the 8k writes doesn't seem to be present on oi. Any thoughts on this? Thanks, Stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From stephan.budach at jvm.de Mon Sep 11 07:24:04 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Mon, 11 Sep 2017 09:24:04 +0200 (CEST) Subject: [OmniOS-discuss] Ang: COMSTAR and blocksizes In-Reply-To: <471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de> References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local> <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local> <471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: <1384013344.979.1505114641278.JavaMail.stephan.budach@stephan.budach.wlan.jvm.de> > > ----- Urspr?ngliche Mail ----- > > > Von: "Johan Kragsterman" > > > An: "Stephan Budach" > > > CC: omnios-discuss at lists.omniti.com > > > Gesendet: Donnerstag, 7. September 2017 12:08:45 > > > Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes > > > > > > > > > Hi! > > > > > > -----"OmniOS-discuss" > > > skrev: ----- > > > Till: omnios-discuss at lists.omniti.com > > > Fr?n: Stephan Budach > > > S?nt av: "OmniOS-discuss" > > > Datum: 2017-09-07 10:38 > > > ?rende: [OmniOS-discuss] COMSTAR and blocksizes > > > > > > Hi, > > > > > > I am having trouble getting an issue sorted out, where omniOS > > > 151020 > > > complaints about mismatched blocksizes on some COMSTAR iSCSI > > > LUNS, > > > like this: > > > > > > Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned > > > with > > > 8192 disk sector size in 10 seconds. They are handled through > > > Read > > > Modify Write but the performance is very low! > > > Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] > > > WARNING: > > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > > > Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned > > > with > > > 8192 disk sector size in 10 seconds. They are handled through > > > Read > > > Modify Write but the performance is very low! > > > Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] > > > WARNING: > > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): > > > Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned > > > with > > > 8192 disk sector size in 10 seconds. They are handled through > > > Read > > > Modify Write but the performance is very low! > > > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] > > > WARNING: > > > /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): > > > Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned > > > with > > > 8192 disk sector size in 10 seconds. They are handled through > > > Read > > > Modify Write but the performance is very low! > > > Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] > > > WARNING: > > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): > > > Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned > > > with > > > 8192 disk sector size in 10 seconds. They are handled through > > > Read > > > Modify Write but the performance is very low! > > > > > > These COMSTAR LUNs are configured to export a blocksize 8k like > > > this: > > > > > > LU Name: 600144F0564D504F4F4C3037534C3132 > > > Operational Status: Online > > > Provider Name : sbd > > > Alias : nfsvmpool07Slot12 > > > View Entry Count : 1 > > > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > > > Meta File : not set > > > Size : 200042414080 > > > Block Size : 8192 > > > Management URL : not set > > > Vendor ID : SUN > > > Product ID : COMSTAR > > > Serial Num : not set > > > Write Protect : Disabled > > > Writeback Cache : Enabled > > > Access State : Active > > > > > > Now, the system seems to recognize the 8k, but for whatever > > > reason, > > > doesn't adjust the block size accordingly. It does that for the > > > 4k > > > LUNs, however, so I am unsure, on how to tackle this? Any tipp, > > > anyone could share? > > > > > > Thanks, > > > Stephan > > > > > > > > > > > > > > > > > > > > > What's in the bottom here? > > > > > > Data File : /dev/rdsk/c3t50015178F364A264d0p1 > > > > > > It dosesn't look like a zvol? > > > > > > /Johan > > > > It isn't one - it's a raw partition, an Intel SSD in this case. > > > > Cheers, > > Stephan > > > > Hi all, > > I just exported this zpool from my omniOS 020 and imported it on a > recent oi box and the issue with the 8k writes doesn't seem to be > present on oi. Any thoughts on this? > > Thanks, > Stephan I have the exact same behaviour on a freshly installed and updated omniosce 151022o: root at vsm02:~# cat /etc/release OmniOS v11 r151022o Copyright 2017 OmniTI Computer Consulting, Inc. All rights reserved. Copyright 2017 OmniOS Community Edition (OmniOSce) Association. All rights reserved. Use is subject to license terms. root at vsm02:~# tail -f /var/adm/messages Sep 11 11:21:42 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd29): Sep 11 11:21:42 vsm02 18 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 11 11:21:47 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd13): Sep 11 11:21:47 vsm02 19 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 11 11:21:48 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd7): Sep 11 11:21:48 vsm02 18 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 11 11:21:50 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3131 (sd6): Sep 11 11:21:50 vsm02 9 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Sep 11 11:21:52 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd29): Sep 11 11:21:52 vsm02 9 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! Is this even a COMSTAR/iSCSI Initiator thing, or is this something else? /stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From sylvain at yesik.it Mon Sep 11 16:27:12 2017 From: sylvain at yesik.it (Sylvain Leroux) Date: Mon, 11 Sep 2017 18:27:12 +0200 Subject: [OmniOS-discuss] Supported architectures Message-ID: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it> I was asked about the supported architecture for OmniOSce. As far as I can tell, this was not mentioned on the http://www.omniosce.org website. So, I assume x86_64. But, I *think* OmniOS did support x86_32 too. Is this the case for OmniOSce? What about the SPARC architecture? - Sylvain -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it -- https://yesik.it -- -- Follow me on Google+! -- https://plus.google.com/+YesikIT -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From danmcd at kebe.com Mon Sep 11 16:48:28 2017 From: danmcd at kebe.com (Dan McDonald) Date: Mon, 11 Sep 2017 12:48:28 -0400 Subject: [OmniOS-discuss] Supported architectures In-Reply-To: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it> References: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it> Message-ID: <20170911164828.GA35134@everywhere.local> On Mon, Sep 11, 2017 at 06:27:12PM +0200, Sylvain Leroux wrote: > I was asked about the supported architecture for OmniOSce. > > As far as I can tell, this was not mentioned on the > http://www.omniosce.org website. So, I assume x86_64. > > But, I *think* OmniOS did support x86_32 too. Is this the case for > OmniOSce? What about the SPARC architecture? Pre-ce, OmniOS could boot on 32-bit x86, but OmniTI would not provide support for 32-bit boots. SPARC was always unsupported and unbuilt. I assume the new community management will not change any of that. Your 32-bit x86 apps, BTW, will continue to run on an amd64/x64/x86_64 boot just fine. Dan From jboren at drakecooper.com Wed Sep 13 01:27:41 2017 From: jboren at drakecooper.com (Joseph Boren) Date: Tue, 12 Sep 2017 19:27:41 -0600 Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE install Message-ID: Greetings, Had a strange thing happen with a new server build today. Installing r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash drive image, burned to a 64Gb USB flash drive, installed just fine, rebooted, all is well. Installed napp-it, which creates a new BE and sets it active. Rebooted and let it try to boot into the new BE and got this: Loading now.... can't find 'now' can't find 'now' Loading /boot/defualts/loader.conf Loading now..... can't find 'now' can't find 'now' Type ? for a list of commands *blah blah blah* ok Rebooted and selected the BE from before the install and it booted fine. So I created a new BE with beadmin, set it active, and rebooted and got the same error. So I set the old (working) BE active and rebooted again, and now I'm getting that same error on that BE too. Rebooted yet again, selected the new, manually created BE and it booted just fine. Set it active and rebooted again - now that BE is broken too. Rebooted for the millionth time and selected the other (old - not active) BE and lo and behold it boots just fine. So it appears that whatever BE is set active will not boot. I believe that's actual irony right there. Any non-active BE will boot just fine if selected in the loader screen. Anyone seen this problem before? Any Ideas what may be going on? It's probably my fault somehow.... Thanks, Joseph Boren IT Specialist +c: 208.891.2128 + 416 S 8th St., Boise, ID 83702 + drakecooper.com *Read our minds *Blog Facebook , Twitter , Linkedin , Instagram -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdg117 at elvis.arl.psu.edu Wed Sep 13 14:08:31 2017 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Wed, 13 Sep 2017 10:08:31 -0400 Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE install In-Reply-To: Your message of "Tue, 12 Sep 2017 19:27:41 MDT." References: Message-ID: <201709131408.v8DE8VDa029615@elvis.arl.psu.edu> In message , Joseph Boren writes: >Loading now.... >can't find 'now' >can't find 'now' Has the stench of reboot(1M) now boot_arguments An optional boot_arguments specifies arguments to the uadmin(2) function that are passed to the boot program and kernel upon restart. The form and list of arguments is described in the boot(1M) and kernel(1M) man pages.. If the arguments are specified, whitespace between them is replaced by single spaces unless the whitespace is quoted for the shell. If the boot_arguments begin with a hyphen, they must be preceded by the -- delimiter (two hyphens) to denote the end of the reboot argument list. I find shutdown(1M) and init(1M) are safer when moving between OSs. John groenveld at acm.org From omnios at citrus-it.net Wed Sep 13 18:29:53 2017 From: omnios at citrus-it.net (Andy Fiddaman) Date: Wed, 13 Sep 2017 18:29:53 +0000 (UTC) Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE install In-Reply-To: References: Message-ID: On Tue, 12 Sep 2017, Joseph Boren wrote: ; Greetings, ; ; Had a strange thing happen with a new server build today. Installing ; r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash ; drive image, burned to a 64Gb USB flash drive, installed just fine, ; rebooted, all is well. Installed napp-it, which creates a new BE and sets ; it active. Rebooted and let it try to boot into the new BE and got this: ; ; Loading now.... ; can't find 'now' ; can't find 'now' How are you rebooting? If you are using 'reboot now' then it will result in this behaviour. Use just 'reboot' or 'init 6' - it isn't Linux : ) Andy -- Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From jboren at drakecooper.com Wed Sep 13 19:07:00 2017 From: jboren at drakecooper.com (Joseph Boren) Date: Wed, 13 Sep 2017 13:07:00 -0600 Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE install In-Reply-To: References: Message-ID: Arghhh, are you freaking kidding me?!? At least I was right about it being something I was doing. :) Thank you Andy, I appreciate the reminder. Embarrassing, but I guess it's not wasted time if I learned something. Or re-learned it in this case... Thanks again! Cheers! Joseph Boren IT Specialist +c: 208.891.2128 + 416 S 8th St., Boise, ID 83702 + drakecooper.com *Read our minds *Blog Facebook , Twitter , Linkedin , Instagram On Wed, Sep 13, 2017 at 12:29 PM, Andy Fiddaman wrote: > > On Tue, 12 Sep 2017, Joseph Boren wrote: > > ; Greetings, > ; > ; Had a strange thing happen with a new server build today. Installing > ; r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash > ; drive image, burned to a 64Gb USB flash drive, installed just fine, > ; rebooted, all is well. Installed napp-it, which creates a new BE and > sets > ; it active. Rebooted and let it try to boot into the new BE and got this: > ; > ; Loading now.... > ; can't find 'now' > ; can't find 'now' > > How are you rebooting? > If you are using 'reboot now' then it will result in this behaviour. > > Use just 'reboot' or 'init 6' - it isn't Linux : ) > > Andy > > -- > Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk > Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ > Registered in England and Wales | Company number 4899123 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirk.willems at exitas.be Thu Sep 14 12:26:13 2017 From: dirk.willems at exitas.be (Dirk Willems) Date: Thu, 14 Sep 2017 14:26:13 +0200 Subject: [OmniOS-discuss] questions Message-ID: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> Hello, I'm trying to understand something let me explain. Oracle always told to me that if you create a etherstub switch it has infiniband speed 40GB/s. But I have a customer running on Solaris (Yeah I know but let me explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told him to to use etherstub). The copy witch is performed for a Oracle database with sql command, the DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60 % busy the speed is 30 mb/s. So I did some test just to see and understand if it's the database or the system, but with doing my tests I get very confused ??? On another Solaris at my work copy over etherstub switch => copy speed is 185MB/s expected much more of infiniband speed ??? root at test1:/export/home/Admin# scp test10G Admin at 192.168.1.2:/export/home/Admin/ Password: test10G 100% |****************************************************************| 10240 MB 00:59 root at test2:~# dlstat -i 2 LINK IPKTS RBYTES OPKTS OBYTES net1 25.76K 185.14M 10.08K 2.62M net1 27.04K 187.16M 11.23K 3.22M net1 26.97K 186.37M 11.24K 3.23M net1 26.63K 187.67M 10.82K 2.99M net1 27.94K 186.65M 12.17K 3.75M net1 27.45K 187.46M 11.70K 3.47M net1 26.01K 181.95M 10.63K 2.99M net1 27.95K 188.19M 12.14K 3.69M net1 27.91K 188.36M 12.08K 3.64M The disks are all separate luns with all separated pools => disk are 20 - 30% busy On my OmniOSce at my lab over etherstub root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ Password: test10G 76% 7853MB 116.4MB/s => copy is 116.4 MB/s => expected much more from infiniband speed is just the same as Lan ??? Is not that my disk can not follow 17% busy there sleeping ... extended device statistics r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => rpool 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => rpool 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => data pool 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => data pool 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => data pool 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => data pool 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA root at NGINX:/root# dlstat show-link NGINX1 -i 2 LINK TYPE ID INDEX PKTS BYTES NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 9.26K 692.00K NGINX1 rx local -- 26.00K 216.32M NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 7.01K 531.38K NGINX1 rx local -- 30.65K 253.73M NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 8.95K 669.32K NGINX1 rx local -- 29.10K 241.15M On the other NGZ I receive 250MB/s ???? - So my question is how comes that the speed is equal to Lan 100MB/s on OmniOSce but i receive 250MB/s ? - Why is etherstub so slow if infiniband speed is 40GB/s ??? I'm very confused right now ... And want to know for sure how to understand and see this in the right way, because this customer will be the first customer from my who gonna switch complety over to OmniOSce on production and because this customer is one or the biggest company's in Belgium I really don't want to mess up !!! So any help and clarification will be highly appreciate !!! Thank you very much. Kind Regards, Dirk -- Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From lorban at bitronix.be Thu Sep 14 13:27:36 2017 From: lorban at bitronix.be (Ludovic Orban) Date: Thu, 14 Sep 2017 15:27:36 +0200 Subject: [OmniOS-discuss] questions In-Reply-To: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> Message-ID: Here's a quick thought: if you are copying your files with scp, you might be CPU bound because of all the crypto work. You should try to copy with a much more CPU-lightweight tool. I personally usually use netcat for this kind of stuff. Just my 2 cents. -- Ludovic On Thu, Sep 14, 2017 at 2:26 PM, Dirk Willems wrote: > Hello, > > > I'm trying to understand something let me explain. > > > Oracle always told to me that if you create a etherstub switch it has > infiniband speed 40GB/s. > > But I have a customer running on Solaris (Yeah I know but let me explain) > who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told > him to to use etherstub). > > The copy witch is performed for a Oracle database with sql command, the > DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60 > % busy the speed is 30 mb/s. > > > So I did some test just to see and understand if it's the database or the > system, but with doing my tests I get very confused ??? > > > On another Solaris at my work copy over etherstub switch => copy speed is > 185MB/s expected much more of infiniband speed ??? > > > root at test1:/export/home/Admin# scp test10G Admin at 192.168.1.2:/export/ > home/Admin/ > Password: > test10G 100% |***************************** > ***********************************| 10240 MB 00:59 > > > root at test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M > net1 26.97K 186.37M 11.24K 3.23M > net1 26.63K 187.67M 10.82K 2.99M > net1 27.94K 186.65M 12.17K 3.75M > net1 27.45K 187.46M 11.70K 3.47M > net1 26.01K 181.95M 10.63K 2.99M > net1 27.95K 188.19M 12.14K 3.69M > net1 27.91K 188.36M 12.08K 3.64M > > The disks are all separate luns with all separated pools => disk are 20 - > 30% busy > > > On my OmniOSce at my lab over etherstub > > > root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ > Password: > test10G > 76% 7853MB 116.4MB/s > > > => copy is 116.4 MB/s => expected much more from infiniband speed is just > the same as Lan ??? > > > Is not that my disk can not follow 17% busy there sleeping ... > > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => > rpool > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => > rpool > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => > data pool > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => > data pool > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => > data pool > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => > data pool > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > > root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 7.01K 531.38K > NGINX1 rx local -- 30.65K 253.73M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 8.95K 669.32K > NGINX1 rx local -- 29.10K 241.15M > > > On the other NGZ I receive 250MB/s ???? > > > - So my question is how comes that the speed is equal to Lan 100MB/s on > OmniOSce but i receive 250MB/s ? > > - Why is etherstub so slow if infiniband speed is 40GB/s ??? > > > I'm very confused right now ... > > > And want to know for sure how to understand and see this in the right way, > because this customer will be the first customer from my who gonna switch > complety over to OmniOSce on production and because this customer is one or > the biggest company's in Belgium I really don't want to mess up !!! > > > So any help and clarification will be highly appreciate !!! > > > Thank you very much. > > > Kind Regards, > > > Dirk > > > -- > Dirk Willems > System Engineer > > > +32 (0)3 443 12 38 <03%20443%2012%2038> > Dirk.Willems at exitas.be > > Quality. Passion. Personality > > www.exitas.be | Veldkant 31 | 2550 Kontich > > Illumos OmniOS Installation and Configuration Implementation Specialist. > Oracle Solaris 11 Installation and Configuration Certified Implementation > Specialist. > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From jimklimov at cos.ru Thu Sep 14 14:10:09 2017 From: jimklimov at cos.ru (Jim Klimov) Date: Thu, 14 Sep 2017 14:10:09 +0000 Subject: [OmniOS-discuss] questions In-Reply-To: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> Message-ID: <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems wrote: >Hello, > > >I'm trying to understand something let me explain. > > >Oracle always told to me that if you create a etherstub switch it has >infiniband speed 40GB/s. > >But I have a customer running on Solaris (Yeah I know but let me >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >(I know told him to to use etherstub). > >The copy witch is performed for a Oracle database with sql command, the > >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >- >60 % busy the speed is 30 mb/s. > > >So I did some test just to see and understand if it's the database or >the system, but with doing my tests I get very confused ??? > > >On another Solaris at my work copy over etherstub switch => copy speed >is 185MB/s expected much more of infiniband speed ??? > > >root at test1:/export/home/Admin# scp test10G >Admin at 192.168.1.2:/export/home/Admin/ >Password: >test10G 100% >|****************************************************************| >10240 >MB 00:59 > > >root at test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M > net1 26.97K 186.37M 11.24K 3.23M > net1 26.63K 187.67M 10.82K 2.99M > net1 27.94K 186.65M 12.17K 3.75M > net1 27.45K 187.46M 11.70K 3.47M > net1 26.01K 181.95M 10.63K 2.99M > net1 27.95K 188.19M 12.14K 3.69M > net1 27.91K 188.36M 12.08K 3.64M > >The disks are all separate luns with all separated pools => disk are 20 > >- 30% busy > > >On my OmniOSce at my lab over etherstub > > >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ >Password: >test10G 76% 7853MB 116.4MB/s > > >=> copy is 116.4 MB/s => expected much more from infiniband speed is >just the same as Lan ??? > > >Is not that my disk can not follow 17% busy there sleeping ... > > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => >rpool > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => >rpool > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => >data pool > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => >data pool > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => >data pool > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => >data pool > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > >root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 7.01K 531.38K > NGINX1 rx local -- 30.65K 253.73M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 8.95K 669.32K > NGINX1 rx local -- 29.10K 241.15M > > >On the other NGZ I receive 250MB/s ???? > > >- So my question is how comes that the speed is equal to Lan 100MB/s on > >OmniOSce but i receive 250MB/s ? > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > >I'm very confused right now ... > > >And want to know for sure how to understand and see this in the right >way, because this customer will be the first customer from my who gonna > >switch complety over to OmniOSce on production and because this >customer >is one or the biggest company's in Belgium I really don't want to mess >up !!! > > >So any help and clarification will be highly appreciate !!! > > >Thank you very much. > > >Kind Regards, > > >Dirk I am not sure where the infiniband claim comes from, but copying data disk to disk, you involve the slow layers like disk, skewed by faster layers like cache of already-read data and delayed writes :) If you have a wide pipe that you may fill, it doesn't mean you do have the means to fill it with a few disks. To estimate the speeds, try pure UDP streams from process to process (no disk), large-packet floodping, etc. I believe etherstub is not constrained artificially, and defaults to jumbo frames. Going to LAN and back can in fact use external hardware (IIRC there may be a system option to disable that, not sure) and so is constrained by that. Jim -- Typos courtesy of K-9 Mail on my Android From ikaufman at eng.ucsd.edu Thu Sep 14 15:07:06 2017 From: ikaufman at eng.ucsd.edu (Ian Kaufman) Date: Thu, 14 Sep 2017 08:07:06 -0700 Subject: [OmniOS-discuss] questions In-Reply-To: <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> Message-ID: Some other things you need to take into account: QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. That is also a theoretical maximum throughput, there is some overhead. In reality, you will never see 40Gbps. My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the theoretical max rates to 18Gbps and 36Gbps respectively. If you are getting 185MB/s, you are seeing 1.48Gbps. Keep your B's and b's straight. Did you play with your frame size at all? Ian On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov wrote: > On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems < > dirk.willems at exitas.be> wrote: > >Hello, > > > > > >I'm trying to understand something let me explain. > > > > > >Oracle always told to me that if you create a etherstub switch it has > >infiniband speed 40GB/s. > > > >But I have a customer running on Solaris (Yeah I know but let me > >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan > >(I know told him to to use etherstub). > > > >The copy witch is performed for a Oracle database with sql command, the > > > >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 > >- > >60 % busy the speed is 30 mb/s. > > > > > >So I did some test just to see and understand if it's the database or > >the system, but with doing my tests I get very confused ??? > > > > > >On another Solaris at my work copy over etherstub switch => copy speed > >is 185MB/s expected much more of infiniband speed ??? > > > > > >root at test1:/export/home/Admin# scp test10G > >Admin at 192.168.1.2:/export/home/Admin/ > >Password: > >test10G 100% > >|****************************************************************| > >10240 > >MB 00:59 > > > > > >root at test2:~# dlstat -i 2 > > > > LINK IPKTS RBYTES OPKTS OBYTES > > net1 25.76K 185.14M 10.08K 2.62M > > net1 27.04K 187.16M 11.23K 3.22M > > net1 26.97K 186.37M 11.24K 3.23M > > net1 26.63K 187.67M 10.82K 2.99M > > net1 27.94K 186.65M 12.17K 3.75M > > net1 27.45K 187.46M 11.70K 3.47M > > net1 26.01K 181.95M 10.63K 2.99M > > net1 27.95K 188.19M 12.14K 3.69M > > net1 27.91K 188.36M 12.08K 3.64M > > > >The disks are all separate luns with all separated pools => disk are 20 > > > >- 30% busy > > > > > >On my OmniOSce at my lab over etherstub > > > > > >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ > >Password: > >test10G 76% 7853MB 116.4MB/s > > > > > >=> copy is 116.4 MB/s => expected much more from infiniband speed is > >just the same as Lan ??? > > > > > >Is not that my disk can not follow 17% busy there sleeping ... > > > > extended device statistics > > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => > >rpool > > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => > >rpool > > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => > >data pool > > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => > >data pool > > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => > >data pool > > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => > >data pool > > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > > > > > >root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > > > LINK TYPE ID INDEX PKTS BYTES > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 9.26K 692.00K > > NGINX1 rx local -- 26.00K 216.32M > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 7.01K 531.38K > > NGINX1 rx local -- 30.65K 253.73M > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 8.95K 669.32K > > NGINX1 rx local -- 29.10K 241.15M > > > > > >On the other NGZ I receive 250MB/s ???? > > > > > >- So my question is how comes that the speed is equal to Lan 100MB/s on > > > >OmniOSce but i receive 250MB/s ? > > > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > > > > >I'm very confused right now ... > > > > > >And want to know for sure how to understand and see this in the right > >way, because this customer will be the first customer from my who gonna > > > >switch complety over to OmniOSce on production and because this > >customer > >is one or the biggest company's in Belgium I really don't want to mess > >up !!! > > > > > >So any help and clarification will be highly appreciate !!! > > > > > >Thank you very much. > > > > > >Kind Regards, > > > > > >Dirk > > I am not sure where the infiniband claim comes from, but copying data disk > to disk, you involve the slow layers like disk, skewed by faster layers > like cache of already-read data and delayed writes :) > > If you have a wide pipe that you may fill, it doesn't mean you do have the > means to fill it with a few disks. > > To estimate the speeds, try pure UDP streams from process to process (no > disk), large-packet floodping, etc. > > I believe etherstub is not constrained artificially, and defaults to jumbo > frames. Going to LAN and back can in fact use external hardware (IIRC there > may be a system option to disable that, not sure) and so is constrained by > that. > > Jim > -- > Typos courtesy of K-9 Mail on my Android > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirk.willems at exitas.be Thu Sep 14 16:13:30 2017 From: dirk.willems at exitas.be (Dirk Willems) Date: Thu, 14 Sep 2017 18:13:30 +0200 Subject: [OmniOS-discuss] questions In-Reply-To: References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> Message-ID: <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> Thank you all, the water is already clearing up :) So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they not take a standaard and set everything in GB/s or MB/s ? A lot of people make a lot of mistakes between them, me too ... If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s throughput. Little difference 40 or 5 :) So Ian you have the full blow with 36Gbps very cool looks more like it :) Did I play with the frame size, not really sure what you mean by that sorry but I think its default on 9000 Backend_Switch0 etherstub 9000 up Do understand that if we use UDP streams from process to process it will be much quicker over the etherstub gonna need more test to do. We used for a customer Mbuffer with zfs send over Lan that is also very quick sometimes I also use it at my home very good prog. But still do not understand how it is that I copy from 1 NGZ with 100MB/s, I receive on the other NGZ 250MB/s very strange ? the command dlstat difference between OmniOSce and Solaris ? RBYTES => receiving OBYTES => sending root at test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M BYTES => receiving and sending ? But then still if the copy is not running I have 0 so doesn't explain why I see 216 MB where come the rest of the 116 MB from is it compression ? root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M Thank you all for your feedback much appreciations ! Kind Regards, Dirk On 14-09-17 17:07, Ian Kaufman wrote: > Some other things you need to take into account: > > QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 > difference. That is also a theoretical maximum throughput, there is > some overhead. In reality, you will never see 40Gbps. > > My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR > (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the > theoretical max rates to 18Gbps and 36Gbps respectively. > > If you are getting 185MB/s, you are seeing 1.48Gbps. > > Keep your B's and b's straight. Did you play with your frame size at all? > > Ian > > On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov > wrote: > > On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems > > wrote: > >Hello, > > > > > >I'm trying to understand something let me explain. > > > > > >Oracle always told to me that if you create a etherstub switch it has > >infiniband speed 40GB/s. > > > >But I have a customer running on Solaris (Yeah I know but let me > >explain) who is copy from 1 NGZ to another NGZ on the same GZ > over Lan > >(I know told him to to use etherstub). > > > >The copy witch is performed for a Oracle database with sql > command, the > > > >DBA witch have 5 streams say it's waiting on the disk, the disk > are 50 > >- > >60 % busy the speed is 30 mb/s. > > > > > >So I did some test just to see and understand if it's the database or > >the system, but with doing my tests I get very confused ??? > > > > > >On another Solaris at my work copy over etherstub switch => copy > speed > >is 185MB/s expected much more of infiniband speed ??? > > > > > >root at test1:/export/home/Admin# scp test10G > >Admin at 192.168.1.2:/export/home/Admin/ > >Password: > >test10G 100% > >|****************************************************************| > >10240 > >MB 00:59 > > > > > >root at test2:~# dlstat -i 2 > > > > LINK IPKTS RBYTES OPKTS OBYTES > > net1 25.76K 185.14M 10.08K 2.62M > > net1 27.04K 187.16M 11.23K 3.22M > > net1 26.97K 186.37M 11.24K 3.23M > > net1 26.63K 187.67M 10.82K 2.99M > > net1 27.94K 186.65M 12.17K 3.75M > > net1 27.45K 187.46M 11.70K 3.47M > > net1 26.01K 181.95M 10.63K 2.99M > > net1 27.95K 188.19M 12.14K 3.69M > > net1 27.91K 188.36M 12.08K 3.64M > > > >The disks are all separate luns with all separated pools => disk > are 20 > > > >- 30% busy > > > > > >On my OmniOSce at my lab over etherstub > > > > > >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ > >Password: > >test10G 76% 7853MB 116.4MB/s > > > > > >=> copy is 116.4 MB/s => expected much more from infiniband speed is > >just the same as Lan ??? > > > > > >Is not that my disk can not follow 17% busy there sleeping ... > > > > extended device statistics > > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => > >rpool > > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => > >rpool > > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => > >data pool > > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => > >data pool > > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => > >data pool > > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => > >data pool > > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > > > > > >root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > > > LINK TYPE ID INDEX PKTS BYTES > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 9.26K 692.00K > > NGINX1 rx local -- 26.00K 216.32M > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 7.01K 531.38K > > NGINX1 rx local -- 30.65K 253.73M > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 8.95K 669.32K > > NGINX1 rx local -- 29.10K 241.15M > > > > > >On the other NGZ I receive 250MB/s ???? > > > > > >- So my question is how comes that the speed is equal to Lan > 100MB/s on > > > >OmniOSce but i receive 250MB/s ? > > > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > > > > >I'm very confused right now ... > > > > > >And want to know for sure how to understand and see this in the right > >way, because this customer will be the first customer from my who > gonna > > > >switch complety over to OmniOSce on production and because this > >customer > >is one or the biggest company's in Belgium I really don't want to > mess > >up !!! > > > > > >So any help and clarification will be highly appreciate !!! > > > > > >Thank you very much. > > > > > >Kind Regards, > > > > > >Dirk > > I am not sure where the infiniband claim comes from, but copying > data disk to disk, you involve the slow layers like disk, skewed > by faster layers like cache of already-read data and delayed writes :) > > If you have a wide pipe that you may fill, it doesn't mean you do > have the means to fill it with a few disks. > > To estimate the speeds, try pure UDP streams from process to > process (no disk), large-packet floodping, etc. > > I believe etherstub is not constrained artificially, and defaults > to jumbo frames. Going to LAN and back can in fact use external > hardware (IIRC there may be a system option to disable that, not > sure) and so is constrained by that. > > Jim > -- > Typos courtesy of K-9 Mail on my Android > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From ikaufman at eng.ucsd.edu Thu Sep 14 16:26:27 2017 From: ikaufman at eng.ucsd.edu (Ian Kaufman) Date: Thu, 14 Sep 2017 09:26:27 -0700 Subject: [OmniOS-discuss] questions In-Reply-To: <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> Message-ID: Networking has always used *bps - that's been the standard for many years. Megabits, Gigabits ... Disk tools have always measured in bytes since that is how the capacity is defined. How did you create your etherstub? I know you can set a maxbw (maximum bandiwdth), but I don't know what the default behavior is. Ian On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems wrote: > Thank you all, the water is already clearing up :) > > > So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they > not take a standaard and set everything in GB/s or MB/s ? > > A lot of people make a lot of mistakes between them, me too ... > > If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s > throughput. > > Little difference 40 or 5 :) > > So Ian you have the full blow with 36Gbps very cool looks more like it :) > > Did I play with the frame size, not really sure what you mean by that > sorry but I think its default on 9000 > > Backend_Switch0 etherstub 9000 up > > > Do understand that if we use UDP streams from process to process it will > be much quicker over the etherstub gonna need more test to do. > > We used for a customer Mbuffer with zfs send over Lan that is also very > quick sometimes I also use it at my home very good prog. > > But still do not understand how it is that I copy from 1 NGZ with 100MB/s, > I receive on the other NGZ 250MB/s very strange ? > > > the command dlstat difference between OmniOSce and Solaris ? > > RBYTES => receiving > > OBYTES => sending > > root at test2:~# dlstat -i 2 > > > > LINK IPKTS RBYTES OPKTS OBYTES > > net1 25.76K 185.14M 10.08K 2.62M > > net1 27.04K 187.16M 11.23K 3.22M > > > > BYTES => receiving and sending ? > > But then still if the copy is not running I have 0 so doesn't explain why > I see 216 MB where come the rest of the 116 MB from is it compression ? > > root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > > > LINK TYPE ID INDEX PKTS BYTES > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 9.26K 692.00K > > NGINX1 rx local -- 26.00K 216.32M > > > Thank you all for your feedback much appreciations ! > > > Kind Regards, > > > Dirk > > > > On 14-09-17 17:07, Ian Kaufman wrote: > > Some other things you need to take into account: > > QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. > That is also a theoretical maximum throughput, there is some overhead. In > reality, you will never see 40Gbps. > > My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR > (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the > theoretical max rates to 18Gbps and 36Gbps respectively. > > If you are getting 185MB/s, you are seeing 1.48Gbps. > > Keep your B's and b's straight. Did you play with your frame size at all? > > Ian > > On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov wrote: > >> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems < >> dirk.willems at exitas.be> wrote: >> >Hello, >> > >> > >> >I'm trying to understand something let me explain. >> > >> > >> >Oracle always told to me that if you create a etherstub switch it has >> >infiniband speed 40GB/s. >> > >> >But I have a customer running on Solaris (Yeah I know but let me >> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >> >(I know told him to to use etherstub). >> > >> >The copy witch is performed for a Oracle database with sql command, the >> > >> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >> >- >> >60 % busy the speed is 30 mb/s. >> > >> > >> >So I did some test just to see and understand if it's the database or >> >the system, but with doing my tests I get very confused ??? >> > >> > >> >On another Solaris at my work copy over etherstub switch => copy speed >> >is 185MB/s expected much more of infiniband speed ??? >> > >> > >> >root at test1:/export/home/Admin# scp test10G >> >Admin at 192.168.1.2:/export/home/Admin/ >> >Password: >> >test10G 100% >> >|****************************************************************| >> >10240 >> >MB 00:59 >> > >> > >> >root at test2:~# dlstat -i 2 >> > >> > LINK IPKTS RBYTES OPKTS OBYTES >> > net1 25.76K 185.14M 10.08K 2.62M >> > net1 27.04K 187.16M 11.23K 3.22M >> > net1 26.97K 186.37M 11.24K 3.23M >> > net1 26.63K 187.67M 10.82K 2.99M >> > net1 27.94K 186.65M 12.17K 3.75M >> > net1 27.45K 187.46M 11.70K 3.47M >> > net1 26.01K 181.95M 10.63K 2.99M >> > net1 27.95K 188.19M 12.14K 3.69M >> > net1 27.91K 188.36M 12.08K 3.64M >> > >> >The disks are all separate luns with all separated pools => disk are 20 >> > >> >- 30% busy >> > >> > >> >On my OmniOSce at my lab over etherstub >> > >> > >> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ >> >Password: >> >test10G 76% 7853MB 116.4MB/s >> > >> > >> >=> copy is 116.4 MB/s => expected much more from infiniband speed is >> >just the same as Lan ??? >> > >> > >> >Is not that my disk can not follow 17% busy there sleeping ... >> > >> > extended device statistics >> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device >> > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 >> > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => >> >rpool >> > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => >> >rpool >> > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => >> >data pool >> > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => >> >data pool >> > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => >> >data pool >> > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => >> >data pool >> > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool >> > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA >> > >> > >> > >> >root at NGINX:/root# dlstat show-link NGINX1 -i 2 >> > >> > LINK TYPE ID INDEX PKTS BYTES >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 9.26K 692.00K >> > NGINX1 rx local -- 26.00K 216.32M >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 7.01K 531.38K >> > NGINX1 rx local -- 30.65K 253.73M >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 8.95K 669.32K >> > NGINX1 rx local -- 29.10K 241.15M >> > >> > >> >On the other NGZ I receive 250MB/s ???? >> > >> > >> >- So my question is how comes that the speed is equal to Lan 100MB/s on >> > >> >OmniOSce but i receive 250MB/s ? >> > >> >- Why is etherstub so slow if infiniband speed is 40GB/s ??? >> > >> > >> >I'm very confused right now ... >> > >> > >> >And want to know for sure how to understand and see this in the right >> >way, because this customer will be the first customer from my who gonna >> > >> >switch complety over to OmniOSce on production and because this >> >customer >> >is one or the biggest company's in Belgium I really don't want to mess >> >up !!! >> > >> > >> >So any help and clarification will be highly appreciate !!! >> > >> > >> >Thank you very much. >> > >> > >> >Kind Regards, >> > >> > >> >Dirk >> >> I am not sure where the infiniband claim comes from, but copying data >> disk to disk, you involve the slow layers like disk, skewed by faster >> layers like cache of already-read data and delayed writes :) >> >> If you have a wide pipe that you may fill, it doesn't mean you do have >> the means to fill it with a few disks. >> >> To estimate the speeds, try pure UDP streams from process to process (no >> disk), large-packet floodping, etc. >> >> I believe etherstub is not constrained artificially, and defaults to >> jumbo frames. Going to LAN and back can in fact use external hardware (IIRC >> there may be a system option to disable that, not sure) and so is >> constrained by that. >> >> Jim >> -- >> Typos courtesy of K-9 Mail on my Android >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu > > > _______________________________________________ > OmniOS-discuss mailing listOmniOS-discuss at lists.omniti.comhttp://lists.omniti.com/mailman/listinfo/omnios-discuss > > > -- > Dirk Willems > System Engineer > > > +32 (0)3 443 12 38 <+32%203%20443%2012%2038> > Dirk.Willems at exitas.be > > Quality. Passion. Personality > > www.exitas.be | Veldkant 31 | 2550 Kontich > > Illumos OmniOS Installation and Configuration Implementation Specialist. > Oracle Solaris 11 Installation and Configuration Certified Implementation > Specialist. > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From dirk.willems at exitas.be Thu Sep 14 16:41:23 2017 From: dirk.willems at exitas.be (Dirk Willems) Date: Thu, 14 Sep 2017 18:41:23 +0200 Subject: [OmniOS-discuss] questions In-Reply-To: References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> Message-ID: just execute => dladm create-etherstub Backend_Switch0 and having => Backend_Switch0 etherstub 9000 up On 14-09-17 18:26, Ian Kaufman wrote: > Networking has always used *bps - that's been the standard for many > years. Megabits, Gigabits ... > > Disk tools have always measured in bytes since that is how the > capacity is defined. > > How did you create your etherstub? I know you can set a maxbw (maximum > bandiwdth), but I don't know what the default behavior is. > > Ian > > > > On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems > wrote: > > Thank you all, the water is already clearing up :) > > > So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps > why they not take a standaard and set everything in GB/s or MB/s ? > > A lot of people make a lot of mistakes between them, me too ... > > If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s > throughput. > > Little difference 40 or 5 :) > > So Ian you have the full blow with 36Gbps very cool looks more > like it :) > > Did I play with the frame size, not really sure what you mean by > that sorry but I think its default on 9000 > > Backend_Switch0 etherstub 9000 up > > > Do understand that if we use UDP streams from process to process > it will be much quicker over the etherstub gonna need more test to do. > > We used for a customer Mbuffer with zfs send over Lan that is also > very quick sometimes I also use it at my home very good prog. > > But still do not understand how it is that I copy from 1 NGZ with > 100MB/s, I receive on the other NGZ 250MB/s very strange ? > > > the command dlstat difference between OmniOSce and Solaris ? > > RBYTES => receiving > > OBYTES => sending > > root at test2:~# dlstat -i 2 > > > > LINK IPKTS RBYTES OPKTS OBYTES > > net1 25.76K 185.14M 10.08K 2.62M > > net1 27.04K 187.16M 11.23K 3.22M > > > > BYTES => receiving and sending ? > > But then still if the copy is not running I have 0 so doesn't > explain why I see 216 MB where come the rest of the 116 MB from is > it compression ? > > root at NGINX:/root# dlstat show-link NGINX1 -i 2 > > > > LINK TYPE ID INDEX PKTS BYTES > > NGINX1 rx bcast -- 0 0 > > NGINX1 rx sw -- 0 0 > > NGINX1 tx bcast -- 0 0 > > NGINX1 tx sw -- 9.26K 692.00K > > NGINX1 rx local -- 26.00K 216.32M > > > Thank you all for your feedback much appreciations ! > > > Kind Regards, > > > Dirk > > > > On 14-09-17 17:07, Ian Kaufman wrote: >> Some other things you need to take into account: >> >> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 >> difference. That is also a theoretical maximum throughput, there >> is some overhead. In reality, you will never see 40Gbps. >> >> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with >> DDR (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops >> the theoretical max rates to 18Gbps and 36Gbps respectively. >> >> If you are getting 185MB/s, you are seeing 1.48Gbps. >> >> Keep your B's and b's straight. Did you play with your frame size >> at all? >> >> Ian >> >> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov > > wrote: >> >> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems >> > wrote: >> >Hello, >> > >> > >> >I'm trying to understand something let me explain. >> > >> > >> >Oracle always told to me that if you create a etherstub >> switch it has >> >infiniband speed 40GB/s. >> > >> >But I have a customer running on Solaris (Yeah I know but let me >> >explain) who is copy from 1 NGZ to another NGZ on the same >> GZ over Lan >> >(I know told him to to use etherstub). >> > >> >The copy witch is performed for a Oracle database with sql >> command, the >> > >> >DBA witch have 5 streams say it's waiting on the disk, the >> disk are 50 >> >- >> >60 % busy the speed is 30 mb/s. >> > >> > >> >So I did some test just to see and understand if it's the >> database or >> >the system, but with doing my tests I get very confused ??? >> > >> > >> >On another Solaris at my work copy over etherstub switch => >> copy speed >> >is 185MB/s expected much more of infiniband speed ??? >> > >> > >> >root at test1:/export/home/Admin# scp test10G >> >Admin at 192.168.1.2:/export/ >> home/Admin/ >> >Password: >> >test10G 100% >> >|****************************************************************| >> >10240 >> >MB 00:59 >> > >> > >> >root at test2:~# dlstat -i 2 >> > >> > LINK IPKTS RBYTES OPKTS OBYTES >> > net1 25.76K 185.14M 10.08K 2.62M >> > net1 27.04K 187.16M 11.23K 3.22M >> > net1 26.97K 186.37M 11.24K 3.23M >> > net1 26.63K 187.67M 10.82K 2.99M >> > net1 27.94K 186.65M 12.17K 3.75M >> > net1 27.45K 187.46M 11.70K 3.47M >> > net1 26.01K 181.95M 10.63K 2.99M >> > net1 27.95K 188.19M 12.14K 3.69M >> > net1 27.91K 188.36M 12.08K 3.64M >> > >> >The disks are all separate luns with all separated pools => >> disk are 20 >> > >> >- 30% busy >> > >> > >> >On my OmniOSce at my lab over etherstub >> > >> > >> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/ >> home/witte/ >> >Password: >> >test10G 76% 7853MB 116.4MB/s >> > >> > >> >=> copy is 116.4 MB/s => expected much more from infiniband >> speed is >> >just the same as Lan ??? >> > >> > >> >Is not that my disk can not follow 17% busy there sleeping ... >> > >> > extended device statistics >> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w >> %b device >> > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 >> > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 >> c1t0d0 => >> >rpool >> > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 >> c1t1d0 => >> >rpool >> > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 >> c1t2d0 => >> >data pool >> > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 >> c1t3d0 => >> >data pool >> > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 >> c1t4d0 => >> >data pool >> > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 >> c1t5d0 => >> >data pool >> > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 >> rpool >> > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 >> DATA >> > >> > >> > >> >root at NGINX:/root# dlstat show-link NGINX1 -i 2 >> > >> > LINK TYPE ID INDEX PKTS BYTES >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 9.26K 692.00K >> > NGINX1 rx local -- 26.00K 216.32M >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 7.01K 531.38K >> > NGINX1 rx local -- 30.65K 253.73M >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 8.95K 669.32K >> > NGINX1 rx local -- 29.10K 241.15M >> > >> > >> >On the other NGZ I receive 250MB/s ???? >> > >> > >> >- So my question is how comes that the speed is equal to Lan >> 100MB/s on >> > >> >OmniOSce but i receive 250MB/s ? >> > >> >- Why is etherstub so slow if infiniband speed is 40GB/s ??? >> > >> > >> >I'm very confused right now ... >> > >> > >> >And want to know for sure how to understand and see this in >> the right >> >way, because this customer will be the first customer from >> my who gonna >> > >> >switch complety over to OmniOSce on production and because this >> >customer >> >is one or the biggest company's in Belgium I really don't >> want to mess >> >up !!! >> > >> > >> >So any help and clarification will be highly appreciate !!! >> > >> > >> >Thank you very much. >> > >> > >> >Kind Regards, >> > >> > >> >Dirk >> >> I am not sure where the infiniband claim comes from, but >> copying data disk to disk, you involve the slow layers like >> disk, skewed by faster layers like cache of already-read data >> and delayed writes :) >> >> If you have a wide pipe that you may fill, it doesn't mean >> you do have the means to fill it with a few disks. >> >> To estimate the speeds, try pure UDP streams from process to >> process (no disk), large-packet floodping, etc. >> >> I believe etherstub is not constrained artificially, and >> defaults to jumbo frames. Going to LAN and back can in fact >> use external hardware (IIRC there may be a system option to >> disable that, not sure) and so is constrained by that. >> >> Jim >> -- >> Typos courtesy of K-9 Mail on my Android >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> >> >> >> >> -- >> Ian Kaufman >> Research Systems Administrator >> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > > -- > Dirk Willems > System Engineer > > > +32 (0)3 443 12 38 > Dirk.Willems at exitas.be > > Quality. Passion. Personality > > www.exitas.be | Veldkant 31 | 2550 Kontich > > Illumos OmniOS Installation and Configuration Implementation > Specialist. > Oracle Solaris 11 Installation and Configuration Certified > Implementation Specialist. > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu -- Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From ikaufman at eng.ucsd.edu Thu Sep 14 16:46:40 2017 From: ikaufman at eng.ucsd.edu (Ian Kaufman) Date: Thu, 14 Sep 2017 09:46:40 -0700 Subject: [OmniOS-discuss] questions In-Reply-To: References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> Message-ID: What about the attached vnics? Can you do: dladm show-linkprop vnic# for the vnics connected to the etherstub? There may be a maxbw setting ... On Thu, Sep 14, 2017 at 9:41 AM, Dirk Willems wrote: > just execute => dladm create-etherstub Backend_Switch0 > > > and having => Backend_Switch0 etherstub 9000 up > > On 14-09-17 18:26, Ian Kaufman wrote: > > Networking has always used *bps - that's been the standard for many years. > Megabits, Gigabits ... > > Disk tools have always measured in bytes since that is how the capacity is > defined. > > How did you create your etherstub? I know you can set a maxbw (maximum > bandiwdth), but I don't know what the default behavior is. > > Ian > > > > On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems > wrote: > >> Thank you all, the water is already clearing up :) >> >> >> So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they >> not take a standaard and set everything in GB/s or MB/s ? >> >> A lot of people make a lot of mistakes between them, me too ... >> >> If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s >> throughput. >> >> Little difference 40 or 5 :) >> >> So Ian you have the full blow with 36Gbps very cool looks more like it :) >> >> Did I play with the frame size, not really sure what you mean by that >> sorry but I think its default on 9000 >> >> Backend_Switch0 etherstub 9000 up >> >> >> Do understand that if we use UDP streams from process to process it will >> be much quicker over the etherstub gonna need more test to do. >> >> We used for a customer Mbuffer with zfs send over Lan that is also very >> quick sometimes I also use it at my home very good prog. >> >> But still do not understand how it is that I copy from 1 NGZ with >> 100MB/s, I receive on the other NGZ 250MB/s very strange ? >> >> >> the command dlstat difference between OmniOSce and Solaris ? >> >> RBYTES => receiving >> >> OBYTES => sending >> >> root at test2:~# dlstat -i 2 >> > >> > LINK IPKTS RBYTES OPKTS OBYTES >> > net1 25.76K 185.14M 10.08K 2.62M >> > net1 27.04K 187.16M 11.23K 3.22M >> >> >> >> BYTES => receiving and sending ? >> >> But then still if the copy is not running I have 0 so doesn't explain why >> I see 216 MB where come the rest of the 116 MB from is it compression ? >> >> root at NGINX:/root# dlstat show-link NGINX1 -i 2 >> > >> > LINK TYPE ID INDEX PKTS BYTES >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 9.26K 692.00K >> > NGINX1 rx local -- 26.00K 216.32M >> >> >> Thank you all for your feedback much appreciations ! >> >> >> Kind Regards, >> >> >> Dirk >> >> >> >> On 14-09-17 17:07, Ian Kaufman wrote: >> >> Some other things you need to take into account: >> >> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. >> That is also a theoretical maximum throughput, there is some overhead. In >> reality, you will never see 40Gbps. >> >> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR >> (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the >> theoretical max rates to 18Gbps and 36Gbps respectively. >> >> If you are getting 185MB/s, you are seeing 1.48Gbps. >> >> Keep your B's and b's straight. Did you play with your frame size at all? >> >> Ian >> >> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov wrote: >> >>> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems < >>> dirk.willems at exitas.be> wrote: >>> >Hello, >>> > >>> > >>> >I'm trying to understand something let me explain. >>> > >>> > >>> >Oracle always told to me that if you create a etherstub switch it has >>> >infiniband speed 40GB/s. >>> > >>> >But I have a customer running on Solaris (Yeah I know but let me >>> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >>> >(I know told him to to use etherstub). >>> > >>> >The copy witch is performed for a Oracle database with sql command, the >>> > >>> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >>> >- >>> >60 % busy the speed is 30 mb/s. >>> > >>> > >>> >So I did some test just to see and understand if it's the database or >>> >the system, but with doing my tests I get very confused ??? >>> > >>> > >>> >On another Solaris at my work copy over etherstub switch => copy speed >>> >is 185MB/s expected much more of infiniband speed ??? >>> > >>> > >>> >root at test1:/export/home/Admin# scp test10G >>> >Admin at 192.168.1.2:/export/home/Admin/ >>> >Password: >>> >test10G 100% >>> >|****************************************************************| >>> >10240 >>> >MB 00:59 >>> > >>> > >>> >root at test2:~# dlstat -i 2 >>> > >>> > LINK IPKTS RBYTES OPKTS OBYTES >>> > net1 25.76K 185.14M 10.08K 2.62M >>> > net1 27.04K 187.16M 11.23K 3.22M >>> > net1 26.97K 186.37M 11.24K 3.23M >>> > net1 26.63K 187.67M 10.82K 2.99M >>> > net1 27.94K 186.65M 12.17K 3.75M >>> > net1 27.45K 187.46M 11.70K 3.47M >>> > net1 26.01K 181.95M 10.63K 2.99M >>> > net1 27.95K 188.19M 12.14K 3.69M >>> > net1 27.91K 188.36M 12.08K 3.64M >>> > >>> >The disks are all separate luns with all separated pools => disk are 20 >>> > >>> >- 30% busy >>> > >>> > >>> >On my OmniOSce at my lab over etherstub >>> > >>> > >>> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ >>> >Password: >>> >test10G 76% 7853MB 116.4MB/s >>> > >>> > >>> >=> copy is 116.4 MB/s => expected much more from infiniband speed is >>> >just the same as Lan ??? >>> > >>> > >>> >Is not that my disk can not follow 17% busy there sleeping ... >>> > >>> > extended device statistics >>> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device >>> > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 >>> > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => >>> >rpool >>> > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => >>> >rpool >>> > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => >>> >data pool >>> > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => >>> >data pool >>> > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => >>> >data pool >>> > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => >>> >data pool >>> > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool >>> > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA >>> > >>> > >>> > >>> >root at NGINX:/root# dlstat show-link NGINX1 -i 2 >>> > >>> > LINK TYPE ID INDEX PKTS BYTES >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 9.26K 692.00K >>> > NGINX1 rx local -- 26.00K 216.32M >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 7.01K 531.38K >>> > NGINX1 rx local -- 30.65K 253.73M >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 8.95K 669.32K >>> > NGINX1 rx local -- 29.10K 241.15M >>> > >>> > >>> >On the other NGZ I receive 250MB/s ???? >>> > >>> > >>> >- So my question is how comes that the speed is equal to Lan 100MB/s on >>> > >>> >OmniOSce but i receive 250MB/s ? >>> > >>> >- Why is etherstub so slow if infiniband speed is 40GB/s ??? >>> > >>> > >>> >I'm very confused right now ... >>> > >>> > >>> >And want to know for sure how to understand and see this in the right >>> >way, because this customer will be the first customer from my who gonna >>> > >>> >switch complety over to OmniOSce on production and because this >>> >customer >>> >is one or the biggest company's in Belgium I really don't want to mess >>> >up !!! >>> > >>> > >>> >So any help and clarification will be highly appreciate !!! >>> > >>> > >>> >Thank you very much. >>> > >>> > >>> >Kind Regards, >>> > >>> > >>> >Dirk >>> >>> I am not sure where the infiniband claim comes from, but copying data >>> disk to disk, you involve the slow layers like disk, skewed by faster >>> layers like cache of already-read data and delayed writes :) >>> >>> If you have a wide pipe that you may fill, it doesn't mean you do have >>> the means to fill it with a few disks. >>> >>> To estimate the speeds, try pure UDP streams from process to process (no >>> disk), large-packet floodping, etc. >>> >>> I believe etherstub is not constrained artificially, and defaults to >>> jumbo frames. Going to LAN and back can in fact use external hardware (IIRC >>> there may be a system option to disable that, not sure) and so is >>> constrained by that. >>> >>> Jim >>> -- >>> Typos courtesy of K-9 Mail on my Android >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >> >> >> >> -- >> Ian Kaufman >> Research Systems Administrator >> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu >> >> >> _______________________________________________ >> OmniOS-discuss mailing listOmniOS-discuss at lists.omniti.comhttp://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> >> -- >> Dirk Willems >> System Engineer >> >> >> +32 (0)3 443 12 38 <+32%203%20443%2012%2038> >> Dirk.Willems at exitas.be >> >> Quality. Passion. Personality >> >> www.exitas.be | Veldkant 31 >> | 2550 >> Kontich >> >> Illumos OmniOS Installation and Configuration Implementation Specialist. >> Oracle Solaris 11 Installation and Configuration Certified Implementation >> Specialist. >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu > > > -- > Dirk Willems > System Engineer > > > +32 (0)3 443 12 38 <+32%203%20443%2012%2038> > Dirk.Willems at exitas.be > > Quality. Passion. Personality > > www.exitas.be | Veldkant 31 > | 2550 > Kontich > > Illumos OmniOS Installation and Configuration Implementation Specialist. > Oracle Solaris 11 Installation and Configuration Certified Implementation > Specialist. > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From dirk.willems at exitas.be Thu Sep 14 16:55:51 2017 From: dirk.willems at exitas.be (Dirk Willems) Date: Thu, 14 Sep 2017 18:55:51 +0200 Subject: [OmniOS-discuss] questions In-Reply-To: References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru> <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be> Message-ID: Actuality have it test with this etherstub => NGINX_Switch0 But all the etherstub switches are created excellently the same But idd on the etherstub the mtu on Default is standing on 1500 but the value is on 9000 so 9000 is in use ? root at OmniOS:/root# dladm show-linkprop NGINX_Switch0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE NGINX_Switch0 autopush rw -- -- -- NGINX_Switch0 zone rw -- -- -- NGINX_Switch0 state r- up up up,down NGINX_Switch0 mtu rw 9000 1500 576-9000 NGINX_Switch0 maxbw rw -- -- -- NGINX_Switch0 cpus rw -- -- -- NGINX_Switch0 cpus-effective r- -- -- -- NGINX_Switch0 pool rw -- -- -- NGINX_Switch0 pool-effective r- -- -- -- NGINX_Switch0 priority rw high high low,medium,high NGINX_Switch0 forward rw 1 1 1,0 NGINX_Switch0 default_tag rw 1 1 -- NGINX_Switch0 learn_limit rw 1000 1000 -- NGINX_Switch0 learn_decay rw 200 200 -- NGINX_Switch0 stp rw 1 1 1,0 NGINX_Switch0 stp_priority rw 128 128 -- NGINX_Switch0 stp_cost rw auto auto -- NGINX_Switch0 stp_edge rw 1 1 1,0 NGINX_Switch0 stp_p2p rw auto auto true,false,auto NGINX_Switch0 stp_mcheck rw 0 0 1,0 NGINX_Switch0 protection rw -- -- mac-nospoof, restricted, ip-nospoof, dhcp-nospoof NGINX_Switch0 allowed-ips rw -- -- -- NGINX_Switch0 allowed-dhcp-cids rw -- -- -- NGINX_Switch0 rxrings rw -- -- -- NGINX_Switch0 rxrings-effective r- -- -- -- NGINX_Switch0 txrings rw -- -- -- NGINX_Switch0 txrings-effective r- -- -- -- NGINX_Switch0 txrings-available r- 0 -- -- NGINX_Switch0 rxrings-available r- 0 -- -- NGINX_Switch0 rxhwclnt-available r- 0 -- -- NGINX_Switch0 txhwclnt-available r- 0 -- -- root at OmniOS:/root# dladm show-linkprop NGINX1 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE NGINX1 autopush rw -- -- -- NGINX1 zone rw NGINX -- -- NGINX1 state r- up up up,down NGINX1 mtu rw 9000 9000 576-9000 NGINX1 secondary-macs rw -- -- -- NGINX1 maxbw rw -- -- -- NGINX1 cpus rw -- -- -- NGINX1 cpus-effective r- 0,23 -- -- NGINX1 pool rw -- -- -- NGINX1 pool-effective r- -- -- -- NGINX1 priority rw high high low,medium,high NGINX1 tagmode rw vlanonly vlanonly normal,vlanonly NGINX1 protection rw ip-nospoof -- mac-nospoof, restricted, ip-nospoof, dhcp-nospoof NGINX1 promisc-filtered rw on on off,on NGINX1 allowed-ips rw 192.168.20.3/32 -- -- NGINX1 allowed-dhcp-cids rw -- -- -- NGINX1 rxrings rw -- -- -- NGINX1 rxrings-effective r- -- -- -- NGINX1 txrings rw -- -- -- NGINX1 txrings-effective r- -- -- -- NGINX1 txrings-available r- 0 -- -- NGINX1 rxrings-available r- 0 -- -- NGINX1 rxhwclnt-available r- 0 -- -- NGINX1 txhwclnt-available r- 0 -- -- root at OmniOS:/root# dladm show-linkprop GNUHealth0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE GNUHealth0 autopush rw -- -- -- GNUHealth0 zone rw GNUHealth -- -- GNUHealth0 state r- up up up,down GNUHealth0 mtu rw 9000 9000 576-9000 GNUHealth0 secondary-macs rw -- -- -- GNUHealth0 maxbw rw -- -- -- GNUHealth0 cpus rw -- -- -- GNUHealth0 cpus-effective r- 21-22 -- -- GNUHealth0 pool rw -- -- -- GNUHealth0 pool-effective r- -- -- -- GNUHealth0 priority rw high high low,medium,high GNUHealth0 tagmode rw vlanonly vlanonly normal,vlanonly GNUHealth0 protection rw ip-nospoof -- mac-nospoof, restricted, ip-nospoof, dhcp-nospoof GNUHealth0 promisc-filtered rw on on off,on GNUHealth0 allowed-ips rw 192.168.20.5/32 -- -- GNUHealth0 allowed-dhcp-cids rw -- -- -- GNUHealth0 rxrings rw -- -- -- GNUHealth0 rxrings-effective r- -- -- -- GNUHealth0 txrings rw -- -- -- GNUHealth0 txrings-effective r- -- -- -- GNUHealth0 txrings-available r- 0 -- -- GNUHealth0 rxrings-available r- 0 -- -- GNUHealth0 rxhwclnt-available r- 0 -- -- GNUHealth0 txhwclnt-available r- 0 -- -- root at OmniOS:/root# On 14-09-17 18:46, Ian Kaufman wrote: > What about the attached vnics? > > Can you do: > > dladm show-linkprop vnic# for the vnics connected to the etherstub? > There may be a maxbw setting ... > > On Thu, Sep 14, 2017 at 9:41 AM, Dirk Willems > wrote: > > just execute => dladm create-etherstub Backend_Switch0 > > > and having => Backend_Switch0 etherstub 9000 up > > > On 14-09-17 18:26, Ian Kaufman wrote: >> Networking has always used *bps - that's been the standard for >> many years. Megabits, Gigabits ... >> >> Disk tools have always measured in bytes since that is how the >> capacity is defined. >> >> How did you create your etherstub? I know you can set a maxbw >> (maximum bandiwdth), but I don't know what the default behavior is. >> >> Ian >> >> >> >> On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems >> > wrote: >> >> Thank you all, the water is already clearing up :) >> >> >> So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s >> Gbps why they not take a standaard and set everything in GB/s >> or MB/s ? >> >> A lot of people make a lot of mistakes between them, me too ... >> >> If it is 40 Gbps a factor of 8 then we theoretical have max 5 >> GB/s throughput. >> >> Little difference 40 or 5 :) >> >> So Ian you have the full blow with 36Gbps very cool looks >> more like it :) >> >> Did I play with the frame size, not really sure what you mean >> by that sorry but I think its default on 9000 >> >> Backend_Switch0 etherstub 9000 up >> >> >> Do understand that if we use UDP streams from process to >> process it will be much quicker over the etherstub gonna need >> more test to do. >> >> We used for a customer Mbuffer with zfs send over Lan that is >> also very quick sometimes I also use it at my home very good >> prog. >> >> But still do not understand how it is that I copy from 1 NGZ >> with 100MB/s, I receive on the other NGZ 250MB/s very strange ? >> >> >> the command dlstat difference between OmniOSce and Solaris ? >> >> RBYTES => receiving >> >> OBYTES => sending >> >> root at test2:~# dlstat -i 2 >> > >> > LINK IPKTS RBYTES OPKTS OBYTES >> > net1 25.76K 185.14M 10.08K 2.62M >> > net1 27.04K 187.16M 11.23K 3.22M >> >> >> >> BYTES => receiving and sending ? >> >> But then still if the copy is not running I have 0 so doesn't >> explain why I see 216 MB where come the rest of the 116 MB >> from is it compression ? >> >> root at NGINX:/root# dlstat show-link NGINX1 -i 2 >> > >> > LINK TYPE ID INDEX PKTS BYTES >> > NGINX1 rx bcast -- 0 0 >> > NGINX1 rx sw -- 0 0 >> > NGINX1 tx bcast -- 0 0 >> > NGINX1 tx sw -- 9.26K 692.00K >> > NGINX1 rx local -- 26.00K 216.32M >> >> >> Thank you all for your feedback much appreciations ! >> >> >> Kind Regards, >> >> >> Dirk >> >> >> >> On 14-09-17 17:07, Ian Kaufman wrote: >>> Some other things you need to take into account: >>> >>> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 >>> difference. That is also a theoretical maximum throughput, >>> there is some overhead. In reality, you will never see 40Gbps. >>> >>> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, >>> with DDR (20Gbps) nodes and a QDR (40Gbps) storage server. >>> IPoIB drops the theoretical max rates to 18Gbps and 36Gbps >>> respectively. >>> >>> If you are getting 185MB/s, you are seeing 1.48Gbps. >>> >>> Keep your B's and b's straight. Did you play with your frame >>> size at all? >>> >>> Ian >>> >>> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov >>> > wrote: >>> >>> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems >>> > >>> wrote: >>> >Hello, >>> > >>> > >>> >I'm trying to understand something let me explain. >>> > >>> > >>> >Oracle always told to me that if you create a etherstub >>> switch it has >>> >infiniband speed 40GB/s. >>> > >>> >But I have a customer running on Solaris (Yeah I know >>> but let me >>> >explain) who is copy from 1 NGZ to another NGZ on the >>> same GZ over Lan >>> >(I know told him to to use etherstub). >>> > >>> >The copy witch is performed for a Oracle database with >>> sql command, the >>> > >>> >DBA witch have 5 streams say it's waiting on the disk, >>> the disk are 50 >>> >- >>> >60 % busy the speed is 30 mb/s. >>> > >>> > >>> >So I did some test just to see and understand if it's >>> the database or >>> >the system, but with doing my tests I get very confused ??? >>> > >>> > >>> >On another Solaris at my work copy over etherstub >>> switch => copy speed >>> >is 185MB/s expected much more of infiniband speed ??? >>> > >>> > >>> >root at test1:/export/home/Admin# scp test10G >>> >Admin at 192.168.1.2:/export/ >>> home/Admin/ >>> >Password: >>> >test10G 100% >>> >|****************************************************************| >>> >10240 >>> >MB 00:59 >>> > >>> > >>> >root at test2:~# dlstat -i 2 >>> > >>> > LINK IPKTS RBYTES OPKTS OBYTES >>> > net1 25.76K 185.14M 10.08K 2.62M >>> > net1 27.04K 187.16M 11.23K 3.22M >>> > net1 26.97K 186.37M 11.24K 3.23M >>> > net1 26.63K 187.67M 10.82K 2.99M >>> > net1 27.94K 186.65M 12.17K 3.75M >>> > net1 27.45K 187.46M 11.70K 3.47M >>> > net1 26.01K 181.95M 10.63K 2.99M >>> > net1 27.95K 188.19M 12.14K 3.69M >>> > net1 27.91K 188.36M 12.08K 3.64M >>> > >>> >The disks are all separate luns with all separated >>> pools => disk are 20 >>> > >>> >- 30% busy >>> > >>> > >>> >On my OmniOSce at my lab over etherstub >>> > >>> > >>> >root at GNUHealth:~# scp test10G >>> witte at 192.168.20.3:/export/ >>> home/witte/ >>> >Password: >>> >test10G 76% 7853MB 116.4MB/s >>> > >>> > >>> >=> copy is 116.4 MB/s => expected much more from >>> infiniband speed is >>> >just the same as Lan ??? >>> > >>> > >>> >Is not that my disk can not follow 17% busy there >>> sleeping ... >>> > >>> > extended device statistics >>> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t >>> %w %b device >>> > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 >>> 102 c1 >>> > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 >>> 17 c1t0d0 => >>> >rpool >>> > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 >>> 17 c1t1d0 => >>> >rpool >>> > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 >>> 17 c1t2d0 => >>> >data pool >>> > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 >>> 17 c1t3d0 => >>> >data pool >>> > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 >>> 18 c1t4d0 => >>> >data pool >>> > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 >>> 17 c1t5d0 => >>> >data pool >>> > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 >>> 18 rpool >>> > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 >>> 20 DATA >>> > >>> > >>> > >>> >root at NGINX:/root# dlstat show-link NGINX1 -i 2 >>> > >>> > LINK TYPE ID INDEX PKTS BYTES >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 9.26K 692.00K >>> > NGINX1 rx local -- 26.00K 216.32M >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 7.01K 531.38K >>> > NGINX1 rx local -- 30.65K 253.73M >>> > NGINX1 rx bcast -- 0 0 >>> > NGINX1 rx sw -- 0 0 >>> > NGINX1 tx bcast -- 0 0 >>> > NGINX1 tx sw -- 8.95K 669.32K >>> > NGINX1 rx local -- 29.10K 241.15M >>> > >>> > >>> >On the other NGZ I receive 250MB/s ???? >>> > >>> > >>> >- So my question is how comes that the speed is equal >>> to Lan 100MB/s on >>> > >>> >OmniOSce but i receive 250MB/s ? >>> > >>> >- Why is etherstub so slow if infiniband speed is >>> 40GB/s ??? >>> > >>> > >>> >I'm very confused right now ... >>> > >>> > >>> >And want to know for sure how to understand and see >>> this in the right >>> >way, because this customer will be the first customer >>> from my who gonna >>> > >>> >switch complety over to OmniOSce on production and >>> because this >>> >customer >>> >is one or the biggest company's in Belgium I really >>> don't want to mess >>> >up !!! >>> > >>> > >>> >So any help and clarification will be highly appreciate !!! >>> > >>> > >>> >Thank you very much. >>> > >>> > >>> >Kind Regards, >>> > >>> > >>> >Dirk >>> >>> I am not sure where the infiniband claim comes from, but >>> copying data disk to disk, you involve the slow layers >>> like disk, skewed by faster layers like cache of >>> already-read data and delayed writes :) >>> >>> If you have a wide pipe that you may fill, it doesn't >>> mean you do have the means to fill it with a few disks. >>> >>> To estimate the speeds, try pure UDP streams from >>> process to process (no disk), large-packet floodping, etc. >>> >>> I believe etherstub is not constrained artificially, and >>> defaults to jumbo frames. Going to LAN and back can in >>> fact use external hardware (IIRC there may be a system >>> option to disable that, not sure) and so is constrained >>> by that. >>> >>> Jim >>> -- >>> Typos courtesy of K-9 Mail on my Android >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >>> >>> >>> >>> >>> -- >>> Ian Kaufman >>> Research Systems Administrator >>> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd >>> DOT edu >>> >>> >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >> >> -- >> Dirk Willems >> System Engineer >> >> >> +32 (0)3 443 12 38 >> Dirk.Willems at exitas.be >> >> Quality. Passion. Personality >> >> www.exitas.be | Veldkant 31 >> >> | 2550 Kontich >> >> Illumos OmniOS Installation and Configuration Implementation >> Specialist. >> Oracle Solaris 11 Installation and Configuration Certified >> Implementation Specialist. >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> >> >> >> >> -- >> Ian Kaufman >> Research Systems Administrator >> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu > > -- > Dirk Willems > System Engineer > > > +32 (0)3 443 12 38 > Dirk.Willems at exitas.be > > Quality. Passion. Personality > > www.exitas.be | Veldkant 31 > | > 2550 Kontich > > Illumos OmniOS Installation and Configuration Implementation > Specialist. > Oracle Solaris 11 Installation and Configuration Certified > Implementation Specialist. > > > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu -- Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From danmcd at kebe.com Fri Sep 15 11:30:00 2017 From: danmcd at kebe.com (Dan McDonald) Date: Fri, 15 Sep 2017 07:30:00 -0400 Subject: [OmniOS-discuss] Fwd: Invitation to the SmartOS/OmniOS/OI/illumos user meeting at Erlangen, Germany on October 12 References: <7713BD48-D528-486E-A34A-344E7C5F7371@peterkelm.com> Message-ID: <944515EE-242E-4C4D-9029-B9A7C59FF2C5@kebe.com> Peter asked me to forward this to the list. Dan ===================== Von: Peter Kelm > Betreff: Invitation to the SmartOS/OmniOS/OI/illumos user meeting at Erlangen, Germany on October 12 Datum: 15. September 2017 um 11:15:28 MESZ An: openindiana-discuss-owner at openindiana.org , smartos-discuss at lists.smartos.org , omnios-discuss at lists.omniti.com All, We?d like to invite you to the SmartOS/OmniOS/OI/illumos user meeting at Erlangen on Oct 12, 2017. Location: KelmMoyles office (at the IGZ Erlangen) Am Weichselgarten 7 91058 Erlangen Germany Date/Time: Oct 12, 2017, starting at 6:00pm There is no firm agenda yet but please let us know what topics you?d be interested in to make this meeting as productive and useful as it can be. Your contributions are highly appreciated! I?d expect that some of the questions below might come up: - How do you use SmartOS/OmniOS/OI/illumos today or how do you intend to use it ?tomorrow?? - What are the factors that drive your interest? Where do you see strengths and weaknesses (vs. competing solutions)? - What are the ?best practices? that you?d like to share? (E.g. how do you handle maintenance and updates for your deployment?) Participation is of course free but we?d like to ask you to register via eMail to: illumos-meetup at kelmmoyles.com FYI: The timing coincides with the IT-SA fair at Nuremberg, so you might want to consider spending the day there before heading over to us: https://www.it-sa.de We are very open to additional thoughts and suggestions, so please let us know. Looking forward to seeing you in October! Peter P.S. We haven?t setup a ?meetup" group yet. Any volunteers? Dipl.-Ing. Peter Kelm KelmMoyles - Tailored Engineering Marketing Am Weichselgarten 7 91058 Erlangen T +49 (9132) 9060595 F +49 (9132) 9060596 E peter.kelm at kelmmoyles.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rjahnel at ellipseinc.com Fri Sep 15 14:10:06 2017 From: rjahnel at ellipseinc.com (Richard Jahnel) Date: Fri, 15 Sep 2017 14:10:06 +0000 Subject: [OmniOS-discuss] questions In-Reply-To: References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be> Message-ID: <65DC5816D4BEE043885A89FD54E273FC781396F9@MAIL101.Ellipseinc.com> /agreed for internal network transfers, netcat is the way to go. I have used it in the past for ZFS sends between internal machines with SSD backed volumes. All the crypto work from ssh/scp and other secure copy methods are the bottleneck in these cases. Obviously if you transfers leave the secure network you?ll want to suck it up and take the CPU hit. ;) From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Ludovic Orban Sent: Thursday, September 14, 2017 8:28 AM To: Dirk Willems Cc: omnios-discuss Subject: Re: [OmniOS-discuss] questions Here's a quick thought: if you are copying your files with scp, you might be CPU bound because of all the crypto work. You should try to copy with a much more CPU-lightweight tool. I personally usually use netcat for this kind of stuff. Just my 2 cents. -- Ludovic On Thu, Sep 14, 2017 at 2:26 PM, Dirk Willems > wrote: Hello, I'm trying to understand something let me explain. Oracle always told to me that if you create a etherstub switch it has infiniband speed 40GB/s. But I have a customer running on Solaris (Yeah I know but let me explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told him to to use etherstub). The copy witch is performed for a Oracle database with sql command, the DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60 % busy the speed is 30 mb/s. So I did some test just to see and understand if it's the database or the system, but with doing my tests I get very confused ??? On another Solaris at my work copy over etherstub switch => copy speed is 185MB/s expected much more of infiniband speed ??? root at test1:/export/home/Admin# scp test10G Admin at 192.168.1.2:/export/home/Admin/ Password: test10G 100% |****************************************************************| 10240 MB 00:59 root at test2:~# dlstat -i 2 LINK IPKTS RBYTES OPKTS OBYTES net1 25.76K 185.14M 10.08K 2.62M net1 27.04K 187.16M 11.23K 3.22M net1 26.97K 186.37M 11.24K 3.23M net1 26.63K 187.67M 10.82K 2.99M net1 27.94K 186.65M 12.17K 3.75M net1 27.45K 187.46M 11.70K 3.47M net1 26.01K 181.95M 10.63K 2.99M net1 27.95K 188.19M 12.14K 3.69M net1 27.91K 188.36M 12.08K 3.64M The disks are all separate luns with all separated pools => disk are 20 - 30% busy On my OmniOSce at my lab over etherstub root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/ Password: test10G 76% 7853MB 116.4MB/s => copy is 116.4 MB/s => expected much more from infiniband speed is just the same as Lan ??? Is not that my disk can not follow 17% busy there sleeping ... extended device statistics r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => rpool 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => rpool 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => data pool 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => data pool 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => data pool 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => data pool 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA root at NGINX:/root# dlstat show-link NGINX1 -i 2 LINK TYPE ID INDEX PKTS BYTES NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 9.26K 692.00K NGINX1 rx local -- 26.00K 216.32M NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 7.01K 531.38K NGINX1 rx local -- 30.65K 253.73M NGINX1 rx bcast -- 0 0 NGINX1 rx sw -- 0 0 NGINX1 tx bcast -- 0 0 NGINX1 tx sw -- 8.95K 669.32K NGINX1 rx local -- 29.10K 241.15M On the other NGZ I receive 250MB/s ???? - So my question is how comes that the speed is equal to Lan 100MB/s on OmniOSce but i receive 250MB/s ? - Why is etherstub so slow if infiniband speed is 40GB/s ??? I'm very confused right now ... And want to know for sure how to understand and see this in the right way, because this customer will be the first customer from my who gonna switch complety over to OmniOSce on production and because this customer is one or the biggest company's in Belgium I really don't want to mess up !!! So any help and clarification will be highly appreciate !!! Thank you very much. Kind Regards, Dirk -- [http://signatures.sidekick.be/exitas/images/placeholder-exitas.jpg] Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. [cid:image001.png at 01D32D3B.E8AE3E10] _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4648 bytes Desc: image001.png URL: From jim.oltman at gmail.com Fri Sep 15 18:10:19 2017 From: jim.oltman at gmail.com (Jim Oltman) Date: Fri, 15 Sep 2017 12:10:19 -0600 Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error In-Reply-To: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> References: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> Message-ID: On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald wrote: > Three swaps are three mountpoints using tmpfs. Please share "zfs list" > and "beadm list" again, please? > > Dan > > Sent from my iPhone (typos, autocorrect, and all) > > Sorry for the super late reply. Been busy at home and work. Here's Dan's requested output: joltman at OmniOS-NAS:/export/home/joltman$ zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 24.1G 37.4M 31K /rpool rpool/ROOT 7.90G 37.4M 23K legacy rpool/ROOT/20170826_01_Pre_OmniOS-CE 101K 37.4M 2.67G / rpool/ROOT/20170826_Pre_OmniOS-CE 2.16M 37.4M 2.56G / rpool/ROOT/20170826_Pre_OmniOS-CE-1 7.82G 37.4M 2.55G / rpool/ROOT/OmniOS-NAS_20170324_1226 125K 37.4M 2.62G / rpool/ROOT/omnios 58.9M 37.4M 2.46G / rpool/ROOT/omnios-1 1.08M 37.4M 2.52G / rpool/ROOT/omnios-2 3.22M 37.4M 2.56G / rpool/ROOT/omnios-2-backup-1 166K 37.4M 2.64G / rpool/ROOT/omnios-2-backup-2 163K 37.4M 2.53G / rpool/ROOT/omnios-2-backup-3 164K 37.4M 2.54G / rpool/ROOT/omnios-backup-1 220K 37.4M 1.72G / rpool/ROOT/omnios-backup-2 226K 37.4M 1.75G / rpool/ROOT/omniosvar 23K 37.4M 23K legacy rpool/ROOT/pre_download_17.01free_1490380649 1K 37.4M 2.61G / rpool/ROOT/pre_napp-it-16.07f 220K 37.4M 1.73G / rpool/ROOT/pre_napp-it-17.01free 166K 37.4M 2.54G / rpool/ROOT/r151020_upgrade_151022 1.89M 37.4M 2.56G / rpool/ROOT/r151020_upgrade_151022-backup-1 12.9M 37.4M 2.55G / rpool/ROOT/r151020_upgrade_151022-backup-2 150K 37.4M 2.55G / rpool/dump 12.0G 37.4M 12.0G - rpool/export 26.9M 37.4M 23K /export rpool/export/home 26.8M 37.4M 26.8M /export/home rpool/swap 4.13G 3.89G 276M - tank 2.34T 1.17T 36K /tank tank/backups 77.4G 1.17T 77.4G /tank/backups ... tank/joltman 93.7G 1.17T 60.9G /tank/joltman ... tank/mythtv 339G 1.17T 339G /tank/mythtv tank/pictures 34.4G 1.17T 34.3G /tank/pictures tank/public 3.62G 1.17T 3.60G /tank/public tank/seccams 20K 1.17T 20K /tank/seccams tank/software 73.9G 1.17T 73.9G /tank/software tank/videos 52.5K 1.17T 52.5K /tank/videos tank/voltman 55.3G 1.17T 55.2G /tank/voltman tank_00 14.0T 41.3G 272K /tank_00 tank_00/videos 14.0T 41.3G 14.0T /tank_00/videos tank_01 13.7T 355G 272K /tank_01 tank_01/videos 13.7T 355G 13.7T /tank_01/videos joltman at OmniOS-NAS:/export/home/joltman$ joltman at OmniOS-NAS:/export/home/joltman$ beadm list BE Active Mountpoint Space Policy Created omnios - - 58.9M static 2016-11-04 15:18 omniosvar - - 23.0K static 2016-11-04 15:18 omnios-backup-1 - - 220K static 2016-11-11 15:38 pre_napp-it-16.07f - - 220K static 2016-11-11 15:40 omnios-backup-2 - - 226K static 2016-11-11 15:40 omnios-1 - - 1.08M static 2016-11-21 15:28 OmniOS-NAS_20170324_1226 - - 125K static 2017-03-24 12:27 omnios-2 - - 3.22M static 2017-03-24 12:27 pre_download_17.01free_1490380649 - - 1.00K static 2017-03-24 12:37 omnios-2-backup-1 - - 166K static 2017-03-28 17:06 omnios-2-backup-2 - - 163K static 2017-04-16 18:33 omnios-2-backup-3 - - 164K static 2017-05-13 22:45 r151020_upgrade_151022 - - 1.89M static 2017-05-13 22:57 pre_napp-it-17.01free - - 166K static 2017-05-18 18:05 r151020_upgrade_151022-backup-1 N / 12.9M static 2017-08-26 12:29 r151020_upgrade_151022-backup-2 - - 150K static 2017-08-26 12:30 20170826_Pre_OmniOS-CE - - 2.16M static 2017-08-26 12:37 20170826_01_Pre_OmniOS-CE - - 101K static 2017-08-26 15:02 20170826_Pre_OmniOS-CE-1 R - 9.05G static 2017-08-26 15:02 joltman at OmniOS-NAS:/export/home/joltman$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasslerd at gmx.li Fri Sep 15 18:21:41 2017 From: hasslerd at gmx.li (Dominik Hassler) Date: Fri, 15 Sep 2017 20:21:41 +0200 Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error In-Reply-To: References: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> Message-ID: <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> It looks like your rpool is running out of free space. On 09/15/2017 08:10 PM, Jim Oltman wrote: > On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald > wrote: > > Three swaps are three mountpoints using tmpfs. Please share "zfs > list" and "beadm list" again, please? > > Dan > > Sent from my iPhone (typos, autocorrect, and all) > > > Sorry for the super late reply. Been busy at home and work. Here's > Dan's requested output: > > joltman at OmniOS-NAS:/export/home/joltman$ zfs list > NAME USED AVAIL REFER > MOUNTPOINT > rpool 24.1G 37.4M 31K /rpool > rpool/ROOT 7.90G 37.4M 23K legacy > rpool/ROOT/20170826_01_Pre_OmniOS-CE 101K 37.4M 2.67G / > rpool/ROOT/20170826_Pre_OmniOS-CE 2.16M 37.4M 2.56G / > rpool/ROOT/20170826_Pre_OmniOS-CE-1 7.82G 37.4M 2.55G / > rpool/ROOT/OmniOS-NAS_20170324_1226 125K 37.4M 2.62G / > rpool/ROOT/omnios 58.9M 37.4M 2.46G / > rpool/ROOT/omnios-1 1.08M 37.4M 2.52G / > rpool/ROOT/omnios-2 3.22M 37.4M 2.56G / > rpool/ROOT/omnios-2-backup-1 166K 37.4M 2.64G / > rpool/ROOT/omnios-2-backup-2 163K 37.4M 2.53G / > rpool/ROOT/omnios-2-backup-3 164K 37.4M 2.54G / > rpool/ROOT/omnios-backup-1 220K 37.4M 1.72G / > rpool/ROOT/omnios-backup-2 226K 37.4M 1.75G / > rpool/ROOT/omniosvar 23K 37.4M 23K legacy > rpool/ROOT/pre_download_17.01free_1490380649 1K 37.4M 2.61G / > rpool/ROOT/pre_napp-it-16.07f 220K 37.4M 1.73G / > rpool/ROOT/pre_napp-it-17.01free 166K 37.4M 2.54G / > rpool/ROOT/r151020_upgrade_151022 1.89M 37.4M 2.56G / > rpool/ROOT/r151020_upgrade_151022-backup-1 12.9M 37.4M 2.55G / > rpool/ROOT/r151020_upgrade_151022-backup-2 150K 37.4M 2.55G / > rpool/dump 12.0G 37.4M 12.0G - > rpool/export 26.9M 37.4M 23K /export > rpool/export/home 26.8M 37.4M 26.8M > /export/home > rpool/swap 4.13G 3.89G 276M - > tank 2.34T 1.17T 36K /tank > tank/backups 77.4G 1.17T 77.4G > /tank/backups > ... > tank/joltman 93.7G 1.17T 60.9G > /tank/joltman > ... > tank/mythtv 339G 1.17T 339G > /tank/mythtv > tank/pictures 34.4G 1.17T 34.3G > /tank/pictures > tank/public 3.62G 1.17T 3.60G > /tank/public > tank/seccams 20K 1.17T 20K > /tank/seccams > tank/software 73.9G 1.17T 73.9G > /tank/software > tank/videos 52.5K 1.17T 52.5K > /tank/videos > tank/voltman 55.3G 1.17T 55.2G > /tank/voltman > tank_00 14.0T 41.3G 272K /tank_00 > tank_00/videos 14.0T 41.3G 14.0T > /tank_00/videos > tank_01 13.7T 355G 272K /tank_01 > tank_01/videos 13.7T 355G 13.7T > /tank_01/videos > joltman at OmniOS-NAS:/export/home/joltman$ > > joltman at OmniOS-NAS:/export/home/joltman$ beadm list > BE Active Mountpoint Space Policy Created > omnios - - 58.9M static > 2016-11-04 15:18 > omniosvar - - 23.0K static > 2016-11-04 15:18 > omnios-backup-1 - - 220K static > 2016-11-11 15:38 > pre_napp-it-16.07f - - 220K static > 2016-11-11 15:40 > omnios-backup-2 - - 226K static > 2016-11-11 15:40 > omnios-1 - - 1.08M static > 2016-11-21 15:28 > OmniOS-NAS_20170324_1226 - - 125K static > 2017-03-24 12:27 > omnios-2 - - 3.22M static > 2017-03-24 12:27 > pre_download_17.01free_1490380649 - - 1.00K static > 2017-03-24 12:37 > omnios-2-backup-1 - - 166K static > 2017-03-28 17:06 > omnios-2-backup-2 - - 163K static > 2017-04-16 18:33 > omnios-2-backup-3 - - 164K static > 2017-05-13 22:45 > r151020_upgrade_151022 - - 1.89M static > 2017-05-13 22:57 > pre_napp-it-17.01free - - 166K static > 2017-05-18 18:05 > r151020_upgrade_151022-backup-1 N / 12.9M static > 2017-08-26 12:29 > r151020_upgrade_151022-backup-2 - - 150K static > 2017-08-26 12:30 > 20170826_Pre_OmniOS-CE - - 2.16M static > 2017-08-26 12:37 > 20170826_01_Pre_OmniOS-CE - - 101K static > 2017-08-26 15:02 > 20170826_Pre_OmniOS-CE-1 R - 9.05G static > 2017-08-26 15:02 > joltman at OmniOS-NAS:/export/home/joltman$ > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > From jim.oltman at gmail.com Fri Sep 15 18:46:35 2017 From: jim.oltman at gmail.com (Jim Oltman) Date: Fri, 15 Sep 2017 12:46:35 -0600 Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error In-Reply-To: <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> References: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> Message-ID: Dominik, Yes I saw that too, but I'm not sure why 12GB is being taken up by rpool/dump. And there's really not that much used by BEs. Is there a reason that dumps has to be so large or should I allocate more space to rpool? Thanks! Jim On Fri, Sep 15, 2017 at 12:21 PM, Dominik Hassler wrote: > It looks like your rpool is running out of free space. > > On 09/15/2017 08:10 PM, Jim Oltman wrote: > >> On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald > danmcd at kebe.com>> wrote: >> >> Three swaps are three mountpoints using tmpfs. Please share "zfs >> list" and "beadm list" again, please? >> >> Dan >> >> Sent from my iPhone (typos, autocorrect, and all) >> >> >> Sorry for the super late reply. Been busy at home and work. Here's >> Dan's requested output: >> >> joltman at OmniOS-NAS:/export/home/joltman$ zfs list >> NAME USED AVAIL REFER >> MOUNTPOINT >> rpool 24.1G 37.4M 31K /rpool >> rpool/ROOT 7.90G 37.4M 23K legacy >> rpool/ROOT/20170826_01_Pre_OmniOS-CE 101K 37.4M 2.67G / >> rpool/ROOT/20170826_Pre_OmniOS-CE 2.16M 37.4M 2.56G / >> rpool/ROOT/20170826_Pre_OmniOS-CE-1 7.82G 37.4M 2.55G / >> rpool/ROOT/OmniOS-NAS_20170324_1226 125K 37.4M 2.62G / >> rpool/ROOT/omnios 58.9M 37.4M 2.46G / >> rpool/ROOT/omnios-1 1.08M 37.4M 2.52G / >> rpool/ROOT/omnios-2 3.22M 37.4M 2.56G / >> rpool/ROOT/omnios-2-backup-1 166K 37.4M 2.64G / >> rpool/ROOT/omnios-2-backup-2 163K 37.4M 2.53G / >> rpool/ROOT/omnios-2-backup-3 164K 37.4M 2.54G / >> rpool/ROOT/omnios-backup-1 220K 37.4M 1.72G / >> rpool/ROOT/omnios-backup-2 226K 37.4M 1.75G / >> rpool/ROOT/omniosvar 23K 37.4M 23K legacy >> rpool/ROOT/pre_download_17.01free_1490380649 1K 37.4M 2.61G / >> rpool/ROOT/pre_napp-it-16.07f 220K 37.4M 1.73G / >> rpool/ROOT/pre_napp-it-17.01free 166K 37.4M 2.54G / >> rpool/ROOT/r151020_upgrade_151022 1.89M 37.4M 2.56G / >> rpool/ROOT/r151020_upgrade_151022-backup-1 12.9M 37.4M 2.55G / >> rpool/ROOT/r151020_upgrade_151022-backup-2 150K 37.4M 2.55G / >> rpool/dump 12.0G 37.4M 12.0G - >> rpool/export 26.9M 37.4M 23K /export >> rpool/export/home 26.8M 37.4M 26.8M >> /export/home >> rpool/swap 4.13G 3.89G 276M - >> tank 2.34T 1.17T 36K /tank >> tank/backups 77.4G 1.17T 77.4G >> /tank/backups >> ... >> tank/joltman 93.7G 1.17T 60.9G >> /tank/joltman >> ... >> tank/mythtv 339G 1.17T 339G >> /tank/mythtv >> tank/pictures 34.4G 1.17T 34.3G >> /tank/pictures >> tank/public 3.62G 1.17T 3.60G >> /tank/public >> tank/seccams 20K 1.17T 20K >> /tank/seccams >> tank/software 73.9G 1.17T 73.9G >> /tank/software >> tank/videos 52.5K 1.17T 52.5K >> /tank/videos >> tank/voltman 55.3G 1.17T 55.2G >> /tank/voltman >> tank_00 14.0T 41.3G 272K >> /tank_00 >> tank_00/videos 14.0T 41.3G 14.0T >> /tank_00/videos >> tank_01 13.7T 355G 272K >> /tank_01 >> tank_01/videos 13.7T 355G 13.7T >> /tank_01/videos >> joltman at OmniOS-NAS:/export/home/joltman$ >> >> joltman at OmniOS-NAS:/export/home/joltman$ beadm list >> BE Active Mountpoint Space Policy Created >> omnios - - 58.9M static >> 2016-11-04 15:18 >> omniosvar - - 23.0K static >> 2016-11-04 15:18 >> omnios-backup-1 - - 220K static >> 2016-11-11 15:38 >> pre_napp-it-16.07f - - 220K static >> 2016-11-11 15:40 >> omnios-backup-2 - - 226K static >> 2016-11-11 15:40 >> omnios-1 - - 1.08M static >> 2016-11-21 15:28 >> OmniOS-NAS_20170324_1226 - - 125K static >> 2017-03-24 12:27 >> omnios-2 - - 3.22M static >> 2017-03-24 12:27 >> pre_download_17.01free_1490380649 - - 1.00K static >> 2017-03-24 12:37 >> omnios-2-backup-1 - - 166K static >> 2017-03-28 17:06 >> omnios-2-backup-2 - - 163K static >> 2017-04-16 18:33 >> omnios-2-backup-3 - - 164K static >> 2017-05-13 22:45 >> r151020_upgrade_151022 - - 1.89M static >> 2017-05-13 22:57 >> pre_napp-it-17.01free - - 166K static >> 2017-05-18 18:05 >> r151020_upgrade_151022-backup-1 N / 12.9M static >> 2017-08-26 12:29 >> r151020_upgrade_151022-backup-2 - - 150K static >> 2017-08-26 12:30 >> 20170826_Pre_OmniOS-CE - - 2.16M static >> 2017-08-26 12:37 >> 20170826_01_Pre_OmniOS-CE - - 101K static >> 2017-08-26 15:02 >> 20170826_Pre_OmniOS-CE-1 R - 9.05G static >> 2017-08-26 15:02 >> joltman at OmniOS-NAS:/export/home/joltman$ >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From cks at cs.toronto.edu Fri Sep 15 20:13:43 2017 From: cks at cs.toronto.edu (Chris Siebenmann) Date: Fri, 15 Sep 2017 16:13:43 -0400 Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error In-Reply-To: jim.oltman's message of Fri, 15 Sep 2017 12:46:35 -0600. Message-ID: <20170915201343.DD0D45A0170@apps1.cs.toronto.edu> > Yes I saw that too, but I'm not sure why 12GB is being taken up by > rpool/dump. And there's really not that much used by BEs. Is there a > reason that dumps has to be so large or should I allocate more space > to rpool? Thanks! If I remember correctly, by default rpool/dump is sized during install based on the machine's physical memory. On machines with any decent amount of memory this can wind up using up an annoying amount of space, especially if you're installing on to not-that-huge system SSDs. My further understanding is that during crash dumps the kernel will use whatever free space is available in the rpool (in addition to whatever is reserved for rpool/dump), so that if you have the free space at the time you can capture even relatively large crash dumps. All that the rpool/dump reservation does is either guarantee or increase the chances that you can capture such large dumps. Our habitual practice is to resize rpool/dump down significantly on our OmniOS machines. I believe we reserve 4 GB by default, and even that might be considered generous. If you're short of space I wouldn't hesitate to either delete it entirely or shrink it drastically. (There are environments where reliably obtaining crash dumps if your kernel does go down is sufficiently important that it's worth paying for the extra disk space needed. My personal suspicion is that most people are not deploying OmniOS in such an environment.) - cks From johan.kragsterman at capvert.se Sat Sep 16 04:38:17 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sat, 16 Sep 2017 06:38:17 +0200 Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm error In-Reply-To: References: , <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> Message-ID: Hi! -----"OmniOS-discuss" skrev: ----- Till: Dominik Hassler Fr?n: Jim Oltman S?nt av: "OmniOS-discuss" Datum: 2017-09-15 20:48 Kopia: omnios-discuss ?rende: Re: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error Dominik, Yes I saw that too, but I'm not sure why 12GB is being taken up by rpool/dump. And there's really not that much used by BEs. Is there a reason that dumps has to be so large or should I allocate more space to rpool? Thanks! Jim Just move your rpool/dump to tank/dump instead. If you're concerned of the space of tank, use a sparse volume. You can move rpool/swap as well. Then you regain 16 GB of rpool space. Rgrds Johan joltman at OmniOS-NAS:/export/home/joltman$ zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 24.1G 37.4M 31K /rpool rpool/ROOT 7.90G 37.4M 23K legacy rpool/ROOT/20170826_01_Pre_OmniOS-CE 101K 37.4M 2.67G / rpool/ROOT/20170826_Pre_OmniOS-CE 2.16M 37.4M 2.56G / rpool/ROOT/20170826_Pre_OmniOS-CE-1 7.82G 37.4M 2.55G / rpool/ROOT/OmniOS-NAS_20170324_1226 125K 37.4M 2.62G / rpool/ROOT/omnios 58.9M 37.4M 2.46G / rpool/ROOT/omnios-1 1.08M 37.4M 2.52G / rpool/ROOT/omnios-2 3.22M 37.4M 2.56G / rpool/ROOT/omnios-2-backup-1 166K 37.4M 2.64G / rpool/ROOT/omnios-2-backup-2 163K 37.4M 2.53G / rpool/ROOT/omnios-2-backup-3 164K 37.4M 2.54G / rpool/ROOT/omnios-backup-1 220K 37.4M 1.72G / rpool/ROOT/omnios-backup-2 226K 37.4M 1.75G / rpool/ROOT/omniosvar 23K 37.4M 23K legacy rpool/ROOT/pre_download_17.01free_1490380649 1K 37.4M 2.61G / rpool/ROOT/pre_napp-it-16.07f 220K 37.4M 1.73G / rpool/ROOT/pre_napp-it-17.01free 166K 37.4M 2.54G / rpool/ROOT/r151020_upgrade_151022 1.89M 37.4M 2.56G / rpool/ROOT/r151020_upgrade_151022-backup-1 12.9M 37.4M 2.55G / rpool/ROOT/r151020_upgrade_151022-backup-2 150K 37.4M 2.55G / rpool/dump 12.0G 37.4M 12.0G - rpool/export 26.9M 37.4M 23K /export rpool/export/home 26.8M 37.4M 26.8M /export/home rpool/swap 4.13G 3.89G 276M - tank 2.34T 1.17T 36K /tank tank/backups 77.4G 1.17T 77.4G /tank/backups ... tank/joltman 93.7G 1.17T 60.9G /tank/joltman ... tank/mythtv 339G 1.17T 339G /tank/mythtv tank/pictures 34.4G 1.17T 34.3G /tank/pictures tank/public 3.62G 1.17T 3.60G /tank/public tank/seccams 20K 1.17T 20K /tank/seccams tank/software 73.9G 1.17T 73.9G /tank/software tank/videos 52.5K 1.17T 52.5K /tank/videos tank/voltman 55.3G 1.17T 55.2G /tank/voltman tank_00 14.0T 41.3G 272K /tank_00 tank_00/videos 14.0T 41.3G 14.0T /tank_00/videos tank_01 13.7T 355G 272K /tank_01 tank_01/videos 13.7T 355G 13.7T /tank_01/videos joltman at OmniOS-NAS:/export/home/joltman$ joltman at OmniOS-NAS:/export/home/joltman$ beadm list BE Active Mountpoint Space Policy Created omnios - - 58.9M static 2016-11-04 15:18 omniosvar - - 23.0K static 2016-11-04 15:18 omnios-backup-1 - - 220K static 2016-11-11 15:38 pre_napp-it-16.07f - - 220K static 2016-11-11 15:40 omnios-backup-2 - - 226K static 2016-11-11 15:40 omnios-1 - - 1.08M static 2016-11-21 15:28 OmniOS-NAS_20170324_1226 - - 125K static 2017-03-24 12:27 omnios-2 - - 3.22M static 2017-03-24 12:27 pre_download_17.01free_1490380649 - - 1.00K static 2017-03-24 12:37 omnios-2-backup-1 - - 166K static 2017-03-28 17:06 omnios-2-backup-2 - - 163K static 2017-04-16 18:33 omnios-2-backup-3 - - 164K static 2017-05-13 22:45 r151020_upgrade_151022 - - 1.89M static 2017-05-13 22:57 pre_napp-it-17.01free - - 166K static 2017-05-18 18:05 r151020_upgrade_151022-backup-1 N / 12.9M static 2017-08-26 12:29 r151020_upgrade_151022-backup-2 - - 150K static 2017-08-26 12:30 20170826_Pre_OmniOS-CE - - 2.16M static 2017-08-26 12:37 20170826_01_Pre_OmniOS-CE - - 101K static 2017-08-26 15:02 20170826_Pre_OmniOS-CE-1 R - 9.05G static 2017-08-26 15:02 joltman at OmniOS-NAS:/export/home/joltman$ _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From jimklimov at cos.ru Sat Sep 16 06:15:00 2017 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 16 Sep 2017 06:15:00 +0000 Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm error In-Reply-To: References: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> Message-ID: <80BA71CC-D1F9-4053-9BD2-2F54A860F07F@cos.ru> On September 16, 2017 6:38:17 AM GMT+02:00, Johan Kragsterman wrote: >Hi! > > >-----"OmniOS-discuss" skrev: >----- >Till: Dominik Hassler >Fr?n: Jim Oltman >S?nt av: "OmniOS-discuss" >Datum: 2017-09-15 20:48 >Kopia: omnios-discuss >?rende: Re: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error > >Dominik, > >Yes I saw that too, but I'm not sure why 12GB is being taken up by >rpool/dump. And there's really not that much used by BEs. Is there a >reason that dumps has to be so large or should I allocate more space to >rpool? Thanks! > >Jim > > > > > >Just move your rpool/dump to tank/dump instead. If you're concerned of >the space of tank, use a sparse volume. You can move rpool/swap as >well. Then you regain 16 GB of rpool space. > > >Rgrds Johan > > > > > > > > >joltman at OmniOS-NAS:/export/home/joltman$ zfs list >NAME USED AVAIL REFER >MOUNTPOINT >rpool 24.1G 37.4M 31K >/rpool >rpool/ROOT 7.90G 37.4M 23K >legacy >rpool/ROOT/20170826_01_Pre_OmniOS-CE 101K 37.4M 2.67G / >rpool/ROOT/20170826_Pre_OmniOS-CE 2.16M 37.4M 2.56G / >rpool/ROOT/20170826_Pre_OmniOS-CE-1 7.82G 37.4M 2.55G / >rpool/ROOT/OmniOS-NAS_20170324_1226 125K 37.4M 2.62G / >rpool/ROOT/omnios 58.9M 37.4M 2.46G / >rpool/ROOT/omnios-1 1.08M 37.4M 2.52G / >rpool/ROOT/omnios-2 3.22M 37.4M 2.56G / >rpool/ROOT/omnios-2-backup-1 166K 37.4M 2.64G / >rpool/ROOT/omnios-2-backup-2 163K 37.4M 2.53G / >rpool/ROOT/omnios-2-backup-3 164K 37.4M 2.54G / >rpool/ROOT/omnios-backup-1 220K 37.4M 1.72G / >rpool/ROOT/omnios-backup-2 226K 37.4M 1.75G / >rpool/ROOT/omniosvar 23K 37.4M 23K >legacy >rpool/ROOT/pre_download_17.01free_1490380649 1K 37.4M 2.61G / >rpool/ROOT/pre_napp-it-16.07f 220K 37.4M 1.73G / >rpool/ROOT/pre_napp-it-17.01free 166K 37.4M 2.54G / >rpool/ROOT/r151020_upgrade_151022 1.89M 37.4M 2.56G / >rpool/ROOT/r151020_upgrade_151022-backup-1 12.9M 37.4M 2.55G / >rpool/ROOT/r151020_upgrade_151022-backup-2 150K 37.4M 2.55G / >rpool/dump 12.0G 37.4M 12.0G - >rpool/export 26.9M 37.4M 23K >/export >rpool/export/home 26.8M 37.4M 26.8M >/export/home >rpool/swap 4.13G 3.89G 276M - >tank 2.34T 1.17T 36K >/tank >tank/backups 77.4G 1.17T 77.4G >/tank/backups >... >tank/joltman 93.7G 1.17T 60.9G >/tank/joltman >... >tank/mythtv 339G 1.17T 339G >/tank/mythtv >tank/pictures 34.4G 1.17T 34.3G >/tank/pictures >tank/public 3.62G 1.17T 3.60G >/tank/public >tank/seccams 20K 1.17T 20K >/tank/seccams >tank/software 73.9G 1.17T 73.9G >/tank/software >tank/videos 52.5K 1.17T 52.5K >/tank/videos >tank/voltman 55.3G 1.17T 55.2G >/tank/voltman >tank_00 14.0T 41.3G 272K >/tank_00 >tank_00/videos 14.0T 41.3G 14.0T >/tank_00/videos >tank_01 13.7T 355G 272K >/tank_01 >tank_01/videos 13.7T 355G 13.7T >/tank_01/videos >joltman at OmniOS-NAS:/export/home/joltman$ > >joltman at OmniOS-NAS:/export/home/joltman$ beadm list >BE Active Mountpoint Space Policy >Created >omnios - - 58.9M static >2016-11-04 15:18 >omniosvar - - 23.0K static >2016-11-04 15:18 >omnios-backup-1 - - 220K static >2016-11-11 15:38 >pre_napp-it-16.07f - - 220K static >2016-11-11 15:40 >omnios-backup-2 - - 226K static >2016-11-11 15:40 >omnios-1 - - 1.08M static >2016-11-21 15:28 >OmniOS-NAS_20170324_1226 - - 125K static >2017-03-24 12:27 >omnios-2 - - 3.22M static >2017-03-24 12:27 >pre_download_17.01free_1490380649 - - 1.00K static >2017-03-24 12:37 >omnios-2-backup-1 - - 166K static >2017-03-28 17:06 >omnios-2-backup-2 - - 163K static >2017-04-16 18:33 >omnios-2-backup-3 - - 164K static >2017-05-13 22:45 >r151020_upgrade_151022 - - 1.89M static >2017-05-13 22:57 >pre_napp-it-17.01free - - 166K static >2017-05-18 18:05 >r151020_upgrade_151022-backup-1 N / 12.9M static >2017-08-26 12:29 >r151020_upgrade_151022-backup-2 - - 150K static >2017-08-26 12:30 >20170826_Pre_OmniOS-CE - - 2.16M static >2017-08-26 12:37 >20170826_01_Pre_OmniOS-CE - - 101K static >2017-08-26 15:02 >20170826_Pre_OmniOS-CE-1 R - 9.05G static >2017-08-26 15:02 >joltman at OmniOS-NAS:/export/home/joltman$ > > >_______________________________________________ >OmniOS-discuss mailing list >OmniOS-discuss at lists.omniti.com >http://lists.omniti.com/mailman/listinfo/omnios-discuss > > >_______________________________________________ >OmniOS-discuss mailing list >OmniOS-discuss at lists.omniti.com >http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > >_______________________________________________ >OmniOS-discuss mailing list >OmniOS-discuss at lists.omniti.com >http://lists.omniti.com/mailman/listinfo/omnios-discuss But note there's dumpadm and /etc/vfstab to take care of to reconfigure the system's dumping and swapping locations. Jim -- Typos courtesy of K-9 Mail on my Android From vab at bb-c.de Sat Sep 16 09:32:35 2017 From: vab at bb-c.de (Volker A. Brandt) Date: Sat, 16 Sep 2017 11:32:35 +0200 Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm error In-Reply-To: References: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com> <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li> Message-ID: <22972.61363.317755.917107@shelob.bb-c.de> > Just move your rpool/dump to tank/dump instead. If you're concerned of the > space of tank, use a sparse volume. You can move rpool/swap as well. Then > you regain 16 GB of rpool space. I would not do that. If the system crashes because it cannot mount the "tank" pool, then you will not have a crash dump. :-) Note that "dumpadm -e" will tell you how much space it thinks it needs. Regards -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Oracle Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim, GERMANY Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 46 Gesch?ftsf?hrer: Rainer J.H. Brandt und Volker A. Brandt "When logic and proportion have fallen sloppy dead" From dirk.willems at exitas.be Sat Sep 16 14:59:35 2017 From: dirk.willems at exitas.be (Dirk Willems) Date: Sat, 16 Sep 2017 16:59:35 +0200 Subject: [OmniOS-discuss] upgrade Debian in LXZones Message-ID: Hello, Tried to upgrade my Debian LXZone but getting following ... So installed a new clean Debian LXZone and tried with that one and having exactly the same error Also don't see my interface in the new LXZones but is working root at debian-8:~# ifconfig -a OnlyOffice_vnic: error fetching interface information: No such device or address root at debian-8:~# ping 192.168.20.3 PING 192.168.20.3 (192.168.20.3) 56(84) bytes of data. 64 bytes from 192.168.20.3: icmp_seq=1 ttl=255 time=0.300 ms 64 bytes from 192.168.20.3: icmp_seq=2 ttl=255 time=0.111 ms 64 bytes from 192.168.20.3: icmp_seq=3 ttl=255 time=0.131 ms Any ideas ? Setting up udev (215-17+deb8u7) ... addgroup: The group `input' already exists as a system group. Exiting. udev does not support containers, not started. insserv: Service udev has to be enabled to start service udev-finish insserv: exiting now! update-rc.d: error: insserv rejected the script header dpkg: error processing package udev (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: udev E: Sub-process /usr/bin/dpkg returned an error code (1) root at debian-8:~# Thanks, Dirk -- Dirk Willems System Engineer +32 (0)3 443 12 38 Dirk.Willems at exitas.be Quality. Passion. Personality www.exitas.be | Veldkant 31 | 2550 Kontich Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SmallPhoenixLogotypeRGB.png Type: image/png Size: 4648 bytes Desc: not available URL: From johan.kragsterman at capvert.se Sun Sep 17 05:58:32 2017 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sun, 17 Sep 2017 07:58:32 +0200 Subject: [OmniOS-discuss] Ang: Re: [developer] ZFS cache device allocation and free seem strange on my OmniOS r151022 system. In-Reply-To: References: , <59AD7975.7010000@btconnect.com> Message-ID: Hi! -----Andy Fiddaman skrev: ----- Till: illumos-developer Fr?n: Andy Fiddaman Datum: 2017-09-17 00:32 Kopia: Andriy Gapon ?rende: Re: [developer] ZFS cache device allocation and free seem strange on my OmniOS r151022 system. On Thu, 14 Sep 2017, Matthew Ahrens wrote: ; The following commit should address the symptom you're describing; is it in ; your version of omnios? ; ; commit 6de76ce2a90f54fecb0dba46dca08c99cef7aa08 ; Author: Andriy Gapon ; Date: Mon Feb 27 14:48:16 2017 -0800 ; ; 7867 ARC space accounting leak ; Reviewed by: Matthew Ahrens ; Reviewed by: Dan Kimmel ; Approved by: Dan McDonald That commit was merged to illumos-omnios on the 7th of March, so is definitely in OmniOS r151022. Andy So apparently, that commit doesn't solve the problem...? It's a bit worrying...I got the exact same problem. Haven't hit me with any consequences yet, though, but that server doesn't have a lot of load. I remember when this first showed up, someone had substantial problems, and we were all recommended to remove any cache devices until the problem was solved. And now it seems the problem isn't solved...? So what is the recommendation now...? Remove it again? Is someone looking into this? Is it reopened as an issue again? Or is it marked as solved and closed? Rgrds Johan ------------------------------------------ illumos-developer Archives: https://illumos.topicbox.com/groups/developer/discussions/T737ffd52f97590b7-M831caefa636559cd9b878164 Powered by Topicbox: https://topicbox.com From danmcd at kebe.com Mon Sep 18 21:34:42 2017 From: danmcd at kebe.com (Dan McDonald) Date: Mon, 18 Sep 2017 17:34:42 -0400 Subject: [OmniOS-discuss] Apache update on omniti-ms? Message-ID: <20170918213442.GA46127@everywhere.local> Some of us still use omniti-ms for things. I know it's not a supported publisher, but I'm curious about this Apache bug: https://blog.fuzzing-project.org/60-Optionsbleed-HTTP-OPTIONS-method-can-leak-Apaches-server-memory.html And whether or not the apache24 package will get a bump to fix this? Thanks, Dan From vab at bb-c.de Tue Sep 19 05:47:50 2017 From: vab at bb-c.de (Volker A. Brandt) Date: Tue, 19 Sep 2017 07:47:50 +0200 Subject: [OmniOS-discuss] Apache update on omniti-ms? In-Reply-To: <20170918213442.GA46127@everywhere.local> References: <20170918213442.GA46127@everywhere.local> Message-ID: <22976.44934.28297.757373@shelob.bb-c.de> Dan McDonald writes: > Some of us still use omniti-ms for things. I know it's not a supported > publisher, but I'm curious about this Apache bug: > > https://blog.fuzzing-project.org/60-Optionsbleed-HTTP-OPTIONS-method-can-leak-Apaches-server-memory.html > > And whether or not the apache24 package will get a bump to fix this? There already is an "extra" repository for OmniOS CE. While I don't like yet another path ("/opt/ooce"), this is the right place for it. Now "someone" must do the work. :-) Where would I find patches etc. that were used to build the 2.2 in omniti-ms? Regards -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Oracle Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim, GERMANY Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 46 Gesch?ftsf?hrer: Rainer J.H. Brandt und Volker A. Brandt "When logic and proportion have fallen sloppy dead" From tobi at oetiker.ch Thu Sep 21 21:43:54 2017 From: tobi at oetiker.ch (Tobias Oetiker) Date: Thu, 21 Sep 2017 23:43:54 +0200 (CEST) Subject: [OmniOS-discuss] ANNOUNCEMENT OmniOS 151022s (Security Update) Message-ID: <780059164.637092.1506030234028.JavaMail.zimbra@oetiker.ch> Release Notes for OmniOSce 151022s ---------------------------------- Early weekly release for w/c 25th of September 2017, uname -a shows omnios-r151022-eb9d5cb557 Since this release contains security fixes we urge people to upgrade as soon as possible. ** This update requires a reboot. ** * Security updates for in-kernel CIFS client & server ?* 8662 SMB server ioctls should be appropriately sized ?* 8663 SMB client assumes serialized ioctls * Perl fixes: ??* CVE-2017-12837 ??* CVE-2017-12883 * Other changes ??* 8651 loader: fix problem where No rootfs module provided, aborting could appear on some systems. ??* IPsec observability improvements. Instructions for updating from OmniTI OmniOS r151022 are available on our web site http://www.omniosce.org/setup/switch Full Release Notes https://github.com/omniosorg/omnios-build/blob/r151022/doc/ReleaseNotes.md Due to the fix to the loader, new release media will be built for this release and will be available tomorrow. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From stephan.budach at jvm.de Fri Sep 22 18:18:30 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Fri, 22 Sep 2017 20:18:30 +0200 (CEST) Subject: [OmniOS-discuss] Can't destroy ZFS In-Reply-To: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> Hi, after having received a zvol from my old S11 box vis zfs send/recv and being unable to re-import the LUN via stmfadm, I decided to remove that zvol/ZFS and start over. However, I cannot remove that particular ZFS and trying to do so yields this strange error: root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists The zpool currently looks like this? (by the way, no snaps whatsoever on it): root at omnios:~# zfs list -r vmpool NAME USED AVAIL REFER MOUNTPOINT vmpool 348G 3.17T 23K /vmpool vmpool/esxi 39.4G 1.96T 39.4G /vmpool/esxi vmpool/iSCSI-Targets 238G 3.17T 23K /tank/iSCSI-Targets vmpool/iSCSI-Targets/EyeTV 238G 3.17T 238G - vmpool/nfsCloudData 70.3G 186G 70.3G /vmpool/nfsCloudData vmpool/nfsZimbraData 41K 100G 41K /vmpool/nfsZimbraData I have scrubbed the web a bit for this particular error, but all the reports seem to relate to either snapshots or clones. Any idea is greatly appeciated. Cheers, stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From peter.tribble at gmail.com Fri Sep 22 18:35:21 2017 From: peter.tribble at gmail.com (Peter Tribble) Date: Fri, 22 Sep 2017 19:35:21 +0100 Subject: [OmniOS-discuss] Can't destroy ZFS In-Reply-To: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> References: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de> <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: On Fri, Sep 22, 2017 at 7:18 PM, Stephan Budach wrote: > Hi, > > after having received a zvol from my old S11 box vis zfs send/recv and > being unable to re-import the LUN via stmfadm, I decided to remove that > zvol/ZFS and start over. However, I cannot remove that particular ZFS and > trying to do so yields this strange error: > > root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV > cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists > Does anything in this help: https://serverfault.com/questions/66414/cannot-destroy-zfs-snapshot-dataset-already-exists > The zpool currently looks like this? (by the way, no snaps whatsoever on > it): > > root at omnios:~# zfs list -r vmpool > NAME USED AVAIL REFER MOUNTPOINT > vmpool 348G 3.17T 23K /vmpool > vmpool/esxi 39.4G 1.96T 39.4G /vmpool/esxi > vmpool/iSCSI-Targets 238G 3.17T 23K /tank/iSCSI-Targets > vmpool/iSCSI-Targets/EyeTV 238G 3.17T 238G - > vmpool/nfsCloudData 70.3G 186G 70.3G /vmpool/nfsCloudData > vmpool/nfsZimbraData 41K 100G 41K /vmpool/nfsZimbraData > > > I have scrubbed the web a bit for this particular error, but all the > reports seem to relate to either snapshots or clones. > > Any idea is greatly appeciated. > > Cheers, > stephan > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephan.budach at jvm.de Fri Sep 22 18:37:59 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Fri, 22 Sep 2017 20:37:59 +0200 (CEST) Subject: [OmniOS-discuss] [SOLVED] Re: Can't destroy ZFS In-Reply-To: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> References: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: <764590023.49.1506105474726.JavaMail.stephan.budach@budy.stephanbudach.de> Shoot? ;) Right after hitting the send button, I issued this: root at omnios:~# zdb -d vmpool Dataset mos [META], ID 0, cr_txg 4, 20.1M, 309 objects Dataset vmpool/nfsZimbraData [ZPL], ID 82, cr_txg 29169, 41.0K, 16 objects Dataset vmpool/esxi [ZPL], ID 49, cr_txg 20, 40.7G, 119 objects Dataset vmpool/iSCSI-Targets/EyeTV/iSCSI-Targets [ZPL], ID 268, cr_txg 296266, 23.0K, 7 objects Dataset vmpool/iSCSI-Targets/EyeTV [ZVOL], ID 262, cr_txg 296222, 238G, 2 objects Dataset vmpool/iSCSI-Targets [ZPL], ID 256, cr_txg 296164, 23.0K, 9 objects Dataset vmpool/nfsCloudData [ZPL], ID 94, cr_txg 30110, 70.3G, 280046 objects Dataset vmpool [ZPL], ID 21, cr_txg 1, 23.0K, 10 objects Verified large_blocks feature refcount of 0 is correct Verified sha512 feature refcount of 0 is correct Verified skein feature refcount of 0 is correct Verified edonr feature refcount of 0 is correct After destroying this "invisible" Dataset vmpool/iSCSI-Targets/EyeTV/iSCSI-Targets dataset, I was able to delete it's parent datasets. Sorry for the noise. Cheers, stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From jbs at lean-on.com Sat Sep 23 09:49:27 2017 From: jbs at lean-on.com (=?utf-8?B?SmFrb2IgQi4gU8O4cmVuc2Vu?=) Date: Sat, 23 Sep 2017 09:49:27 +0000 Subject: [OmniOS-discuss] Can't destroy ZFS In-Reply-To: References: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de><1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: Hi there ? we work all the time with temporary datasets and sometimes it all locks up. Not to be cruel?. But ! Create a OI usb stick Use Gparted in the OI desktop ? delete it all. More cruel? Boot a W7 stick ? use the diskmanager - it will leave the drives in a unformatted state. If the disks are write protected state Use diskpart ? use the command attributes diskpart attributes remove read only then you are good to go ? and the disk are reset all he best Jakob Fra: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] P? vegne af Peter Tribble Sendt: 22. september 2017 20:35 Til: Stephan Budach Cc: omnios-discuss Emne: Re: [OmniOS-discuss] Can't destroy ZFS On Fri, Sep 22, 2017 at 7:18 PM, Stephan Budach > wrote: Hi, after having received a zvol from my old S11 box vis zfs send/recv and being unable to re-import the LUN via stmfadm, I decided to remove that zvol/ZFS and start over. However, I cannot remove that particular ZFS and trying to do so yields this strange error: root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists Does anything in this help: https://serverfault.com/questions/66414/cannot-destroy-zfs-snapshot-dataset-already-exists The zpool currently looks like this? (by the way, no snaps whatsoever on it): root at omnios:~# zfs list -r vmpool NAME USED AVAIL REFER MOUNTPOINT vmpool 348G 3.17T 23K /vmpool vmpool/esxi 39.4G 1.96T 39.4G /vmpool/esxi vmpool/iSCSI-Targets 238G 3.17T 23K /tank/iSCSI-Targets vmpool/iSCSI-Targets/EyeTV 238G 3.17T 238G - vmpool/nfsCloudData 70.3G 186G 70.3G /vmpool/nfsCloudData vmpool/nfsZimbraData 41K 100G 41K /vmpool/nfsZimbraData I have scrubbed the web a bit for this particular error, but all the reports seem to relate to either snapshots or clones. Any idea is greatly appeciated. Cheers, stephan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From apenner.it at gmail.com Mon Sep 25 08:22:28 2017 From: apenner.it at gmail.com (Artem Penner) Date: Mon, 25 Sep 2017 08:22:28 +0000 Subject: [OmniOS-discuss] scsi_vhci. Command Timeout on path Message-ID: Hi, all. I ran into the following problem. One of the zfs data disk is become unavailable and solaris marked it as bad for a too long time. As I understand, disk must be mark offline, if disk timeout grater than *10 seconds (scsi_timeout) * scsi_retry == 30 seconds*, but in log I see that I/O was degraded more than 5 minutes! How can I fix It? Why settings in sd.conf do not apply to all disks? "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2", *Actual sd_io_time and retries-timeout * root at znstor5-n1:~# echo "sd_io_time::print" | mdb -k 0xa # Per disk timeout and retry count root at znstor4-n1:~# echo "::walk sd_state | ::grep '.!=0' | ::sd_state" | mdb -k | egrep "^un|un_retry_count|un_cmd_timeout" un: ffffa1c0afbe9d00 un_retry_count = 5 un_cmd_timeout = 0xa un: ffffa1c0afbea980 un_retry_count = 5 un_cmd_timeout = 0xa un: ffffa1c0afbea340 un_retry_count = 5 un_cmd_timeout = 0xa un: ffffa1c13d99a980 un_retry_count = 3 un_cmd_timeout = 0xa un: ffffa1c13d2ca680 un_retry_count = 3 un_cmd_timeout = 0xa un: ffffa1c1443a4cc0 un_retry_count = 3 un_cmd_timeout = 0xa ........ ........ ----- *MY CONFIGURATION* *scsi_vhci.conf * # File is managed by Ansible name="scsi_vhci" class="root"; load-balance="round-robin"; auto-failback="enable"; ddi-forceload = "misc/scsi_vhci/scsi_vhci_f_asym_sun", "misc/scsi_vhci/scsi_vhci_f_asym_lsi", "misc/scsi_vhci/scsi_vhci_f_asym_emc", "misc/scsi_vhci/scsi_vhci_f_sym_emc", "misc/scsi_vhci/scsi_vhci_f_sym_hds", "misc/scsi_vhci/scsi_vhci_f_sym", "misc/scsi_vhci/scsi_vhci_f_tpgs"; scsi-vhci-failover-override = "HGST HUSMH8020BSS204", "f_sym", "HGST HUC101890CS4204", "f_sym", "TOSHIBA PX04SHB020", "f_sym", "TOSHIBA PX05SHB080", "f_sym"; *sd.conf* # File is managed by Ansible name="sd" class="scsi" target=0 lun=0; name="sd" class="scsi" target=1 lun=0; name="sd" class="scsi" target=2 lun=0; name="sd" class="scsi" target=3 lun=0; name="sd" class="scsi" target=4 lun=0; name="sd" class="scsi" target=5 lun=0; name="sd" class="scsi" target=6 lun=0; name="sd" class="scsi" target=7 lun=0; name="sd" class="scsi" target=8 lun=0; name="sd" class="scsi" target=9 lun=0; name="sd" class="scsi" target=10 lun=0; name="sd" class="scsi" target=11 lun=0; name="sd" class="scsi" target=12 lun=0; name="sd" class="scsi" target=13 lun=0; name="sd" class="scsi" target=14 lun=0; name="sd" class="scsi" target=15 lun=0; name="sd" class="scsi-self-identifying"; sd-config-list= "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2", "HGST HUC101890CS4204", "physical-block-size:4096", "HGST HUSMH8020BSS204", "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096", "TOSHIBA PX05SMB080", "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096", "TOSHIBA PX04SHB020", "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096"; sd_retry_on_reservation_conflict=0; *Pool configuration* config: NAME STATE READ WRITE CKSUM dpool2 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 c0t5000CCA02C417C30d0 ONLINE 0 0 0 c0t5000CCA02C3F3430d0 ONLINE 0 0 0 c0t5000CCA02C400064d0 ONLINE 0 0 0 c0t5000CCA02C417B10d0 ONLINE 0 0 0 spare-4 DEGRADED 0 0 0 c0t5000CCA02C3F5A78d0 UNAVAIL 141 4.59K 0 c0t5000CCA02C4015E4d0 ONLINE 0 0 0 c0t5000CCA02C417AA0d0 ONLINE 0 0 0 c0t5000CCA02C3E3C6Cd0 ONLINE 0 0 0 c0t5000CCA02C4170BCd0 ONLINE 0 0 0 c0t5000CCA02C41448Cd0 ONLINE 0 0 0 c0t5000CCA02C404B34d0 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c0t5000CCA02C3F0148d0 ONLINE 0 0 0 c0t5000CCA02C417D64d0 ONLINE 0 0 0 c0t5000CCA02C417E80d0 ONLINE 0 0 0 c0t5000CCA02C3F6A58d0 ONLINE 0 0 0 c0t5000CCA02C4113D8d0 ONLINE 0 0 0 c0t5000CCA02C411114d0 ONLINE 0 0 0 c0t5000CCA02C414D0Cd0 ONLINE 0 0 0 c0t5000CCA02C416FD0d0 ONLINE 0 0 0 c0t5000CCA02C3E7634d0 ONLINE 0 0 0 c0t5000CCA02C4143B8d0 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 c0t5000CCA02C416FA8d0 ONLINE 0 0 0 c0t5000CCA02C414CB0d0 ONLINE 0 0 0 c0t5000CCA02C414514d0 ONLINE 0 0 0 c0t5000CCA02C414260d0 ONLINE 0 0 0 c0t5000CCA02C3F6BE0d0 ONLINE 0 0 0 c0t5000CCA02C417EA4d0 ONLINE 0 0 0 c0t5000CCA02C414368d0 ONLINE 0 0 0 c0t5000CCA02C414014d0 ONLINE 0 0 0 c0t5000CCA02C416FACd0 ONLINE 0 0 0 c0t5000CCA02C409FB8d0 ONLINE 0 0 0 raidz2-3 ONLINE 0 0 0 c0t5000CCA02C41712Cd0 ONLINE 0 0 0 c0t5000CCA02C41436Cd0 ONLINE 0 0 0 c0t5000CCA02C414520d0 ONLINE 0 0 0 c0t5000CCA02C417AECd0 ONLINE 0 0 0 c0t5000CCA02C3F28FCd0 ONLINE 0 0 0 c0t5000CCA02C40ED00d0 ONLINE 0 0 0 c0t5000CCA02C416F94d0 ONLINE 0 0 0 c0t5000CCA02C403310d0 ONLINE 0 0 0 c0t5000CCA02C417AE0d0 ONLINE 0 0 0 c0t5000CCA02C4078C8d0 ONLINE 0 0 0 raidz2-6 ONLINE 0 0 0 c0t5000CCA02C2EADE8d0 ONLINE 0 0 0 c0t5000CCA02C2E1FD4d0 ONLINE 0 0 0 c0t5000CCA02C2E2044d0 ONLINE 0 0 0 c0t5000CCA02C2E6988d0 ONLINE 0 0 0 c0t5000CCA02C2E0E18d0 ONLINE 0 0 0 c0t5000CCA02C2C4C28d0 ONLINE 0 0 0 c0t5000CCA02C2CED00d0 ONLINE 0 0 0 c0t5000CCA02C2BF270d0 ONLINE 0 0 0 c0t5000CCA02C2C2954d0 ONLINE 0 0 0 c0t5000CCA02C2E1F90d0 ONLINE 0 0 0 raidz2-7 ONLINE 0 0 0 c0t5000CCA02C2E9428d0 ONLINE 0 0 0 c0t5000CCA02C2E5D1Cd0 ONLINE 0 0 0 c0t5000CCA02C2E8038d0 ONLINE 0 0 0 c0t5000CCA02C2C3C54d0 ONLINE 0 0 0 c0t5000CCA02C2E5F00d0 ONLINE 0 0 0 c0t5000CCA02C2E7838d0 ONLINE 0 0 0 c0t5000CCA02C2E38B8d0 ONLINE 0 0 0 c0t5000CCA02C2E8538d0 ONLINE 0 0 0 c0t5000CCA02C2E82D4d0 ONLINE 0 0 0 c0t5000CCA02C2E9978d0 ONLINE 0 0 0 raidz2-8 ONLINE 0 0 0 c0t5000CCA02C2E1FB8d0 ONLINE 0 0 0 c0t5000CCA02C2E1F10d0 ONLINE 0 0 0 c0t5000CCA02C2E1EF8d0 ONLINE 0 0 0 c0t5000CCA02C2E30A8d0 ONLINE 0 0 0 c0t5000CCA02C2E1FC0d0 ONLINE 0 0 0 c0t5000CCA02C2DD6C4d0 ONLINE 0 0 0 c0t5000CCA02C2E204Cd0 ONLINE 0 0 0 c0t5000CCA02C2E2054d0 ONLINE 0 0 0 c0t5000CCA03103B274d0 ONLINE 0 0 0 c0t5000CCA03103B2C8d0 ONLINE 0 0 0 raidz2-9 ONLINE 0 0 0 c0t5000CCA03103B34Cd0 ONLINE 0 0 0 c0t5000CCA03104AE18d0 ONLINE 0 0 0 c0t5000CCA03104AE54d0 ONLINE 0 0 0 c0t5000CCA03103B2D4d0 ONLINE 0 0 0 c0t5000CCA03103B26Cd0 ONLINE 0 0 0 c0t5000CCA03103B3ACd0 ONLINE 0 0 0 c0t5000CCA02C2E8344d0 ONLINE 0 0 0 c0t5000CCA02C2E8018d0 ONLINE 0 0 0 c0t5000CCA02C2E1EE4d0 ONLINE 0 0 0 c0t5000CCA081016134d0 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 c0t500003979C88FF55d0 ONLINE 0 0 0 c0t58CE38E06C897DD5d0 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 c0t500003973C88B6F9d0 ONLINE 0 0 0 c0t500003973C88B709d0 ONLINE 0 0 0 mirror-10 ONLINE 0 0 0 c0t58CE38E06C899C49d0 ONLINE 0 0 0 c0t500003979C88FF91d0 ONLINE 0 0 0 mirror-11 ONLINE 0 0 0 c0t58CE38E06C899C09d0 ONLINE 0 0 0 c0t500003979C88FF5Dd0 ONLINE 0 0 0 cache c0t5000CCA04B129374d0 ONLINE 0 0 0 c0t5000CCA04B12A1E4d0 ONLINE 0 0 0 c0t5000CCA04E27F1E8d0 ONLINE 0 0 0 c0t5000CCA04B12952Cd0 ONLINE 0 0 0 spares c0t5000CCA02C4015E4d0 INUSE c0t5000CCA02C416F7Cd0 AVAIL c0t5000CCA02C2E7850d0 AVAIL c0t5000CCA02C2E71F4d0 AVAIL *dmesg* Sep 24 18:03:59 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 18:03:59 znstor4-n1 /scsi_vhci/disk at g500003979c88ff55 (sd224): Command Timeout on path lsc5/disk at w500003979c88ff57,0 Sep 24 20:18:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:18:48 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:09 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:09 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:29 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:29 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:20:10 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:20:10 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:20:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:20:30 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:20:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:20:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:21:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:21:21 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:21:51 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:21:51 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:22:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:22:21 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:22:41 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:22:41 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:23:02 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:23:02 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:23:22 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:23:22 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:23:43 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:23:43 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:24:13 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:24:13 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:24:34 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:24:34 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:24 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:24 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:25:44 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:25:44 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:26:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:26:25 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:26:45 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:26:45 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:27:05 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:27:05 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:27:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:27:25 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:27:46 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:27:46 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:28:06 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:28:06 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:28:37 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:28:37 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:28:57 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:28:57 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:29:18 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:29:18 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:29:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:29:48 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:30:28 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:30:28 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:30:58 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:30:58 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 Sep 24 20:48:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 20:48:30 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0 Sep 24 20:51:44 znstor4-n1 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Sep 24 20:51:44 znstor4-n1 drive offline Sep 24 21:44:01 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 21:44:01 znstor4-n1 /scsi_vhci/disk at g500003973c88b709 (sd229): Command Timeout on path lsc4/disk at w500003973c88b70b,0 Sep 24 21:48:31 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 24 21:48:31 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From omnios at citrus-it.net Mon Sep 25 23:31:46 2017 From: omnios at citrus-it.net (Andy Fiddaman) Date: Mon, 25 Sep 2017 23:31:46 +0000 (UTC) Subject: [OmniOS-discuss] OmniOS bloody update - counting down to r151024 Message-ID: It's now around two weeks until we plan to freeze OmniOS bloody as we prepare for a r151024 release in November - as per the release schedule that we announced back in August ( https://www.omniosce.org/schedule ). In preparation for this, we have just updated our bloody IPS repository with all of the latest bits and would appreciate your help in putting them through their paces. If you have some spare cycles please grab the ISO and set up a bloody system for testing, or do a test upgrade of an r151022 system to a bloody boot-environment. The open VM tools package has been brought right up-to-date so it would also be good to get some feedback about running this version under different versions of VMware. Downloads can be found at https://downloads.omniosce.org/media/bloody/ More details and draft release notes for r151024 are at https://www.omniosce.org/article/bloody-20170925 Thanks in advance from the OmniOSce team. Please join us at https://gitter.im/omniosorg/Lobby to discuss this update. Regards, Andy -- Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From apenner.it at gmail.com Tue Sep 26 12:17:25 2017 From: apenner.it at gmail.com (Artem Penner) Date: Tue, 26 Sep 2017 12:17:25 +0000 Subject: [OmniOS-discuss] scsi_vhci. Command Timeout on path In-Reply-To: References: Message-ID: I reproduced issue. I have a pool with problem disks and I/o on this pool is extremely slow. here is *zpool iostat output:* root at znstor5-n2:~# zpool iostat dpool7 1 capacity operations bandwidth pool alloc free read write read write ------ ----- ----- ----- ----- ----- ----- dpool7 6.24T 124T 485 359 5.55M 3.23M dpool7 6.24T 124T 0 1 0 7.97K dpool7 6.24T 124T 83 44 1002K 360K dpool7 6.24T 124T 47 28 572K 246K dpool7 6.24T 124T 286 1.04K 3.35M 9.75M dpool7 6.24T 124T 783 532 9.18M 3.98M dpool7 6.24T 124T 5 2 71.5K 23.8K ^C iostat report 100% busy on two disks: *iostat -xnmMpz 1* extended device statistics r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 0.0 12.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 c3t50000397B831FBA6d0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t50000397B831FBA6d0s0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 c3t50000397B831FBA6d0s1 0.0 12.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 c1t50000397B83011BEd0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t50000397B83011BEd0s0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 c1t50000397B83011BEd0s1 0.0 12.0 0.0 0.0 0.0 0.0 0.0 2.5 0 3 c2t50000397B8307346d0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t50000397B8307346d0s0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 5.1 0 3 c2t50000397B8307346d0s1 0.0 0.0 0.0 0.0 7.0 0.9 0.0 0.0 100 91 c0t5000CCA02C51CFB0d0 0.0 0.0 0.0 0.0 7.0 0.9 0.0 0.0 100 91 c0t5000CCA02C51CFB0d0s0 0.0 0.0 0.0 0.0 9.0 0.9 0.0 0.0 100 89 c0t5000CCA02C51AB38d0 0.0 0.0 0.0 0.0 9.0 0.9 0.0 0.0 100 89 c0t5000CCA02C51AB38d0s0 in /var/adm/messages I see timeouts from disks: Sep 26 15:01:58 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:01:58 znstor5-n2 /scsi_vhci/disk at g5000cca02c51eb00 (sd50): Command Timeout on path lsc4/disk at w5000cca02c51eb02,0 Sep 26 15:02:09 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:09 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ea68 (sd52): Command Timeout on path lsc4/disk at w5000cca02c51ea6a,0 Sep 26 15:02:13 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:13 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ab38 (sd108): Command Timeout on path lsc3/disk at w5000cca02c51ab39,0 Sep 26 15:02:14 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:14 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ea54 (sd103): Command Timeout on path lsc2/disk at w5000cca02c51ea55,0 Sep 26 15:02:27 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:27 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ab38 (sd108): Command Timeout on path lsc3/disk at w5000cca02c51ab39,0 Sep 26 15:02:38 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:38 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ab38 (sd108): Command Timeout on path lsc3/disk at w5000cca02c51ab39,0 Sep 26 15:02:49 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): Sep 26 15:02:49 znstor5-n2 /scsi_vhci/disk at g5000cca02c51ab38 (sd108): Command Timeout on path lsc3/disk at w5000cca02c51ab39,0 My SCSI settings is follow: *echo "*sd_state::walk softstate | ::print -d struct sd_lun un_sd un_f_disksort_disabled un_f_suppress_cache_flush un_throttle un_min_throttle un_cmd_timeout un_retry_count un_victim_retry_count" | sudo mdb -k* un_sd = 0xffffa1c143aac6f8 un_f_disksort_disabled = 1 un_f_suppress_cache_flush = 1 un_throttle = 0t256 un_min_throttle = 8 un_cmd_timeout = 5 un_retry_count = 1 un_victim_retry_count = 2 *fmdump -eV *report following errors: Sep 26 2017 15:04:05.142507316 ereport.io.scsi.cmd.disk.tran nvlist version: 0 class = ereport.io.scsi.cmd.disk.tran ena = 0x582bda1c3c809c01 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev cna_dev = 0x59ca325100000064 device-path = /pci at 0,0/pci8086,6f08 at 3/pci1137,14c at 0/iport at f /disk at w5000cca02c51eb76,0 (end detector) devid = id1,sd at n5000cca02c51eb74 driver-assessment = fail op-code = 0x2a cdb = 0x2a 0x0 0x2 0x9f 0xc5 0x28 0x0 0x0 0xe 0x0 pkt-reason = 0x6 pkt-state = 0x17 pkt-stats = 0x50 __ttl = 0x1 __tod = 0x59ca4235 0x87e7d34 Sep 26 2017 15:04:07.392324757 ereport.io.scsi.cmd.disk.tran nvlist version: 0 class = ereport.io.scsi.cmd.disk.tran ena = 0x58219b9ffda04c1d detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev cna_dev = 0x59ca32510000010f device-path = /pci at 0,0/pci8086,6f08 at 3/pci1137,14c at 0/iport at f0 /disk at w5000cca02c51190a,0 (end detector) devid = id1,sd at n5000cca02c511908 driver-assessment = fail op-code = 0x28 cdb = 0x28 0x0 0x3 0xaa 0x10 0xec 0x0 0x0 0x1 0x0 pkt-reason = 0x4 pkt-state = 0x17 pkt-stats = 0x10 __ttl = 0x1 __tod = 0x59ca4237 0x17626695 ??, 25 ????. 2017 ?. ? 11:22, Artem Penner : > Hi, all. > > I ran into the following problem. One of the zfs data disk is become > unavailable and solaris marked it as bad for a too long time. > > As I understand, disk must be mark offline, if disk timeout grater than *10 > seconds (scsi_timeout) * scsi_retry == 30 seconds*, but in log I see > that I/O was degraded more than 5 minutes! > > How can I fix It? > > > Why settings in sd.conf do not apply to all disks? > > "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2", > > *Actual sd_io_time and retries-timeout * > > > root at znstor5-n1:~# echo "sd_io_time::print" | mdb -k > > 0xa > > > # Per disk timeout and retry count > > root at znstor4-n1:~# echo "::walk sd_state | ::grep '.!=0' | ::sd_state" | > mdb -k | egrep "^un|un_retry_count|un_cmd_timeout" > > un: ffffa1c0afbe9d00 > > un_retry_count = 5 > > un_cmd_timeout = 0xa > > un: ffffa1c0afbea980 > > un_retry_count = 5 > > un_cmd_timeout = 0xa > > un: ffffa1c0afbea340 > > un_retry_count = 5 > > un_cmd_timeout = 0xa > > un: ffffa1c13d99a980 > > un_retry_count = 3 > > un_cmd_timeout = 0xa > > un: ffffa1c13d2ca680 > > un_retry_count = 3 > > un_cmd_timeout = 0xa > > un: ffffa1c1443a4cc0 > > un_retry_count = 3 > > un_cmd_timeout = 0xa > > ........ > > ........ > > > ----- > > > *MY CONFIGURATION* > *scsi_vhci.conf * > > # File is managed by Ansible > > > > name="scsi_vhci" class="root"; > > > > load-balance="round-robin"; > > > > auto-failback="enable"; > > ddi-forceload = > > "misc/scsi_vhci/scsi_vhci_f_asym_sun", > > "misc/scsi_vhci/scsi_vhci_f_asym_lsi", > > "misc/scsi_vhci/scsi_vhci_f_asym_emc", > > "misc/scsi_vhci/scsi_vhci_f_sym_emc", > > "misc/scsi_vhci/scsi_vhci_f_sym_hds", > > "misc/scsi_vhci/scsi_vhci_f_sym", > > "misc/scsi_vhci/scsi_vhci_f_tpgs"; > > > > scsi-vhci-failover-override = > > "HGST HUSMH8020BSS204", "f_sym", > > "HGST HUC101890CS4204", "f_sym", > > "TOSHIBA PX04SHB020", "f_sym", > > "TOSHIBA PX05SHB080", "f_sym"; > > > *sd.conf* > > # File is managed by Ansible > > name="sd" class="scsi" target=0 lun=0; > > name="sd" class="scsi" target=1 lun=0; > > name="sd" class="scsi" target=2 lun=0; > > name="sd" class="scsi" target=3 lun=0; > > name="sd" class="scsi" target=4 lun=0; > > name="sd" class="scsi" target=5 lun=0; > > name="sd" class="scsi" target=6 lun=0; > > name="sd" class="scsi" target=7 lun=0; > > name="sd" class="scsi" target=8 lun=0; > > name="sd" class="scsi" target=9 lun=0; > > name="sd" class="scsi" target=10 lun=0; > > name="sd" class="scsi" target=11 lun=0; > > name="sd" class="scsi" target=12 lun=0; > > name="sd" class="scsi" target=13 lun=0; > > name="sd" class="scsi" target=14 lun=0; > > name="sd" class="scsi" target=15 lun=0; > > > > name="sd" class="scsi-self-identifying"; > > sd-config-list= > > "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2", > > "HGST HUC101890CS4204", "physical-block-size:4096", > > "HGST HUSMH8020BSS204", > "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096", > > "TOSHIBA PX05SMB080", > "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096", > > "TOSHIBA PX04SHB020", > "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096"; > > sd_retry_on_reservation_conflict=0; > > > *Pool configuration* > > config: > > > > NAME STATE READ WRITE CKSUM > > dpool2 DEGRADED 0 0 0 > > raidz2-0 DEGRADED 0 0 0 > > c0t5000CCA02C417C30d0 ONLINE 0 0 0 > > c0t5000CCA02C3F3430d0 ONLINE 0 0 0 > > c0t5000CCA02C400064d0 ONLINE 0 0 0 > > c0t5000CCA02C417B10d0 ONLINE 0 0 0 > > spare-4 DEGRADED 0 0 0 > > c0t5000CCA02C3F5A78d0 UNAVAIL 141 4.59K 0 > > c0t5000CCA02C4015E4d0 ONLINE 0 0 0 > > c0t5000CCA02C417AA0d0 ONLINE 0 0 0 > > c0t5000CCA02C3E3C6Cd0 ONLINE 0 0 0 > > c0t5000CCA02C4170BCd0 ONLINE 0 0 0 > > c0t5000CCA02C41448Cd0 ONLINE 0 0 0 > > c0t5000CCA02C404B34d0 ONLINE 0 0 0 > > raidz2-1 ONLINE 0 0 0 > > c0t5000CCA02C3F0148d0 ONLINE 0 0 0 > > c0t5000CCA02C417D64d0 ONLINE 0 0 0 > > c0t5000CCA02C417E80d0 ONLINE 0 0 0 > > c0t5000CCA02C3F6A58d0 ONLINE 0 0 0 > > c0t5000CCA02C4113D8d0 ONLINE 0 0 0 > > c0t5000CCA02C411114d0 ONLINE 0 0 0 > > c0t5000CCA02C414D0Cd0 ONLINE 0 0 0 > > c0t5000CCA02C416FD0d0 ONLINE 0 0 0 > > c0t5000CCA02C3E7634d0 ONLINE 0 0 0 > > c0t5000CCA02C4143B8d0 ONLINE 0 0 0 > > raidz2-2 ONLINE 0 0 0 > > c0t5000CCA02C416FA8d0 ONLINE 0 0 0 > > c0t5000CCA02C414CB0d0 ONLINE 0 0 0 > > c0t5000CCA02C414514d0 ONLINE 0 0 0 > > c0t5000CCA02C414260d0 ONLINE 0 0 0 > > c0t5000CCA02C3F6BE0d0 ONLINE 0 0 0 > > c0t5000CCA02C417EA4d0 ONLINE 0 0 0 > > c0t5000CCA02C414368d0 ONLINE 0 0 0 > > c0t5000CCA02C414014d0 ONLINE 0 0 0 > > c0t5000CCA02C416FACd0 ONLINE 0 0 0 > > c0t5000CCA02C409FB8d0 ONLINE 0 0 0 > > raidz2-3 ONLINE 0 0 0 > > c0t5000CCA02C41712Cd0 ONLINE 0 0 0 > > c0t5000CCA02C41436Cd0 ONLINE 0 0 0 > > c0t5000CCA02C414520d0 ONLINE 0 0 0 > > c0t5000CCA02C417AECd0 ONLINE 0 0 0 > > c0t5000CCA02C3F28FCd0 ONLINE 0 0 0 > > c0t5000CCA02C40ED00d0 ONLINE 0 0 0 > > c0t5000CCA02C416F94d0 ONLINE 0 0 0 > > c0t5000CCA02C403310d0 ONLINE 0 0 0 > > c0t5000CCA02C417AE0d0 ONLINE 0 0 0 > > c0t5000CCA02C4078C8d0 ONLINE 0 0 0 > > raidz2-6 ONLINE 0 0 0 > > c0t5000CCA02C2EADE8d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1FD4d0 ONLINE 0 0 0 > > c0t5000CCA02C2E2044d0 ONLINE 0 0 0 > > c0t5000CCA02C2E6988d0 ONLINE 0 0 0 > > c0t5000CCA02C2E0E18d0 ONLINE 0 0 0 > > c0t5000CCA02C2C4C28d0 ONLINE 0 0 0 > > c0t5000CCA02C2CED00d0 ONLINE 0 0 0 > > c0t5000CCA02C2BF270d0 ONLINE 0 0 0 > > c0t5000CCA02C2C2954d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1F90d0 ONLINE 0 0 0 > > raidz2-7 ONLINE 0 0 0 > > c0t5000CCA02C2E9428d0 ONLINE 0 0 0 > > c0t5000CCA02C2E5D1Cd0 ONLINE 0 0 0 > > c0t5000CCA02C2E8038d0 ONLINE 0 0 0 > > c0t5000CCA02C2C3C54d0 ONLINE 0 0 0 > > c0t5000CCA02C2E5F00d0 ONLINE 0 0 0 > > c0t5000CCA02C2E7838d0 ONLINE 0 0 0 > > c0t5000CCA02C2E38B8d0 ONLINE 0 0 0 > > c0t5000CCA02C2E8538d0 ONLINE 0 0 0 > > c0t5000CCA02C2E82D4d0 ONLINE 0 0 0 > > c0t5000CCA02C2E9978d0 ONLINE 0 0 0 > > raidz2-8 ONLINE 0 0 0 > > c0t5000CCA02C2E1FB8d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1F10d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1EF8d0 ONLINE 0 0 0 > > c0t5000CCA02C2E30A8d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1FC0d0 ONLINE 0 0 0 > > c0t5000CCA02C2DD6C4d0 ONLINE 0 0 0 > > c0t5000CCA02C2E204Cd0 ONLINE 0 0 0 > > c0t5000CCA02C2E2054d0 ONLINE 0 0 0 > > c0t5000CCA03103B274d0 ONLINE 0 0 0 > > c0t5000CCA03103B2C8d0 ONLINE 0 0 0 > > raidz2-9 ONLINE 0 0 0 > > c0t5000CCA03103B34Cd0 ONLINE 0 0 0 > > c0t5000CCA03104AE18d0 ONLINE 0 0 0 > > c0t5000CCA03104AE54d0 ONLINE 0 0 0 > > c0t5000CCA03103B2D4d0 ONLINE 0 0 0 > > c0t5000CCA03103B26Cd0 ONLINE 0 0 0 > > c0t5000CCA03103B3ACd0 ONLINE 0 0 0 > > c0t5000CCA02C2E8344d0 ONLINE 0 0 0 > > c0t5000CCA02C2E8018d0 ONLINE 0 0 0 > > c0t5000CCA02C2E1EE4d0 ONLINE 0 0 0 > > c0t5000CCA081016134d0 ONLINE 0 0 0 > > logs > > mirror-4 ONLINE 0 0 0 > > c0t500003979C88FF55d0 ONLINE 0 0 0 > > c0t58CE38E06C897DD5d0 ONLINE 0 0 0 > > mirror-5 ONLINE 0 0 0 > > c0t500003973C88B6F9d0 ONLINE 0 0 0 > > c0t500003973C88B709d0 ONLINE 0 0 0 > > mirror-10 ONLINE 0 0 0 > > c0t58CE38E06C899C49d0 ONLINE 0 0 0 > > c0t500003979C88FF91d0 ONLINE 0 0 0 > > mirror-11 ONLINE 0 0 0 > > c0t58CE38E06C899C09d0 ONLINE 0 0 0 > > c0t500003979C88FF5Dd0 ONLINE 0 0 0 > > cache > > c0t5000CCA04B129374d0 ONLINE 0 0 0 > > c0t5000CCA04B12A1E4d0 ONLINE 0 0 0 > > c0t5000CCA04E27F1E8d0 ONLINE 0 0 0 > > c0t5000CCA04B12952Cd0 ONLINE 0 0 0 > > spares > > c0t5000CCA02C4015E4d0 INUSE > > c0t5000CCA02C416F7Cd0 AVAIL > > c0t5000CCA02C2E7850d0 AVAIL > > c0t5000CCA02C2E71F4d0 AVAIL > > > > *dmesg* > > Sep 24 18:03:59 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 18:03:59 znstor4-n1 /scsi_vhci/disk at g500003979c88ff55 > (sd224): Command Timeout on path lsc5/disk at w500003979c88ff57,0 > > Sep 24 20:18:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:18:48 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:09 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:09 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:29 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:29 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:19:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:20:10 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:20:10 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:20:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:20:30 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:20:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:20:50 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:21:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:21:21 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:21:51 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:21:51 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:22:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:22:21 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:22:41 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:22:41 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:23:02 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:23:02 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:23:22 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:23:22 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:23:43 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:23:43 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:24:13 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:24:13 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:24:34 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:24:34 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:04 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:24 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:24 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:25:44 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:25:44 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:26:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:26:25 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:26:45 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:26:45 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:27:05 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:27:05 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:27:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:27:25 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:27:46 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:27:46 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:28:06 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:28:06 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:28:37 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:28:37 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:28:57 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:28:57 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:29:18 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:29:18 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:29:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:29:48 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:30:28 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:30:28 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:30:58 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:30:58 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0 > > Sep 24 20:48:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 20:48:30 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0 > > Sep 24 20:51:44 znstor4-n1 scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk at g5000cca02c3f5a78 (sd218): > > Sep 24 20:51:44 znstor4-n1 drive offline > > Sep 24 21:44:01 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 21:44:01 znstor4-n1 /scsi_vhci/disk at g500003973c88b709 > (sd229): Command Timeout on path lsc4/disk at w500003973c88b70b,0 > > Sep 24 21:48:31 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING: > /scsi_vhci (scsi_vhci0): > > Sep 24 21:48:31 znstor4-n1 /scsi_vhci/disk at g5000cca02c3f5a78 > (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0 > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icoomnios at gmail.com Wed Sep 27 07:57:28 2017 From: icoomnios at gmail.com (anthony omnios) Date: Wed, 27 Sep 2017 09:57:28 +0200 Subject: [OmniOS-discuss] write amplification zvol Message-ID: Hi, i have a problem, i used many ISCSI zvol (for each vm), network traffic is 2MB/s between kvm host and filer but i write on disks many more than that. I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung 850 evo 1To zpool status pool: filervm2 state: ONLINE scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017 config: NAME STATE READ WRITE CKSUM filervm2 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c7t5002538D41657AAFd0 ONLINE 0 0 0 c7t5002538D41F85C0Dd0 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 c7t5002538D41CC7105d0 ONLINE 0 0 0 c7t5002538D41CC7127d0 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 c7t5002538D41CD83FDd0 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 c4t2d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 i used correct ashift of 13 for samsung 850 evo zdb|grep ashift : ashift: 13 ashift: 13 ashift: 13 ashift: 13 ashift: 13 But i write a lot on ssd every 5 seconds (many more than the network traffic of 2 MB/s) iostat -xn -d 1 : r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0 I used zvol of 64k, i try with 8k and problem is the same. zfs get all filervm2/hdd-110022a : NAME PROPERTY VALUE SOURCE filervm2/hdd-110022a type volume - filervm2/hdd-110022a creation Tue May 16 10:24 2017 - filervm2/hdd-110022a used 5.26G - filervm2/hdd-110022a available 2.90T - filervm2/hdd-110022a referenced 5.24G - filervm2/hdd-110022a compressratio 3.99x - filervm2/hdd-110022a reservation none default filervm2/hdd-110022a volsize 25G local filervm2/hdd-110022a volblocksize 64K - filervm2/hdd-110022a checksum on default filervm2/hdd-110022a compression lz4 local filervm2/hdd-110022a readonly off default filervm2/hdd-110022a copies 1 default filervm2/hdd-110022a refreservation none default filervm2/hdd-110022a primarycache all default filervm2/hdd-110022a secondarycache all default filervm2/hdd-110022a usedbysnapshots 15.4M - filervm2/hdd-110022a usedbydataset 5.24G - filervm2/hdd-110022a usedbychildren 0 - filervm2/hdd-110022a usedbyrefreservation 0 - filervm2/hdd-110022a logbias latency default filervm2/hdd-110022a dedup off default filervm2/hdd-110022a mlslabel none default filervm2/hdd-110022a sync standard local filervm2/hdd-110022a refcompressratio 3.99x - filervm2/hdd-110022a written 216K - filervm2/hdd-110022a logicalused 20.9G - filervm2/hdd-110022a logicalreferenced 20.9G - filervm2/hdd-110022a snapshot_limit none default filervm2/hdd-110022a snapshot_count none default filervm2/hdd-110022a redundant_metadata all default Sorry for my bad english. What can be the problem ? thanks Best regards, Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From apenner.it at gmail.com Wed Sep 27 10:56:13 2017 From: apenner.it at gmail.com (Artem Penner) Date: Wed, 27 Sep 2017 10:56:13 +0000 Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: References: Message-ID: Use https://github.com/richardelling/tools/blob/master/iscsisvrtop to observe iscsi I/O ??, 27 ????. 2017 ?. ? 11:06, anthony omnios : > Hi, > > i have a problem, i used many ISCSI zvol (for each vm), network traffic is > 2MB/s between kvm host and filer but i write on disks many more than that. > I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung > 850 evo 1To > > zpool status > pool: filervm2 > state: ONLINE > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017 > config: > > NAME STATE READ WRITE CKSUM > filervm2 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > c7t5002538D41657AAFd0 ONLINE 0 0 0 > c7t5002538D41F85C0Dd0 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > c7t5002538D41CC7105d0 ONLINE 0 0 0 > c7t5002538D41CC7127d0 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 > c7t5002538D41CD83FDd0 ONLINE 0 0 0 > mirror-4 ONLINE 0 0 0 > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > c4t2d0 ONLINE 0 0 0 > c4t4d0 ONLINE 0 0 0 > > i used correct ashift of 13 for samsung 850 evo > zdb|grep ashift : > > ashift: 13 > ashift: 13 > ashift: 13 > ashift: 13 > ashift: 13 > > But i write a lot on ssd every 5 seconds (many more than the network > traffic of 2 MB/s) > > iostat -xn -d 1 : > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41657AAFd0 > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41CC7127d0 > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 > c7t5002538D41CC7105d0 > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41F85C0Dd0 > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 > c7t5002538D41CD83FDd0 > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 > c7t5002538D41CD7F7Ed0 > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 > c7t5002538D41CD7F7Dd0 > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 > c7t5002538D41CD7F7Ad0 > > I used zvol of 64k, i try with 8k and problem is the same. > > zfs get all filervm2/hdd-110022a : > > NAME PROPERTY VALUE SOURCE > filervm2/hdd-110022a type volume - > filervm2/hdd-110022a creation Tue May 16 10:24 2017 - > filervm2/hdd-110022a used 5.26G - > filervm2/hdd-110022a available 2.90T - > filervm2/hdd-110022a referenced 5.24G - > filervm2/hdd-110022a compressratio 3.99x - > filervm2/hdd-110022a reservation none default > filervm2/hdd-110022a volsize 25G local > filervm2/hdd-110022a volblocksize 64K - > filervm2/hdd-110022a checksum on default > filervm2/hdd-110022a compression lz4 local > filervm2/hdd-110022a readonly off default > filervm2/hdd-110022a copies 1 default > filervm2/hdd-110022a refreservation none default > filervm2/hdd-110022a primarycache all default > filervm2/hdd-110022a secondarycache all default > filervm2/hdd-110022a usedbysnapshots 15.4M - > filervm2/hdd-110022a usedbydataset 5.24G - > filervm2/hdd-110022a usedbychildren 0 - > filervm2/hdd-110022a usedbyrefreservation 0 - > filervm2/hdd-110022a logbias latency default > filervm2/hdd-110022a dedup off default > filervm2/hdd-110022a mlslabel none default > filervm2/hdd-110022a sync standard local > filervm2/hdd-110022a refcompressratio 3.99x - > filervm2/hdd-110022a written 216K - > filervm2/hdd-110022a logicalused 20.9G - > filervm2/hdd-110022a logicalreferenced 20.9G - > filervm2/hdd-110022a snapshot_limit none default > filervm2/hdd-110022a snapshot_count none default > filervm2/hdd-110022a redundant_metadata all default > > Sorry for my bad english. > > What can be the problem ? thanks > > Best regards, > > Anthony > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icoomnios at gmail.com Wed Sep 27 15:12:25 2017 From: icoomnios at gmail.com (anthony omnios) Date: Wed, 27 Sep 2017 17:12:25 +0200 Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: References: Message-ID: thanks, this is the result of the test: ./iscsisvrtop 1 30 >> /tmp/iscsisvrtop.txt more /tmp/iscsisvrtop.txt : Tracing... Please wait. 2017 Sep 27 17:01:48 load: 0.22 read_KB: 345 write_KB: 56 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 4 0 0 0 0 0 0 0 0 0 0 0 1.1.193.250 105 91 1 0 345 56 3 56 4 756 0 100 all 109 91 1 0 345 56 3 56 4 756 0 0 2017 Sep 27 17:01:49 load: 0.22 read_KB: 163 write_KB: 41 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 32 26 2 0 117 41 4 20 6 417 0 100 1.1.193.250 42 34 0 0 46 0 1 0 7 0 0 0 all 74 60 2 0 163 41 2 20 7 417 0 0 2017 Sep 27 17:01:50 load: 0.22 read_KB: 499 write_KB: 232 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 45 40 3 0 210 196 5 65 5 763 0 100 1.1.193.250 77 65 2 0 288 36 4 18 4 439 0 100 all 122 105 5 0 499 232 4 46 4 634 0 0 2017 Sep 27 17:01:51 load: 0.22 read_KB: 314 write_KB: 84 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 3 1 0 0 0 0 0 0 2 0 0 0 1.1.193.250 100 88 4 0 313 84 3 21 4 396 0 100 all 103 89 4 0 314 84 3 21 4 396 0 0 2017 Sep 27 17:01:52 load: 0.22 read_KB: 184 write_KB: 104 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 23 17 1 0 59 88 3 88 5 871 0 100 1.1.193.250 50 44 1 0 125 16 2 16 8 445 0 100 all 73 61 2 0 184 104 3 52 7 658 0 0 2017 Sep 27 17:01:53 load: 0.22 read_KB: 250 write_KB: 1920 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 7 6 0 0 12 0 2 0 6 0 0 0 1.1.193.250 71 44 16 0 263 1920 5 120 6 2531 0 100 all 78 50 16 0 276 1920 5 120 6 2531 0 0 2017 Sep 27 17:01:54 load: 0.22 read_KB: 93 write_KB: 0 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 7 0 0 0 0 0 0 0 0 0 0 0 1.1.193.250 38 28 0 0 70 0 2 0 6 0 0 0 all 45 28 0 0 70 0 2 0 6 0 0 0 2017 Sep 27 17:01:55 load: 0.22 read_KB: 467 write_KB: 156 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 23 21 0 0 23 0 1 0 6 0 0 0 1.1.193.250 115 106 4 0 441 156 4 39 5 538 0 100 all 138 127 4 0 464 156 3 39 5 538 0 0 2017 Sep 27 17:01:56 load: 0.22 read_KB: 485 write_KB: 152 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 16 13 0 0 22 0 1 0 2 0 0 0 1.1.193.250 133 119 4 0 462 152 3 38 4 427 0 100 all 149 132 4 0 485 152 3 38 3 427 0 0 2017 Sep 27 17:01:57 load: 0.22 read_KB: 804 write_KB: 248 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 36 33 1 0 137 104 4 104 6 1064 0 100 1.1.193.250 133 131 2 0 667 144 5 72 5 885 0 100 all 169 164 3 0 804 248 4 82 5 945 0 0 2017 Sep 27 17:01:58 load: 0.22 read_KB: 631 write_KB: 36 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 91 87 0 0 373 0 4 0 2 0 0 0 1.1.193.250 93 75 2 0 257 36 3 18 4 252 0 100 all 184 162 2 0 631 36 3 18 3 252 0 0 2017 Sep 27 17:01:59 load: 0.21 read_KB: 1472 write_KB: 764 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.250 76 68 6 0 281 636 4 106 4 803 0 100 1.1.193.247 265 262 2 0 1191 128 4 64 3 482 0 100 all 341 330 8 0 1472 764 4 95 3 723 0 0 2017 Sep 27 17:02:00 load: 0.21 read_KB: 3559 write_KB: 376 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 83 82 0 0 270 0 3 0 1 0 0 0 1.1.193.250 541 524 8 0 3289 376 6 47 6 359 0 100 all 624 606 8 0 3559 376 5 47 5 359 0 0 2017 Sep 27 17:02:01 load: 0.21 read_KB: 2079 write_KB: 232 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 120 118 0 0 612 0 5 0 2 0 0 0 1.1.193.250 418 416 2 0 1476 232 3 116 4 765 0 100 all 538 534 2 0 2088 232 3 116 4 765 0 0 2017 Sep 27 17:02:02 load: 0.21 read_KB: 2123 write_KB: 168 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 84 80 3 0 317 168 3 56 4 292 0 100 1.1.193.250 367 366 0 0 1812 0 4 0 6 0 0 0 all 451 446 3 0 2129 168 4 56 6 292 0 0 2017 Sep 27 17:02:03 load: 0.21 read_KB: 307 write_KB: 484 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 14 14 0 0 19 0 1 0 5 0 0 0 1.1.193.250 90 85 5 0 273 484 3 96 1 302 0 100 all 104 99 5 0 292 484 2 96 2 302 0 0 2017 Sep 27 17:02:04 load: 0.22 read_KB: 298 write_KB: 0 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 10 4 0 0 2 0 0 0 9 0 0 0 1.1.193.250 85 70 0 0 296 0 4 0 5 0 0 0 all 95 74 0 0 298 0 4 0 5 0 0 0 2017 Sep 27 17:02:05 load: 0.22 read_KB: 296 write_KB: 420 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 1 0 0 0 0 0 0 0 0 0 0 0 1.1.193.250 86 76 5 0 306 420 4 84 6 739 0 100 all 87 76 5 0 306 420 4 84 6 739 0 0 2017 Sep 27 17:02:06 load: 0.22 read_KB: 1149 write_KB: 379 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 66 56 3 0 310 75 5 25 4 538 0 100 1.1.193.250 182 166 5 0 828 304 4 60 4 581 0 100 all 248 222 8 0 1138 379 5 47 4 565 0 0 2017 Sep 27 17:02:07 load: 0.23 read_KB: 615 write_KB: 164 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 77 75 2 0 374 28 4 14 4 399 0 100 1.1.193.250 89 82 2 0 241 136 2 68 3 266 0 100 all 166 157 4 0 615 164 3 41 3 333 0 0 2017 Sep 27 17:02:08 load: 0.23 read_KB: 1505 write_KB: 9712 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 9 6 0 0 7 0 1 0 5 0 0 0 1.1.193.250 302 166 124 0 1978 14288 11 115 3 2037 0 100 all 311 172 124 0 1985 14288 11 115 11 2037 0 0 2017 Sep 27 17:02:09 load: 0.23 read_KB: 4267 write_KB: 61980 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 12 7 0 0 15 0 2 0 4 0 0 0 1.1.193.250 644 156 484 0 3772 57404 24 118 1 1728 0 100 all 656 163 484 0 3787 57404 23 118 2 1728 0 0 2017 Sep 27 17:02:10 load: 0.24 read_KB: 610 write_KB: 48 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 13 10 1 0 95 16 9 16 4 374 0 100 1.1.193.250 116 107 1 0 514 32 4 32 6 495 0 100 all 129 117 2 0 610 48 5 24 5 435 0 0 2017 Sep 27 17:02:11 load: 0.24 read_KB: 684 write_KB: 68 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 26 20 1 0 59 32 2 32 5 545 0 100 1.1.193.250 169 158 2 0 624 36 3 18 4 451 0 100 all 195 178 3 0 684 68 3 22 4 482 0 0 2017 Sep 27 17:02:12 load: 0.24 read_KB: 154 write_KB: 176 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 14 12 1 0 46 96 3 96 5 854 0 100 1.1.193.250 43 35 1 0 492 80 14 80 28 947 0 100 all 57 47 2 0 538 176 11 88 22 900 0 0 2017 Sep 27 17:02:13 load: 0.25 read_KB: 1134 write_KB: 12 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.250 36 24 1 0 191 12 7 12 16 469 0 100 1.1.193.247 122 117 0 0 558 0 4 0 5 0 0 0 all 158 141 1 0 750 12 5 12 7 469 0 0 2017 Sep 27 17:02:14 load: 0.25 read_KB: 6357 write_KB: 90908 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 5 2 0 0 1 0 0 0 9 0 0 0 1.1.193.250 1003 233 762 0 6357 90908 27 119 14 4844 0 100 all 1008 235 762 0 6358 90908 27 119 14 4844 0 0 2017 Sep 27 17:02:15 load: 0.25 read_KB: 243 write_KB: 0 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 23 18 0 0 37 0 2 0 2 0 0 0 1.1.193.250 70 58 0 0 207 0 3 0 4 0 0 0 all 93 76 0 0 244 0 3 0 3 0 0 0 2017 Sep 27 17:02:16 load: 0.25 read_KB: 382 write_KB: 16 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 42 38 0 0 191 0 5 0 5 0 0 0 1.1.193.250 59 50 1 0 189 16 3 16 8 427 0 100 all 101 88 1 0 381 16 4 16 7 427 0 0 2017 Sep 27 17:02:17 load: 0.25 read_KB: 23 write_KB: 0 client ops reads writes nops rd_bw wr_bw ard_sz awr_sz rd_t wr_t nop_t align% 1.1.193.247 6 3 0 0 1 0 0 0 7 0 0 0 1.1.193.250 21 13 0 0 21 0 1 0 5 0 0 0 all 27 16 0 0 23 0 1 0 6 0 0 0 How can i have 2MB/s network traffic (verified on omnios filer and also in kvm host) and write on disk many more than that ? 2017-09-27 12:56 GMT+02:00 Artem Penner : > Use https://github.com/richardelling/tools/blob/master/iscsisvrtop to > observe iscsi I/O > > > ??, 27 ????. 2017 ?. ? 11:06, anthony omnios : > >> Hi, >> >> i have a problem, i used many ISCSI zvol (for each vm), network traffic >> is 2MB/s between kvm host and filer but i write on disks many more than >> that. I used a pool with separated mirror zil (intel s3710) and 8 ssd >> samsung 850 evo 1To >> >> zpool status >> pool: filervm2 >> state: ONLINE >> scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017 >> config: >> >> NAME STATE READ WRITE CKSUM >> filervm2 ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> c7t5002538D41657AAFd0 ONLINE 0 0 0 >> c7t5002538D41F85C0Dd0 ONLINE 0 0 0 >> mirror-2 ONLINE 0 0 0 >> c7t5002538D41CC7105d0 ONLINE 0 0 0 >> c7t5002538D41CC7127d0 ONLINE 0 0 0 >> mirror-3 ONLINE 0 0 0 >> c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 >> c7t5002538D41CD83FDd0 ONLINE 0 0 0 >> mirror-4 ONLINE 0 0 0 >> c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 >> c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 >> logs >> mirror-1 ONLINE 0 0 0 >> c4t2d0 ONLINE 0 0 0 >> c4t4d0 ONLINE 0 0 0 >> >> i used correct ashift of 13 for samsung 850 evo >> zdb|grep ashift : >> >> ashift: 13 >> ashift: 13 >> ashift: 13 >> ashift: 13 >> ashift: 13 >> >> But i write a lot on ssd every 5 seconds (many more than the network >> traffic of 2 MB/s) >> >> iostat -xn -d 1 : >> >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 >> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool >> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 >> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 >> 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 >> 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 >> 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 >> c7t5002538D41657AAFd0 >> 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 >> c7t5002538D41CC7127d0 >> 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 >> c7t5002538D41CC7105d0 >> 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 >> c7t5002538D41F85C0Dd0 >> 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 >> c7t5002538D41CD83FDd0 >> 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 >> c7t5002538D41CD7F7Ed0 >> 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 >> c7t5002538D41CD7F7Dd0 >> 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 >> c7t5002538D41CD7F7Ad0 >> >> I used zvol of 64k, i try with 8k and problem is the same. >> >> zfs get all filervm2/hdd-110022a : >> >> NAME PROPERTY VALUE SOURCE >> filervm2/hdd-110022a type volume - >> filervm2/hdd-110022a creation Tue May 16 10:24 2017 - >> filervm2/hdd-110022a used 5.26G - >> filervm2/hdd-110022a available 2.90T - >> filervm2/hdd-110022a referenced 5.24G - >> filervm2/hdd-110022a compressratio 3.99x - >> filervm2/hdd-110022a reservation none default >> filervm2/hdd-110022a volsize 25G local >> filervm2/hdd-110022a volblocksize 64K - >> filervm2/hdd-110022a checksum on default >> filervm2/hdd-110022a compression lz4 local >> filervm2/hdd-110022a readonly off default >> filervm2/hdd-110022a copies 1 default >> filervm2/hdd-110022a refreservation none default >> filervm2/hdd-110022a primarycache all default >> filervm2/hdd-110022a secondarycache all default >> filervm2/hdd-110022a usedbysnapshots 15.4M - >> filervm2/hdd-110022a usedbydataset 5.24G - >> filervm2/hdd-110022a usedbychildren 0 - >> filervm2/hdd-110022a usedbyrefreservation 0 - >> filervm2/hdd-110022a logbias latency default >> filervm2/hdd-110022a dedup off default >> filervm2/hdd-110022a mlslabel none default >> filervm2/hdd-110022a sync standard local >> filervm2/hdd-110022a refcompressratio 3.99x - >> filervm2/hdd-110022a written 216K - >> filervm2/hdd-110022a logicalused 20.9G - >> filervm2/hdd-110022a logicalreferenced 20.9G - >> filervm2/hdd-110022a snapshot_limit none default >> filervm2/hdd-110022a snapshot_count none default >> filervm2/hdd-110022a redundant_metadata all default >> >> Sorry for my bad english. >> >> What can be the problem ? thanks >> >> Best regards, >> >> Anthony >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergey57 at gmail.com Wed Sep 27 19:31:05 2017 From: sergey57 at gmail.com (sergey ivanov) Date: Wed, 27 Sep 2017 15:31:05 -0400 Subject: [OmniOS-discuss] OmniOS based redundant NFS Message-ID: Hi, as end-of-life of r151014 approaches, we are planning upgrade for our NFS servers. I'm thinking about 2 servers providing ISCSI targets, and 2 another OmniOS servers using these ISCSI block devices in mirrored ZPOOL setup. IP address for NFS service can be a floating IP between those 2 servers. I have the following questions: 1. Are there any advantages to have separate ISCSI target servers and NFS servers or I should better combine one ISCSI target and NFS server on each of 2 hosts? 2. I do not want snapshots, checksums, and other ZFS features for block devices at the level where they are exported as ISCSI targets, - I would prefer these features at the level where these block devices are combined into mirror Zpools. Maybe it's better to have these ISCSI target servers running some not so advanced OS and have 2 Linux boxes? 3. But if I have SSD for intent log and for cache, - maybe they can improve performance for ZVOLs used as block devices for ISCSI targets? Does anybody have experience setting up such redundant NFS servers? -- Regards, Sergey Ivanov From stephan.budach at jvm.de Wed Sep 27 20:34:14 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Wed, 27 Sep 2017 22:34:14 +0200 (CEST) Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: Message-ID: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> Hi Sergey, ----- Urspr?ngliche Mail ----- > Von: "sergey ivanov" > An: "omnios-discuss" > Gesendet: Mittwoch, 27. September 2017 21:31:05 > Betreff: [OmniOS-discuss] OmniOS based redundant NFS > > Hi, > as end-of-life of r151014 approaches, we are planning upgrade for our > NFS servers. > I'm thinking about 2 servers providing ISCSI targets, and 2 another > OmniOS servers using these ISCSI block devices in mirrored ZPOOL > setup. IP address for NFS service can be a floating IP between those > 2 > servers. This is quite the same setup, as we have it. I am running two omniOS hosts as iSCSI targets and RSF-1 on two other omniOS hosts as NFS heads, where the NFS VIPs are failling over between. This setup has been very stable for quite some time and failovers have been occurring on a couple of occasions. > I have the following questions: > 1. Are there any advantages to have separate ISCSI target servers and > NFS servers or I should better combine one ISCSI target and NFS > server on each of 2 hosts? The reason to use two x two seperate servers is, that the mirrored zpool's vdevs look the same on each NFS head. This makes for a very straight forward setup, but on the other hand brings some interesting decisions to the table, when it comes to the design of the iSCSI targets. We all know, that ZFS works best when presented with raw devices and the closest to that would be iSCSI to raw devices/partitions. However, this will leave you to decide how you want to arrange your targets. If you chose to have fewer targets with more LUNs, you will face some interesting challenges when it comes to device failures, which will have you to offline that whole target on your NFS heads, leaving you running with a degraded zpool on your NFS head. I just went through that and I can tell you that you will need to prepare for such a case, and the more drives you are using the more challenging it becomes. > 2. I do not want snapshots, checksums, and other ZFS features for > block devices at the level where they are exported as ISCSI targets, > - > I would prefer these features at the level where these block devices > are combined into mirror Zpools. Maybe it's better to have these > ISCSI > target servers running some not so advanced OS and have 2 Linux > boxes? Me neither, but I chose omniOS as my iSCSI targets nevertheless. I do make heavy use of ZFS' features like snapshots and clones on my NFS heads and I am very comfortable with that. No bells and whistles on my target nodes. > 3. But if I have SSD for intent log and for cache, - maybe they can > improve performance for ZVOLs used as block devices for ISCSI > targets? > It will depend on your workload. I do also have some S3700s for ZIL on some of my iSCSI-based ZPOOls. > Does anybody have experience setting up such redundant NFS servers? Well? yes, and afaik, there are also some other people around this list, who are using similar setups, short of the iSCS target nodes, providing non-ZFS based LUNs, that seems to be more exotic? ;) > -- > Regards, > Sergey Ivanov Cheers, Stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From sergey57 at gmail.com Wed Sep 27 21:15:49 2017 From: sergey57 at gmail.com (sergey ivanov) Date: Wed, 27 Sep 2017 17:15:49 -0400 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> References: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: Thanks, Stephan! Please, explain "The reason to use two x two separate servers is, that the mirrored zpool's vdevs look the same on each NFS head". I understand that, if I want to have the same zpool based on iscsi devices, I should not mix local disks with iscsi target disks. But I think I can have 2 computers, each exporting a set of local disks as iscsi targets. And to have iscsi initiators on the same computers importing these targets to build zpools. Also, looking at sbdadm, I think I can 'create lu /dev/rdsk/c0t0d3s2'. Ok, I think I would better try it and report how it goes. -- Regards, Sergey. -- Regards, Sergey Ivanov On Wed, Sep 27, 2017 at 4:34 PM, Stephan Budach wrote: > Hi Sergey, > > ----- Urspr?ngliche Mail ----- >> Von: "sergey ivanov" >> An: "omnios-discuss" >> Gesendet: Mittwoch, 27. September 2017 21:31:05 >> Betreff: [OmniOS-discuss] OmniOS based redundant NFS >> >> Hi, >> as end-of-life of r151014 approaches, we are planning upgrade for our >> NFS servers. >> I'm thinking about 2 servers providing ISCSI targets, and 2 another >> OmniOS servers using these ISCSI block devices in mirrored ZPOOL >> setup. IP address for NFS service can be a floating IP between those >> 2 >> servers. > > This is quite the same setup, as we have it. I am running two omniOS hosts as iSCSI targets and RSF-1 on two other omniOS hosts as NFS heads, where the NFS VIPs are failling over between. This setup has been very stable for quite some time and failovers have been occurring on a couple of occasions. > >> I have the following questions: >> 1. Are there any advantages to have separate ISCSI target servers and >> NFS servers or I should better combine one ISCSI target and NFS >> server on each of 2 hosts? > > The reason to use two x two seperate servers is, that the mirrored zpool's vdevs look the same on each NFS head. > This makes for a very straight forward setup, but on the other hand brings some interesting decisions to the table, when it comes to the design of the iSCSI targets. > > We all know, that ZFS works best when presented with raw devices and the closest to that would be iSCSI to raw devices/partitions. However, this will leave you to decide how you want to arrange your targets. If you chose to have fewer targets with more LUNs, you will face some interesting challenges when it comes to device failures, which will have you to offline that whole target on your NFS heads, leaving you running with a degraded zpool on your NFS head. I just went through that and I can tell you that you will need to prepare for such a case, and the more drives you are using the more challenging it becomes. > > >> 2. I do not want snapshots, checksums, and other ZFS features for >> block devices at the level where they are exported as ISCSI targets, >> - >> I would prefer these features at the level where these block devices >> are combined into mirror Zpools. Maybe it's better to have these >> ISCSI >> target servers running some not so advanced OS and have 2 Linux >> boxes? > > Me neither, but I chose omniOS as my iSCSI targets nevertheless. I do make heavy use of ZFS' features like snapshots and clones on my NFS heads and I am very comfortable with that. No bells and whistles on my target nodes. > >> 3. But if I have SSD for intent log and for cache, - maybe they can >> improve performance for ZVOLs used as block devices for ISCSI >> targets? >> > > It will depend on your workload. I do also have some S3700s for ZIL on some of my iSCSI-based ZPOOls. > >> Does anybody have experience setting up such redundant NFS servers? > > Well? yes, and afaik, there are also some other people around this list, who are using similar setups, short of the iSCS target nodes, providing non-ZFS based LUNs, that seems to be more exotic? ;) > >> -- >> Regards, >> Sergey Ivanov > > Cheers, > Stephan From richard.elling at richardelling.com Wed Sep 27 23:29:52 2017 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 27 Sep 2017 16:29:52 -0700 Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: References: Message-ID: Comment below... > On Sep 27, 2017, at 12:57 AM, anthony omnios wrote: > > Hi, > > i have a problem, i used many ISCSI zvol (for each vm), network traffic is 2MB/s between kvm host and filer but i write on disks many more than that. I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung 850 evo 1To > > zpool status > pool: filervm2 > state: ONLINE > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017 > config: > > NAME STATE READ WRITE CKSUM > filervm2 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > c7t5002538D41657AAFd0 ONLINE 0 0 0 > c7t5002538D41F85C0Dd0 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > c7t5002538D41CC7105d0 ONLINE 0 0 0 > c7t5002538D41CC7127d0 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 > c7t5002538D41CD83FDd0 ONLINE 0 0 0 > mirror-4 ONLINE 0 0 0 > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > c4t2d0 ONLINE 0 0 0 > c4t4d0 ONLINE 0 0 0 > > i used correct ashift of 13 for samsung 850 evo > zdb|grep ashift : > > ashift: 13 > ashift: 13 > ashift: 13 > ashift: 13 > ashift: 13 > > But i write a lot on ssd every 5 seconds (many more than the network traffic of 2 MB/s) > > iostat -xn -d 1 : > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 filervm2 is seeing 3067 writes per second. This is the interface to the upper layers. These writes are small. > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 The log devices are seeing 552 writes per second and since sync=standard that means that the upper layers are requesting syncs. > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0 > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0 > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0 > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0 > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0 > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0 > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0 > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0 The pool disks see 1989 writes per second total or 994 writes per second logically. It seems to me that reducing 3067 requested writes to 994 logical writes is the opposite of amplification. What do you expect? -- richard > > I used zvol of 64k, i try with 8k and problem is the same. > > zfs get all filervm2/hdd-110022a : > > NAME PROPERTY VALUE SOURCE > filervm2/hdd-110022a type volume - > filervm2/hdd-110022a creation Tue May 16 10:24 2017 - > filervm2/hdd-110022a used 5.26G - > filervm2/hdd-110022a available 2.90T - > filervm2/hdd-110022a referenced 5.24G - > filervm2/hdd-110022a compressratio 3.99x - > filervm2/hdd-110022a reservation none default > filervm2/hdd-110022a volsize 25G local > filervm2/hdd-110022a volblocksize 64K - > filervm2/hdd-110022a checksum on default > filervm2/hdd-110022a compression lz4 local > filervm2/hdd-110022a readonly off default > filervm2/hdd-110022a copies 1 default > filervm2/hdd-110022a refreservation none default > filervm2/hdd-110022a primarycache all default > filervm2/hdd-110022a secondarycache all default > filervm2/hdd-110022a usedbysnapshots 15.4M - > filervm2/hdd-110022a usedbydataset 5.24G - > filervm2/hdd-110022a usedbychildren 0 - > filervm2/hdd-110022a usedbyrefreservation 0 - > filervm2/hdd-110022a logbias latency default > filervm2/hdd-110022a dedup off default > filervm2/hdd-110022a mlslabel none default > filervm2/hdd-110022a sync standard local > filervm2/hdd-110022a refcompressratio 3.99x - > filervm2/hdd-110022a written 216K - > filervm2/hdd-110022a logicalused 20.9G - > filervm2/hdd-110022a logicalreferenced 20.9G - > filervm2/hdd-110022a snapshot_limit none default > filervm2/hdd-110022a snapshot_count none default > filervm2/hdd-110022a redundant_metadata all default > > Sorry for my bad english. > > What can be the problem ? thanks > > Best regards, > > Anthony > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From stephan.budach at jvm.de Thu Sep 28 04:49:25 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Thu, 28 Sep 2017 06:49:25 +0200 (CEST) Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de> Hi Sergey, ----- Urspr?ngliche Mail ----- > Von: "sergey ivanov" > An: "Stephan Budach" > CC: "omnios-discuss" > Gesendet: Mittwoch, 27. September 2017 23:15:49 > Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS > > Thanks, Stephan! > > Please, explain "The reason to use two x two separate servers is, > that > the mirrored zpool's vdevs look the same on each NFS head". > > I understand that, if I want to have the same zpool based on iscsi > devices, I should not mix local disks with iscsi target disks. > > But I think I can have 2 computers, each exporting a set of local > disks as iscsi targets. And to have iscsi initiators on the same > computers importing these targets to build zpools. > > Also, looking at sbdadm, I think I can 'create lu > /dev/rdsk/c0t0d3s2'. > > Ok, I think I would better try it and report how it goes. Actually, things can become quite complex, I'd like to reduce the "mental" involvement to the absolute minimum, mainly because we often faced a situation where something would suddenly break, which had been running for a long time without problems. This is when peeple start? well maybe not panicking, but having to recap what the current setup was like and what they had to do to tackle this. So, uniformity is a great deal of help on such systems - at least for us. Technically, there is no issue with mixing local and remote iSCST targets on the same node, which serves as an iSCSI target and a NFS head. Also, if one of the nodes really goes down, you will be loosing your failover NFS head as well, maybe not a big deal and depending on your requirements okay. I do have such a setup as well, although only for an archive ZPOOL, where I can tolerate this reduced redundancy for the benefit of a more lightweight setup. Cheers, Stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From apenner.it at gmail.com Thu Sep 28 07:25:15 2017 From: apenner.it at gmail.com (Artem Penner) Date: Thu, 28 Sep 2017 07:25:15 +0000 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: Message-ID: Hi, Sergey. My be following information will be useful for you. We use solaris as HA NAS in following configuration: DATA_DISKS: HUC101818CS4204 / HUH728080AL5204 SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204 (PX04SHB020 has best performance https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html) L2ARC: HUSMH8080BSS204 / PX05SMB080 Servers: 2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS, 2 x UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter) JBODS: 216BE2C-R741JBOD - for SFF disks SC847E2C-R1K28JBOD - for LFF disks We have two HA variant: 1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware) 2) OmniOS + RSF-1 (as Cinder for Openstack) If you need some additional info, I'll be glad to share any information that I have. ??, 27 ????. 2017 ?. ? 22:39, sergey ivanov : > Hi, > as end-of-life of r151014 approaches, we are planning upgrade for our > NFS servers. > I'm thinking about 2 servers providing ISCSI targets, and 2 another > OmniOS servers using these ISCSI block devices in mirrored ZPOOL > setup. IP address for NFS service can be a floating IP between those 2 > servers. > I have the following questions: > 1. Are there any advantages to have separate ISCSI target servers and > NFS servers or I should better combine one ISCSI target and NFS > server on each of 2 hosts? > 2. I do not want snapshots, checksums, and other ZFS features for > block devices at the level where they are exported as ISCSI targets, - > I would prefer these features at the level where these block devices > are combined into mirror Zpools. Maybe it's better to have these ISCSI > target servers running some not so advanced OS and have 2 Linux boxes? > 3. But if I have SSD for intent log and for cache, - maybe they can > improve performance for ZVOLs used as block devices for ISCSI targets? > > Does anybody have experience setting up such redundant NFS servers? > -- > Regards, > Sergey Ivanov > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icoomnios at gmail.com Thu Sep 28 07:56:42 2017 From: icoomnios at gmail.com (anthony omnios) Date: Thu, 28 Sep 2017 09:56:42 +0200 Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: References: Message-ID: Thanks Richard for your help. My problem is that i have a network ISCSI traffic of 2 MB/s, each 5 seconds i need to write on disks 10 MB of network traffic but on pool filervm2 I am writing much more that, approximatively 60 MB each 5 seconds. Each ssd of filervm2 is writting 15 MB every 5 second. When i check with smartmootools every ssd is writing approximatively 250 GB of data each day. How can i reduce amont of data writting on each ssd ? i have try to reduce block size of zvol but it change nothing. Anthony 2017-09-28 1:29 GMT+02:00 Richard Elling : > Comment below... > > > On Sep 27, 2017, at 12:57 AM, anthony omnios > wrote: > > > > Hi, > > > > i have a problem, i used many ISCSI zvol (for each vm), network traffic > is 2MB/s between kvm host and filer but i write on disks many more than > that. I used a pool with separated mirror zil (intel s3710) and 8 ssd > samsung 850 evo 1To > > > > zpool status > > pool: filervm2 > > state: ONLINE > > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 > 2017 > > config: > > > > NAME STATE READ WRITE CKSUM > > filervm2 ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > c7t5002538D41657AAFd0 ONLINE 0 0 0 > > c7t5002538D41F85C0Dd0 ONLINE 0 0 0 > > mirror-2 ONLINE 0 0 0 > > c7t5002538D41CC7105d0 ONLINE 0 0 0 > > c7t5002538D41CC7127d0 ONLINE 0 0 0 > > mirror-3 ONLINE 0 0 0 > > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 > > c7t5002538D41CD83FDd0 ONLINE 0 0 0 > > mirror-4 ONLINE 0 0 0 > > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 > > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 > > logs > > mirror-1 ONLINE 0 0 0 > > c4t2d0 ONLINE 0 0 0 > > c4t4d0 ONLINE 0 0 0 > > > > i used correct ashift of 13 for samsung 850 evo > > zdb|grep ashift : > > > > ashift: 13 > > ashift: 13 > > ashift: 13 > > ashift: 13 > > ashift: 13 > > > > But i write a lot on ssd every 5 seconds (many more than the network > traffic of 2 MB/s) > > > > iostat -xn -d 1 : > > > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 > > filervm2 is seeing 3067 writes per second. This is the interface to the > upper layers. > These writes are small. > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 > > The log devices are seeing 552 writes per second and since sync=standard > that > means that the upper layers are requesting syncs. > > > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41657AAFd0 > > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41CC7127d0 > > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 > c7t5002538D41CC7105d0 > > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 > c7t5002538D41F85C0Dd0 > > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 > c7t5002538D41CD83FDd0 > > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 > c7t5002538D41CD7F7Ed0 > > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 > c7t5002538D41CD7F7Dd0 > > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 > c7t5002538D41CD7F7Ad0 > > The pool disks see 1989 writes per second total or 994 writes per second > logically. > > It seems to me that reducing 3067 requested writes to 994 logical writes > is the opposite > of amplification. What do you expect? > -- richard > > > > > I used zvol of 64k, i try with 8k and problem is the same. > > > > zfs get all filervm2/hdd-110022a : > > > > NAME PROPERTY VALUE SOURCE > > filervm2/hdd-110022a type volume - > > filervm2/hdd-110022a creation Tue May 16 10:24 2017 - > > filervm2/hdd-110022a used 5.26G - > > filervm2/hdd-110022a available 2.90T - > > filervm2/hdd-110022a referenced 5.24G - > > filervm2/hdd-110022a compressratio 3.99x - > > filervm2/hdd-110022a reservation none > default > > filervm2/hdd-110022a volsize 25G local > > filervm2/hdd-110022a volblocksize 64K - > > filervm2/hdd-110022a checksum on > default > > filervm2/hdd-110022a compression lz4 local > > filervm2/hdd-110022a readonly off > default > > filervm2/hdd-110022a copies 1 > default > > filervm2/hdd-110022a refreservation none > default > > filervm2/hdd-110022a primarycache all > default > > filervm2/hdd-110022a secondarycache all > default > > filervm2/hdd-110022a usedbysnapshots 15.4M - > > filervm2/hdd-110022a usedbydataset 5.24G - > > filervm2/hdd-110022a usedbychildren 0 - > > filervm2/hdd-110022a usedbyrefreservation 0 - > > filervm2/hdd-110022a logbias latency > default > > filervm2/hdd-110022a dedup off > default > > filervm2/hdd-110022a mlslabel none > default > > filervm2/hdd-110022a sync standard local > > filervm2/hdd-110022a refcompressratio 3.99x - > > filervm2/hdd-110022a written 216K - > > filervm2/hdd-110022a logicalused 20.9G - > > filervm2/hdd-110022a logicalreferenced 20.9G - > > filervm2/hdd-110022a snapshot_limit none > default > > filervm2/hdd-110022a snapshot_count none > default > > filervm2/hdd-110022a redundant_metadata all > default > > > > Sorry for my bad english. > > > > What can be the problem ? thanks > > > > Best regards, > > > > Anthony > > > > > > _______________________________________________ > > OmniOS-discuss mailing list > > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephan.budach at jvm.de Thu Sep 28 08:33:17 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Thu, 28 Sep 2017 10:33:17 +0200 (CEST) Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: References: Message-ID: <1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local> ----- Urspr?ngliche Mail ----- > Von: "anthony omnios" > An: "Richard Elling" > CC: omnios-discuss at lists.omniti.com > Gesendet: Donnerstag, 28. September 2017 09:56:42 > Betreff: Re: [OmniOS-discuss] write amplification zvol > Thanks Richard for your help. > My problem is that i have a network ISCSI traffic of 2 MB/s, each 5 > seconds i need to write on disks 10 MB of network traffic but on > pool filervm2 I am writing much more that, approximatively 60 MB > each 5 seconds. Each ssd of filervm2 is writting 15 MB every 5 > second. When i check with smartmootools every ssd is writing > approximatively 250 GB of data each day. > How can i reduce amont of data writting on each ssd ? i have try to > reduce block size of zvol but it change nothing. > Anthony > 2017-09-28 1:29 GMT+02:00 Richard Elling < > richard.elling at richardelling.com > : > > Comment below... > > > > On Sep 27, 2017, at 12:57 AM, anthony omnios < > > > icoomnios at gmail.com > > > > wrote: > > > > > > > > Hi, > > > > > > > > i have a problem, i used many ISCSI zvol (for each vm), network > > > traffic is 2MB/s between kvm host and filer but i write on disks > > > many more than that. I used a pool with separated mirror zil > > > (intel s3710) and 8 ssd samsung 850 evo 1To > > > > > > > > zpool status > > > > pool: filervm2 > > > > state: ONLINE > > > > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 > > > 15:45:48 > > > 2017 > > > > config: > > > > > > > > NAME STATE READ WRITE CKSUM > > > > filervm2 ONLINE 0 0 0 > > > > mirror-0 ONLINE 0 0 0 > > > > c7t5002538D41657AAFd0 ONLINE 0 0 0 > > > > c7t5002538D41F85C0Dd0 ONLINE 0 0 0 > > > > mirror-2 ONLINE 0 0 0 > > > > c7t5002538D41CC7105d0 ONLINE 0 0 0 > > > > c7t5002538D41CC7127d0 ONLINE 0 0 0 > > > > mirror-3 ONLINE 0 0 0 > > > > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 > > > > c7t5002538D41CD83FDd0 ONLINE 0 0 0 > > > > mirror-4 ONLINE 0 0 0 > > > > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 > > > > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 > > > > logs > > > > mirror-1 ONLINE 0 0 0 > > > > c4t2d0 ONLINE 0 0 0 > > > > c4t4d0 ONLINE 0 0 0 > > > > > > > > i used correct ashift of 13 for samsung 850 evo > > > > zdb|grep ashift : > > > > > > > > ashift: 13 > > > > ashift: 13 > > > > ashift: 13 > > > > ashift: 13 > > > > ashift: 13 > > > > > > > > But i write a lot on ssd every 5 seconds (many more than the > > > network traffic of 2 MB/s) > > > > > > > > iostat -xn -d 1 : > > > > > > > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > > > > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 > > > filervm2 is seeing 3067 writes per second. This is the interface to > > the upper layers. > > > These writes are small. > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 > > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 > > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 > > > The log devices are seeing 552 writes per second and since > > sync=standard that > > > means that the upper layers are requesting syncs. > > > > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0 > > > > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0 > > > > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0 > > > > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0 > > > > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0 > > > > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0 > > > > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0 > > > > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0 > > > The pool disks see 1989 writes per second total or 994 writes per > > second logically. > > > It seems to me that reducing 3067 requested writes to 994 logical > > writes is the opposite > > > of amplification. What do you expect? > > > -- richard > > > > > > > > I used zvol of 64k, i try with 8k and problem is the same. > > > > > > > > zfs get all filervm2/hdd-110022a : > > > > > > > > NAME PROPERTY VALUE SOURCE > > > > filervm2/hdd-110022a type volume - > > > > filervm2/hdd-110022a creation Tue May 16 10:24 2017 - > > > > filervm2/hdd-110022a used 5.26G - > > > > filervm2/hdd-110022a available 2.90T - > > > > filervm2/hdd-110022a referenced 5.24G - > > > > filervm2/hdd-110022a compressratio 3.99x - > > > > filervm2/hdd-110022a reservation none default > > > > filervm2/hdd-110022a volsize 25G local > > > > filervm2/hdd-110022a volblocksize 64K - > > > > filervm2/hdd-110022a checksum on default > > > > filervm2/hdd-110022a compression lz4 local > > > > filervm2/hdd-110022a readonly off default > > > > filervm2/hdd-110022a copies 1 default > > > > filervm2/hdd-110022a refreservation none default > > > > filervm2/hdd-110022a primarycache all default > > > > filervm2/hdd-110022a secondarycache all default > > > > filervm2/hdd-110022a usedbysnapshots 15.4M - > > > > filervm2/hdd-110022a usedbydataset 5.24G - > > > > filervm2/hdd-110022a usedbychildren 0 - > > > > filervm2/hdd-110022a usedbyrefreservation 0 - > > > > filervm2/hdd-110022a logbias latency default > > > > filervm2/hdd-110022a dedup off default > > > > filervm2/hdd-110022a mlslabel none default > > > > filervm2/hdd-110022a sync standard local > > > > filervm2/hdd-110022a refcompressratio 3.99x - > > > > filervm2/hdd-110022a written 216K - > > > > filervm2/hdd-110022a logicalused 20.9G - > > > > filervm2/hdd-110022a logicalreferenced 20.9G - > > > > filervm2/hdd-110022a snapshot_limit none default > > > > filervm2/hdd-110022a snapshot_count none default > > > > filervm2/hdd-110022a redundant_metadata all default > > > > > > > > Sorry for my bad english. > > > > > > > > What can be the problem ? thanks > > > > > > > > Best regards, > > > > > > > > Anthony > How did you setup your LUNs? Especially, what is the block size for those LUNs. Could it be, that you went with the default of 512b blocks, where the drives do have 4k or even 8k blocks? Cheers, Stephan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From icoomnios at gmail.com Thu Sep 28 09:48:18 2017 From: icoomnios at gmail.com (anthony omnios) Date: Thu, 28 Sep 2017 11:48:18 +0200 Subject: [OmniOS-discuss] write amplification zvol In-Reply-To: <1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local> References: <1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local> Message-ID: Thanks for you help Stephan. i have tried differents LUN with default of 512b and 4096: LU Name: 600144F04D4F0600000059A588910001 Operational Status: Online Provider Name : sbd Alias : /dev/zvol/rdsk/filervm2/hdd-110002b View Entry Count : 1 Data File : /dev/zvol/rdsk/filervm2/hdd-110002b Meta File : not set Size : 26843545600 Block Size : 4096 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active Problem is the same. Cheers, Anthony 2017-09-28 10:33 GMT+02:00 Stephan Budach : > ----- Urspr?ngliche Mail ----- > > > Von: "anthony omnios" > > An: "Richard Elling" > > CC: omnios-discuss at lists.omniti.com > > Gesendet: Donnerstag, 28. September 2017 09:56:42 > > Betreff: Re: [OmniOS-discuss] write amplification zvol > > > Thanks Richard for your help. > > > My problem is that i have a network ISCSI traffic of 2 MB/s, each 5 > > seconds i need to write on disks 10 MB of network traffic but on > > pool filervm2 I am writing much more that, approximatively 60 MB > > each 5 seconds. Each ssd of filervm2 is writting 15 MB every 5 > > second. When i check with smartmootools every ssd is writing > > approximatively 250 GB of data each day. > > > How can i reduce amont of data writting on each ssd ? i have try to > > reduce block size of zvol but it change nothing. > > > Anthony > > > 2017-09-28 1:29 GMT+02:00 Richard Elling < > > richard.elling at richardelling.com > : > > > > Comment below... > > > > > > > On Sep 27, 2017, at 12:57 AM, anthony omnios < > > > > icoomnios at gmail.com > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > i have a problem, i used many ISCSI zvol (for each vm), network > > > > traffic is 2MB/s between kvm host and filer but i write on disks > > > > many more than that. I used a pool with separated mirror zil > > > > (intel s3710) and 8 ssd samsung 850 evo 1To > > > > > > > > > > > > zpool status > > > > > > pool: filervm2 > > > > > > state: ONLINE > > > > > > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 > > > > 15:45:48 > > > > 2017 > > > > > > config: > > > > > > > > > > > > NAME STATE READ WRITE CKSUM > > > > > > filervm2 ONLINE 0 0 0 > > > > > > mirror-0 ONLINE 0 0 0 > > > > > > c7t5002538D41657AAFd0 ONLINE 0 0 0 > > > > > > c7t5002538D41F85C0Dd0 ONLINE 0 0 0 > > > > > > mirror-2 ONLINE 0 0 0 > > > > > > c7t5002538D41CC7105d0 ONLINE 0 0 0 > > > > > > c7t5002538D41CC7127d0 ONLINE 0 0 0 > > > > > > mirror-3 ONLINE 0 0 0 > > > > > > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0 > > > > > > c7t5002538D41CD83FDd0 ONLINE 0 0 0 > > > > > > mirror-4 ONLINE 0 0 0 > > > > > > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0 > > > > > > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0 > > > > > > logs > > > > > > mirror-1 ONLINE 0 0 0 > > > > > > c4t2d0 ONLINE 0 0 0 > > > > > > c4t4d0 ONLINE 0 0 0 > > > > > > > > > > > > i used correct ashift of 13 for samsung 850 evo > > > > > > zdb|grep ashift : > > > > > > > > > > > > ashift: 13 > > > > > > ashift: 13 > > > > > > ashift: 13 > > > > > > ashift: 13 > > > > > > ashift: 13 > > > > > > > > > > > > But i write a lot on ssd every 5 seconds (many more than the > > > > network traffic of 2 MB/s) > > > > > > > > > > > > iostat -xn -d 1 : > > > > > > > > > > > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > > > > > > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2 > > > > > > filervm2 is seeing 3067 writes per second. This is the interface to > > > the upper layers. > > > > > These writes are small. > > > > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool > > > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0 > > > > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0 > > > > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0 > > > > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0 > > > > > > The log devices are seeing 552 writes per second and since > > > sync=standard that > > > > > means that the upper layers are requesting syncs. > > > > > > > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0 > > > > > > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0 > > > > > > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0 > > > > > > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0 > > > > > > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0 > > > > > > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0 > > > > > > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0 > > > > > > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0 > > > > > > The pool disks see 1989 writes per second total or 994 writes per > > > second logically. > > > > > > It seems to me that reducing 3067 requested writes to 994 logical > > > writes is the opposite > > > > > of amplification. What do you expect? > > > > > -- richard > > > > > > > > > > > > > I used zvol of 64k, i try with 8k and problem is the same. > > > > > > > > > > > > zfs get all filervm2/hdd-110022a : > > > > > > > > > > > > NAME PROPERTY VALUE SOURCE > > > > > > filervm2/hdd-110022a type volume - > > > > > > filervm2/hdd-110022a creation Tue May 16 10:24 2017 - > > > > > > filervm2/hdd-110022a used 5.26G - > > > > > > filervm2/hdd-110022a available 2.90T - > > > > > > filervm2/hdd-110022a referenced 5.24G - > > > > > > filervm2/hdd-110022a compressratio 3.99x - > > > > > > filervm2/hdd-110022a reservation none default > > > > > > filervm2/hdd-110022a volsize 25G local > > > > > > filervm2/hdd-110022a volblocksize 64K - > > > > > > filervm2/hdd-110022a checksum on default > > > > > > filervm2/hdd-110022a compression lz4 local > > > > > > filervm2/hdd-110022a readonly off default > > > > > > filervm2/hdd-110022a copies 1 default > > > > > > filervm2/hdd-110022a refreservation none default > > > > > > filervm2/hdd-110022a primarycache all default > > > > > > filervm2/hdd-110022a secondarycache all default > > > > > > filervm2/hdd-110022a usedbysnapshots 15.4M - > > > > > > filervm2/hdd-110022a usedbydataset 5.24G - > > > > > > filervm2/hdd-110022a usedbychildren 0 - > > > > > > filervm2/hdd-110022a usedbyrefreservation 0 - > > > > > > filervm2/hdd-110022a logbias latency default > > > > > > filervm2/hdd-110022a dedup off default > > > > > > filervm2/hdd-110022a mlslabel none default > > > > > > filervm2/hdd-110022a sync standard local > > > > > > filervm2/hdd-110022a refcompressratio 3.99x - > > > > > > filervm2/hdd-110022a written 216K - > > > > > > filervm2/hdd-110022a logicalused 20.9G - > > > > > > filervm2/hdd-110022a logicalreferenced 20.9G - > > > > > > filervm2/hdd-110022a snapshot_limit none default > > > > > > filervm2/hdd-110022a snapshot_count none default > > > > > > filervm2/hdd-110022a redundant_metadata all default > > > > > > > > > > > > Sorry for my bad english. > > > > > > > > > > > > What can be the problem ? thanks > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Anthony > > > > How did you setup your LUNs? Especially, what is the block size for those > LUNs. Could it be, that you went with the default of 512b blocks, where the > drives do have 4k or even 8k blocks? > > Cheers, > Stephan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergey57 at gmail.com Fri Sep 29 20:14:02 2017 From: sergey57 at gmail.com (sergey ivanov) Date: Fri, 29 Sep 2017 16:14:02 -0400 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: Message-ID: Thanks, Artem, it is a very good list of suitable hardware! I have a question about ZIL. I read that SSD for it should be about 8G. I did not see more than one gigabyte allocated for ZIL on our servers. Can you comment it? -- Sergey. Regards, Sergey Ivanov On Thu, Sep 28, 2017 at 3:25 AM, Artem Penner wrote: > Hi, Sergey. > My be following information will be useful for you. > We use solaris as HA NAS in following configuration: > > DATA_DISKS: HUC101818CS4204 / HUH728080AL5204 > SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204 (PX04SHB020 has best > performance > https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html) > L2ARC: HUSMH8080BSS204 / PX05SMB080 > > Servers: > 2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS, 2 x > UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter) > > JBODS: > 216BE2C-R741JBOD - for SFF disks > SC847E2C-R1K28JBOD - for LFF disks > > We have two HA variant: > 1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware) > 2) OmniOS + RSF-1 (as Cinder for Openstack) > > If you need some additional info, I'll be glad to share any information that > I have. > > ??, 27 ????. 2017 ?. ? 22:39, sergey ivanov : >> >> Hi, >> as end-of-life of r151014 approaches, we are planning upgrade for our >> NFS servers. >> I'm thinking about 2 servers providing ISCSI targets, and 2 another >> OmniOS servers using these ISCSI block devices in mirrored ZPOOL >> setup. IP address for NFS service can be a floating IP between those 2 >> servers. >> I have the following questions: >> 1. Are there any advantages to have separate ISCSI target servers and >> NFS servers or I should better combine one ISCSI target and NFS >> server on each of 2 hosts? >> 2. I do not want snapshots, checksums, and other ZFS features for >> block devices at the level where they are exported as ISCSI targets, - >> I would prefer these features at the level where these block devices >> are combined into mirror Zpools. Maybe it's better to have these ISCSI >> target servers running some not so advanced OS and have 2 Linux boxes? >> 3. But if I have SSD for intent log and for cache, - maybe they can >> improve performance for ZVOLs used as block devices for ISCSI targets? >> >> Does anybody have experience setting up such redundant NFS servers? >> -- >> Regards, >> Sergey Ivanov >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss From apenner.it at gmail.com Fri Sep 29 20:30:09 2017 From: apenner.it at gmail.com (Artem Penner) Date: Fri, 29 Sep 2017 20:30:09 +0000 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: Message-ID: As you see, we use Toshiba PX04SHB020 as SLOG devices. It's a 200gb in size, but we reduce size of the disk to extend their life (increase DWPD) we do it as follows: root at znstor1-n1:~# sg_format /dev/rdsk/c0t500003979C88FF79d0 TOSHIBA PX04SHB020 0106 peripheral_type: disk [0x0] << supports protection information>> Mode Sense (block descriptor) data, prior to changes: Number of blocks=390721968 [0x1749f1b0] Block size=512 [0x200] Read Capacity (10) results: Number of logical blocks=390721968 Logical block size=512 bytes root at znstor1-n1:~# sg_format --size 4096 --format /dev/rdsk/c0t500003979C88FF79d0 TOSHIBA PX04SHB020 0106 peripheral_type: disk [0x0] << supports protection information>> Mode Sense (block descriptor) data, prior to changes: Number of blocks=390721968 [0x1749f1b0] Block size=512 [0x200] A FORMAT will commence in 10 seconds ALL data on /dev/rdsk/c0t500003979C88FF79d0 will be DESTROYED Press control-C to abort A FORMAT will commence in 5 seconds ALL data on /dev/rdsk/c0t500003979C88FF79d0 will be DESTROYED Press control-C to abort Format has started FORMAT Complete root at znstor1-n1:~# sg_format --resize --count=0x1749f1b /dev/rdsk/c0t500003979C88FF79d0 TOSHIBA PX04SHB020 0106 peripheral_type: disk [0x0] << supports protection information>> Mode Sense (block descriptor) data, prior to changes: Number of blocks=48840246 [0x2e93e36] Block size=4096 [0x1000] Resize operation seems to have been successful root at znstor1-n1:~# update_drv -f sd Every jbod has following layout (24 disk total) 2 disk for mirror slog 1 disk for l2arc 1 disk for spare 2 x raidz2 (8D + 2P) I didn't observe used space on every SLOG mirror group more than 3gb. ??, 29 ????. 2017 ?. ? 23:14, sergey ivanov : > Thanks, Artem, it is a very good list of suitable hardware! > I have a question about ZIL. > I read that SSD for it should be about 8G. I did not see more than one > gigabyte allocated for ZIL on our servers. Can you comment it? > -- > Sergey. > > Regards, > Sergey Ivanov > > > On Thu, Sep 28, 2017 at 3:25 AM, Artem Penner > wrote: > > Hi, Sergey. > > My be following information will be useful for you. > > We use solaris as HA NAS in following configuration: > > > > DATA_DISKS: HUC101818CS4204 / HUH728080AL5204 > > SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204 (PX04SHB020 has best > > performance > > https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html) > > L2ARC: HUSMH8080BSS204 / PX05SMB080 > > > > Servers: > > 2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS, > 2 x > > UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter) > > > > JBODS: > > 216BE2C-R741JBOD - for SFF disks > > SC847E2C-R1K28JBOD - for LFF disks > > > > We have two HA variant: > > 1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware) > > 2) OmniOS + RSF-1 (as Cinder for Openstack) > > > > If you need some additional info, I'll be glad to share any information > that > > I have. > > > > ??, 27 ????. 2017 ?. ? 22:39, sergey ivanov : > >> > >> Hi, > >> as end-of-life of r151014 approaches, we are planning upgrade for our > >> NFS servers. > >> I'm thinking about 2 servers providing ISCSI targets, and 2 another > >> OmniOS servers using these ISCSI block devices in mirrored ZPOOL > >> setup. IP address for NFS service can be a floating IP between those 2 > >> servers. > >> I have the following questions: > >> 1. Are there any advantages to have separate ISCSI target servers and > >> NFS servers or I should better combine one ISCSI target and NFS > >> server on each of 2 hosts? > >> 2. I do not want snapshots, checksums, and other ZFS features for > >> block devices at the level where they are exported as ISCSI targets, - > >> I would prefer these features at the level where these block devices > >> are combined into mirror Zpools. Maybe it's better to have these ISCSI > >> target servers running some not so advanced OS and have 2 Linux boxes? > >> 3. But if I have SSD for intent log and for cache, - maybe they can > >> improve performance for ZVOLs used as block devices for ISCSI targets? > >> > >> Does anybody have experience setting up such redundant NFS servers? > >> -- > >> Regards, > >> Sergey Ivanov > >> _______________________________________________ > >> OmniOS-discuss mailing list > >> OmniOS-discuss at lists.omniti.com > >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergey57 at gmail.com Fri Sep 29 20:30:31 2017 From: sergey57 at gmail.com (sergey ivanov) Date: Fri, 29 Sep 2017 16:30:31 -0400 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de> References: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: Thanks, Stephan. I did a simple test with creating lu over physical disks for use as ISSI targets, and it worked well. I am going to directly connect 2 servers and export their disks as separate I SCSI targets. Or maybe different LUNs in a target. And then on the active server start initiator to get these targets, and combine them into a pool of 2-way mirrors so that it stays degraded but working if one of the servers dies. So, manual failover for this configuration will be the following. If the server to be disabled is still active, stop NFS, export zpool on it, stop iscsiadm, release shared IP. On the other server: import zpool and start NFS, activate shared IP. I read once there are some tricks which make clients do not recognize NFS server is changed underneath all mounts, but I never tried it. -- Regards, Sergey Ivanov. Regards, Sergey Ivanov On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach wrote: > Hi Sergey, > > ----- Urspr?ngliche Mail ----- >> Von: "sergey ivanov" >> An: "Stephan Budach" >> CC: "omnios-discuss" >> Gesendet: Mittwoch, 27. September 2017 23:15:49 >> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS >> >> Thanks, Stephan! >> >> Please, explain "The reason to use two x two separate servers is, >> that >> the mirrored zpool's vdevs look the same on each NFS head". >> >> I understand that, if I want to have the same zpool based on iscsi >> devices, I should not mix local disks with iscsi target disks. >> >> But I think I can have 2 computers, each exporting a set of local >> disks as iscsi targets. And to have iscsi initiators on the same >> computers importing these targets to build zpools. >> >> Also, looking at sbdadm, I think I can 'create lu >> /dev/rdsk/c0t0d3s2'. >> >> Ok, I think I would better try it and report how it goes. > > Actually, things can become quite complex, I'd like to reduce the "mental" involvement to the absolute minimum, mainly because we often faced a situation where something would suddenly break, which had been running for a long time without problems. This is when peeple start? well maybe not panicking, but having to recap what the current setup was like and what they had to do to tackle this. > > So, uniformity is a great deal of help on such systems - at least for us. Technically, there is no issue with mixing local and remote iSCST targets on the same node, which serves as an iSCSI target and a NFS head. > > Also, if one of the nodes really goes down, you will be loosing your failover NFS head as well, maybe not a big deal and depending on your requirements okay. I do have such a setup as well, although only for an archive ZPOOL, where I can tolerate this reduced redundancy for the benefit of a more lightweight setup. > > Cheers, > Stephan From jdg117 at elvis.arl.psu.edu Fri Sep 29 20:37:16 2017 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Fri, 29 Sep 2017 16:37:16 -0400 Subject: [OmniOS-discuss] Apache Serf Message-ID: <201709292037.v8TKbGMx019519@elvis.arl.psu.edu> Anyone gotten serf to build on OmniOS? $ env PATH=/opt/subversion/apache24/bin:/usr/bin:/usr/sbin:/usr/ccs/bin:/usr/gnu/bin:/usr/sfw/bin /usr/bin/python2.7 /tmp/scons.py APR=/opt/subversion/apache24 APU=/opt/subversion/apache24 OPENSSL=/usr PREFIX=/opt/subversion/serf CC="gcc" CFLAGS="-fPIC -D__EXTENSIONS__" LINKFLAGS="-G -shared -m64 -R/opt/subversion/apache24/lib" check == Testing test/testcases/chunked-empty.response == ERROR: test case test/testcases/chunked-empty.response failed scons: *** [check] Error 1 scons: building terminated because of errors. $ pstack core core 'core' of 11900: test/serf_response fffffd7fff3ee939 main () + 219 0000000000000001 ???????? () pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be available John groenveld at acm.org From bfriesen at simple.dallas.tx.us Fri Sep 29 22:55:40 2017 From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn) Date: Fri, 29 Sep 2017 17:55:40 -0500 (CDT) Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: Message-ID: On Fri, 29 Sep 2017, sergey ivanov wrote: > Thanks, Artem, it is a very good list of suitable hardware! > I have a question about ZIL. > I read that SSD for it should be about 8G. I did not see more than one > gigabyte allocated for ZIL on our servers. Can you comment it? It only needs to be large enough to contain the maximum amount of synchronous write data to be committed in one transaction group. This means that it is dependent on the I/O performance of the primary storage devices in the pool, the amount of main memory, and the maximum time allowed without writing a transaction group, all of which are factors when deciding when a TXG should be written. The amount of data to be stored also depends on how large the synchronous writes are since large writes (e.g. 128k) by-pass the dedicated ZIL and go directly to main store. If there was one TXG per second, then the 8GB SSD would provide for almost 8GB of synchronous writes per second (assuming that writes are small), which is quite a lot. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From stephan.budach at jvm.de Sat Sep 30 06:45:58 2017 From: stephan.budach at jvm.de (Stephan Budach) Date: Sat, 30 Sep 2017 08:45:58 +0200 (CEST) Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: References: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: <150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de> Hi Sergey, ----- Urspr?ngliche Mail ----- > Von: "sergey ivanov" > An: "Stephan Budach" > CC: "omnios-discuss" > Gesendet: Freitag, 29. September 2017 22:30:31 > Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS > > Thanks, Stephan. > I did a simple test with creating lu over physical disks for use as > ISSI targets, and it worked well. I am going to directly connect 2 > servers and export their disks as separate I SCSI targets. Or maybe > different LUNs in a target. And then on the active server start > initiator to get these targets, and combine them into a pool of 2-way > mirrors so that it stays degraded but working if one of the servers > dies. Well, different targets mean, that you will be able to service single disks on one node, without having to degrade the whole zpool, but only the affected vdevs. On the other hand, there is more complexity since, you will have of course quite a big number of iSCSI targets to login to. This may be ok, if the number doesn't get too hight, but going with hundreds of disks, I chose to use fewer targets with more LUNs. One thing to keep in mind is, that stmfad allows you to create the guuid to your liking. That is that you can freely choose the last 20 bytes to be anything you want. I used that to ascii-code the node name and slot into the guid, such as that it displays on my NFS heads, when running format. This helps a lot in mapping the LUNs to drives. > So, manual failover for this configuration will be the following. If > the server to be disabled is still active, stop NFS, export zpool on > it, stop iscsiadm, release shared IP. On the other server: import > zpool and start NFS, activate shared IP. I am using the sharenfs properties of ZFS, but you will likely have to run zpool export -f if you want to fail over the service, since the zpool is still busy. Also, you'd better set zpool failmode to panic instead of wait, such as that an issue triggers a reboot, rather than keeping you NFS head waiting. > I read once there are some tricks which make clients do not recognize > NFS server is changed underneath all mounts, but I never tried it. The only issue I came across was, when I deliberatley failed over the NFS service forth and back within the a too short period, which causes the NFSd on the former primary node to re-use the tcp packets numbers, insisting on reusing it's old NFS connections to the clients. I solved that by resetting the NFSd each time a service starts on any NFS head. The currently connected NFS clients are not affected by that and this solved this particular issue for me. Cheers, Stephan > -- > Regards, > Sergey Ivanov. > > Regards, > Sergey Ivanov > > > On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach > wrote: > > Hi Sergey, > > > > ----- Urspr?ngliche Mail ----- > >> Von: "sergey ivanov" > >> An: "Stephan Budach" > >> CC: "omnios-discuss" > >> Gesendet: Mittwoch, 27. September 2017 23:15:49 > >> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS > >> > >> Thanks, Stephan! > >> > >> Please, explain "The reason to use two x two separate servers is, > >> that > >> the mirrored zpool's vdevs look the same on each NFS head". > >> > >> I understand that, if I want to have the same zpool based on iscsi > >> devices, I should not mix local disks with iscsi target disks. > >> > >> But I think I can have 2 computers, each exporting a set of local > >> disks as iscsi targets. And to have iscsi initiators on the same > >> computers importing these targets to build zpools. > >> > >> Also, looking at sbdadm, I think I can 'create lu > >> /dev/rdsk/c0t0d3s2'. > >> > >> Ok, I think I would better try it and report how it goes. > > > > Actually, things can become quite complex, I'd like to reduce the > > "mental" involvement to the absolute minimum, mainly because we > > often faced a situation where something would suddenly break, > > which had been running for a long time without problems. This is > > when peeple start? well maybe not panicking, but having to recap > > what the current setup was like and what they had to do to tackle > > this. > > > > So, uniformity is a great deal of help on such systems - at least > > for us. Technically, there is no issue with mixing local and > > remote iSCST targets on the same node, which serves as an iSCSI > > target and a NFS head. > > > > Also, if one of the nodes really goes down, you will be loosing > > your failover NFS head as well, maybe not a big deal and depending > > on your requirements okay. I do have such a setup as well, > > although only for an archive ZPOOL, where I can tolerate this > > reduced redundancy for the benefit of a more lightweight setup. > > > > Cheers, > > Stephan > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5546 bytes Desc: not available URL: From alka at hfg-gmuend.de Sat Sep 30 08:17:11 2017 From: alka at hfg-gmuend.de (=?UTF-8?Q?G=c3=bcnther_Alka?=) Date: Sat, 30 Sep 2017 10:17:11 +0200 Subject: [OmniOS-discuss] OmniOS based redundant NFS In-Reply-To: <150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de> References: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de> <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de> <150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de> Message-ID: I have added a similar solution with full service and storage failover of two identical storage servers last year into my napp-it web ui as an experimental solution. Main difference is that I do nor use single disks as targets but the whole ZFS pools on two servers to mirror them over the network and share the resulting network mirror. This will allow a head to function with full redundancy if the other server fails. As you only need one LUN per server this reduces complexity.? I also switch target ip and NFS/SMB service ip for an instant service failover, see my solution under http://www.napp-it.org/doc/downloads/z-raid.pdf Gea @napp-it.org Am 30.09.2017 um 08:45 schrieb Stephan Budach: > Hi Sergey, > > ----- Urspr?ngliche Mail ----- >> Von: "sergey ivanov" >> An: "Stephan Budach" >> CC: "omnios-discuss" >> Gesendet: Freitag, 29. September 2017 22:30:31 >> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS >> >> Thanks, Stephan. >> I did a simple test with creating lu over physical disks for use as >> ISSI targets, and it worked well. I am going to directly connect 2 >> servers and export their disks as separate I SCSI targets. Or maybe >> different LUNs in a target. And then on the active server start >> initiator to get these targets, and combine them into a pool of 2-way >> mirrors so that it stays degraded but working if one of the servers >> dies. > Well, different targets mean, that you will be able to service single disks on one node, without having to degrade the whole zpool, but only the affected vdevs. On the other hand, there is more complexity since, you will have of course quite a big number of iSCSI targets to login to. This may be ok, if the number doesn't get too hight, but going with hundreds of disks, I chose to use fewer targets with more LUNs. > > One thing to keep in mind is, that stmfad allows you to create the guuid to your liking. That is that you can freely choose the last 20 bytes to be anything you want. I used that to ascii-code the node name and slot into the guid, such as that it displays on my NFS heads, when running format. This helps a lot in mapping the LUNs to drives. > >> So, manual failover for this configuration will be the following. If >> the server to be disabled is still active, stop NFS, export zpool on >> it, stop iscsiadm, release shared IP. On the other server: import >> zpool and start NFS, activate shared IP. > I am using the sharenfs properties of ZFS, but you will likely have to run zpool export -f if you want to fail over the service, since the zpool is still busy. Also, you'd better set zpool failmode to panic instead of wait, such as that an issue triggers a reboot, rather than keeping you NFS head waiting. > >> I read once there are some tricks which make clients do not recognize >> NFS server is changed underneath all mounts, but I never tried it. > The only issue I came across was, when I deliberatley failed over the NFS service forth and back within the a too short period, which causes the NFSd on the former primary node to re-use the tcp packets numbers, insisting on reusing it's old NFS connections to the clients. I solved that by resetting the NFSd each time a service starts on any NFS head. The currently connected NFS clients are not affected by that and this solved this particular issue for me. > > Cheers, > Stephan > >> -- >> Regards, >> Sergey Ivanov. >> >> Regards, >> Sergey Ivanov >> >> >> On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach >> wrote: >>> Hi Sergey, >>> >>> ----- Urspr?ngliche Mail ----- >>>> Von: "sergey ivanov" >>>> An: "Stephan Budach" >>>> CC: "omnios-discuss" >>>> Gesendet: Mittwoch, 27. September 2017 23:15:49 >>>> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS >>>> >>>> Thanks, Stephan! >>>> >>>> Please, explain "The reason to use two x two separate servers is, >>>> that >>>> the mirrored zpool's vdevs look the same on each NFS head". >>>> >>>> I understand that, if I want to have the same zpool based on iscsi >>>> devices, I should not mix local disks with iscsi target disks. >>>> >>>> But I think I can have 2 computers, each exporting a set of local >>>> disks as iscsi targets. And to have iscsi initiators on the same >>>> computers importing these targets to build zpools. >>>> >>>> Also, looking at sbdadm, I think I can 'create lu >>>> /dev/rdsk/c0t0d3s2'. >>>> >>>> Ok, I think I would better try it and report how it goes. >>> Actually, things can become quite complex, I'd like to reduce the >>> "mental" involvement to the absolute minimum, mainly because we >>> often faced a situation where something would suddenly break, >>> which had been running for a long time without problems. This is >>> when peeple start? well maybe not panicking, but having to recap >>> what the current setup was like and what they had to do to tackle >>> this. >>> >>> So, uniformity is a great deal of help on such systems - at least >>> for us. Technically, there is no issue with mixing local and >>> remote iSCST targets on the same node, which serves as an iSCSI >>> target and a NFS head. >>> >>> Also, if one of the nodes really goes down, you will be loosing >>> your failover NFS head as well, maybe not a big deal and depending >>> on your requirements okay. I do have such a setup as well, >>> although only for an archive ZPOOL, where I can tolerate this >>> reduced redundancy for the benefit of a more lightweight setup. >>> >>> Cheers, >>> Stephan >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: