[OmniOS-discuss] esxi 5.5 to omnios r151014 nfs server issue

Fri Apr 10 14:10:44 UTC 2015

This information certainly puts the pause on me moving to r151014 for my
systems serving ESXi.

My next suspicion would be that this would be an NFS locking issue.  There
were updates to the lock manager in r151014.   Tcpdumps will reveal if that
is the problem.

I've seen numerous lockd problems in Illumos NFSv3.  Fortunately ESXi
connections has not been one of them.    Might be the end of my luck on
that.

-Chip

On Fri, Apr 10, 2015 at 8:55 AM, Hafiz Rafiyev <rafibeyli at gmail.com> wrote:

> Chip thank you for quick answer,I think you mean these Oracle parameters:
>
> I changed them as yours and as oracle's but same result on r151014 system:(
>
> again datasources on r151014 connecting randomly,and some not connected
>
> TABLE 6. RECOMMENDED NFS AND TCP/IP ADVANCED SETTINGS FOR VMWARE VSPHERE
> 5.1 DATASTORES ON ORACLE ZFS STORAGE APPLIANCE
> OPTION
> VALUE
> NFS.HeartbeatTimeout
> 5
> Nfs.Sendbuffersize
> 264
> Nfs.Receivebuffersize
> 256
> Nfs.MaxVolumes
> 256
> Net.TcpipHeapMax
> 128
> Net.TcpipHeapsize
> 32
> Nfs.heartbeatfrequency
> 20
> Nfs.heartbeatdelta
> 12
> Nfs.heartbeatmaxfailures
> 10
>
> ----- Original Message -----
> From: "Schweiss, Chip" <chip at innovates.com>
> To: "Hafiz Rafibeyli" <rafibeyli at gmail.com>
> Cc: "omnios-discuss" <omnios-discuss at lists.omniti.com>, "Günther Alka" <
> alka at hfg-gmuend.de>
> Sent: Friday, 10 April, 2015 16:11:49
> Subject: Re: [OmniOS-discuss] esxi 5.5 to omnios r151014 nfs server issue
>
> On Fri, Apr 10, 2015 at 7:51 AM, Hafiz Rafiyev < rafibeyli at gmail.com >
> wrote:
>
>
> I tested all suggested solutions(reset filesystems everyone@=modify,edited
> hosts files on esxi and omnios) but nothing changed ,
>
>
> I have same random not connected NFS share problem(also pached esxi 5.5 to
> last release 2638301),
>
> I think something changed in r151014 nfs stack,because when I restored to
> r151012 everythink running smooth
>
> Now I'm back to r151012,
>
> Hafiz
>
>
>
>
> Have you adjusted the NFS heartbeats on ESXi? I can't seem to located the
> best practices paper I found a few years ago. I think it was on Oracle or
> Nexenta, but you need to adjust your heart beats from ESXi or it will think
> your storage is offline. Here's my settings:
>
>
>
> My data stores are still on r151012, so I can't say for sure this has
> anything to do with the problem you are seeing. But I have definitely seen
> your problems before. I figured out the heart beat problem by analyzing
> tcpdumps while the datastores went off and online.
>
> -Chip
>
> BQ_BEGIN
>
>
>
> ----- Original Message -----
> From: omnios-discuss-request at lists.omniti.com
> To: "omnios-discuss" < omnios-discuss at lists.omniti.com >
> Sent: Monday, 6 April, 2015 16:41:22
> Subject: OmniOS-discuss Digest, Vol 37, Issue 16
>
> Send OmniOS-discuss mailing list submissions to
> omnios-discuss at lists.omniti.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> or, via email, send a message with subject or body 'help' to
> omnios-discuss-request at lists.omniti.com
>
> You can reach the person managing the list at
> omnios-discuss-owner at lists.omniti.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of OmniOS-discuss digest..."
>
>
> Today's Topics:
>
> 1. r151014 KVM crash (Johan Kragsterman)
> 2. Re: OmniOS r151014 is now out! (Natxo Asenjo)
> 3. pkgrecv r151014 (Al Slater)
> 4. esxi 5.5 to omnios r151014 nfs server issue (Hafiz Rafiyev)
> 5. Re: pkgrecv r151014 (Al Slater)
> 6. Re: esxi 5.5 to omnios r151014 nfs server issue (G?nther Alka)
> 7. Re: All SSD pool advice (Chris Nagele)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 6 Apr 2015 10:20:56 +0200
> From: Johan Kragsterman < johan.kragsterman at capvert.se >
> To: " omnios-discuss at lists.omniti.com "
> < omnios-discuss at lists.omniti.com >
> Subject: [OmniOS-discuss] r151014 KVM crash
> Message-ID:
> < OF8C4EC57F.10747AAD-ONC1257E1F.0027F11B-C1257E1F.002DDCA2 at inse.com >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi!
>
> I switched one of my development ?machines over to r151014. On that
> machine I got a few KVM VM's.
>
> One of them is a Linux terminal server, and when I wanted to
> update/upgrade it, both the general OS and the chroot environments I got in
> it, it crashed. I tried several times, and every time I did it, it crashed.
> It seems to run without problems when I don't do any heavy work on it, but
> with this update/upgrade, it runs for about ~5 min, then it crashes. It
> can't get started again, until I reboot the server.
>
> The following msg is from /var/adm/messages:
>
>
> 40b0000, id=1, base_msr= fee00000 PRIx64 base_address=fee00000
> Apr ?4 20:45:45 omni2 kvm: [ID 710719 kern.info ] vmcs revision_id = f
> Apr ?4 20:45:45 omni2 kvm: [ID 420667 kern.info ] kvm_lapic_reset:
> vcpu=ffffff06140a8000
> , id=2, base_msr= fee00000 PRIx64 base_address=fee00000
> Apr ?4 20:45:45 omni2 kvm: [ID 710719 kern.info ] vmcs revision_id = f
> Apr ?4 20:45:45 omni2 kvm: [ID 420667 kern.info ] kvm_lapic_reset:
> vcpu=ffffff0614236000
> , id=3, base_msr= fee00000 PRIx64 base_address=fee00000
> Apr ?4 20:45:45 omni2 kvm: [ID 710719 kern.info ] vmcs revision_id = f
> Apr ?4 20:45:52 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x1010101 data fffffd
> 7fffdfe8e0
> Apr ?4 20:45:52 omni2 last message repeated 3 times
> Apr ?4 20:45:52 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0xff3d0f9c data fffff
> d7fffdfe8b0
> Apr ?4 20:45:52 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr: 0x0
> data 0
> Apr ?4 20:45:52 omni2 last message repeated 6 times
> Apr ?4 20:45:52 omni2 kvm: [ID 291337 kern.info ] vcpu 1 received sipi
> with vector # 10
> Apr ?4 20:45:52 omni2 kvm: [ID 291337 kern.info ] vcpu 2 received sipi
> with vector # 10
> Apr ?4 20:45:52 omni2 kvm: [ID 291337 kern.info ] vcpu 3 received sipi
> with vector # 10
> Apr ?4 20:45:52 omni2 kvm: [ID 420667 kern.info ] kvm_lapic_reset:
> vcpu=ffffff06140b0000
> , id=1, base_msr= fee00800 PRIx64 base_address=fee00000
> Apr ?4 20:45:52 omni2 kvm: [ID 420667 kern.info ] kvm_lapic_reset:
> vcpu=ffffff06140a8000
> , id=2, base_msr= fee00800 PRIx64 base_address=fee00000
>
>
> Then it goes on like this:
>
>
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:25 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:25 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 8000000
> 01
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
> 001
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
> 001
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
> 001
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
> 001
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
> 001
> Apr ?4 20:46:34 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:46:34 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x525f2f data 2000000
>
>
>
> And like this:
>
>
>
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 8
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 8
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 8
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 8
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 8
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr ?4 20:50:45 omni2 kvm: [ID 713435 kern.info ] unhandled rdmsr:
> 0xff311c4c
> Apr ?4 20:50:45 omni2 kvm: [ID 391722 kern.info ] unhandled wrmsr:
> 0x526835 data 10
> Apr
>
>
> I switched back to r151012, and there everything is working fine...
>
> I do a rollback of the volumes I used for the chroots in the VM, because
> they've been messed up of the repetedly interupted upgrade attemts, so I
> run new updates/upgrades on the chroots, and even build new ones, and no
> problems here in r151012.
>
> So the problem seem to be exclusively in r151014.
>
> I got some messages on the omnios console after the VM crashes that I
> didn't record, unfortunatly. What I remember was that it was complaining
> about a bus, and it was also complains about either ps or pthread as well.
>
> I will go back to r151014 again, and run more tests like this, to get this
> clarified, and record the exact msg on the consol.
>
> Any suggestion?
>
>
>
> Best regards from/Med v?nliga h?lsningar fr?n
>
> Johan Kragsterman
>
> Capvert
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 6 Apr 2015 11:32:16 +0200
> From: Natxo Asenjo < natxo.asenjo at gmail.com >
> To: omnios-discuss < omnios-discuss at lists.omniti.com >
> Subject: Re: [OmniOS-discuss] OmniOS r151014 is now out!
> Message-ID:
> < CAHBEJzUN7-4L0AyYxBLTxUXQbUsjmb0N7oOtWpx0uGPL0i7S5Q at mail.gmail.com >
> Content-Type: text/plain; charset="utf-8"
>
> On Fri, Apr 3, 2015 at 3:58 AM, Dan McDonald < danmcd at omniti.com > wrote:
>
> > Say hello to OmniOS r151014:
> >
> > http://omnios.omniti.com/wiki.php/ReleaseNotes/r151014
>
>
> upgrade succesful on my home microserver :-)
>
> Congrats on the good work!
>
> --
> regards,
> natxo
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://omniosce.org/ml-archive/attachments/20150406/ef8ee9b8/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 3
> Date: Mon, 06 Apr 2015 11:03:47 +0100
> From: Al Slater < al.slater at scluk.com >
> To: omnios-discuss at lists.omniti.com
> Subject: [OmniOS-discuss] pkgrecv r151014
> Message-ID: < 55225A03.7020102 at scluk.com >
> Content-Type: text/plain; charset=utf-8
>
> Hi,
>
> I am trying to pkgrecv r151014 into my own repository and keep bumping
> into this:
>
> pkgrecv: Invalid contentpath opt/sunstudio12.1/prod/lib/sys/libsunir.so:
> chash failure: expected: b251c238070b6fdbf392194e85319e2c954a5384
> computed: 17d9899f959ac5835569e8870f7e02eb14607242. (happened 4 times)
>
> Is there a problem with this package in the repository?
>
> --
> Al Slater
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 6 Apr 2015 12:50:00 +0300 (EEST)
> From: Hafiz Rafiyev < rafibeyli at gmail.com >
> To: omnios-discuss < omnios-discuss at lists.omniti.com >
> Subject: [OmniOS-discuss] esxi 5.5 to omnios r151014 nfs server issue
> Message-ID:
> < 1070639246.2109450.1428313800852.JavaMail.zimbra at cantekstil.com.tr >
> Content-Type: text/plain; charset=windows-1254
>
>
> After upgrade from r151012 to r151014 i have issue with nfs server,
> after upgrade, some of Esxi 5.5 nfs datastores connecting and some not,
>
> and it's being randomly,after omnios restart again some datastores
> connected and some not
>
> when looking omnios side,nfs server up and running,
>
> note:before upgrade all esxi datastores were connected and running,omnios
> running as VM,disks connected with HBA passthruogh mode
>
> only log I see from omnios side is:
>
> nfs4cbd[468]: [ID 867284 daemon.notice] nfsv4 cannot determine local
> hostname binding for transport tcp6 - delegations will not be available on
> this transport
>
>
> regards
>
> Hafiz.
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 06 Apr 2015 11:24:30 +0100
> From: Al Slater < al.slater at scluk.com >
> To: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] pkgrecv r151014
> Message-ID: < 55225EDE.2010902 at scluk.com >
> Content-Type: text/plain; charset=windows-1252
>
> On 06/04/15 11:03, Al Slater wrote:
> > Hi,
> >
> > I am trying to pkgrecv r151014 into my own repository and keep bumping
> > into this:
> >
> > pkgrecv: Invalid contentpath opt/sunstudio12.1/prod/lib/sys/libsunir.so:
> > chash failure: expected: b251c238070b6fdbf392194e85319e2c954a5384
> > computed: 17d9899f959ac5835569e8870f7e02eb14607242. (happened 4 times)
> >
> > Is there a problem with this package in the repository?
>
> Same happens with pkg install...
>
> # pkg install pkg:/developer/sunstudio12.1 at 12.1-0.151014
> Packages to install: 1
> Create boot environment: No
> Create backup boot environment: No
>
> DOWNLOAD PKGS FILES XFER (MB)
> SPEED
> developer/sunstudio12.1 0/1 5042/7006 203.1/256.3
> 3.0M/s
>
>
>
> Errors were encountered while attempting to retrieve package or file
> data for
> the requested operation.
> Details follow:
>
> Invalid contentpath opt/sunstudio12.1/prod/lib/sys/libsunir.so: chash
> failure: expected: b251c238070b6fdbf392194e85319e2c954a5384 computed:
> 17d9899f959ac5835569e8870f7e02eb14607242. (happened 4 times)
>
>
> regards
>
> --
> Al Slater
>
> Technical Director
> SCL
>
> Phone : +44 (0)1273 666607
> Fax : +44 (0)1273 666601
> email : al.slater at scluk.com
>
> Stanton Consultancy Ltd
>
> Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU
>
> Registered in England Company number: 1957652 VAT number: GB 760 2433 55
>
>
>
> ------------------------------
>
> Message: 6
> Date: Mon, 6 Apr 2015 12:34:30 +0200
> From: G?nther Alka < alka at hfg-gmuend.de >
> To: omnios-discuss < omnios-discuss at lists.omniti.com >
> Subject: Re: [OmniOS-discuss] esxi 5.5 to omnios r151014 nfs server
> issue
> Message-ID: < 4700B3B3-2CED-407D-A131-62FE1E392B53 at hfg-gmuend.de >
> Content-Type: text/plain; charset=us-ascii
>
> just to rule out a permission problem
>
> can you recursively reset permissions of that filesystem to a
> everyone@=modify setting.
>
>
>
> > Am 06.04.2015 um 11:50 schrieb Hafiz Rafiyev < rafibeyli at gmail.com >:
> >
> >
> > After upgrade from r151012 to r151014 i have issue with nfs server,
> > after upgrade, some of Esxi 5.5 nfs datastores connecting and some not,
> >
> > and it's being randomly,after omnios restart again some datastores
> connected and some not
> >
> > when looking omnios side,nfs server up and running,
> >
> > note:before upgrade all esxi datastores were connected and
> running,omnios running as VM,disks connected with HBA passthruogh mode
> >
> > only log I see from omnios side is:
> >
> > nfs4cbd[468]: [ID 867284 daemon.notice] nfsv4 cannot determine local
> hostname binding for transport tcp6 - delegations will not be available on
> this transport
> >
> >
> > regards
> >
> > Hafiz.
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
>
> ------------------------------
>
> Message: 7
> Date: Mon, 6 Apr 2015 09:41:19 -0400
> From: Chris Nagele < nagele at wildbit.com >
> To: " omnios-discuss at lists.omniti.com "
> < omnios-discuss at lists.omniti.com >
> Subject: Re: [OmniOS-discuss] All SSD pool advice
> Message-ID:
> < CAHfYOdUN_CWsmPVDCZGRh3pCUoSkRkWThwBj7khkj+ztiwC5Zg at mail.gmail.com >
> Content-Type: text/plain; charset=UTF-8
>
> Thanks everyone. Regarding the expanders, our 4U servers are on the
> following chassis:
>
> http://www.supermicro.com/products/chassis/4U/846/SC846E16-R1200.cfm
>
> We are using all SAS disks, except for the SSDs. How big is the risk
> here when it comes to SAS -> SATA conversion? Our newer servers have
> direct connections on each lane to the disk.
>
> Chris
>
> Chris Nagele
> Co-founder, Wildbit
> Beanstalk, Postmark, dploy.io
>
>
> On Sat, Apr 4, 2015 at 7:18 PM, Doug Hughes < doug at will.to > wrote:
> >
> > We have a couple of machines with all SSD pool (~6-10 Samsung 850 pro is
> the
> > current favorite). They work great for IOPS. Here's my take.
> > 1) you don't need a dedicated zil. Just let the zpool intersperse it
> amongst
> > the existing zpool devices. They are plenty fast enough.
> > 2) you don't need an L2arc for the same reason. a smaller number of
> > dedicated devices would likely cause more of a bottleneck than serving
> off
> > the existing pool devices (unless you were to put it on one of those
> giant
> > RDRAM things or similar, but that adds a lot of expense)
> >
> >
> >
> >
> >
> > On 4/4/2015 3:07 PM, Chris Nagele wrote:
> >
> > We've been running a few 4U Supermicro servers using ZeusRAM for zil and
> > SSDs for L2. The main disks are regular 1TB SAS.
> >
> > I'm considering moving to all SSD since the pricing has dropped so much.
> > What things should I know or do when moving to all SSD pools? I'm
> assuming I
> > don't need L2 and that I should keep the ZeusRAM. Should I only use
> certain
> > types of SSDs?
> >
> > Thanks,
> > Chris
> >
> >
> > --
> >
> > Chris Nagele
> > Co-founder, Wildbit
> > Beanstalk, Postmark, dploy.io
> >
> >
> >
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
> >
> >
> >
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
> >
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
> ------------------------------
>
> End of OmniOS-discuss Digest, Vol 37, Issue 16
> **********************************************
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
> BQ_END
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20150410/6f4557c2/attachment-0001.html>