From henson at acm.org Thu Aug 1 03:22:02 2013 From: henson at acm.org (Paul B. Henson) Date: Wed, 31 Jul 2013 20:22:02 -0700 Subject: [OmniOS-discuss] SSD alignment, EFI label rpool support Message-ID: <51F9D45A.8020203@acm.org> While putting together a linux system, I found the current advice in that ecosystem seems to be to align your SSD partitions not just to the page size, but to the erase block size. Any thoughts on that recommendation for an rpool, l2arc, or zil partition? I'm using a couple Crucial 256G m4 SSD's for my rpool, which I believe have an 8k page size and 512k erase block size. Given the rpool requirement for an SMI label, I would end up having to start the slice on cylinder 15 (which, when added to the one cylinder used by the fdisk label, results in cylinder 16) in order to get the alignment right, which wastes something like 150Mb :(. Not ridiculous on a 256G device, but annoying. Has any progress been made on booting from EFI labeled disks? I vaguely recall somebody was working on grub2 but I'm not sure where it ended up. That would allow alignment without as much waste. Or if erase block alignment doesn't matter that much for ZFS I could just align to the page size? Thanks? From richard.elling at gmail.com Thu Aug 1 10:39:31 2013 From: richard.elling at gmail.com (Richard Elling) Date: Thu, 1 Aug 2013 03:39:31 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51F9D45A.8020203@acm.org> References: <51F9D45A.8020203@acm.org> Message-ID: <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> On Jul 31, 2013, at 8:22 PM, Paul B. Henson wrote: > While putting together a linux system, I found the current advice in that ecosystem seems to be to align your SSD partitions not just to the page size, but to the erase block size. Any thoughts on that recommendation for an rpool, l2arc, or zil partition? > > I'm using a couple Crucial 256G m4 SSD's for my rpool, which I believe have an 8k page size and 512k erase block size. Given the rpool requirement for an SMI label, I would end up having to start the slice on cylinder 15 (which, when added to the one cylinder used by the fdisk label, results in cylinder 16) in order to get the alignment right, which wastes something like 150Mb :(. Eh? Cylinders? The ZFS label already reserves 8KB of space at the front so that it will not clobber an SMI label. The actual data use begins at a 4MB offset, past the ZFS labels and reserved space. In other words, why would you purposefully misalign? -- richard -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Thu Aug 1 12:42:35 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Thu, 01 Aug 2013 14:42:35 +0200 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> Message-ID: <51FA57BB.2090209@cos.ru> On 2013-08-01 12:39, Richard Elling wrote: > The ZFS label already reserves 8KB of space at the front so that it will > not clobber an SMI label. > The actual data use begins at a 4MB offset, past the ZFS labels and > reserved space. > > In other words, why would you purposefully misalign? Well, technically - this is correct, except that this applies to offsets within the the "device" which you gave to ZFS to be a leaf component of the pool. This device may be a classic Solaris slice in a SMI label-table, possibly in MBR-backed partitions on x86, or a whole partition (in either MBR or EFI definitions), or a file - just to be complete. What matters is that this container (slice/partition) usually does not start at HDD sector 0 (and as history has shown, complex devices such as "old-windows-compatible 4K AF drives" or RAID0-backed JBOD LUNs or anything else may lead to the logical 0-offset not being physically well aligned with hardware sectors either). The same applies to cases where you give ZFS a "whole disk" and it creates a EFI partition table according to its rules, and marks "whole-disk usage" in the pool labels - but otherwise this is an ordinary partition table (properly aligned by default, in belief that physical 0 == logical 0). I believe Paul's question, just like any question of this sort, regarded the possible need to realign his partitions - or perhaps a way to verify that they are aligned. In case of SSD, there is a fresh twist regarding page size vs. erase-block size (and not yet asked - a recommendation about recommended ZFS minblocksize for such devices). Now, since 512k is divisible by 8k, an offset of 512k or 1024k for the partition which should contain the rpool should be good for both types of alignment in question (note the next paragraph though). While it may indeed be problematic to carve disks with such precision via fdisk/format, one can use the command-line "parted" to manage disk partitions, including MBR-style ones. When the MBR partition for the rpool with the desired offset (and Solaris or maybe Solaris2 type) is made, it can be sliced with "format" in order to designate a container for rpool. I believe, manually prepared partitions like this can also be used in the Caiman installer, so you don't have to fuss with "format" (the installer will overwrite your slicing anyway). Note that in my sample box which I glanced at while writing this, the zeroth "cylinder" (16065 * 512-byte "blocks" or 7.84MB) is reserved on x86 for "boot", and the rpool starts at cylinder number 1. This may mean that for proper alignment of the rpool, its MBR partition may have to start at, for example, 8Mb-7.84Mb or 16384 - 16065 512b-"blocks" (legacy "sectors", as still used in partitioning terminology), give or take one ;) This way the rpool's slice 0 would start at the physical device's logical sector 16384 which is hopefully properly aligned for the IOs, and ZFS's 4Mb offset further into that would not contradict anything. Note that I've picked 8Mb rather arbitrarily, as a multiple of 1Mb next after this "cylinder" size. The classical MBR layout does only reserve 63 sectors (and yes, the "tracks" have odd sizes) before the first partition, which is what bootloaders should be able to cope with. In my example I give ample room - over 300 sectors ;) Some software (i.e. for low-level disk archiving) may complain about offsets which are not whole tracks, but otherwise this is quite usable. HTH, //Jim Klimov From henson at acm.org Thu Aug 1 19:30:42 2013 From: henson at acm.org (Paul B. Henson) Date: Thu, 01 Aug 2013 12:30:42 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> Message-ID: <51FAB762.8020109@acm.org> On 8/1/2013 3:39 AM, Richard Elling wrote: > Eh? Cylinders? > The ZFS label already reserves 8KB of space at the front so that it will > not clobber an SMI label. > The actual data use begins at a 4MB offset, past the ZFS labels and > reserved space. > > In other words, why would you purposefully misalign? Well, obviously I wouldn't *purposefully* misalign, but I can't rule out the possibility of *incompetently* misaligning ;). I think you are talking about the case when you give zfs an entire disk? In that case, yes, I understand there is no extra magic to be performed. However, I am specifically talking about the case of the rpool, in which you currently cannot give zfs the entire disk, but must give it a slice of an SMI labeled disk. In that case, you have the fdisk partitioning, in which from what I understand the Solaris partition typically starts with a one cylinder offset. So, when you give zfs the first slice of the Solaris partition, it already has an offset. Are you saying zfs refers to the fdisk partition, figures out where on the disk the first slice actually starts, and then just does the right thing? I did not believe that to be the case... From henson at acm.org Thu Aug 1 19:39:10 2013 From: henson at acm.org (Paul B. Henson) Date: Thu, 01 Aug 2013 12:39:10 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51FA57BB.2090209@cos.ru> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> <51FA57BB.2090209@cos.ru> Message-ID: <51FAB95E.6090207@acm.org> On 8/1/2013 5:42 AM, Jim Klimov wrote: > What matters is that this container (slice/partition) usually does > not start at HDD sector 0 Yes, you understood my question exactly, thanks for expanding upon it. > fresh twist regarding page size vs. erase-block size (and not yet > asked - a recommendation about recommended ZFS minblocksize for > such devices). Oh no, I'm almost done tuning my install, not more possible options to have to explore ;). > such precision via fdisk/format, one can use the command-line > "parted" to manage disk partitions, including MBR-style ones. Hmm, if I understand you correctly, you are recommending using a non-illumos tool to create a Solaris MBR partition aligned on an arbitrary sector rather than a cylinder, which would then allow the beginning of slice zero in that partition to be in the right place? I didn't think of that, that seems much simpler than trying to calculate a cylinder offset of the slice relative to the beginning of the partition that lines up. Thanks much? From jimklimov at cos.ru Thu Aug 1 20:21:51 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Thu, 01 Aug 2013 22:21:51 +0200 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51FAB95E.6090207@acm.org> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> <51FA57BB.2090209@cos.ru> <51FAB95E.6090207@acm.org> Message-ID: <51FAC35F.7010404@cos.ru> On 2013-08-01 21:39, Paul B. Henson wrote: > On 8/1/2013 5:42 AM, Jim Klimov wrote: > >> What matters is that this container (slice/partition) usually does >> not start at HDD sector 0 > > Yes, you understood my question exactly, thanks for expanding upon it. > >> fresh twist regarding page size vs. erase-block size (and not yet >> asked - a recommendation about recommended ZFS minblocksize for >> such devices). > > Oh no, I'm almost done tuning my install, not more possible options to > have to explore ;). > >> such precision via fdisk/format, one can use the command-line >> "parted" to manage disk partitions, including MBR-style ones. > > Hmm, if I understand you correctly, you are recommending using a > non-illumos tool to create a Solaris MBR partition aligned on an > arbitrary sector rather than a cylinder, which would then allow the > beginning of slice zero in that partition to be in the right place? I > didn't think of that, that seems much simpler than trying to calculate a > cylinder offset of the slice relative to the beginning of the partition > that lines up. In short - yes. As can be seen in src.illumos.org, the GNU parted (or some derivate thereof) is even part of the illumos-gate, so it is not a non-illumos tool now ;) > Thanks much? You are welcome, and I hope this does help :) //Jim From mahrens at delphix.com Thu Aug 1 20:27:23 2013 From: mahrens at delphix.com (Matthew Ahrens) Date: Thu, 1 Aug 2013 13:27:23 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51F9D45A.8020203@acm.org> References: <51F9D45A.8020203@acm.org> Message-ID: It's hard for me to see how aligning the partition on a greater granularity than the physical sector size (aka "ashift", typically 4k for modern large devices) would help anything. Regardless of partition alignment, ZFS is going to write to whatever sectors it wants. --matt On Wed, Jul 31, 2013 at 8:22 PM, Paul B. Henson wrote: > While putting together a linux system, I found the current advice in that > ecosystem seems to be to align your SSD partitions not just to the page > size, but to the erase block size. Any thoughts on that recommendation for > an rpool, l2arc, or zil partition? > > I'm using a couple Crucial 256G m4 SSD's for my rpool, which I believe > have an 8k page size and 512k erase block size. Given the rpool requirement > for an SMI label, I would end up having to start the slice on cylinder 15 > (which, when added to the one cylinder used by the fdisk label, results in > cylinder 16) in order to get the alignment right, which wastes something > like 150Mb :(. Not ridiculous on a 256G device, but annoying. Has any > progress been made on booting from EFI labeled disks? I vaguely recall > somebody was working on grub2 but I'm not sure where it ended up. That > would allow alignment without as much waste. > > Or if erase block alignment doesn't matter that much for ZFS I could just > align to the page size? > > Thanks? > > > ------------------------------**------------- > illumos-zfs > Archives: https://www.listbox.com/**member/archive/182191/=now > RSS Feed: https://www.listbox.com/**member/archive/rss/182191/** > 21635000-ebd1d460 > Modify Your Subscription: https://www.listbox.com/** > member/?member_id=21635000&id_**secret=21635000-73dc201a > Powered by Listbox: http://www.listbox.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Thu Aug 1 21:13:41 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Thu, 01 Aug 2013 23:13:41 +0200 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: References: <51F9D45A.8020203@acm.org> Message-ID: <51FACF85.1060804@cos.ru> On 2013-08-01 22:27, Matthew Ahrens wrote: > It's hard for me to see how aligning the partition on a greater > granularity than the physical sector size (aka "ashift", typically 4k > for modern large devices) would help anything. Regardless of partition > alignment, ZFS is going to write to whatever sectors it wants. Well, it is my understanding (as detailed in another post) that the partitioning tables (MBR/SMI, GPT/EFI) still count in 512-byte units as the minimum size, over whatever hardware sector sizes. So it is, on one hand, required to do all this accounting in finer granularities than 4k sectors or 8k pages, and on another hand - it is possible to make a mistake in all this. And the goal of alignment is to not have ZFS block writes span incomplete hardware sectors. One potential for errors, which may break ZFS whole-drive usage with its default EFI label generation, is the alleged existence of 4k AF drives with a DIP-switch or something like that which shifts the LBAs by one legacy sector, so that the first partition which starts at LBA sector number 63 would in fact start on a physical 4k sector boundary. Apparently, this helps optimal usage of new hardware from old OSes like Windows XP by default, which would format the disk with a 4k clustered NTFS or FAT32 and use the hardware sectors wholly and well aligned, to store their FS clusters, and not care or know about either alignment or non-512b sector sizing. My 2c, //Jim Klimov From daleg at omniti.com Thu Aug 1 22:14:26 2013 From: daleg at omniti.com (Dale Ghent) Date: Thu, 1 Aug 2013 18:14:26 -0400 Subject: [OmniOS-discuss] Updated release: r151006n Message-ID: <5BAED73C-33C1-4A61-810C-4A7544FDAF7E@omniti.com> This week brings a new release: r151006n This release includes a security fix for for the kernel and BIND, as well as other bug fixes and minor improvements, namely with the ncurses library (and apps which use it) and SVM/LVM/DiskSuite. As there is a kernel change (specifically, in the ip driver), a reboot is required. Full release notes are available here: http://omnios.omniti.com/wiki.php/ReleaseNotes#r151006n As this is an update which addresses security issues, fresh r151006n install media has also been generated: http://omnios.omniti.com/wiki.php/Installation /dale -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hakansom at ohsu.edu Fri Aug 2 00:02:06 2013 From: hakansom at ohsu.edu (Marion Hakanson) Date: Thu, 01 Aug 2013 17:02:06 -0700 Subject: [OmniOS-discuss] R720 reset magic? Message-ID: <201308020002.r720265f001086@kyklops.ohsu.edu> Greetings, I'm trying out OmniOS 151006l on a new R720, and have noticed that a "shutdown -i6" style shutdown does not reboot the server. It goes through the "syncing disks..." and "done" phases, then just stops there. Power remains on. An IPMI or DRAC "reset" gets things going again. Note that a "shutdown -i5" style shutdown does power off the system; And "shutdown -i6" works fine on all our older R710's (one OmniOS, one oi151a7). I've disabled the fast-reboot flag on the R720 as well, with no change in behavior, via: svccfg -s system/boot-config:default setprop config/fastreboot_default=false Anyone else out there with experience running OmniOS on an R720? Have we tripped over some new BIOS default somewhere? Suggestions on debugging what's probably an acpi-related thing would be welcome. Thanks and regards, Marion From henson at acm.org Fri Aug 2 02:24:22 2013 From: henson at acm.org (Paul B. Henson) Date: Thu, 01 Aug 2013 19:24:22 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51FAC35F.7010404@cos.ru> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> <51FA57BB.2090209@cos.ru> <51FAB95E.6090207@acm.org> <51FAC35F.7010404@cos.ru> Message-ID: <51FB1856.5060903@acm.org> On 8/1/2013 1:21 PM, Jim Klimov wrote: > As can be seen in src.illumos.org, the GNU parted (or some derivate > thereof) is even part of the illumos-gate, so it is not a non-illumos > tool now ;) Cool; I didn't know that. It looks like Solaris 11 supports booting from an EFI-labled rpool now; given grub is GPL, presumably they had to release the modifications to do that? I wonder how cleanly they would integrate into illumos, it would be nice to get away from the requirement for SMI labels on rpool. From henson at acm.org Fri Aug 2 02:30:20 2013 From: henson at acm.org (Paul B. Henson) Date: Thu, 01 Aug 2013 19:30:20 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: References: <51F9D45A.8020203@acm.org> Message-ID: <51FB19BC.9080509@acm.org> On 8/1/2013 1:27 PM, Matthew Ahrens wrote: > It's hard for me to see how aligning the partition on a greater > granularity than the physical sector size (aka "ashift", typically 4k > for modern large devices) would help anything. Regardless of partition > alignment, ZFS is going to write to whatever sectors it wants. Well, as I originally indicated, this suggestion came from the linux world, and was more perhaps intended for ext3/4 use cases, they discuss setting the stride and stripe-width options for ext filesystems to optimize for the page/erase block size. I wasn't really sure if that was relevant for ZFS, which is why I asked :). Thanks? From thibault.vincent at smartjog.com Fri Aug 2 00:24:27 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Fri, 02 Aug 2013 00:24:27 -0000 (UTC) Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <201308020002.r720265f001086@kyklops.ohsu.edu> Message-ID: > I'm trying out OmniOS 151006l on a new R720, and have noticed that a > "shutdown -i6" style shutdown does not reboot the server. It goes > through the "syncing disks..." and "done" phases, then just stops there. > Power remains on. Hello Marion, I confirm I get the same behaviour with two R720xd all firmwares up-to-date. Power also stays on after "reboot -c" and most of the time after a kernel crash dump. But I have no solution and did not investigate yet because my servers currently have way more serious problems with mpath. Cheers -- Thibault VINCENT - Infrastructure Operations Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From danmcd at nexenta.com Fri Aug 2 03:06:32 2013 From: danmcd at nexenta.com (Dan McDonald) Date: Thu, 1 Aug 2013 23:06:32 -0400 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <51FB1856.5060903@acm.org> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> <51FA57BB.2090209@cos.ru> <51FAB95E.6090207@acm.org> <51FAC35F.7010404@cos.ru> <51FB1856.5060903@acm.org> Message-ID: <547672CF-9CED-4527-B7E1-CF38A30D0FCF@nexenta.com> On Aug 1, 2013, at 10:24 PM, Paul B. Henson wrote: > > It looks like Solaris 11 supports booting from an EFI-labled rpool now; given grub is GPL, presumably they had to release the modifications to do that? I wonder how cleanly they would integrate into illumos, it would be nice to get away from the requirement for SMI labels on rpool. ISTR S11 switched to GRUB2. That would be interesting, but a non-trivial testing task, to say the least. Dan From jimklimov at cos.ru Fri Aug 2 14:44:09 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 02 Aug 2013 16:44:09 +0200 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) Message-ID: <51FBC5B9.9090804@cos.ru> Hello, I decided to give OmniOS a practical shot - or at least try something other than glare at it from a distance; but one aspect that is not yet clear to me - even after reading the installation docs, blogs, etc. - is a simple question: how different is it (for OS administration) from OpenIndiana? From the docs and examples I see that the CLI syntax for network and zone management is quite the same, packaging seems to be IPS, though the installers are somewhat different - but hey, you only need them once in a lifetime, and I know too well how imperfect Caiman is for my taste ;) So, in layman's terms for a newcomer to this platform - can it be said that OmniOS is like OI, only without desktop aspirations and with more frequent updates (in bloody at least)? Can the two OSes be noticeably distinguished while using them for server tasks via SSH or serial consoles? On a side note, does OmniOS still include *some* X11 good enough to fire up a VNC server for some of those programs which do require a graphics interface but need nothing fancy (some enterprise software installers, java consoles, etc.) Similarly, is there enough relation between the two (OmniOS and OI) so as to be able to easily migrate from one to another by changing the package repo provider and updating a new BE, or are things too diverged for that? :) Can the OmniOS package repositories be easily cloned (and/or built) in order to have a replica in (closed) LAN - like can be done for OI? If I build my own ZFS-root image for the networked installer, can it involve a hierarchy of several datasets or just one root FS dataset? Sorry if this was all answered before, then my google-foo and ability to walk links on OmniOS Wiki is seriously impaired by the brain-melting heat in central Europe ;) If not - well, I think answers to this letter deserve a prominent spot on the project's web site in order to lure some potential converts like myself ;) Thanks, //Jim Klimov From jesus at omniti.com Fri Aug 2 15:20:40 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 2 Aug 2013 08:20:40 -0700 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) In-Reply-To: <51FBC5B9.9090804@cos.ru> References: <51FBC5B9.9090804@cos.ru> Message-ID: On Fri, Aug 2, 2013 at 7:44 AM, Jim Klimov wrote: > Hello, > > I decided to give OmniOS a practical shot - or at least try something > other than glare at it from a distance; but one aspect that is not yet > clear to me - even after reading the installation docs, blogs, etc. - > is a simple question: how different is it (for OS administration) from > OpenIndiana? > That's a subjective question. I'd imagine it looks quite similar to some and starkly different to others. > From the docs and examples I see that the CLI syntax for network and > zone management is quite the same, packaging seems to be IPS, though the > installers are somewhat different - but hey, you only need them once in > a lifetime, and I know too well how imperfect Caiman is for my taste ;) > > So, in layman's terms for a newcomer to this platform - can it be said > that OmniOS is like OI, only without desktop aspirations and with more > frequent updates (in bloody at least)? Can the two OSes be noticeably > distinguished while using them for server tasks via SSH or serial > consoles? On a side note, does OmniOS still include *some* X11 good > enough to fire up a VNC server for some of those programs which do > require a graphics interface but need nothing fancy (some enterprise > software installers, java consoles, etc.) > No shipped X at all, not enough to run X stuff in anyway. You can, however, get X pkgs from pkgsrc. > Similarly, is there enough relation between the two (OmniOS and OI) > so as to be able to easily migrate from one to another by changing the > package repo provider and updating a new BE, or are things too diverged > for that? :) > Nope. Too different for that. You can image-create into a new BE, but that's certainly not an update. > Can the OmniOS package repositories be easily cloned (and/or built) > in order to have a replica in (closed) LAN - like can be done for OI? > Yes. pkgrecv them. > If I build my own ZFS-root image for the networked installer, can it > involve a hierarchy of several datasets or just one root FS dataset? > Not as a baked in feature, but there is post-install scripting available and you could easily make that happen without modifying kayak. (just zfs recv the extra data sets over the network after the rpool is recv'd). I think you'll be pleasantly surprised by the simplicity and power of kayak. > Sorry if this was all answered before, then my google-foo and ability > to walk links on OmniOS Wiki is seriously impaired by the brain-melting > heat in central Europe ;) If not - well, I think answers to this letter > deserve a prominent spot on the project's web site in order to lure > some potential converts like myself ;) > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Fri Aug 2 16:12:16 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 02 Aug 2013 18:12:16 +0200 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) In-Reply-To: References: <51FBC5B9.9090804@cos.ru> Message-ID: <51FBDA60.1060503@cos.ru> Hello Theo, thanks for the prompt reply! On 2013-08-02 17:20, Theo Schlossnagle wrote: > On Fri, Aug 2, 2013 at 7:44 AM, Jim Klimov > wrote: > That's a subjective question. I'd imagine it looks quite similar to > some and starkly different to others. Right... well, to put it another way: I have rehearsed the procedure to update older OpenSolaris SXCE servers to OpenIndiana before that project stalled (and then became too experimental), and while some servers can wait a few years more, others would quite benefit from an OS upgrade. If I've figured out the needed procedure for OI, would I likely easily adapt and reuse it for OmniOS, or should I expect to learn a lot of new stuff and restart rehearsing from scratch? :^) > No shipped X at all, not enough to run X stuff in anyway. You can, > however, get X pkgs from pkgsrc. Uh-huh... and is there an install-ready IPS repo of pkgsrc provided by OmniTI (or at least recommended, even if maintained by some other team), along with occasional updates of software in the repo? Sorry for this particular lame question, I guess I just did not get too far through the docs yet :\ > If I build my own ZFS-root image for the networked installer, can it > involve a hierarchy of several datasets or just one root FS dataset? > > > Not as a baked in feature, but there is post-install scripting available > and you could easily make that happen without modifying kayak. (just zfs > recv the extra data sets over the network after the rpool is recv'd). I > think you'll be pleasantly surprised by the simplicity and power of kayak. And is that all received as a "zfs-send" stream packed into a file? I mean, are there any limitations in Kayak today that would limit an ability to receive a "replication" stream which already includes a hierarchy of datasets? In a simple case - a hierarchy rooted under one common root filesystem dataset - for example, rpool/ROOT/rootfs and rpool/ROOT/rootfs/var (and maybe some split usr/ and opt/)?.. I would likely need scripted receiving for my other half of the general setup - the rpool/SHARED/var/* datasets (logs, mail, etc.) to be common across different BEs. Thanks, //Jim Klimov From jimklimov at cos.ru Fri Aug 2 16:34:05 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 02 Aug 2013 18:34:05 +0200 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) In-Reply-To: <51FBC5B9.9090804@cos.ru> References: <51FBC5B9.9090804@cos.ru> Message-ID: <51FBDF7D.8080501@cos.ru> ... and a few more questions: 1) Does the stable or bloody ISO include relatively recent illumos gate features such as LZ4 compression for rpools? I am a bit alarmed that the filename timestamp for bloody involves February 2013, if I read it right... ;) 2) Is there an out-of-the-box or via simple hackery a way to get the omnios rootfs onto an existing rpool, perhaps one of an older version (think SXCE/Sol10)? 3) Installing additional SVR4 packages in GZ and LZs should not be a problem, right? :) Thanks again, //Jim From garrett.damore at dey-sys.com Fri Aug 2 16:22:26 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Fri, 2 Aug 2013 09:22:26 -0700 Subject: [OmniOS-discuss] [zfs] SSD alignment, EFI label rpool support In-Reply-To: <547672CF-9CED-4527-B7E1-CF38A30D0FCF@nexenta.com> References: <51F9D45A.8020203@acm.org> <380A4A44-37A4-4BAD-90F5-BFEB4A6AA39B@gmail.com> <51FA57BB.2090209@cos.ru> <51FAB95E.6090207@acm.org> <51FAC35F.7010404@cos.ru> <51FB1856.5060903@acm.org> <547672CF-9CED-4527-B7E1-CF38A30D0FCF@nexenta.com> Message-ID: <1F396483-C9C4-4927-90FA-DAC73D341018@dey-sys.com> On Aug 1, 2013, at 8:06 PM, Dan McDonald wrote: > > On Aug 1, 2013, at 10:24 PM, Paul B. Henson wrote: >> >> It looks like Solaris 11 supports booting from an EFI-labled rpool now; given grub is GPL, presumably they had to release the modifications to do that? I wonder how cleanly they would integrate into illumos, it would be nice to get away from the requirement for SMI labels on rpool. > > ISTR S11 switched to GRUB2. That would be interesting, but a non-trivial testing task, to say the least. Thats correct. I believe Seth did a large chunk of that work. I'd like to investigate another alternative altogether -- the BSD licensed loader that FreeBSD uses. I just haven't had cycles. :-) - Garrett > > Dan > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From jesus at omniti.com Fri Aug 2 17:36:03 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 2 Aug 2013 10:36:03 -0700 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) In-Reply-To: <51FBDA60.1060503@cos.ru> References: <51FBC5B9.9090804@cos.ru> <51FBDA60.1060503@cos.ru> Message-ID: On Fri, Aug 2, 2013 at 9:12 AM, Jim Klimov wrote: > Hello Theo, thanks for the prompt reply! > > > On 2013-08-02 17:20, Theo Schlossnagle wrote: > >> On Fri, Aug 2, 2013 at 7:44 AM, Jim Klimov > > wrote: >> That's a subjective question. I'd imagine it looks quite similar to >> some and starkly different to others. >> > > Right... well, to put it another way: I have rehearsed the procedure > to update older OpenSolaris SXCE servers to OpenIndiana before that > project stalled (and then became too experimental), and while some > servers can wait a few years more, others would quite benefit from > an OS upgrade. If I've figured out the needed procedure for OI, would > I likely easily adapt and reuse it for OmniOS, or should I expect to > learn a lot of new stuff and restart rehearsing from scratch? :^) We don't use incorporations and consolidations the same way. So, the procedure will most certainly be different. > No shipped X at all, not enough to run X stuff in anyway. You can, >> however, get X pkgs from pkgsrc. >> > > Uh-huh... and is there an install-ready IPS repo of pkgsrc provided > by OmniTI (or at least recommended, even if maintained by some other > team), along with occasional updates of software in the repo? > > Sorry for this particular lame question, I guess I just did not get > too far through the docs yet :\ http://comments.gmane.org/gmane.os.omnios.general/640 > > If I build my own ZFS-root image for the networked installer, can >> it >> involve a hierarchy of several datasets or just one root FS dataset? >> >> >> Not as a baked in feature, but there is post-install scripting available >> and you could easily make that happen without modifying kayak. (just zfs >> recv the extra data sets over the network after the rpool is recv'd). I >> think you'll be pleasantly surprised by the simplicity and power of kayak. >> > > And is that all received as a "zfs-send" stream packed into a file? > I mean, are there any limitations in Kayak today that would limit an > ability to receive a "replication" stream which already includes a > hierarchy of datasets? In a simple case - a hierarchy rooted under > one common root filesystem dataset - for example, rpool/ROOT/rootfs > and rpool/ROOT/rootfs/var (and maybe some split usr/ and opt/)?.. > I would likely need scripted receiving for my other half of the > general setup - the rpool/SHARED/var/* datasets (logs, mail, etc.) > to be common across different BEs. I've never tried. I see no reason it couldn't work. -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesus at omniti.com Fri Aug 2 17:37:53 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 2 Aug 2013 10:37:53 -0700 Subject: [OmniOS-discuss] Hello world, and some newcomers questions :) In-Reply-To: <51FBDF7D.8080501@cos.ru> References: <51FBC5B9.9090804@cos.ru> <51FBDF7D.8080501@cos.ru> Message-ID: On Fri, Aug 2, 2013 at 9:34 AM, Jim Klimov wrote: > ... and a few more questions: > > 1) Does the stable or bloody ISO include relatively recent illumos > gate features such as LZ4 compression for rpools? I am a bit alarmed > that the filename timestamp for bloody involves February 2013, if I > read it right... ;) > Both stable and bloody support lz4 as the default and if you kayak install, it uses compression. > 2) Is there an out-of-the-box or via simple hackery a way to get > the omnios rootfs onto an existing rpool, perhaps one of an older > version (think SXCE/Sol10)? > I believe you can create a new BE anddo a fresh pkg image-create into it. > 3) Installing additional SVR4 packages in GZ and LZs should not be > a problem, right? :) > No problems there, I do that all the time. -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Sat Aug 3 06:55:12 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 03 Aug 2013 08:55:12 +0200 Subject: [OmniOS-discuss] [discuss] pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <20130730141043.GD631@joyent.com> References: <20130730141043.GD631@joyent.com> Message-ID: <51FCA950.2030608@cos.ru> Hello Jonathan, I was reading through your release notes (blog entry) here: http://www.perkin.org.uk/posts/whats-new-in-pkgsrc-2013Q2.html and stumbled upon this bit of information: : Also on OmniOS 'od' is located in a different location $ sed -i -e 's,/usr/bin/od,/usr/gnu/bin/od,' /opt/local/bin/startx I believe (didn't try yet) that /opt/local/bin/startx is a script; wouldn't it be more correct to have it handle setting the proper environment - either as force-prepending its PATH defaults (and not using full paths to commands), or perhaps defining variables for the full paths to binaries? A pattern like this might be used for this particular example: ### Note we don't preset an OD="" so that one can be inherited from env for F in /usr/bin/od /usr/gnu/bin/od ; do [ x"$OD" = x -a -x "$F" ] && OD="$F" ### An ultimate test might be to try executing the program with ### an expected zero-return, like "$OD < /dev/null > /dev/null" ### to filter out broken binaries, missing dynamic libs, etc. done ### The "which" command can be used to fall back to a binary in PATH: [ x"$OD" = x ] && F="`which od`" && case "$F" in /*) [ -x "$F" ] && OD="$F" ;; esac ### Sanity check: we have a good variable [ x"$OD" = x -o ! -x "$OD" ] && \ echo "ERROR: OD not available: '$OD'" >&2 && exit 1 This way you have your $OD set to a sane value, detection of which you might further enhance by logic that would consider per-distro nuances. And this would be a least-surprise (and less manual work) for users on various platforms. With some "eval" trickery this pattern can be turned into a shell function (Bourne sh, bash, maybe others) to set the provided shell variable to the matched one of provided binaries. HTH, //Jim Klimov From jimklimov at cos.ru Sat Aug 3 07:44:50 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 03 Aug 2013 09:44:50 +0200 Subject: [OmniOS-discuss] [discuss] pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <51FCB173.9050303@oracle.com> References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> <51FCB173.9050303@oracle.com> Message-ID: <51FCB4F2.2050208@cos.ru> On 2013-08-03 09:29, Alan Coopersmith wrote: > No, it would be more correct to have a real /usr/bin/od command on the > system, even if it's just a symlink to /usr/gnu/bin/od - not having one > is just asking to break scripts. I'm not going to take a long convoluted > hack upstream at X.Org to work around such incompatible behavior for one > distro. I see, thanks for the reply. Well, *if* OmniOS indeed (again, I did not yet try it in practice) does not provide even a symlink at "standard" Solarisish locations, especially if it provides the binary anyway, this is the problem to fix. However I might envision a scenario like what happened with OpenSolaris and later OI, where the GNU userland got priority in default PATHs and sometimes was incompatible with Solaris equivalent commands in either command-line switches or parsable output, which could make interactive admins' life uneasy and old scripts - broken. Though for exactly that particular case, just fixing the PATH back helped ;) in cases similar to that, the hacks like mine might help... //Jim From jimklimov at cos.ru Sat Aug 3 07:50:27 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 03 Aug 2013 09:50:27 +0200 Subject: [OmniOS-discuss] [discuss] pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <51FCB173.9050303@oracle.com> References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> <51FCB173.9050303@oracle.com> Message-ID: <51FCB643.3080004@cos.ru> On 2013-08-03 09:29, Alan Coopersmith wrote: > I'm not going to take a long convoluted > hack upstream at X.Org to work around such incompatible behavior for one > distro. PS: and it is not that long nor convoluted ;) Just a bit of experience from writing scripts that ought to work across many different platforms which themselves to not adhere to a single standard... Though X has more battle-scars in this area - and here I'd trust your expertise ;) From jperkin at joyent.com Sat Aug 3 07:54:53 2013 From: jperkin at joyent.com (Jonathan Perkin) Date: Sat, 3 Aug 2013 08:54:53 +0100 Subject: [OmniOS-discuss] [discuss] Re: pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> <51FCB173.9050303@oracle.com> <51FCB4F2.2050208@cos.ru> Message-ID: <20130803075453.GC33687@joyent.com> * On 2013-08-03 at 08:52 BST, Richard PALO wrote: > Even more strange, I seem to have /usr/bin/od on OmniOS! Maybe it's part of an additional package - I tested on a clean OmniOS "bloody", so if there are additional packages I need to install let me know and I'll add it to the instructions. Thanks, -- Jonathan Perkin - Joyent, Inc. - www.joyent.com From jesus at omniti.com Sat Aug 3 13:59:53 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Sat, 3 Aug 2013 06:59:53 -0700 Subject: [OmniOS-discuss] [discuss] Re: pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <20130803075453.GC33687@joyent.com> References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> <51FCB173.9050303@oracle.com> <51FCB4F2.2050208@cos.ru> <20130803075453.GC33687@joyent.com> Message-ID: /usr/bin/od is part of package: pkg:/developer/linker On Sat, Aug 3, 2013 at 12:54 AM, Jonathan Perkin wrote: > * On 2013-08-03 at 08:52 BST, Richard PALO wrote: > > > Even more strange, I seem to have /usr/bin/od on OmniOS! > > Maybe it's part of an additional package - I tested on a clean OmniOS > "bloody", so if there are additional packages I need to install let me > know and I'll add it to the instructions. > > Thanks, > > -- > Jonathan Perkin - Joyent, Inc. - www.joyent.com > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.vincent at smartjog.com Tue Aug 6 09:41:51 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Tue, 06 Aug 2013 11:41:51 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 Message-ID: <5200C4DF.6020805@smartjog.com> Hello, In a server with lots of disks, I kept getting FMA faults about excessive temperature (DISK-8000-12) of almost every drive and it triggered retirements each time. First I tried cooling down the JBOD but some drives were still beeing retired, and I ended up disabling the io-retire.so plugin of fmd (crappy workaround). Later I found that our Seagate ST4000NM0023 drives had a suprisingly low "trip temperature" and now I'm pretty sure that's the value interpreted by fmd. The trip is at 40?C and is indeed exceeded sometimes. So I have two questions : * how would you disable disk retirement in FMA without crippling modules ? or maybe enforce the temperature threshold ? * is anyone working with ST4000NM0023 or have a clue about fixing the trip temperature in smart ? Thanks a lot, -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From jimklimov at cos.ru Tue Aug 6 09:55:27 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Tue, 06 Aug 2013 11:55:27 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 In-Reply-To: <5200C4DF.6020805@smartjog.com> References: <5200C4DF.6020805@smartjog.com> Message-ID: <5200C80F.2030502@cos.ru> On 2013-08-06 11:41, Thibault VINCENT wrote: > * is anyone working with ST4000NM0023 or have a clue about fixing the > trip temperature in smart ? Unfortunately, I can't really help about the main subject; but I can speculate that the vendor knows about "fragility" of the disks against temperature and spec'ed their self-diags accordingly? After all, that is what they are for?.. That is, even if you find ways to override the failsafes, you might substantially reduce the lifetime of devices... Perhaps, investment in an air-conditioner would pay off better? ;) //Jim From franz.fabian at ipvs.uni-stuttgart.de Tue Aug 6 12:28:59 2013 From: franz.fabian at ipvs.uni-stuttgart.de (Franz Fabian) Date: Tue, 6 Aug 2013 12:28:59 +0000 (UTC) Subject: [OmniOS-discuss] Updated release: r151006l References: <7BA1151A-35C5-45D6-A3B0-5A13BC7A9C1D@omniti.com> Message-ID: Dale Ghent writes: > > > > This week brings a new release: r151006l?This release entails only a security fix for library/libxml2, updating its version from 2.9.0 to 2.9.1.? The /usr/include/amd64/libxml2/... subtree is missing in the updated package. pkg contents libxml2 | grep usr/include/amd64/libxml2 returns NOTHING and there is no longer an /usr/include/amd64/libxml2 directory but /usr/bin/amd64/xml2-config --cflags ==> -I/usr/include/amd64/libxml2 Which one is wrong: xml2-config or the package itself??? From thibault.vincent at smartjog.com Tue Aug 6 12:48:01 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Tue, 06 Aug 2013 14:48:01 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 In-Reply-To: <5200C80F.2030502@cos.ru> References: <5200C4DF.6020805@smartjog.com> <5200C80F.2030502@cos.ru> Message-ID: <5200F081.80807@smartjog.com> On 08/06/2013 11:55 AM, Jim Klimov wrote: > Unfortunately, I can't really help about the main subject; but I can > speculate that the vendor knows about "fragility" of the disks against > temperature and spec'ed their self-diags accordingly? After all, that > is what they are for?.. That's right but 40?C or lower is impossible to maintain for a busy disk in a dense environment and even with good A/C. That value can't be right, and looking at the specs from Seagate you'll see the disk operation can range from 5?C to 60?C. That makes sense because the trip temperature from other Seagate disk is actually 60?C not 40. This was also verified under Linux with no expander and other controller. So there's something wrong going on with the ST4000NM0023 (firmware 003) and I've opened a ticket at Seagate. I'll let you know when I have news. > That is, even if you find ways to override the failsafes, you might > substantially reduce the lifetime of devices... > Perhaps, investment in an air-conditioner would pay off better? ;) Well my lab room doesn't have datacenter class A/C but it's cold enough for any other system, and to get sick :) As I said the conditions are good and faults should not happen here. I invite you looking at the report from Google "Failure Trends in a Large Disk Drive Population" (2007) in which they found 45?C was the best compromise for lifetime. Going lower or higher will increase different kind of failure. Colder is not the best! Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From thibault.vincent at smartjog.com Tue Aug 6 12:54:11 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Tue, 06 Aug 2013 14:54:11 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 In-Reply-To: <8F773229-9B4A-4377-B100-E958729061DE@RichardElling.com> References: <5200C4DF.6020805@smartjog.com> <8F773229-9B4A-4377-B100-E958729061DE@RichardElling.com> Message-ID: <5200F1F3.6010905@smartjog.com> On 08/06/2013 02:22 PM, Richard Elling wrote: > Can you send the "fmdump -eV" output for one of these events? > It is possible to disable specific ereports, but as you are aware, the > documentation for FMA is limited. Of course, you'll find it bellow. I'd like to know more if you can point at some ways to disable reports. Nexenta has certified this disk model with the same firmware revision, so I wonder how they handle it, maybe with an other monitoring of theirs. ------------ Aug 06 2013 12:45:51.536140077 ereport.io.scsi.disk.over-temperature nvlist version: 0 class = ereport.io.scsi.disk.over-temperature version = 0x0 ena = 0x8a33541069e03401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = hc hc-root = serial = Z1Z0EADC000093332198 part = SEAGATE-ST4000NM0023 revision = 0003 authority = (embedded nvlist) nvlist version: 0 product-id = LSI-SAS2X36 server-id = chassis-id = 50030480004de83f (end authority) hc-list = (array of embedded nvlists) (start hc-list[0]) nvlist version: 0 hc-name = ses-enclosure hc-id = 1 (end hc-list[0]) (start hc-list[1]) nvlist version: 0 hc-name = bay hc-id = 8 (end hc-list[1]) (start hc-list[2]) nvlist version: 0 hc-name = disk hc-id = 0 (end hc-list[2]) (end detector) current-temperature = 0x29 threshold-temperature = 0x28 __ttl = 0x1 __tod = 0x5200efff 0x1ff4d92d -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From skiselkov.ml at gmail.com Tue Aug 6 12:57:48 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Tue, 06 Aug 2013 13:57:48 +0100 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 In-Reply-To: <5200F081.80807@smartjog.com> References: <5200C4DF.6020805@smartjog.com> <5200C80F.2030502@cos.ru> <5200F081.80807@smartjog.com> Message-ID: <5200F2CC.8070705@gmail.com> On 8/6/13 1:48 PM, Thibault VINCENT wrote: > I invite you looking at the report from Google "Failure Trends in a > Large Disk Drive Population" (2007) in which they found 45?C was the > best compromise for lifetime. Going lower or higher will increase > different kind of failure. Colder is not the best! While Google's paper on this is highly informative, keep also in mind that it was written with 2007 and earlier technology in mind. 2007 was when the maximum capacity HDDs had was 1TB with consumer-grade reliability. The drive we're talking about here has four times the data density and presumably enterprise-grade reliability. Mechanically it's still fairly similar, but the nanotechnology that goes into making the patters and tracks can be very, very different. Anyways, hope you get through to Seagate and find out what's going on. Cheers, -- Saso From alan.coopersmith at oracle.com Sat Aug 3 07:29:55 2013 From: alan.coopersmith at oracle.com (Alan Coopersmith) Date: Sat, 03 Aug 2013 00:29:55 -0700 Subject: [OmniOS-discuss] [discuss] pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <51FCA950.2030608@cos.ru> References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> Message-ID: <51FCB173.9050303@oracle.com> On 08/ 2/13 11:55 PM, Jim Klimov wrote: > Hello Jonathan, > > I was reading through your release notes (blog entry) here: > http://www.perkin.org.uk/posts/whats-new-in-pkgsrc-2013Q2.html > and stumbled upon this bit of information: > > : Also on OmniOS 'od' is located in a different location > $ sed -i -e 's,/usr/bin/od,/usr/gnu/bin/od,' /opt/local/bin/startx > > I believe (didn't try yet) that /opt/local/bin/startx is a script; It is. > wouldn't it be more correct to have it handle setting the proper > environment - either as force-prepending its PATH defaults (and > not using full paths to commands), or perhaps defining variables > for the full paths to binaries? No, it would be more correct to have a real /usr/bin/od command on the system, even if it's just a symlink to /usr/gnu/bin/od - not having one is just asking to break scripts. I'm not going to take a long convoluted hack upstream at X.Org to work around such incompatible behavior for one distro. On the other hand, just before seeing this mail I'd sent a patch off for code review upstream at X.Org that will prefer the openssl command over /bin/od /dev/urandom... on Solarish platforms anyway: http://patchwork.freedesktop.org/patch/14235/ so the problem may get solved as a side effect of that. -- -Alan Coopersmith- alan.coopersmith at oracle.com Oracle Solaris Engineering - http://blogs.oracle.com/alanc From Joerg.Schilling at fokus.fraunhofer.de Sat Aug 3 10:08:08 2013 From: Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) Date: Sat, 3 Aug 2013 12:08:08 +0200 Subject: [OmniOS-discuss] [discuss] pkgsrc-2013Q2 binary packages for illumos now available In-Reply-To: <51FCB173.9050303@oracle.com> References: <20130730141043.GD631@joyent.com> <51FCA950.2030608@cos.ru> <51FCB173.9050303@oracle.com> Message-ID: <51fcd688.9qAVNT3kfgYR/QPR%Joerg.Schilling@fokus.fraunhofer.de> Alan Coopersmith wrote: > On 08/ 2/13 11:55 PM, Jim Klimov wrote: > > : Also on OmniOS 'od' is located in a different location > > $ sed -i -e 's,/usr/bin/od,/usr/gnu/bin/od,' /opt/local/bin/startx > > > > I believe (didn't try yet) that /opt/local/bin/startx is a script; > > It is. > > > wouldn't it be more correct to have it handle setting the proper > > environment - either as force-prepending its PATH defaults (and > > not using full paths to commands), or perhaps defining variables > > for the full paths to binaries? > > No, it would be more correct to have a real /usr/bin/od command on the > system, even if it's just a symlink to /usr/gnu/bin/od - not having one > is just asking to break scripts. I'm not going to take a long convoluted > hack upstream at X.Org to work around such incompatible behavior for one > distro. I don't understand why where is a problem. There is a POSIXly correct, fast and SunOS compatible OSS "od" implementation in SchilliX-ON since 31 months. The visible difference to the closed source SunOS version is that it is much faster, that it does not implement the obvious bugs from the SunOS implementation and that /usr/xpg4/bin/od is a hard link to /usr/bin/od as it implements the XPG4 differences at runtime. I can understand that people don't like to include the bugy od from Illumos, but there is a real OSS alternative licensed under the CDDL. Let me repeat my proposal from Thursday: It seems that we need to set up some kind of register about who is working on what for OpenSolaris in order to work against the negative effects of the fragmentation of the OpenSolaris development. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From thibault.vincent at smartjog.com Mon Aug 5 12:57:58 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Mon, 05 Aug 2013 14:57:58 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 Message-ID: <51FFA156.1040604@smartjog.com> Hello, In a server with lots of disks, I kept getting FMA faults about excessive temperature (DISK-8000-12) of almost every drive and it triggered retirements each time. First I tried cooling down the JBOD but some drives were still beeing retired, and I ended up disabling the io-retire.so plugin of fmd (crappy workaround). Later I found that our Seagate ST4000NM0023 drives had a suprisingly low "trip temperature" and now I'm pretty sure that's the value interpreted by fmd. The trip is at 40?C and is indeed exceeded sometimes. So I have two questions : * how would you disable disk retirement in FMA without crippling modules ? or maybe enforce the temperature threshold ? * is anyone working with ST4000NM0023 or have a clue about fixing the trip temperature in smart ? Thanks a lot, -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From daleg at omniti.com Tue Aug 6 14:56:50 2013 From: daleg at omniti.com (Dale Ghent) Date: Tue, 6 Aug 2013 10:56:50 -0400 Subject: [OmniOS-discuss] Updated release: r151006l In-Reply-To: References: <7BA1151A-35C5-45D6-A3B0-5A13BC7A9C1D@omniti.com> Message-ID: On Aug 6, 2013, at 8:28 AM, Franz Fabian wrote: > Dale Ghent writes: >> >> >> >> This week brings a new release: r151006l This release entails only a > security fix for library/libxml2, updating its version from 2.9.0 to 2.9.1. > > The /usr/include/amd64/libxml2/... subtree is missing in the updated package. > > pkg contents libxml2 | grep usr/include/amd64/libxml2 returns NOTHING > and there is no longer an /usr/include/amd64/libxml2 directory > > but > /usr/bin/amd64/xml2-config --cflags ==> -I/usr/include/amd64/libxml2 > > Which one is wrong: xml2-config or the package itself??? xml2-config is wrong. I'll open a ticket and get a fixed version out this week. /dale -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tobi at oetiker.ch Wed Aug 7 15:43:19 2013 From: tobi at oetiker.ch (Tobias Oetiker) Date: Wed, 7 Aug 2013 17:43:19 +0200 (CEST) Subject: [OmniOS-discuss] /usr/bin/chmod and the set-gid-bit Message-ID: Hi, /usr/bin/chmod seems to ignore the set-group-id bit when called with an octal permission string. try this on a zfs filesystem export PATH=/usr/bin umask 022 mkdir gd ls -ld gd chmod 2770 gd ls -ld gd chmod g+s gd ls -ld gd chmod 0770 gd ls -ld gd running this with truss reveals that I should call chmod 042770 gd instead, and indeed it works I just don't know why ... the man page does not provide any insight into this behaviour ... ah and btw, /usr/gnu/bin/chmod work the way legacy unix tools do :-) can someone please enlighten me ? cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900 From tim at multitalents.net Wed Aug 7 17:14:03 2013 From: tim at multitalents.net (Tim Rice) Date: Wed, 7 Aug 2013 10:14:03 -0700 (PDT) Subject: [OmniOS-discuss] /usr/bin/chmod and the set-gid-bit In-Reply-To: References: Message-ID: On Wed, 7 Aug 2013, Tobias Oetiker wrote: > Hi, > > /usr/bin/chmod seems to ignore the set-group-id bit when called > with an octal permission string. try this on a zfs filesystem That behavior seems to be intentional for directories and is documented in the man page. It has been that way a long time. Probably since the SVR4.0 days. > > export PATH=/usr/bin > umask 022 > mkdir gd > ls -ld gd > chmod 2770 gd > ls -ld gd > chmod g+s gd > ls -ld gd > chmod 0770 gd > ls -ld gd > > running this with truss reveals that I should call > > chmod 042770 gd > > instead, and indeed it works I just don't know why ... the man page > does not provide any insight into this behaviour ... > > ah and btw, /usr/gnu/bin/chmod work the way legacy unix tools do :-) > > can someone please enlighten me ? > > cheers > tobi > > -- Tim Rice Multitalents tim at multitalents.net From lists at mcintyreweb.com Wed Aug 7 23:30:09 2013 From: lists at mcintyreweb.com (Hugh McIntyre) Date: Wed, 07 Aug 2013 16:30:09 -0700 Subject: [OmniOS-discuss] /usr/bin/chmod and the set-gid-bit In-Reply-To: References: Message-ID: <5202D881.8060904@mcintyreweb.com> On 8/7/13 10:14 AM, Tim Rice wrote: > On Wed, 7 Aug 2013, Tobias Oetiker wrote: > >> Hi, >> >> /usr/bin/chmod seems to ignore the set-group-id bit when called >> with an octal permission string. try this on a zfs filesystem > > That behavior seems to be intentional for directories and is documented > in the man page. > It has been that way a long time. Probably since the SVR4.0 days. I always this assumed this was because of the typical usage on directories of setting the SGID bit at the top level of a directory and expecting the bit to then be set (and stay set) on all subsequently created directories all the way down. Including in the presence of older software that just said "chmod 755". FWIW, Linux (e.g. Ubuntu 12.04) also does not allow you to clear the bit with octal mode, although they do allow you to set the SGID bit this way. Arguably this is a better solution, but whether people will want to change the Illumos chmod at this point is debatable. From Linux: chmod preserves a directory's set-user-ID and set-group-ID bits unless you explicitly specify otherwise. You can set or clear the bits with symbolic modes like u+s and g-s, and you can set (but not clear) the bits with a numeric mode. Hugh. From alexander.r.eremin at gmail.com Sat Aug 10 12:56:17 2013 From: alexander.r.eremin at gmail.com (Alexander Eremin) Date: Sat, 10 Aug 2013 16:56:17 +0400 Subject: [OmniOS-discuss] plumb ifaces automatically after install Message-ID: Hi all just installed bloody on VBox and has only loopback iface (cause sysidtool-net script is missed) If sysidtool service was not added by some reason, may be it's functionality must be added to some own omnios smf script: whether there are here any plans? Thanks, Alexander From jesus at omniti.com Sat Aug 10 13:29:23 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Sat, 10 Aug 2013 06:29:23 -0700 Subject: [OmniOS-discuss] plumb ifaces automatically after install In-Reply-To: References: Message-ID: In OmniOS, we opted for a single way to add an interface instead of the "one way during install and another way generally." http://omnios.omniti.com/wiki.php/GeneralAdministration If you are using Kayak to install, you can make this happen post-boot via the initial-boot SMF bits. On Sat, Aug 10, 2013 at 5:56 AM, Alexander Eremin < alexander.r.eremin at gmail.com> wrote: > Hi all > just installed bloody on VBox and has only loopback iface (cause > sysidtool-net script is missed) > If sysidtool service was not added by some reason, may be it's > functionality must be added to some own omnios smf script: whether there > are here any plans? > > Thanks, > Alexander > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.r.eremin at gmail.com Sat Aug 10 13:49:06 2013 From: alexander.r.eremin at gmail.com (Alexander Eremin) Date: Sat, 10 Aug 2013 17:49:06 +0400 Subject: [OmniOS-discuss] plumb ifaces automatically after install In-Reply-To: References: Message-ID: Well, I'm happy with manual configuration (like on Slackware many years ago) :). May be then go further and minimize list of running smf services, like netcfg, dns/client, ...? Alexander On 10.08.2013, at 17:29, Theo Schlossnagle wrote: > In OmniOS, we opted for a single way to add an interface instead of the "one way during install and another way generally." > > http://omnios.omniti.com/wiki.php/GeneralAdministration > > If you are using Kayak to install, you can make this happen post-boot via the initial-boot SMF bits. > > > On Sat, Aug 10, 2013 at 5:56 AM, Alexander Eremin wrote: >> Hi all >> just installed bloody on VBox and has only loopback iface (cause sysidtool-net script is missed) >> If sysidtool service was not added by some reason, may be it's functionality must be added to some own omnios smf script: whether there are here any plans? >> >> Thanks, >> Alexander >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > -- > Theo Schlossnagle > > http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From cardenas12 at gmail.com Sat Aug 10 16:37:12 2013 From: cardenas12 at gmail.com (Carlos Cardenas) Date: Sat, 10 Aug 2013 09:37:12 -0700 Subject: [OmniOS-discuss] Switching syslog with rsyslog In-Reply-To: <762596BC2411406D908FEE8E1C83A920@gmail.com> References: <12FA49CA183D4402B05F59DC62C29A53@gmail.com> <20130716030524.GU98158@eschaton.local> <51e4bf86.8358440a.1473.385f@mx.google.com> <250C5563A9C74B918C8B152090616C2A@gmail.com> <982CC057AC1542D2B195B5658FB32D47@gmail.com> <762596BC2411406D908FEE8E1C83A920@gmail.com> Message-ID: <299BF76FCC4B4F86A376019C5306E6E6@gmail.com> ping. -- Carlos On Sunday, July 28, 2013 at 12:31 PM, Carlos Cardenas wrote: > I've resolved these compile time issues. All pkgs are 32 and 64 bit in one wad. > > https://github.com/omniti-labs/omnios-build/pull/28 > > -- > Carlos > > > On Saturday, July 20, 2013 at 5:07 PM, Carlos Cardenas wrote: > > > Ok? > > > > I have rsyslog 7.4.3 building/running with Guardtime enabled. As a result of jumping from 7.3.9 to 7.4.3, I had to add several new pkgs (deps): > > * libguardtime (Guardtime C SDK) > > * guardtime (Guardtime Client utility) > > * python26-docutils (build dep, generate man pages) > > > > I've also added rsyslog diag and user tools to allow for the additional functionality of using Guardtime within rsyslog. > > > > The one caveat I have with the latest rsyslog with GT support is that there is bug with the 64bit version of libguardtime linking with curl so rsyslog is 32bit. I'm opening a bug report with GT on this and hope to have this resolved soon. > > > > That said, is 32bit fine or do y'all want to wait until we have 64bit as well? > > > > How do y'all want the PRs done: all in one or one PR per pkg? > > > > Thanks in advance. > > > > -- > > Carlos > > > > > > On Tuesday, July 16, 2013 at 11:12 AM, Carlos Cardenas wrote: > > > > > I'll send an PR later this week{end}. > > > > > > -- > > > Carlos > > > > > > > > > On Tuesday, July 16, 2013 at 6:35 AM, Eric Sproul wrote: > > > > > > > > > > > > > > > > > > > On Mon, Jul 15, 2013 at 11:35 PM, wrote: > > > > > That's awesome Chris, where can I find the package source so I can update to the latest stable version? GitHub or somewhere else? > > > > > > > > The build scripts for ms.omniti.com (http://ms.omniti.com) packages are in the omniti-ms branch of omnios-build: https://github.com/omniti-labs/omnios-build/tree/omniti-ms > > > > _______________________________________________ > > > > OmniOS-discuss mailing list > > > > OmniOS-discuss at lists.omniti.com (mailto:OmniOS-discuss at lists.omniti.com) > > > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aney at ivision.fr Sun Aug 11 14:08:34 2013 From: aney at ivision.fr (Alexandre NEY) Date: Sun, 11 Aug 2013 14:08:34 +0000 Subject: [OmniOS-discuss] HA for ZFS on OmniOS Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> Hello, I'm quite new to Solaris / Illumnos / OmniOS. We chose Omnios because it seemed like the best choice for our ZFS project and is recommended by "Napp-it" Anyway, we are looking into putting some failover capabilities in our ZFS architecture. We are aiming at 2 heads connected by SAS cables to the disk enclosure. The LUNs are presented with COMSTAR iSCSI. >From what I understand there is no built-in failover function in ZFS, we have to rely on additional software like RSF for example (http://www.high-availability.com/zfs-ha-plugin/) I guess my question is : anyone else already running this configuration in a production environment ? How is it performing ? Any recommendation ? I saw this specific bug on the illumnos tracker : https://www.illumos.org/issues/3621 it seems fixed but i don't know how to check if the fix is included in omnios, I did look in the release notes but didn't find anything about it. I know it's probably the holidays for most of you guys but any feedback will be much appreciated ! Thanks ! Alexandre NEY From jesus at omniti.com Sun Aug 11 14:30:05 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Sun, 11 Aug 2013 07:30:05 -0700 Subject: [OmniOS-discuss] HA for ZFS on OmniOS In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> Message-ID: That change is in bloody, but not in 151006. On Sun, Aug 11, 2013 at 7:08 AM, Alexandre NEY wrote: > Hello, > > I'm quite new to Solaris / Illumnos / OmniOS. We chose Omnios because it > seemed like the best choice for our ZFS project and is recommended by > "Napp-it" > > Anyway, we are looking into putting some failover capabilities in our ZFS > architecture. We are aiming at 2 heads connected by SAS cables to the disk > enclosure. The LUNs are presented with COMSTAR iSCSI. > > From what I understand there is no built-in failover function in ZFS, we > have to rely on additional software like RSF for example ( > http://www.high-availability.com/zfs-ha-plugin/) > > I guess my question is : anyone else already running this configuration in > a production environment ? How is it performing ? Any recommendation ? > > I saw this specific bug on the illumnos tracker : > https://www.illumos.org/issues/3621 it seems fixed but i don't know how > to check if the fix is included in omnios, I did look in the release notes > but didn't find anything about it. > > I know it's probably the holidays for most of you guys but any feedback > will be much appreciated ! > > Thanks ! > > Alexandre NEY > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From aney at ivision.fr Sun Aug 11 14:39:33 2013 From: aney at ivision.fr (Alexandre NEY) Date: Sun, 11 Aug 2013 14:39:33 +0000 Subject: [OmniOS-discuss] HA for ZFS on OmniOS In-Reply-To: References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> , Message-ID: <80ll2tx6oweif0108eu1m3fh.1376231959411@email.android.com> Thanks for the quick feedback! I guess I just need to wait for the next stable then. Do you have any feedback using RSF? thanks ! George-Cristian B?rzan a ?crit : https://github.com/omniti-labs/illumos-omnios/commits/master/usr/src/uts/common/io/comstar There we go. Also, my heart goes out to you, for hitting that bug. On Sun, Aug 11, 2013 at 5:18 PM, George-Cristian B?rzan > wrote: I was looking into that, it's in bloody, afair, but not in stable. On Sun, Aug 11, 2013 at 5:08 PM, Alexandre NEY > wrote: Hello, I'm quite new to Solaris / Illumnos / OmniOS. We chose Omnios because it seemed like the best choice for our ZFS project and is recommended by "Napp-it" Anyway, we are looking into putting some failover capabilities in our ZFS architecture. We are aiming at 2 heads connected by SAS cables to the disk enclosure. The LUNs are presented with COMSTAR iSCSI. >From what I understand there is no built-in failover function in ZFS, we have to rely on additional software like RSF for example (http://www.high-availability.com/zfs-ha-plugin/) I guess my question is : anyone else already running this configuration in a production environment ? How is it performing ? Any recommendation ? I saw this specific bug on the illumnos tracker : https://www.illumos.org/issues/3621 it seems fixed but i don't know how to check if the fix is included in omnios, I did look in the release notes but didn't find anything about it. I know it's probably the holidays for most of you guys but any feedback will be much appreciated ! Thanks ! Alexandre NEY _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- George-Cristian B?rzan -- George-Cristian B?rzan -------------- next part -------------- An HTML attachment was scrubbed... URL: From skiselkov.ml at gmail.com Sun Aug 11 14:42:40 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sun, 11 Aug 2013 15:42:40 +0100 Subject: [OmniOS-discuss] HA for ZFS on OmniOS In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> Message-ID: <5207A2E0.30209@gmail.com> On 8/11/13 3:08 PM, Alexandre NEY wrote: > Hello, > > I'm quite new to Solaris / Illumnos / OmniOS. We chose Omnios because it seemed like the best choice for our ZFS project and is recommended by "Napp-it" > > Anyway, we are looking into putting some failover capabilities in our ZFS architecture. We are aiming at 2 heads connected by SAS cables to the disk enclosure. The LUNs are presented with COMSTAR iSCSI. > > From what I understand there is no built-in failover function in ZFS, we have to rely on additional software like RSF for example (http://www.high-availability.com/zfs-ha-plugin/) > > I guess my question is : anyone else already running this configuration in a production environment ? How is it performing ? Any recommendation ? Hi Alexandre, I wrote about this specific use case on my blog recently: http://zfs-create.blogspot.com/ I really need to pull myself together for the part 2 explaining iSCSI failover, but part 1 which is already up should help you a long way to get a clustered omnios installation up and running and sharing NFS. Cheers, -- Saso From aney at ivision.fr Sun Aug 11 16:01:47 2013 From: aney at ivision.fr (Alexandre NEY) Date: Sun, 11 Aug 2013 16:01:47 +0000 Subject: [OmniOS-discuss] HA for ZFS on OmniOS In-Reply-To: <80ll2tx6oweif0108eu1m3fh.1376231959411@email.android.com> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> , <80ll2tx6oweif0108eu1m3fh.1376231959411@email.android.com> Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310DCB@ivision-mailbox.Exchange.local> Thanks for this great article it was very instructive ! Waiting for part 2 :) Unfortunately, since we have little experience / time for Solaris I think we might go with the $$$ option of RSF-1 if it's worth it and takes care of all the failover problems we could encounter. We also use iSCSI. What do you think ? Thanks ! Alex ---- Hi Alexandre, I wrote about this specific use case on my blog recently: http://zfs-create.blogspot.com/ I really need to pull myself together for the part 2 explaining iSCSI failover, but part 1 which is already up should help you a long way to get a clustered omnios installation up and running and sharing NFS. Cheers, -- Saso From skiselkov.ml at gmail.com Sun Aug 11 16:28:47 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sun, 11 Aug 2013 17:28:47 +0100 Subject: [OmniOS-discuss] HA for ZFS on OmniOS In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310DCB@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310B0A@ivision-mailbox.Exchange.local> , <80ll2tx6oweif0108eu1m3fh.1376231959411@email.android.com> <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310DCB@ivision-mailbox.Exchange.local> Message-ID: <5207BBBF.8080001@gmail.com> On 8/11/13 5:01 PM, Alexandre NEY wrote: > Unfortunately, since we have little experience / time for Solaris I think we > might go with the $$$ option of RSF-1 if it's worth it and takes care of all > the failover problems we could encounter. We also use iSCSI. What do you think? Sure, if you can afford it, go for it, although I'm unsure why you're going with a DIY storage solution if you don't want to get your hands dirty in Solaris/Illumos stuff. If you just want a minimum-hassle working ZFS-based iSCSI storage solution with a pretty GUI, go for something like Nexenta. They already integrate RSF-1 into their clustered solution and you'll get support for the whole bundle. Going with OmniOS makes sense if you're trying to do something out of the ordinary or on a budget or you just like tinkering with stuff. Cheers, -- Saso From aney at ivision.fr Sun Aug 11 16:52:49 2013 From: aney at ivision.fr (Alexandre NEY) Date: Sun, 11 Aug 2013 16:52:49 +0000 Subject: [OmniOS-discuss] [!! SPAM] Re: HA for ZFS on OmniOS Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310F0C@ivision-mailbox.Exchange.local> I guess we are between the DIY and full hassle free solution ;) We are a Linux / FreeBSD shop so some concepts are familiar and some others..well.. it's different. The truth is we tried Nexenta but their licensing costs just skyrocket when you go up in the TBs forcing you to go all platinum / diamond and being a service provider that didn't quite fit our business model. In any case, I'll give a shot at the open source solution in our lab. Alex -----Message d'origine----- De?: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] De la part de Saso Kiselkov Envoy??: dimanche 11 ao?t 2013 18:29 ??: omnios-discuss at lists.omniti.com Objet?: [!! SPAM] Re: [OmniOS-discuss] HA for ZFS on OmniOS On 8/11/13 5:01 PM, Alexandre NEY wrote: > Unfortunately, since we have little experience / time for Solaris I > think we might go with the $$$ option of RSF-1 if it's worth it and > takes care of all the failover problems we could encounter. We also use iSCSI. What do you think? Sure, if you can afford it, go for it, although I'm unsure why you're going with a DIY storage solution if you don't want to get your hands dirty in Solaris/Illumos stuff. If you just want a minimum-hassle working ZFS-based iSCSI storage solution with a pretty GUI, go for something like Nexenta. They already integrate RSF-1 into their clustered solution and you'll get support for the whole bundle. Going with OmniOS makes sense if you're trying to do something out of the ordinary or on a budget or you just like tinkering with stuff. Cheers, -- Saso _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From skiselkov.ml at gmail.com Sun Aug 11 17:18:48 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sun, 11 Aug 2013 18:18:48 +0100 Subject: [OmniOS-discuss] [!! SPAM] Re: HA for ZFS on OmniOS In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310F0C@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD4310310F0C@ivision-mailbox.Exchange.local> Message-ID: <5207C778.9080705@gmail.com> On 8/11/13 5:52 PM, Alexandre NEY wrote: > I guess we are between the DIY and full hassle free solution ;) We are a Linux / FreeBSD shop so some concepts are familiar and some others..well.. it's different. > > The truth is we tried Nexenta but their licensing costs just skyrocket when you go up in the TBs forcing you to go all platinum / diamond and being a service provider that didn't quite fit our business model. > > In any case, I'll give a shot at the open source solution in our lab. Ah, I see. Well, if you guys have Linux/*BSD experience, picking up OmniOS won't be much work. 95% of the basic tools work the same, and there really is only a handful of concepts that differ. From an ops perspective, the largest difference is SMF, the init replacement. It can seem daunting at first, but the general idea is fairly simple and there are pre-built manifests and method scripts available on the net, so most of the time I myself just grab something that's done and adapt. Cheers, -- Saso From steve at linuxsuite.org Tue Aug 13 15:20:22 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Tue, 13 Aug 2013 11:20:22 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? Message-ID: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Howdy! This is a SuperMicro JBOD with SATA disks. I am aware of the issues of having SATA on SAS, but was wondering just how serious these kinds of errors are.. a scrub of the pool completes without noticable problems.. I did a lot of stress testing earlier and could not get a failure. Disabling NCQ on the controller was a neccessary. What is the practical risk to data?? See below info for iostat / syslog thanx - steve syslog info kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31120303 kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31120436 kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31120303 kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): Blah Blah... kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31120436 kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for target 13. kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for target 13. kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): Output of iostat -En Looks like "Hard Errors" and "No Device" correspond. What does "Transport Error" and "Recoverable" mean. I see no evidence of data corruption/loss, does ZFS deal/recover from these errors in a good/safe way? c5t5000C500489947A8d0 Soft Errors: 0 Hard Errors: 2 Transport Errors: 11 Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F0AAMA Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 2 Recoverable: 0 Illegal Request: 2 Predictive Failure Analysis: 0 c5t5000C500525EB2B9d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 46 Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F0QM5H Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 Illegal Request: 5 Predictive Failure Analysis: 0 c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F09G4Q Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 Illegal Request: 1 Predictive Failure Analysis: 0 From borising at gmail.com Wed Aug 14 08:50:30 2013 From: borising at gmail.com (Bo Agerskov Rising) Date: Wed, 14 Aug 2013 10:50:30 +0200 Subject: [OmniOS-discuss] Updated release: r151006l In-Reply-To: References: <7BA1151A-35C5-45D6-A3B0-5A13BC7A9C1D@omniti.com> Message-ID: Hi Dale, How far have you come with this? Regards, Bo On 06/08/2013, at 16.56, Dale Ghent wrote: > > On Aug 6, 2013, at 8:28 AM, Franz Fabian wrote: > >> Dale Ghent writes: >>> >>> >>> >>> This week brings a new release: r151006l This release entails only a >> security fix for library/libxml2, updating its version from 2.9.0 to 2.9.1. >> >> The /usr/include/amd64/libxml2/... subtree is missing in the updated package. >> >> pkg contents libxml2 | grep usr/include/amd64/libxml2 returns NOTHING >> and there is no longer an /usr/include/amd64/libxml2 directory >> >> but >> /usr/bin/amd64/xml2-config --cflags ==> -I/usr/include/amd64/libxml2 >> >> Which one is wrong: xml2-config or the package itself??? > > xml2-config is wrong. I'll open a ticket and get a fixed version out this week. > > /dale > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From thibault.vincent at smartjog.com Wed Aug 14 11:44:33 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Wed, 14 Aug 2013 13:44:33 +0200 Subject: [OmniOS-discuss] Overheating faults with ST4000NM0023 In-Reply-To: <5200F2CC.8070705@gmail.com> References: <5200C4DF.6020805@smartjog.com> <5200C80F.2030502@cos.ru> <5200F081.80807@smartjog.com> <5200F2CC.8070705@gmail.com> Message-ID: <520B6DA1.9000203@smartjog.com> On 08/06/2013 02:57 PM, Saso Kiselkov wrote: > While Google's paper on this is highly informative, keep also in mind > that it was written with 2007 and earlier technology in mind. Thanks Saso, it's true it might have changed. MTBF of these modern disks is asserted for a case temperature of 40?C which might be the optimal from manufacturer standpoint, and it's not so close to Google's 45 or way off usual DC conditions. Anyway Seagate is currently working on this case and I'll keep the list informed for reference. Cheers, -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From thibault.vincent at smartjog.com Wed Aug 14 23:17:19 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Wed, 14 Aug 2013 23:17:19 -0000 (UTC) Subject: [OmniOS-discuss] NULL pointer in scsi_vhci triggered by mpathadm (vhci_mpapi_sync_lu_oid_list) In-Reply-To: <520B92A2.5040105@smartjog.com> Message-ID: <52a9011e-3178-49b5-84b7-9b1610fa23a9@zimbra.fr.smartjog.net> Sorry this is a cross post, initially sent to illumos developer ML as recommended on #omnios. But this may be good to the OmniOS audience as yet incomplete tests suggest there is no issue under SmartOS. ----------------- Hello, I'm building ZFS servers with multipathed SAS enclosures and experience a very reproducible BAD TRAP when running "mpathadm list lu" in some situations. You'll find an fmdump at the end of this message and a stack trace attached. The core is huge but if required I can set kmem_flags to any value and have it hosted on FTP. Hardware is: * Dell R720xd server with two CPUs and 192GB RAM (all firmwares up-to-date) * 2 * LSI 9207-8e (fw version 16, the latest AFAIK) * Supermicro SC847E26-RJBOD1 JBODs with LSI expanders * Seagate ST4000NM0023 drives and STEC SSDs It's running OmniOS, stable or up-to-date bloody : it crashed in both cases. "stmsboot -D mpt_sas -e" was used to activate mpiox. The NULL dereference happens immediately with "mpathadm list lu" at any time, just after booting or 2 days later. Because the system may run fine with multipathed drives as long as the command is not used. "stmsboot -L" even reports stuff with no problem. But the fault is closely related to the SAS topology : * one JBOD with 45 drives on a single HBA, single cable (no multipath) ==> no crash, "list lu" reports one path which is expected * same JBOD with a second cable, making a path to the same HBA or to an other identical LSI HBA ==> no crash, I see two path and mpathadm is functional * any of the two scenarii above but with a second JBOD daisy chained to the first, raising the topology to 90 drives ==> CRASH in vhci_mpapi_sync_lu_oid_list()/ddi_get_instance() * any scenario involving two JBOD and their 90 drives, such as one on each HBA with a single cable ==> same CRASH Also, it is very interesting that : * the problem was way more serious if using a Dell H310 or H710 controller [mr_sas] for system drives. Then only 6 disks in the SAS topology of LSI HBAs would be enough to trigger the fault. 5 would not... Since I've stopped using Dell controllers and it results in the cases above. * it doesn't crash on a Dell R510, all other peripherals being identical. Even under heavy ZFS load. * I tried removing one CPU, leaving only one RAM module, disabling hyperthreading, disabling NUMA, disabling virtual devices from Dell iDRACs, changing PCI-e slot. * forcing 4k blocks, f_sym module, or the load-balance algo in vhci is not responsible. I tested with or without these. * it happens with or without filesystem imported (even brand new drives). # fmdump -u 110c95e8-1906-cb20-c364-8c27099d8b3c -V TIME UUID SUNW-MSG-ID Aug 14 2013 13:01:41.614295000 110c95e8-1906-cb20-c364-8c27099d8b3c SUNOS-8000-KL TIME CLASS ENA Aug 14 13:01:41.0142 ireport.os.sunos.panic.dump_available 0x0000000000000000 Aug 14 13:01:14.8130 ireport.os.sunos.panic.dump_pending_on_device 0x0000000000000000 nvlist version: 0 version = 0x0 class = list.suspect uuid = 110c95e8-1906-cb20-c364-8c27099d8b3c code = SUNOS-8000-KL diag-time = 1376485301 39052 de = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = fmd authority = (embedded nvlist) nvlist version: 0 version = 0x0 product-id = PowerEdge-R720xd chassis-id = 9FSLFY1 server-id = san-1 (end authority) mod-name = software-diagnosis mod-version = 0.1 (end de) fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = defect.sunos.kernel.panic certainty = 0x64 asru = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = sw object = (embedded nvlist) nvlist version: 0 path = /var/crash/unknown/.110c95e8-1906-cb20-c364-8c27099d8b3c (end object) (end asru) resource = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = sw object = (embedded nvlist) nvlist version: 0 path = /var/crash/unknown/.110c95e8-1906-cb20-c364-8c27099d8b3c (end object) (end resource) savecore-succcess = 1 dump-dir = /var/crash/unknown dump-files = vmdump.14 os-instance-uuid = 110c95e8-1906-cb20-c364-8c27099d8b3c panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff00f50999a0 addr=2c occurred in module "genunix" due to a NULL pointer dereference panicstack = unix:die+df () | unix:trap+db3 () | unix:cmntrap+e6 () | genunix:ddi_get_instance+8 () | scsi_vhci:vhci_mpapi_sync_lu_oid_list+5d () | scsi_vhci:vhci_mpapi_ioctl+a8 () | scsi_vhci:vhci_mpapi_ctl+e3 () | scsi_vhci:vhci_ioctl+5d () | genunix:cdev_ioctl+39 () | specfs:spec_ioctl+60 () | genunix:fop_ioctl+55 () | genunix:ioctl+9b () | unix:brand_sys_sysenter+1c9 () | crashtime = 1376476491 panic-time = Wed Aug 14 10:34:51 2013 UTC (end fault-list[0]) fault-status = 0x1 severity = Major __ttl = 0x1 __tod = 0x520b7fb5 0x249d65d8 Any thoughts about this ? Thanks Best regards, -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company ------------------------------------------- illumos-developer Archives: https://www.listbox.com/member/archive/182179/=now RSS Feed: https://www.listbox.com/member/archive/rss/182179/24872454-6cc5e689 Modify Your Subscription: https://www.listbox.com/member/?member_id=24872454&id_secret=24872454-3a034122 Powered by Listbox: http://www.listbox.com -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fault.txt URL: From gate03 at landcroft.co.uk Thu Aug 15 06:55:43 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Thu, 15 Aug 2013 16:55:43 +1000 Subject: [OmniOS-discuss] filesystems received from send -R Message-ID: <20130815165543.1eb145e3@dickless.landy.net> I recently moved to a new machine: # ssh root at oldmachine 'zfs send -R rpool at M' | zfs recv rpool/bring the idea being: 1. zfs rename my own filesystems such as rpool/bring/home to rpool/home. 2. Copy settings from rpool/bring/omnios and below whilst setting up the new machine. 3. Refer to stuff under rpool/bring/zone/ whilst setting up zones. but curiously, it seems that any filesystem under rpool/bring/omnios or rpool/bring/zone would not honour the 'zfs set mountpoint' command; that is, the command was accepted without error, 'zfs get' shows the value that I set, but in fact the directory doesn't exist. It was necessary to copy the filesystem with send and recv to be able to view the contents. The mountpoint settings of those filesystem are special in some way. My own filesystems, those that I had created myself with zfs create, behaved as expected; i.e., could be 'zfs rename'-d or 'zfs set mountpoint'-ed. This is surely a bug n'est-ce pas ? Michael. From gate03 at landcroft.co.uk Thu Aug 15 07:01:15 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Thu, 15 Aug 2013 17:01:15 +1000 Subject: [OmniOS-discuss] performance penalty of a slow mirror Message-ID: <20130815170115.40e8a400@dickless.landy.net> Hello, as my current OmniOS server doesn't have removable disks, I've added an external USB2 HDD as a mirror, so that in the event of fire or holiday, the external disk can be taken away as a backup. How does OmniOS deal with a very slow mirror ? Does it perform any measurements to work out that it should always write to the other mirror first, or does it do a 'fair sharing' or writes ? This is a SuperMicro 5017C-LF (http://www.supermicro.com/products/system/1u/5017/sys-5017c-lf.cfm ) with 2 x 500 GiB 2.5" drives mirrored internally, plus the external mirror. The machine has only USB-2 sockets, not USB-3. Michael. From tobi at oetiker.ch Thu Aug 15 07:07:32 2013 From: tobi at oetiker.ch (Tobias Oetiker) Date: Thu, 15 Aug 2013 09:07:32 +0200 (CEST) Subject: [OmniOS-discuss] performance penalty of a slow mirror In-Reply-To: <20130815170115.40e8a400@dickless.landy.net> References: <20130815170115.40e8a400@dickless.landy.net> Message-ID: Hi Michael, Today Michael Mounteney wrote: > Hello, as my current OmniOS server doesn't have removable disks, I've > added an external USB2 HDD as a mirror, so that in the event of fire or > holiday, the external disk can be taken away as a backup. > > How does OmniOS deal with a very slow mirror ? Does it perform any > measurements to work out that it should always write to the other > mirror first, or does it do a 'fair sharing' or writes ? > > This is a SuperMicro 5017C-LF > (http://www.supermicro.com/products/system/1u/5017/sys-5017c-lf.cfm ) > with 2 x 500 GiB 2.5" drives mirrored internally, plus the external > mirror. The machine has only USB-2 sockets, not USB-3. wouldn't you rather want to use zfs snapshots with send/recv to keep a copy on your external drive ? cheers tobi > > Michael. > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900 From daleg at omniti.com Thu Aug 15 19:13:54 2013 From: daleg at omniti.com (Dale Ghent) Date: Thu, 15 Aug 2013 15:13:54 -0400 Subject: [OmniOS-discuss] Updated release: r151006p Message-ID: <0F49BA4A-6B96-48A6-ABC5-F865330B2873@omniti.com> This week brings a new release: r151006p This release includes a security fix for for the kernel in the zfs driver, as well as other bug fixes and minor improvements. Full release notes are available here: http://omnios.omniti.com/wiki.php/ReleaseNotes#r151006p As this is an update which addresses security issues, fresh r151006p install media has also been generated: http://omnios.omniti.com/wiki.php/Installation -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From valrhona at gmail.com Thu Aug 15 19:44:45 2013 From: valrhona at gmail.com (Valrhona) Date: Thu, 15 Aug 2013 15:44:45 -0400 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees Message-ID: Thanks to OmniTI for making a fantastic product for the community! I am doing a bunch of backups, and trying to organize data, and have two questions: 1. Is there a better alternative, perhaps in the new package repositories, for gzip-style compression that is multithreaded? I am doing the usualy zfs send to a file, which I then backup to tape. Using gzip makes the process like 100x slower than if I just dump the zfs stream to an uncompressed file, and so it's not practical from a time standpoint. 2. This is a more general UNIX question: I have a lot of directories with mixed files, but I want to extract all of the files with a certain extension, say .xyz, and move only those files to a different zfs filesystem. In the destination, I want to recreate the directory structure of the original tree, but only have the .xyz files in them (these are large, uncompressed raw data). So the source and destination should have the same directory structure; the source would have none of the .xyz files, and the destination would have all of the .xyz files. Is there a simple way to do this with mv, or is another command recommended? I am sure this is obvious to many of the unix gurus around, so any help would be appreciated. Thanks! Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From narayan.desai at gmail.com Fri Aug 16 01:25:08 2013 From: narayan.desai at gmail.com (Narayan Desai) Date: Thu, 15 Aug 2013 20:25:08 -0500 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: We're seeing something similar on the same gear (LSI/supermicro expanders, lsi controllers, sata drives). We've tried standard hardware debugging (cable reseat/replacement, etc) and the problem in our case seems to follow the sas expander backplane. We did a disk by disk migration into a different expander and they stopped. How high are your error counts? (in our case, we were getting about 1500/day/device). Is your performance impacted? (it was in our case) -nld On Tue, Aug 13, 2013 at 10:20 AM, wrote: > > Howdy! > > This is a SuperMicro JBOD with SATA disks. I am aware of the > issues of having > SATA on SAS, but was wondering just how serious these kinds of errors > are.. a scrub of the pool > completes without noticable problems.. I did a lot of stress testing > earlier and could > not get a failure. Disabling NCQ on the controller was a neccessary. > What is the practical risk to data?? > > See below info for iostat / syslog > > thanx - steve > > syslog info > > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: > IOCStatus=0x8000, IOCLogInfo=0x31120303 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: > IOCStatus=0x8000, IOCLogInfo=0x31120436 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: > IOCStatus=0x8000, IOCLogInfo=0x31120303 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > > Blah Blah... > > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: > IOCStatus=0x8000, IOCLogInfo=0x31120436 > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for > target 13. > kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for > target 13. > kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > > Output of iostat -En > > Looks like "Hard Errors" and "No Device" correspond. What > does "Transport Error" and "Recoverable" mean. I see no evidence > of data corruption/loss, does ZFS deal/recover from these errors in a > good/safe > way? > > > c5t5000C500489947A8d0 Soft Errors: 0 Hard Errors: 2 Transport Errors: 11 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: > W1F0AAMA > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 2 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > > c5t5000C500525EB2B9d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 46 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: > W1F0QM5H > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 > Illegal Request: 5 Predictive Failure Analysis: 0 > > c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: > W1F09G4Q > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 > Illegal Request: 1 Predictive Failure Analysis: 0 > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Fri Aug 16 08:20:10 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 16 Aug 2013 10:20:10 +0200 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: <520DE0BA.2050108@cos.ru> On 2013-08-15 21:44, Valrhona wrote: > Thanks to OmniTI for making a fantastic product for the community! > > I am doing a bunch of backups, and trying to organize data, and have two > questions: > > 1. Is there a better alternative, perhaps in the new package > repositories, for gzip-style compression that is multithreaded? I am > doing the usualy zfs send to a file, which I then backup to tape. Using > gzip makes the process like 100x slower than if I just dump the zfs > stream to an uncompressed file, and so it's not practical from a time > standpoint. I am not sure about repositories, but there are projects such as pigz and pbzip2 which are parallelized interfaces to the same compression libraries, and easily compilable. They split the incoming data into chunks of size you specify (like 900kb for best results) and send them off to different cores; then a dispatcher thread collects the results and spews them off in correct order. Archives/streams are format-compatible with other (single-thread) implementations. In some versions of pigz there was a problem with compression of multiple filename arguments (some state was not cleared, so they were processed as if concatenated), so for predictable results better script up loops and call each time for one argument :) > 2. This is a more general UNIX question: I have a lot of directories > with mixed files, but I want to extract all of the files with a certain > extension, say .xyz, and move only those files to a different zfs > filesystem. In the destination, I want to recreate the directory > structure of the original tree, but only have the .xyz files in them > (these are large, uncompressed raw data). So the source and destination > should have the same directory structure; the source would have none of > the .xyz files, and the destination would have all of the .xyz files. > > Is there a simple way to do this with mv, or is another command > recommended? I am sure this is obvious to many of the unix gurus around, > so any help would be appreciated. Thanks! You can look into examples on cpio manpage - it takes results of the find command as input, so you can "find" piles according to the name pattern(s) you need, and as part of their path they would include the directories which contain them. If you need the whole directory structure, try "find -type d" - though you might need to somehow tell cpio to only archive the directories and not their contents recursively. Or if you're doing this locally - just create the structure at new root node with "mkdir -p". For remote sending, you might do the same and then archive this empty structure ;) HTH, //Jim Klimov From jimklimov at cos.ru Fri Aug 16 09:21:00 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 16 Aug 2013 11:21:00 +0200 Subject: [OmniOS-discuss] filesystems received from send -R In-Reply-To: <20130815165543.1eb145e3@dickless.landy.net> References: <20130815165543.1eb145e3@dickless.landy.net> Message-ID: <520DEEFC.5000803@cos.ru> On 2013-08-15 08:55, Michael Mounteney wrote: > I recently moved to a new machine: > > # ssh root at oldmachine 'zfs send -R rpool at M' | zfs recv rpool/bring > > the idea being: > > 1. zfs rename my own filesystems such as rpool/bring/home to > rpool/home. > 2. Copy settings from rpool/bring/omnios and below whilst setting up > the new machine. > 3. Refer to stuff under rpool/bring/zone/ whilst setting up zones. > > but curiously, it seems that any filesystem under rpool/bring/omnios or > rpool/bring/zone would not honour the 'zfs set mountpoint' command; > that is, the command was accepted without error, 'zfs get' shows the > value that I set, but in fact the directory doesn't exist. Verify also the "canmount" property - it may disable automounting in several ways. And it may be possible that just "receiving" the datasets did not mount them - possibly "zfs mount -a" would rectify that then. HTH, //Jim From jimklimov at cos.ru Fri Aug 16 09:47:39 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 16 Aug 2013 11:47:39 +0200 Subject: [OmniOS-discuss] performance penalty of a slow mirror In-Reply-To: <20130815170115.40e8a400@dickless.landy.net> References: <20130815170115.40e8a400@dickless.landy.net> Message-ID: <520DF53B.8000401@cos.ru> On 2013-08-15 09:01, Michael Mounteney wrote: > Hello, as my current OmniOS server doesn't have removable disks, I've > added an external USB2 HDD as a mirror, so that in the event of fire or > holiday, the external disk can be taken away as a backup. > > How does OmniOS deal with a very slow mirror ? Does it perform any > measurements to work out that it should always write to the other > mirror first, or does it do a 'fair sharing' or writes ? > > This is a SuperMicro 5017C-LF > (http://www.supermicro.com/products/system/1u/5017/sys-5017c-lf.cfm ) > with 2 x 500 GiB 2.5" drives mirrored internally, plus the external > mirror. The machine has only USB-2 sockets, not USB-3. AFAIK, at least the general illumos ZFS so far should not have special precautions for this (recently discussed as a case for remote-networked mirrors - that WAN copies should only receive writes and not a fairly shared half of reads). It may be that lags would cause "degradation" of the device in pool terms, but so far likely not balancing of IOs any other than by filling up the request queue. The latter might cause the slow device to process less requests per second than the faster device, but the requests that do get scheduled to this slower disk would suffer a higher latency. Overall, with reports that USB is not a very reliable connection method, it would IMHO be safer to use the external disk as a separate pool and set up regular replication of snapshots (perhaps with a handler script in the zfs-auto-snap SMF services). HTH, //Jim From steve at linuxsuite.org Fri Aug 16 14:41:10 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Fri, 16 Aug 2013 10:41:10 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: > We're seeing something similar on the same gear (LSI/supermicro expanders, > lsi controllers, sata drives). > > We've tried standard hardware debugging (cable reseat/replacement, etc) > and > the problem in our case seems to follow the sas expander backplane. > > We did a disk by disk migration into a different expander and they > stopped. > > How high are your error counts? (in our case, we were getting about > 1500/day/device). Is your performance impacted? (it was in our case) > -nld > Different expander? but still SATA behind SAS expander? On Supermicro 847 chassis? Is your setup stable, ie. works and drives don't drop out as failed? Performance isn't an issue here, but stability is.. It is definately a SATA behind SAS expanders issue. I did lots of testing with pools built on SAS drives they have no errors. I also did a lot of stress testing with 20T SATA pools, they were completely unusable, scrub would always wipe out the pool because drives would "drop out" as failed, but a hardware power cycle of the SuperMicro chassis would bring them all back. Then I turned off NCQ on the LSI controller and everything worked fine. Couldn't get anything to fail no matter how hard I beat on it. I will start to track error rates, we are not moving much data yet.. Would SATA port multipliers be a better solution? Does Solaris/OmniOS support such a hardware config. Just came across this http://www.45drives.com/ Which I think is a SATA port multiplier solution.... Centos/NetBSD?? can it work with OMniOS? -steve > > On Tue, Aug 13, 2013 at 10:20 AM, wrote: > >> >> Howdy! >> >> This is a SuperMicro JBOD with SATA disks. I am aware of the >> issues of having >> SATA on SAS, but was wondering just how serious these kinds of errors >> are.. a scrub of the pool >> completes without noticable problems.. I did a lot of stress testing >> earlier and could >> not get a failure. Disabling NCQ on the controller was a neccessary. >> What is the practical risk to data?? >> >> See below info for iostat / syslog >> >> thanx - steve >> >> syslog info >> >> kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] >> WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: >> IOCStatus=0x8000, IOCLogInfo=0x31120303 >> kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] >> WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: >> IOCStatus=0x8000, IOCLogInfo=0x31120436 >> kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] >> WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: >> IOCStatus=0x8000, IOCLogInfo=0x31120303 >> kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] >> WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> >> Blah Blah... >> >> kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: >> IOCStatus=0x8000, IOCLogInfo=0x31120436 >> kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] >> /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for >> target 13. >> kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, >> ioc_status=0x804b, >> scsi_state=0xc >> kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] >> /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for >> target 13. >> kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, >> ioc_status=0x804b, >> scsi_state=0xc >> kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] >> /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): >> >> Output of iostat -En >> >> Looks like "Hard Errors" and "No Device" correspond. What >> does "Transport Error" and "Recoverable" mean. I see no evidence >> of data corruption/loss, does ZFS deal/recover from these errors in a >> good/safe >> way? >> >> >> c5t5000C500489947A8d0 Soft Errors: 0 Hard Errors: 2 Transport Errors: 11 >> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >> W1F0AAMA >> Size: 3000.59GB <3000592982016 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 2 Recoverable: 0 >> Illegal Request: 2 Predictive Failure Analysis: 0 >> >> c5t5000C500525EB2B9d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 46 >> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >> W1F0QM5H >> Size: 3000.59GB <3000592982016 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 >> Illegal Request: 5 Predictive Failure Analysis: 0 >> >> c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 >> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >> W1F09G4Q >> Size: 3000.59GB <3000592982016 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 >> Illegal Request: 1 Predictive Failure Analysis: 0 >> >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > From esproul at omniti.com Fri Aug 16 15:35:43 2013 From: esproul at omniti.com (Eric Sproul) Date: Fri, 16 Aug 2013 11:35:43 -0400 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: <520DE0BA.2050108@cos.ru> References: <520DE0BA.2050108@cos.ru> Message-ID: On Fri, Aug 16, 2013 at 4:20 AM, Jim Klimov wrote: > I am not sure about repositories, but there are projects such as > pigz and pbzip2 which are parallelized interfaces to the same > compression libraries, and easily compilable. We have pigz and pbzip2 in the ms.omniti.com repo: http://omnios.omniti.com/wiki.php/Packaging#UnofficialExtras Eric From bfriesen at simple.dallas.tx.us Fri Aug 16 15:46:23 2013 From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn) Date: Fri, 16 Aug 2013 10:46:23 -0500 (CDT) Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: On Thu, 15 Aug 2013, Valrhona wrote: > Thanks to OmniTI for making a fantastic product for the community! > I am doing a bunch of backups, and trying to organize data, and have two questions: > > 1. Is there a better alternative, perhaps in the new package repositories, for gzip-style compression that is multithreaded? I am doing the usualy zfs send to a > file, which I then backup to tape. Using gzip makes the process like 100x slower than if I just dump the zfs stream to an uncompressed file, and so it's not > practical from a time standpoint. Use 'lzop' with default compression settings instead. It is much more CPU efficient than gzip. The compression ratio is somewhat less than gzip but the compression rate is vastly better. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From narayan.desai at gmail.com Fri Aug 16 16:06:38 2013 From: narayan.desai at gmail.com (Narayan Desai) Date: Fri, 16 Aug 2013 11:06:38 -0500 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: responses inline On Fri, Aug 16, 2013 at 9:41 AM, wrote: > > We're seeing something similar on the same gear (LSI/supermicro > expanders, > > lsi controllers, sata drives). > > > > We've tried standard hardware debugging (cable reseat/replacement, etc) > > and > > the problem in our case seems to follow the sas expander backplane. > > > > We did a disk by disk migration into a different expander and they > > stopped. > > > > How high are your error counts? (in our case, we were getting about > > 1500/day/device). Is your performance impacted? (it was in our case) > > -nld > > > > Different expander? but still SATA behind SAS expander? On > Supermicro 847 chassis? > Same model of expander, still SATA behind a SAS expander. Still supermicro 847. Is your setup stable, ie. works and drives don't drop out as failed? > Performance isn't an issue here, but stability is.. > We're having extreme performance issues, but no stability problems. The system just does I/O slowly. (less than 50MB/s, when we should be getting 1-2 GB/s for scrubs, etc) Moving the drives to another expander resolved the issue, though we had a second expander start experiencing the same issues at a lower rate, so we have more drive moves to do before we can expect to resolve the lower rate errors. > It is definately a SATA behind SAS expanders issue. I did lots of > testing > with pools built on SAS drives they have no errors. I also did a lot of > stress testing > with 20T SATA pools, they were completely unusable, scrub would > always wipe out the pool because drives would "drop out" as failed, > but a hardware power cycle of the SuperMicro chassis > would bring them all back. Then I turned off NCQ on the LSI controller > and everything worked fine. Couldn't get anything to fail no matter > how hard I beat on it. > We haven't had any issues like this at all. We've got at least 8 of these kinds of systems, with similar configs (SATA drives, SAS expanders, generally not chaining expanders). Each of them has at least 80 spindles, and generally they work reliably and perform well. (Excluding cases where there is a bad drive in a stripe, etc) Getting upwards of 2GB/s in aggregate off of one of these isn't hard. > I will start to track error rates, we are not moving much data > yet.. > > Would SATA port multipliers be a better solution? Does > Solaris/OmniOS > support such a hardware config. > > Just came across this > > http://www.45drives.com/ > > Which I think is a SATA port multiplier solution.... > Centos/NetBSD?? can > it work with OMniOS? > I'd assume it would work as long as there is a driver for those sata controllers, but you know what they say about assuming. These seem a bit expensive, particularly considering they don't include any drives. -nld -------------- next part -------------- An HTML attachment was scrubbed... URL: From mounty at landcroft.com Fri Aug 16 21:10:57 2013 From: mounty at landcroft.com (Entfernt) Date: Sat, 17 Aug 2013 07:10:57 +1000 Subject: [OmniOS-discuss] filesystems received from send -R In-Reply-To: <520DEEFC.5000803@cos.ru> References: <20130815165543.1eb145e3@dickless.landy.net> <520DEEFC.5000803@cos.ru> Message-ID: <20130817071057.0b6699c7fd2922d73f5b4e34@landcroft.com> On Fri, 16 Aug 2013 11:21:00 +0200 Jim Klimov wrote: > Verify also the "canmount" property That was it. Thanks Jim. -- Michael Mounteney From mounty at landcroft.com Fri Aug 16 21:30:10 2013 From: mounty at landcroft.com (Entfernt) Date: Sat, 17 Aug 2013 07:30:10 +1000 Subject: [OmniOS-discuss] performance penalty of a slow mirror In-Reply-To: <520DF53B.8000401@cos.ru> References: <20130815170115.40e8a400@dickless.landy.net> <520DF53B.8000401@cos.ru> Message-ID: <20130817073010.3fb410d9d5a64a5c79d75abf@landcroft.com> On Fri, 16 Aug 2013 11:47:39 +0200 Jim Klimov wrote: > Overall, with reports that USB is not a very reliable connection method, > it would IMHO be safer to use the external disk as a separate pool and > set up regular replication of snapshots (perhaps with a handler script > in the zfs-auto-snap SMF services). Thanks for the reply. I'm attracted to the extra-mirror method because it ensures that the copy is always up to date with no extra work. Backups involving mirrors will always lag behind the live situation. How about either: (1) detaching and re-attaching the USB disk occasionally ? (2) occasional scrubbing ? -- Michael Mounteney From valrhona at gmail.com Sat Aug 17 03:47:24 2013 From: valrhona at gmail.com (Valrhona) Date: Sat, 17 Aug 2013 04:47:24 +0100 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: <520DE0BA.2050108@cos.ru> References: <520DE0BA.2050108@cos.ru> Message-ID: y> I am not sure about repositories, but there are projects such as > pigz and pbzip2 which are parallelized interfaces to the same > compression libraries, and easily compilable Thanks! Looks like pigz and pbzip2 are in the libraries; thanks Eric for the tip! > In some versions of pigz there was a problem with compression of > multiple filename arguments (some state was not cleared, so they > were processed as if concatenated), so for predictable results > better script up loops and call each time for one argument :) At the moment, I am just interested in a single file, piped in from zfs send. Would the problem you mention affect me? > You can look into examples on cpio manpage - it takes results of > the find command as input, so you can "find" piles according to > the name pattern(s) you need, and as part of their path they would > include the directories which contain them. If you need the whole > directory structure, try "find -type d" - though you might need > to somehow tell cpio to only archive the directories and not their > contents recursively. Or if you're doing this locally - just create > the structure at new root node with "mkdir -p". For remote sending, > you might do the same and then archive this empty structure ;) So I found an easier solution: rsync for files with extension .xyz rsync -avgR --include='*/' --include='*.xyz' --exclude='*' --remove-source-files /src /dest This recreates the whole subdirectory structure, including (many) directories which have no files of type .xyz. To remove them, in /dest find . -type d -empty -exec rmdir {} \; You have to run this a few times because only one level of empty directory gets removed each time, so if you have empty directories that are subdirectories that then become empty, etc. etc. Thanks! Peter From valrhona at gmail.com Sat Aug 17 03:53:11 2013 From: valrhona at gmail.com (Valrhona) Date: Sat, 17 Aug 2013 04:53:11 +0100 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: Thanks for the tip. If I understand correctly, lzop is basically a drop-in replacement for gzip that is much faster, but single-threaded. Is there a comparison anywhere between (single-threaded) lzop and (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of RAM so I expect the compression to be IO-bound rather than CPU-bound, but that is just a guess. And for ZFS streams, is there a reason to prefer any one of these programs over another? Thanks! On Fri, Aug 16, 2013 at 4:46 PM, Bob Friesenhahn wrote: > On Thu, 15 Aug 2013, Valrhona wrote: > >> Thanks to OmniTI for making a fantastic product for the community! >> I am doing a bunch of backups, and trying to organize data, and have two >> questions: >> >> 1. Is there a better alternative, perhaps in the new package repositories, >> for gzip-style compression that is multithreaded? I am doing the usualy zfs >> send to a >> file, which I then backup to tape. Using gzip makes the process like 100x >> slower than if I just dump the zfs stream to an uncompressed file, and so >> it's not >> practical from a time standpoint. > > > Use 'lzop' with default compression settings instead. It is much more CPU > efficient than gzip. The compression ratio is somewhat less than gzip but > the compression rate is vastly better. > > Bob > -- > Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From rafibeyli at gmail.com Sat Aug 17 07:51:13 2013 From: rafibeyli at gmail.com (Hafiz Rafibeyli) Date: Sat, 17 Aug 2013 10:51:13 +0300 (EEST) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC Message-ID: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> Hello , anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? I bought 2 of them for replication ,but they does not recognized by Omnios.Any chances? regards. Hafiz. From skiselkov.ml at gmail.com Sat Aug 17 10:55:01 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sat, 17 Aug 2013 11:55:01 +0100 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> References: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> Message-ID: <520F5685.4020906@gmail.com> On 8/17/13 8:51 AM, Hafiz Rafibeyli wrote: > Hello , > > anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? > > I bought 2 of them for replication ,but they does not recognized by Omnios.Any chances? Have a look on Mellanox's website if they publish Solaris drivers. They should work on OmniOS as well. Cheers, -- Saso From jimklimov at cos.ru Sat Aug 17 11:16:14 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 17 Aug 2013 13:16:14 +0200 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: <520DE0BA.2050108@cos.ru> Message-ID: <520F5B7E.7020404@cos.ru> On 2013-08-17 05:47, Valrhona wrote: > y> I am not sure about repositories, but there are projects such as >> pigz and pbzip2 which are parallelized interfaces to the same >> compression libraries, and easily compilable > Thanks! Looks like pigz and pbzip2 are in the libraries; thanks Eric > for the tip! You're welcome =) >> In some versions of pigz there was a problem with compression of >> multiple filename arguments (some state was not cleared, so they >> were processed as if concatenated), so for predictable results >> better script up loops and call each time for one argument :) > At the moment, I am just interested in a single file, piped in from > zfs send. Would the problem you mention affect me? I guess not. The bug was back with version 1.7.1 IIRC, a few years ago; it may be corrected altogether by now. > So I found an easier solution: rsync > for files with extension .xyz > > rsync -avgR --include='*/' --include='*.xyz' --exclude='*' > --remove-source-files /src /dest > > This recreates the whole subdirectory structure, including (many) > directories which have no files of type .xyz. For rsync of dir to dir, it is safer to end them both with a slash like "... src/ dest/". Otherwise it may be too smart and create a subdir in the specified existing target, in some cases. > > To remove them, in /dest Take a look at "rsync -m" - see if it helps? -m, --prune-empty-dirs prune empty directory chains from the file-list > > find . -type d -empty -exec rmdir {} \; > > You have to run this a few times because only one level of empty > directory gets removed each time, so if you have empty directories > that are subdirectories that then become empty, etc. etc. I think you can do a reverse search; you certainly can pipe the output to "sort -r" and then to xargs which would call rmdir - to do it all in one pass, by starting at "leaf" empty directories in the tree. Though this would probably not hit directories which only contain directories - they are not "empty"... I think I did something similar by searching "root" dirs suspected for no files inside with "find $dir -type f" - if the output was empty, the $dir were good for "rm -rf". HTH, //Jim From jimklimov at cos.ru Sat Aug 17 11:33:17 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 17 Aug 2013 13:33:17 +0200 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: <520F5F7D.4050608@cos.ru> On 2013-08-17 05:53, Valrhona wrote: > Thanks for the tip. If I understand correctly, lzop is basically a > drop-in replacement for gzip that is much faster, but single-threaded. > Is there a comparison anywhere between (single-threaded) lzop and > (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of > RAM so I expect the compression to be IO-bound rather than CPU-bound, > but that is just a guess. And for ZFS streams, is there a reason to > prefer any one of these programs over another? Thanks! Just in case, you might also want to consider 7zip - I think it is parallel out of the box, and might offer best compression of them all (if your backups happen to be more constrained by space than IO) though not all versions support stdin|stdout compression. Also note that the common mailing-list wisdom argues against relying on ZFS-send streams in files as a backup method. Unlike ZFS itself, the streams offer little to no protection against bitrot, except that if a stream is corrupt - this is detectable and you can no longer import it. When the streams are used "live", piped to zfs receive to save into a target dataset, the transport error can be detected (at least by the admin or scripts that do the operation) and retried. When the stream file image is your only medium during restoration (especially if it is stored not on ZFS which would protect the file's backend storage) - by Murphy's Law you can bet to find the needed data un-importable. *Possibly*, things like ZIP CRC or RAR "1% redundancy" or some similar ECC-like schemes in other formats (if available) might help you against occasional bitrot (though likely not loss of larger chunks of data like a whole sector-size) - I am not ready to elaborate on this further, maybe some more knowledgeable experts would step in and say if this is just marketing voodoo or something that really works ;) HTH, //Jim From jimklimov at cos.ru Sat Aug 17 12:05:00 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sat, 17 Aug 2013 14:05:00 +0200 Subject: [OmniOS-discuss] performance penalty of a slow mirror In-Reply-To: <20130817073010.3fb410d9d5a64a5c79d75abf@landcroft.com> References: <20130815170115.40e8a400@dickless.landy.net> <520DF53B.8000401@cos.ru> <20130817073010.3fb410d9d5a64a5c79d75abf@landcroft.com> Message-ID: <520F66EC.8020606@cos.ru> On 2013-08-16 23:30, Entfernt wrote: > On Fri, 16 Aug 2013 11:47:39 +0200 > Jim Klimov wrote: > >> Overall, with reports that USB is not a very reliable connection method, >> it would IMHO be safer to use the external disk as a separate pool and >> set up regular replication of snapshots (perhaps with a handler script >> in the zfs-auto-snap SMF services). > > Thanks for the reply. > > I'm attracted to the extra-mirror method because it ensures that the copy is always up to date with no extra work. Backups involving mirrors will always lag behind the live situation. How about either: > (1) detaching and re-attaching the USB disk occasionally ? > (2) occasional scrubbing ? Technically this occasional attachment should work. Things to look out for include: 1) Pools may refuse to work if degraded - which includes removal of a component disk. See "zpool set failmode=continue" as one way to work around this... 2) It is possible that if you connect the USB disk to some other system and import it not read-only, this would cause the other system's ZFS to continue TXG numbering from where it was left when you disconnected the disk. It is likely that upon reattachment to the original pool, the system would at least be confused, and likely require that the disk is wiped and resilvered from scratch. I hope (but can't vouch) that there wouldn't be any more serious possibly hidden damage in such case. 3) If possible, do have a mirror remaining inside the box running at all times (so the USB disk makes it a 3+way mirror), so that the system is still reliable even without it. 4) Depending on hardware (USB is usually cheap and some corners may have been cut by manufacturers), the controller in the external disk enclosure or in your computer may lock up or otherwise misbehave in work, especially under load (i.e. due to overheating if nothing else) so scrubs might actually cause some of the problems they'd find. Not that they are useless or shouldn't be done - just keep an open mind if there are more CKSUM errors found on the USB disk than on others... possibly it needs better cooling or cabling or whatnot. HTH, //Jim Klimov From narayan.desai at gmail.com Sat Aug 17 12:10:22 2013 From: narayan.desai at gmail.com (Narayan Desai) Date: Sat, 17 Aug 2013 07:10:22 -0500 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <520F5685.4020906@gmail.com> References: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> <520F5685.4020906@gmail.com> Message-ID: According to Mellanox, Sun/Oracle has always done their own stack, so they don't have any code available. ConnectX/ConnectX2 adaptors do work out of the box, so I think that if it isn't supported out of the box, it isn't likely to work. -nld On Sat, Aug 17, 2013 at 5:55 AM, Saso Kiselkov wrote: > On 8/17/13 8:51 AM, Hafiz Rafibeyli wrote: > > Hello , > > > > anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? > > > > I bought 2 of them for replication ,but they does not recognized by > Omnios.Any chances? > > Have a look on Mellanox's website if they publish Solaris drivers. They > should work on OmniOS as well. > > Cheers, > -- > Saso > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hafizr at cantekstil.com.tr Sat Aug 17 07:22:20 2013 From: hafizr at cantekstil.com.tr (Hafiz Rafibeyli) Date: Sat, 17 Aug 2013 10:22:20 +0300 (EEST) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <874728.79962.1376723972143.JavaMail.root@cantekstil.com.tr> Message-ID: <20756784.79990.1376724140055.JavaMail.root@cantekstil.com.tr> Hello , anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? I bought 2 of them for replication ,but they does not recognized by Omnios.Any chances? regards. Hafiz. From garrett.damore at dey-sys.com Sat Aug 17 08:26:54 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Sat, 17 Aug 2013 12:26:54 +0400 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> References: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> Message-ID: <23E9DDC8-EF14-4194-90D3-721039896407@dey-sys.com> On Aug 17, 2013, at 11:51 AM, Hafiz Rafibeyli wrote: > Hello , > > anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? > > I bought 2 of them for replication ,but they does not recognized by Omnios.Any chances? Very slim. Oracle was working on drivers for this years ago, but they didn't get it integrated before they closed up the source code. The mellanox parts are incredibly complex (owing to IB heritage), and I doubt you'll see any open source illumos drivers for these ethernet parts anytime soon. Even the latest IB HCAs are probably not supported. Best bet is probably to pick up a pair of Intel 10GB NICs. I'm sure there are other vendors as well (SolarFlare, etc.) - Garrett > > > regards. > > Hafiz. > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From bfriesen at simple.dallas.tx.us Sat Aug 17 14:33:29 2013 From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn) Date: Sat, 17 Aug 2013 09:33:29 -0500 (CDT) Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: On Sat, 17 Aug 2013, Valrhona wrote: > Thanks for the tip. If I understand correctly, lzop is basically a > drop-in replacement for gzip that is much faster, but single-threaded. > Is there a comparison anywhere between (single-threaded) lzop and > (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of > RAM so I expect the compression to be IO-bound rather than CPU-bound, > but that is just a guess. And for ZFS streams, is there a reason to > prefer any one of these programs over another? Thanks! Lzop uses a completely different compression format. Its default compression is a bit less compression than 'gzip -3' but it is much more CPU efficient so it is able to achieve "wire speed" level compression rates on modern CPUs, and without relying on threading. There are some other compressors which are even faster than lzop but their compression formats are often not stable. A problem with most threaded compressors is that more effective compression algorithms (e.g. lzma as used by 'xz' and 'lzip') require very large data chunk sizes so that the compression algorithm works effectively. This means that the input data size needs to be large (hundreds of MB or even gigabytes) in order for the multi-thread chunk size to be large enough. Zfs send streams or tar/cpio archives of large directory trees are likely to be large enough but many/most ordinary files don't qualify. As a result, using threading may result in much less compression. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From thibault.vincent at smartjog.com Sat Aug 17 15:02:04 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sat, 17 Aug 2013 15:02:04 -0000 (UTC) Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <201308020002.r720265f001086@kyklops.ohsu.edu> Message-ID: > I'm trying out OmniOS 151006l on a new R720, and have noticed that a > "shutdown -i6" style shutdown does not reboot the server. It goes > through > the "syncing disks..." and "done" phases, then just stops there. > Power > remains on. An IPMI or DRAC "reset" gets things going again. Reboot to prom issues also affect kernel when handling a fault. A bug is open at illumos, feel free to add new information. https://www.illumos.org/issues/4052 Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From skiselkov.ml at gmail.com Sat Aug 17 16:01:32 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sat, 17 Aug 2013 17:01:32 +0100 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: References: Message-ID: <520F9E5C.2040502@gmail.com> On 8/17/13 4:02 PM, Thibault VINCENT wrote: >> I'm trying out OmniOS 151006l on a new R720, and have noticed that a >> "shutdown -i6" style shutdown does not reboot the server. It goes >> through >> the "syncing disks..." and "done" phases, then just stops there. >> Power >> remains on. An IPMI or DRAC "reset" gets things going again. > > Reboot to prom issues also affect kernel when handling a fault. > A bug is open at illumos, feel free to add new information. > https://www.illumos.org/issues/4052 shutdown(1M) is deprecated. Use reboot(1M), poweroff(1M) and halt(1M) instead. Did you try "reboot -p"? Cheers, -- Saso From thibault.vincent at smartjog.com Sat Aug 17 18:04:35 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sat, 17 Aug 2013 18:04:35 -0000 (UTC) Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <520F9E5C.2040502@gmail.com> Message-ID: <44b7a8d6-5704-4238-81a1-9a0baa039a43@zimbra.fr.smartjog.net> > > Did you try "reboot -p"? Unfortunately only the fast reboot works like a charm. Reboot to prom with -p of after a kernel crash gets stalled on the "Rebooting..." message. This message is printed in the last bit of code involved [1] and at that point everything is designed to make the reboot happen. Yet it doesn't :-( [1] http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/machdep.c#203 -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From skiselkov.ml at gmail.com Sat Aug 17 22:51:03 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Sat, 17 Aug 2013 23:51:03 +0100 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <44b7a8d6-5704-4238-81a1-9a0baa039a43@zimbra.fr.smartjog.net> References: <44b7a8d6-5704-4238-81a1-9a0baa039a43@zimbra.fr.smartjog.net> Message-ID: <520FFE57.4010507@gmail.com> On 8/17/13 7:04 PM, Thibault VINCENT wrote: >> >> Did you try "reboot -p"? > > Unfortunately only the fast reboot works like a charm. > > Reboot to prom with -p of after a kernel crash gets stalled > on the "Rebooting..." message. This message is printed in > the last bit of code involved [1] and at that point > everything is designed to make the reboot happen. Yet it > doesn't :-( > > [1] http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/machdep.c#203 > Which BIOS version are you running? -- Saso From valrhona at gmail.com Sat Aug 17 22:53:45 2013 From: valrhona at gmail.com (Valrhona) Date: Sat, 17 Aug 2013 23:53:45 +0100 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: <520F5B7E.7020404@cos.ru> References: <520DE0BA.2050108@cos.ru> <520F5B7E.7020404@cos.ru> Message-ID: > For rsync of dir to dir, it is safer to end them both with a slash > like "... src/ dest/". Otherwise it may be too smart and create a > subdir in the specified existing target, in some cases. Yes, thanks for the tip. > Take a look at "rsync -m" - see if it helps? > -m, --prune-empty-dirs prune empty directory chains from the file-list Ah, that's great. I really appreciate all of the pointers. From valrhona at gmail.com Sat Aug 17 23:01:30 2013 From: valrhona at gmail.com (Valrhona) Date: Sun, 18 Aug 2013 00:01:30 +0100 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: <520F5F7D.4050608@cos.ru> References: <520F5F7D.4050608@cos.ru> Message-ID: > Just in case, you might also want to consider 7zip - I think it is > parallel out of the box, and might offer best compression of them > all (if your backups happen to be more constrained by space than IO) > though not all versions support stdin|stdout compression. Thanks. I use this for default compression on individual files, so it makes sense to try it here. > Also note that the common mailing-list wisdom argues against relying > on ZFS-send streams in files as a backup method. Unlike ZFS itself, > the streams offer little to no protection against bitrot, except that > if a stream is corrupt - this is detectable and you can no longer > import it. When the streams are used "live", piped to zfs receive > to save into a target dataset, the transport error can be detected > (at least by the admin or scripts that do the operation) and retried. > When the stream file image is your only medium during restoration > (especially if it is stored not on ZFS which would protect the file's > backend storage) - by Murphy's Law you can bet to find the needed data > un-importable. Thanks for the concern. So I didn't elaborate on what I actually do further downstream. I do a zfs send > file.zstream To check integrity, I do a zstreamdump < file.zstream Then before committing to tape, checksum again: md5sum file.zstream > file.md5 I then tar it to tape. Then I destroy the the original filesystem which had the zstream file and create a new zpool, restore the contents of the tape to the files, then rerun the md5sum. This way, if a single bit changes throughout the backup and restore process, I will know. And the checksum is stored on the tape in the file.md5 file, so at any point in the future I can restore and know that the data is good. This is somewhat tedious and takes a while, but fortunately it's just a few commands and I can let it run in the background. But I don't see any better backup solution, because ZFS internally checksums, and I do a manual backup and restore, and checksum before and after. I think this is more intense in terms of verification than most of the other solutions I know. It is also a bit more future-proof, since it relies only on ZFS, tar, and md5sum, all of which are on live-CDs and standard illumos (without any libraries). So with a tape, tape drive, enough space on a hard disk subsystem, I can restore my filesystems without any additional software (don't even need network connectivity), and have end-to-end checksummng. But maybe there are things I could be doing better? From valrhona at gmail.com Sat Aug 17 23:05:40 2013 From: valrhona at gmail.com (Valrhona) Date: Sun, 18 Aug 2013 00:05:40 +0100 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: Message-ID: > Lzop uses a completely different compression format. Its default > compression is a bit less compression than 'gzip -3' but it is much more CPU > efficient so it is able to achieve "wire speed" level compression rates on > modern CPUs, and without relying on threading. There are some other > compressors which are even faster than lzop but their compression formats > are often not stable. > > A problem with most threaded compressors is that more effective compression > algorithms (e.g. lzma as used by 'xz' and 'lzip') require very large data > chunk sizes so that the compression algorithm works effectively. This means > that the input data size needs to be large (hundreds of MB or even > gigabytes) in order for the multi-thread chunk size to be large enough. Zfs > send streams or tar/cpio archives of large directory trees are likely to be > large enough but many/most ordinary files don't qualify. As a result, using > threading may result in much less compression. Thanks for the insights. I basically had a structural question: most compression I know if analyzes the whole file, and then algorithmically looks for redundancies and patterns that can be compressed. For streaming data, obviously how much of a chunk to analyze makes a huge difference. I think probably the most common is video data, but the nice thing about that is that, roughly speaking, most frames are similar to adjacent ones, so the window in time that a video compressor must look can actually be pretty small, and therefore efficient. For a ZFS stream, this seems impractical, especially for terabyte-sized chunks. There isn't any reason to believe that the compressor will be able to find similar regions, depending on file layout. So it's not clear to me how efficient, from an architectural standpoint, threading is, depending on how it is implemented. Do you know, from an architectural standpoint, how LZOP differs from the others have been mentioned, such that there would be a fundamental reason to prefer one format over the other for ZFS stream data? Thanks! Peter From valrhona at gmail.com Sat Aug 17 23:08:21 2013 From: valrhona at gmail.com (Valrhona) Date: Sun, 18 Aug 2013 00:08:21 +0100 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <23E9DDC8-EF14-4194-90D3-721039896407@dey-sys.com> References: <18933589.80406.1376725872964.JavaMail.root@cantekstil.com.tr> <23E9DDC8-EF14-4194-90D3-721039896407@dey-sys.com> Message-ID: Do you guys have preferred Intel 10 GbE NIC models for small installations, that are on the cheaper side? Also, if I just have a few workstations I need to connect to an NFS server over 10 GbE, I was thinking of just getting a couple of adapters and crossover cables, instead of going with a switch, just for cost reasons. Does this make sense, and is there any downside? Switches right now seem to be pretty expensive, and for just a couple of workstations/servers, it seems like crossover cables might do just as well. Thoughts? On Sat, Aug 17, 2013 at 9:26 AM, Garrett D'Amore wrote: > > On Aug 17, 2013, at 11:51 AM, Hafiz Rafibeyli wrote: > >> Hello , >> >> anybody runs Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC on Omnios? >> >> I bought 2 of them for replication ,but they does not recognized by Omnios.Any chances? > > Very slim. Oracle was working on drivers for this years ago, but they didn't get it integrated before they closed up the source code. The mellanox parts are incredibly complex (owing to IB heritage), and I doubt you'll see any open source illumos drivers for these ethernet parts anytime soon. > > Even the latest IB HCAs are probably not supported. > > Best bet is probably to pick up a pair of Intel 10GB NICs. I'm sure there are other vendors as well (SolarFlare, etc.) > > - Garrett >> >> >> regards. >> >> Hafiz. >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From thibault.vincent at smartjog.com Sat Aug 17 23:15:18 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sat, 17 Aug 2013 23:15:18 -0000 (UTC) Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <520FFE57.4010507@gmail.com> Message-ID: > Which BIOS version are you running? > BIOS has version 1.6.0 and iDrac has 1.40.40, servers were updated with Life Cycle Controller few weeks ago and still expose the problem. I haven't found interesting settings to play with in the BIOS. This Dell generation got even better UEFI integration but we're using BIOS boot with OmniOS. I wonder if the bug could be related with UEFI since reset() implements specific routines for it. -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From richard.elling at richardelling.com Sat Aug 17 23:25:49 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Sat, 17 Aug 2013 16:25:49 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: On Aug 13, 2013, at 8:20 AM, steve at linuxsuite.org wrote: > > Howdy! > > This is a SuperMicro JBOD with SATA disks. I am aware of the > issues of having > SATA on SAS, but was wondering just how serious these kinds of errors > are.. a scrub of the pool > completes without noticable problems.. I did a lot of stress testing > earlier and could > not get a failure. Disabling NCQ on the controller was a neccessary. > What is the practical risk to data?? > > See below info for iostat / syslog > > thanx - steve > > syslog info > > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: > IOCStatus=0x8000, IOCLogInfo=0x31120303 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event_sync: > IOCStatus=0x8000, IOCLogInfo=0x31120436 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: > IOCStatus=0x8000, IOCLogInfo=0x31120303 > kern.warning<4>: Aug 13 10:39:10 dfs1 scsi: [ID 243001 kern.warning] > WARNING: /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): These messages are generated by the device and reported by the mpt_sas driver. They can be decoded, but unfortunately the illumos driver leaves it as an exercise for the developer :-( > > Blah Blah... > > kern.warning<4>: Aug 13 10:39:10 dfs1 #011mptsas_handle_event: > IOCStatus=0x8000, IOCLogInfo=0x31120436 > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for > target 13. > kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): > kern.info<6>: Aug 13 10:39:11 dfs1 #011Log info 0x31120303 received for > target 13. > kern.info<6>: Aug 13 10:39:11 dfs1 #011scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc > kern.info<6>: Aug 13 10:39:11 dfs1 scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340d at 6/pci1000,3080 at 0 (mpt_sas0): These (status 0x804b == aborted command) are the result of a device reset. The fact you are seeing a reset means that an I/O timed out somewhere. > > Output of iostat -En > > Looks like "Hard Errors" and "No Device" correspond. What > does "Transport Error" and "Recoverable" mean. I see no evidence > of data corruption/loss, does ZFS deal/recover from these errors in a > good/safe > way? Transport errors are things like non-response to command. Recoverable means that the sd driver can retry. > > c5t5000C500489947A8d0 Soft Errors: 0 Hard Errors: 2 Transport Errors: 11 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F0AAMA > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 2 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > > c5t5000C500525EB2B9d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 46 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F0QM5H > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 > Illegal Request: 5 Predictive Failure Analysis: 0 > > c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 > Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: W1F09G4Q > Size: 3000.59GB <3000592982016 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 > Illegal Request: 1 Predictive Failure Analysis: 0 Something is sick. Unfortunately, it could be one of the many SATA devices causing disruptions to everyone else -- a good reason to not directly attach SATA devices to SAS expanders. Tracking these down and eliminating a bug in the expander itself, is a tedious task. Is there anything in the JBOD other than the Seagate 3TBs? -- richard -- ZFS storage and performance consulting at http://www.RichardElling.com From thibault.vincent at smartjog.com Sat Aug 17 23:27:20 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sat, 17 Aug 2013 23:27:20 -0000 (UTC) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: Message-ID: > Do you guys have preferred Intel 10 GbE NIC models for small > installations, that are on the cheaper side? We're using these cards everywhere with Linux and it works like a charm, under high bandwidth or high pkts/s loads. They have nice offloading features and load is well spread over CPU cores. SFP+ twinax cables are great and cheap but are limited in length, and I would discourage you to use the longest, 5 meters at most. If you go with fiber you'll need a special reference for the cards because Intel is locking up the cheapest. Using Intel optic module is not required (most drivers have a parameter to let them accept any brand). Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From valrhona at gmail.com Sun Aug 18 00:38:13 2013 From: valrhona at gmail.com (Valrhona) Date: Sun, 18 Aug 2013 01:38:13 +0100 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: References: Message-ID: Don't know if you mentioned this elsewhere, but what specific cards have you had good experience under Illumos using? And I would probably go with copper cables. Thanks. Peter On Sun, Aug 18, 2013 at 12:27 AM, Thibault VINCENT wrote: >> Do you guys have preferred Intel 10 GbE NIC models for small >> installations, that are on the cheaper side? > > We're using these cards everywhere with Linux and it works like a charm, under high bandwidth or high pkts/s loads. They have nice offloading features and load is well spread over CPU cores. > SFP+ twinax cables are great and cheap but are limited in length, and I would discourage you to use the longest, 5 meters at most. > If you go with fiber you'll need a special reference for the cards because Intel is locking up the cheapest. Using Intel optic module is not required (most drivers have a parameter to let them accept any brand). > > Cheers > > > -- > Thibault VINCENT - Infrastructure Engineer > SmartJog | T: +33 1 5868 6238 > 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France > www.smartjog.com | a TDF Group company From thibault.vincent at smartjog.com Sun Aug 18 01:23:28 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sun, 18 Aug 2013 01:23:28 -0000 (UTC) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: Message-ID: <384961a5-9475-4e5c-9f75-73acea9b262e@zimbra.fr.smartjog.net> > Don't know if you mentioned this elsewhere, but what specific cards > have you had good experience under Illumos using? And I would > probably go with copper cables. Thanks. Well I mentioned this was a Linux experience because illumos is sort of new grounds, but I'm happy with network performance so far with this kernel and Intel 10Ge. We've made them our standard, it's the X520 familly with various form factors (dual SFP+ PCIe, or dual KR in Dell blade server). On OmniOS boxes I've used up to two cards to make four ports LACP aggregations to a switch. If you go with copper you've got up to 10 meters on twinax cables, but Intel is making an RJ45 version which we didn't test but the link can reach 100 meters on really good cables (you'll need to upgrade patch panels to CAT6 if it's not a single cable from end to end). Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From valrhona at gmail.com Sun Aug 18 01:40:21 2013 From: valrhona at gmail.com (Valrhona) Date: Sun, 18 Aug 2013 02:40:21 +0100 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <384961a5-9475-4e5c-9f75-73acea9b262e@zimbra.fr.smartjog.net> References: <384961a5-9475-4e5c-9f75-73acea9b262e@zimbra.fr.smartjog.net> Message-ID: So you have these working on OmniOS without any issues? Did you have to install any particular drivers? Intel now has the converged X540 adapters out, e.g.: http://www.newegg.com/Product/Product.aspx?Item=N82E16833106144 http://www.newegg.com/Product/Product.aspx?Item=N82E16833106179 The computers are all within a few meters, so cat6 patch cables would be convenient, and length won't be an issue. Also, has anyone had any experience with the Netgear XS708E switch? It seems to be the only one that is reasonably priced for small networks at the moment. On Sun, Aug 18, 2013 at 2:23 AM, Thibault VINCENT wrote: >> Don't know if you mentioned this elsewhere, but what specific cards >> have you had good experience under Illumos using? And I would >> probably go with copper cables. Thanks. > > Well I mentioned this was a Linux experience because illumos is > sort of new grounds, but I'm happy with network performance so far > with this kernel and Intel 10Ge. We've made them our standard, it's > the X520 familly with various form factors (dual SFP+ PCIe, or > dual KR in Dell blade server). On OmniOS boxes I've used up to two > cards to make four ports LACP aggregations to a switch. > > If you go with copper you've got up to 10 meters on twinax cables, > but Intel is making an RJ45 version which we didn't test but the > link can reach 100 meters on really good cables (you'll need to > upgrade patch panels to CAT6 if it's not a single cable from > end to end). > > Cheers > > -- > Thibault VINCENT - Infrastructure Engineer > SmartJog | T: +33 1 5868 6238 > 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France > www.smartjog.com | a TDF Group company From thibault.vincent at smartjog.com Sun Aug 18 01:40:47 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sun, 18 Aug 2013 01:40:47 -0000 (UTC) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: Message-ID: <6aeb55aa-bdfd-4d70-8656-3a2f1b43bbe8@zimbra.fr.smartjog.net> > Also, if I just have a few workstations I need to connect to an NFS > server over 10 GbE, I was thinking of just getting a couple of > adapters and crossover cables, instead of going with a switch, just > for cost reasons. Does this make sense, and is there any downside? > Switches right now seem to be pretty expensive, and for just a couple > of workstations/servers, it seems like crossover cables might do just > as well. Thoughts? A quick comparison on Intel cards and cheap switchs shows the 10Gbe RJ45 port costs double on the adapter. Choose wisely, I'll go with the switch over four workstations. And having lots of NICs in the same server may not scale well : too many interrupts, under usage of ring buffers and queues, lots of kernel tasks. -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From thibault.vincent at smartjog.com Sun Aug 18 01:58:08 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Sun, 18 Aug 2013 01:58:08 -0000 (UTC) Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: Message-ID: <3452b306-fd43-4d50-8f71-eb5a8031ec79@zimbra.fr.smartjog.net> > So you have these working on OmniOS without any issues? Did you have > to install any particular drivers? Yes, it works great out of the box. The only setting you can think of with the ixgbe driver is to deactivate the interrupt throttling but that's only interesting when receiving lots of packets per second (small packets you won't see much on a storage with few clients, more likely when doing routing or LB stuff). Like I said we don't have a super long experience of X520 and OmniOS but all lab tests were done with these adapters and servers have been benchmarked for days on NFS and iSCSI load. It's satisfying and will go into production. -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From jimklimov at cos.ru Sun Aug 18 08:44:33 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sun, 18 Aug 2013 10:44:33 +0200 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: <520F5F7D.4050608@cos.ru> Message-ID: <52108971.3090502@cos.ru> On 2013-08-18 01:01, Valrhona wrote: >> Also note that the common mailing-list wisdom argues against relying >> on ZFS-send streams in files as a backup method. Unlike ZFS itself, >> the streams offer little to no protection against bitrot, except that >> if a stream is corrupt - this is detectable and you can no longer >> import it. When the streams are used "live", piped to zfs receive >> to save into a target dataset, the transport error can be detected >> (at least by the admin or scripts that do the operation) and retried. >> When the stream file image is your only medium during restoration >> (especially if it is stored not on ZFS which would protect the file's >> backend storage) - by Murphy's Law you can bet to find the needed data >> un-importable. > Thanks for the concern. So I didn't elaborate on what I actually do > further downstream. > > I do a zfs send > file.zstream > > To check integrity, I do a zstreamdump < file.zstream > > Then before committing to tape, checksum again: > > md5sum file.zstream > file.md5 > > I then tar it to tape. Then I destroy the the original filesystem > which had the zstream file and create a new zpool, restore the > contents of the tape to the files, then rerun the md5sum. This way, if > a single bit changes throughout the backup and restore process, I will > know. And the checksum is stored on the tape in the file.md5 file, so > at any point in the future I can restore and know that the data is > good. Luckily for me, ZFS stream files (stored on ZFS) did not fail me yet. I believe some of the illumos distros use such images for quick installation of zones and systems, so the approach is not "taboo" - but my experience ends at this and concerns-from-internet step in. Namely, that "ZFS receive" would also know about a single bit-flip, and refuse to use this stream to initiate a dataset at all. That is, it wouldn't be an error limited to some unlucky file in the received dataset, but it would be absence of the dataset at all. On a side note, you can easily stage an experiment by taking a ZFS stream file, corrupting it with "dd", replacing a byte or more with different data (you can also forge the byte to be a bit-off from an originally stored value) and see what happens today upon receive; and in particular - whether error-diags offer any practical remedies. Maybe this is FUD by now and the "problem" was fixed by i.e. allowing an admin to force receipt of a stream, or some ECC/CRC was added into stream format, or... - I just don't know, and hope that people who do would now step in and tell us the state of things now in this area. Also, there was some work about ZFS and backups by integration with NDMP protocol (format?) Here, again, "I heard a tune but don't know the words", so search the internet for these keywords to see if this "something" fits you better :) HTH, //Jim From jimklimov at cos.ru Sun Aug 18 08:48:16 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Sun, 18 Aug 2013 10:48:16 +0200 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: References: Message-ID: <52108A50.6030103@cos.ru> On 2013-08-18 01:15, Thibault VINCENT wrote: >> Which BIOS version are you running? >> > > BIOS has version 1.6.0 and iDrac has 1.40.40, servers were updated with Life Cycle Controller few weeks ago and still expose the problem. I haven't found interesting settings to play with in the BIOS. > > This Dell generation got even better UEFI integration but we're using BIOS boot with OmniOS. I wonder if the bug could be related with UEFI since reset() implements specific routines for it. Does programmatic ungraceful reboot work or also hang? See the "uadmin" command for flags, I think the reboot would be "uadmin 1 2"... This should be basically a hook into the kernel routine that ends the kernel's work, so you can see quickly if it works. Stop all worthy processes beforehand :) HTH, //Jim From holtzhausen.jan at gmail.com Sun Aug 18 10:54:21 2013 From: holtzhausen.jan at gmail.com (Jan Holtzhausen) Date: Sun, 18 Aug 2013 12:54:21 +0200 Subject: [OmniOS-discuss] Omnios, VMware fusion Message-ID: Omni needs a virtual floppy drive to install. In OSX, open a terminal to a directory that fusion can browse to (~/Desktop is fine) ~> dd bs=512 count=2880 if=/dev/zero of=floppy.img There's your blank image. Now add a floppy device in fusion, point it to floppy.img. Go ahead and install on fusion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cnehren+omnios-discuss at omniti.com Sun Aug 18 14:12:32 2013 From: cnehren+omnios-discuss at omniti.com (Chris Nehren) Date: Sun, 18 Aug 2013 10:12:32 -0400 Subject: [OmniOS-discuss] Omnios, VMware fusion In-Reply-To: References: Message-ID: <20130818141232.GA39893@eschaton.local> On Sun, Aug 18, 2013 at 12:54:21 +0200, Jan Holtzhausen wrote: > Omni needs a virtual floppy drive to install. > In OSX, open a terminal to a directory that fusion can browse to (~/Desktop > is fine) > ~> dd bs=512 count=2880 if=/dev/zero of=floppy.img > There's your blank image. > Now add a floppy device in fusion, point it to floppy.img. > Go ahead and install on fusion. This is a known issue. See http://omnios.omniti.com/wiki.php/VMwareNotes for our documentation of it. -- Chris Nehren Site Reliability Engineer, OmniTI From cnehren+omnios-discuss at omniti.com Sun Aug 18 14:43:55 2013 From: cnehren+omnios-discuss at omniti.com (Chris Nehren) Date: Sun, 18 Aug 2013 10:43:55 -0400 Subject: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees In-Reply-To: References: <520F5F7D.4050608@cos.ru> Message-ID: <20130818144355.GA41101@eschaton.local> On Sun, Aug 18, 2013 at 00:01:30 +0100, Valrhona wrote: > This is somewhat tedious and takes a while, but fortunately it's just > a few commands and I can let it run in the background. But I don't see > any better backup solution, because ZFS internally checksums, and I do > a manual backup and restore, and checksum before and after. I think > this is more intense in terms of verification than most of the other > solutions I know. > > It is also a bit more future-proof, since it relies only on ZFS, tar, > and md5sum, all of which are on live-CDs and standard illumos (without > any libraries). So with a tape, tape drive, enough space on a hard > disk subsystem, I can restore my filesystems without any additional > software (don't even need network connectivity), and have end-to-end > checksummng. > > But maybe there are things I could be doing better? You might want to take a look at zetaback (https://labs.omniti.com/labs/zetaback) which we've been using for a while for backups. -- Chris Nehren Site Reliability Engineer, OmniTI From garrett.damore at dey-sys.com Sun Aug 18 16:30:21 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Sun, 18 Aug 2013 09:30:21 -0700 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <6aeb55aa-bdfd-4d70-8656-3a2f1b43bbe8@zimbra.fr.smartjog.net> References: <6aeb55aa-bdfd-4d70-8656-3a2f1b43bbe8@zimbra.fr.smartjog.net> Message-ID: On Aug 17, 2013, at 6:40 PM, Thibault VINCENT wrote: >> Also, if I just have a few workstations I need to connect to an NFS >> server over 10 GbE, I was thinking of just getting a couple of >> adapters and crossover cables, instead of going with a switch, just >> for cost reasons. Does this make sense, and is there any downside? >> Switches right now seem to be pretty expensive, and for just a couple >> of workstations/servers, it seems like crossover cables might do just >> as well. Thoughts? > > A quick comparison on Intel cards and cheap switchs shows the 10Gbe RJ45 > port costs double on the adapter. Choose wisely, I'll go with the switch > over four workstations. And having lots of NICs in the same server may > not scale well : too many interrupts, under usage of ring buffers and > queues, lots of kernel tasks. Actually, because modern 10G hardware all have multiple rings, these wind up behaving much like having multiple NICs? i.e. more interrupts, more kernel tasks, etc. That's generally a *good* thing, because most CPUs can't keep up with 10G load (unless you're using jumbo frames and/or TCP offload.) So I wouldn't assume that CPU resources will be any different for 10 x 1G cards vs 1 x 10 G. They will probably be fairly close. Factors that *should* drive your decisions: 1. B/w needs. If you don't need close to 10G, but just a little more than 1G (say 2G), then multiple cards may be cheaper. 2. Reliability. A single 10G card and link are a single point of failure. But, multiple 1G cards introduce multiple points of failure -- so more likely to encounter a failure, but more likely that the failure doesn't prevent basic functionality. 3. Complexity. A single 10G card is *lots* simpler than arranging for 802.3ad link aggregation. 4. Port costs. Again, 2x1G ports probably inexpensive. But at 10x1G, you may find otherwise. 5. Power consumption. More cards == more power consumption. (10G consumes much more than 1G, but consumes far less than 10x the power. :-) 6. Slot availability in your servers, and port availability on your switches. All things being equal, I'd probably opt for a 10 GbE link unless I can be satisfied by 1GbE. I'd only use link aggregation to give some reliability, and I'd elect to do that with *either* 1G or 10G if I needed the reliability. *Do* consider the upstream bottlenecks too! If you're connected to the internet with a 10Mbps connection, then having 10GbE isn't going to give you faster internet. In your datacenter, if your traffic is a bunch of client machines hitting a single server, it may make sense to have a 10GbE to the server, and 1GbE to the clients. (Assuming you have a switch with a 10GbE uplink port. :-) - Garrett > > -- > Thibault VINCENT - Infrastructure Engineer > SmartJog | T: +33 1 5868 6238 > 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France > www.smartjog.com | a TDF Group company > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From nagele at wildbit.com Sun Aug 18 18:25:05 2013 From: nagele at wildbit.com (Chris Nagele) Date: Sun, 18 Aug 2013 14:25:05 -0400 Subject: [OmniOS-discuss] Mellanox Infiniband 2x 10Gbps MHEL-CF128-TC In-Reply-To: <3452b306-fd43-4d50-8f71-eb5a8031ec79@zimbra.fr.smartjog.net> References: <3452b306-fd43-4d50-8f71-eb5a8031ec79@zimbra.fr.smartjog.net> Message-ID: We've used the X520's for a while on a very active NFS production environment. They work extremely well. We now have a 10G switch, but used cross over SFP+ cables for almost a year. Our primary use is NFS. We did some performance testing and noticed a big difference when tuning the following settings (like Thibault said) in the /kernel/drv/ixgbe.conf file for the 10 GbE driver: default_mtu=9000; intr_throttling=1; (or 0) The last one (intro_throttling) made a big difference in lowering latency, but to be honest, I read that it can have some side affects as well. Chris On Sat, Aug 17, 2013 at 9:58 PM, Thibault VINCENT wrote: >> So you have these working on OmniOS without any issues? Did you have >> to install any particular drivers? > > Yes, it works great out of the box. The only setting you can think of > with the ixgbe driver is to deactivate the interrupt throttling but > that's only interesting when receiving lots of packets per second (small > packets you won't see much on a storage with few clients, more likely > when doing routing or LB stuff). > > Like I said we don't have a super long experience of X520 and OmniOS > but all lab tests were done with these adapters and servers have been > benchmarked for days on NFS and iSCSI load. It's satisfying and will > go into production. > > -- > Thibault VINCENT - Infrastructure Engineer > SmartJog | T: +33 1 5868 6238 > 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France > www.smartjog.com | a TDF Group company > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From holtzhausen.jan at gmail.com Mon Aug 19 06:55:50 2013 From: holtzhausen.jan at gmail.com (Jan Holtzhausen) Date: Mon, 19 Aug 2013 08:55:50 +0200 Subject: [OmniOS-discuss] Omnios, VMware fusion In-Reply-To: <20130818141232.GA39893@eschaton.local> References: <20130818141232.GA39893@eschaton.local> Message-ID: No sweat, I just jotted down a solution for the OSX/fusion crowd in case anyone needed it. On 18 Aug 2013 4:14 PM, "Chris Nehren" wrote: > On Sun, Aug 18, 2013 at 12:54:21 +0200, Jan Holtzhausen wrote: > > Omni needs a virtual floppy drive to install. > > In OSX, open a terminal to a directory that fusion can browse to > (~/Desktop > > is fine) > > ~> dd bs=512 count=2880 if=/dev/zero of=floppy.img > > There's your blank image. > > Now add a floppy device in fusion, point it to floppy.img. > > Go ahead and install on fusion. > > This is a known issue. See http://omnios.omniti.com/wiki.php/VMwareNotes > for our documentation of it. > > -- > Chris Nehren > Site Reliability Engineer, OmniTI > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gate03 at landcroft.co.uk Mon Aug 19 09:52:55 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Mon, 19 Aug 2013 19:52:55 +1000 Subject: [OmniOS-discuss] VBox to qemu Message-ID: <20130819195255.281e1d0b@dickless.landy.net> Hello, I've just moved servers and the new one has the capability of running qemu-system-x86_64 but is it worth changing over ? I've copied all the VirtualBox images over so it would be easiest just to install VirtualBox on the new machine but if qemu is significantly faster I would change over. Also, is there any information anywhere about migrating, in terms of mapping VirtualBox settings to qemu command line arguments ? The re-learning effort is another barrier to moving. Thanks, Michael. From steve at linuxsuite.org Mon Aug 19 16:37:09 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 12:37:09 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> >> SNIP >> >> c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 >> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >> W1F09G4Q >> Size: 3000.59GB <3000592982016 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 >> Illegal Request: 1 Predictive Failure Analysis: 0 > > Something is sick. Unfortunately, it could be one of the many SATA devices > causing disruptions to everyone else -- a good reason to not directly > attach > SATA devices to SAS expanders. Tracking these down and eliminating a bug > in the expander itself, is a tedious task. Is there anything in the JBOD > other than > the Seagate 3TBs? > -- richard Thanks Richard... There are 11 Seagate 3T SATA drives on a single backplane. 10 in a RAIDZ-2 and a hot spare. There are no other devices in the JBOD. ZIL is on a separate SSD in a DELL chassis So a couple of questions i) If there is a single device that has a "large" number of errors compared to the other devices, then perhaps this device is messing up the others? So swapping it out could solve the problem? ii) If I had the SATA devices behind SATA port multipliers, will I have the same issues? ie. a single bad device will mess up the others? iii) Does OmniOS even support SATA HBA connected to SATA port multipliers? Perhaps tracking errors on devices will allow for preventative action... thanx - steve > > -- > > ZFS storage and performance consulting at http://www.RichardElling.com > > > > > > > From steve at linuxsuite.org Mon Aug 19 16:37:08 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 12:37:08 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> Message-ID: <55a2ac85d52226ae8beb9470c1d64cb2.squirrel@emailmg.netfirms.com> >> SNIP >> >> c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 >> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >> W1F09G4Q >> Size: 3000.59GB <3000592982016 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 >> Illegal Request: 1 Predictive Failure Analysis: 0 > > Something is sick. Unfortunately, it could be one of the many SATA devices > causing disruptions to everyone else -- a good reason to not directly > attach > SATA devices to SAS expanders. Tracking these down and eliminating a bug > in the expander itself, is a tedious task. Is there anything in the JBOD > other than > the Seagate 3TBs? > -- richard Thanks Richard... There are 11 Seagate 3T SATA drives on a single backplane. 10 in a RAIDZ-2 and a hot spare. There are no other devices in the JBOD. ZIL is on a separate SSD in a DELL chassis So a couple of questions i) If there is a single device that has a "large" number of errors compared to the other devices, then perhaps this device is messing up the others? So swapping it out could solve the problem? ii) If I had the SATA devices behind SATA port multipliers, will I have the same issues? ie. a single bad device will mess up the others? iii) Does OmniOS even support SATA HBA connected to SATA port multipliers? Perhaps tracking errors on devices will allow for preventative action... thanx - steve > > -- > > ZFS storage and performance consulting at http://www.RichardElling.com > > > > > > > From richard.elling at richardelling.com Mon Aug 19 17:16:31 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Mon, 19 Aug 2013 10:16:31 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> Message-ID: <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> Hi Steve, On Aug 19, 2013, at 9:37 AM, steve at linuxsuite.org wrote: >>> > SNIP >>> >>> c5t5000C50045561CEAd0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 7 >>> Vendor: ATA Product: ST3000DM001-9YN1 Revision: CC4H Serial No: >>> W1F09G4Q >>> Size: 3000.59GB <3000592982016 bytes> >>> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 >>> Illegal Request: 1 Predictive Failure Analysis: 0 >> >> Something is sick. Unfortunately, it could be one of the many SATA devices >> causing disruptions to everyone else -- a good reason to not directly >> attach >> SATA devices to SAS expanders. Tracking these down and eliminating a bug >> in the expander itself, is a tedious task. Is there anything in the JBOD >> other than >> the Seagate 3TBs? >> -- richard > > Thanks Richard... > > There are 11 Seagate 3T SATA drives on a single backplane. 10 > in a RAIDZ-2 and a hot spare. There are no other devices in the JBOD. > ZIL is on a separate SSD in a DELL chassis > > So a couple of questions > > i) If there is a single device that has a "large" number of errors > compared to the other devices, then perhaps this device is > messing up the others? So swapping it out could solve the problem? In my experience, no. The exact errors and their timeline matters. Coincidently, I am looking at methods to approach this problem :-) I'll contact you off-alias for details. > > ii) If I had the SATA devices behind SATA port multipliers, will I > have the same issues? ie. a single bad device will mess up the others? Port multipliers are evil, I avoid them at all costs. The successful solutions I have seen work are, in no particular order: 1. pay the extra $10 and get the SAS version of the disk 2. pay $30 and get a SAS-SATA interposer, hopefully your carrier can accommodate 3. go direct, get a backplane without an expander (eg Supermicro TQ series) -- richard > > iii) Does OmniOS even support SATA HBA connected to SATA port multipliers? > > Perhaps tracking errors on devices will allow for preventative > action... > > thanx - steve > >> >> -- >> >> ZFS storage and performance consulting at http://www.RichardElling.com >> >> >> >> >> >> >> > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at linuxsuite.org Mon Aug 19 20:35:53 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 16:35:53 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> Message-ID: <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> >> Thanks Richard... >> >> There are 11 Seagate 3T SATA drives on a single backplane. 10 >> in a RAIDZ-2 and a hot spare. There are no other devices in the JBOD. >> ZIL is on a separate SSD in a DELL chassis >> >> So a couple of questions >> >> i) If there is a single device that has a "large" number of errors >> compared to the other devices, then perhaps this device is >> messing up the others? So swapping it out could solve the problem? > > In my experience, no. The exact errors and their timeline matters. > Coincidently, I am looking at methods to approach this problem :-) > I'll contact you off-alias for details. > >> >> ii) If I had the SATA devices behind SATA port multipliers, will I >> have the same issues? ie. a single bad device will mess up the >> others? > > Port multipliers are evil, I avoid them at all costs. > > The successful solutions I have seen work are, in no particular order: > 1. pay the extra $10 and get the SAS version of the disk > 2. pay $30 and get a SAS-SATA interposer, hopefully your carrier can > accommodate > 3. go direct, get a backplane without an expander (eg Supermicro TQ > series) > Hahah, you jokin right?? You know were to get 3T/4T SAS disks that are only $10 more than the SATA version??? Everywhere I look SAS is around twice the price?? Nearline is same. If I could get SAS for $10 more I would be in Storage Heaven. thanx very much - steve > -- richard > >> >> iii) Does OmniOS even support SATA HBA connected to SATA port >> multipliers? >> >> Perhaps tracking errors on devices will allow for preventative >> action... >> >> thanx - steve >> >>> >>> -- >>> >>> ZFS storage and performance consulting at http://www.RichardElling.com >>> >>> >>> >>> >>> >>> >>> >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- > > Richard.Elling at RichardElling.com > +1-760-896-4422 > > > > From jdg117 at elvis.arl.psu.edu Mon Aug 19 20:51:52 2013 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Mon, 19 Aug 2013 16:51:52 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: Your message of "Mon, 19 Aug 2013 16:35:53 EDT." <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> Message-ID: <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> In message <9c89f21353b8385e37a7c96f2dd13962.squirrel at emailmg.netfirms.com>, st eve at linuxsuite.org writes: > Hahah, you jokin right?? You know were to get 3T/4T SAS disks >that are only $10 more than the SATA version??? Everywhere >I look SAS is around twice the price?? Nearline is same. If I could >get SAS for $10 more I would be in Storage Heaven. John groenveld at acm.org From steve at linuxsuite.org Mon Aug 19 21:35:55 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 17:35:55 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> Message-ID: <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> > In message > <9c89f21353b8385e37a7c96f2dd13962.squirrel at emailmg.netfirms.com>, st > eve at linuxsuite.org writes: >> Hahah, you jokin right?? You know were to get 3T/4T SAS disks >>that are only $10 more than the SATA version??? Everywhere >>I look SAS is around twice the price?? Nearline is same. If I could >>get SAS for $10 more I would be in Storage Heaven. > > > > > 4T SAS drives as per above are $359 4T SATA here http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 are $179 359 - 179 = 180 180 != 10 -steve > John > groenveld at acm.org > > From esproul at omniti.com Mon Aug 19 21:47:11 2013 From: esproul at omniti.com (Eric Sproul) Date: Mon, 19 Aug 2013 17:47:11 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> Message-ID: On Mon, Aug 19, 2013 at 5:35 PM, wrote: > 4T SATA here > > http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 > > are $179 That's a desktop drive, not nearline enterprise, so it's apples-oranges. You get what you pay for. Desktop drives can take much, much longer to respond to commands, leading the HBA/expander to declare them dead and reset them, and the rest is history. Avoid desktop parts. :) Eric From jdg117 at elvis.arl.psu.edu Mon Aug 19 21:55:12 2013 From: jdg117 at elvis.arl.psu.edu (John D Groenveld) Date: Mon, 19 Aug 2013 17:55:12 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: Your message of "Mon, 19 Aug 2013 17:47:11 EDT." References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> Message-ID: <201308192155.r7JLtCLk029255@elvis.arl.psu.edu> In message , Eric Sproul writes: >Avoid desktop parts. :) Looks like Amazon is charging $0 difference between Seagate's "Enterprise" 4TB SAS and SATA drives John groenveld at acm.org From steve at linuxsuite.org Mon Aug 19 23:25:43 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 19:25:43 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> Message-ID: <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> > Can't agree more here. Desktop firmwares are designed to try harder to > never return an error. While this is what home users want, it's an > anathema to large configurations with redundancy where you would prefer to > just get the error so you can handle it - usually by doing the op on > another drive. > > Use enterprise drives if you love your data and your ability to access it. Yes I know, I know.... Desktop hardware sucks. Didn't know there was Enterprise (ie none Desktop) SATA. Perhaps the solution is to partition the data and do "deep" storage that never gets read on large bulk SATA and then have a smaller SAS pool for the often read/written stuff. Anyway, large storage rollouts will always be looking for the cheapest solution. Even if it isn't perfect. I do backups to a SATA pool (behind SAS expanders) with zfs send,.. no problems errors whatever YET!! (cross fingers) -steve > > Sent from my iPhone > > On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: > >> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>> 4T SATA here >>> >>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>> >>> are $179 >> >> That's a desktop drive, not nearline enterprise, so it's >> apples-oranges. You get what you pay for. Desktop drives can take >> much, much longer to respond to commands, leading the HBA/expander to >> declare them dead and reset them, and the rest is history. >> >> Avoid desktop parts. :) >> >> Eric >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > From steve at linuxsuite.org Mon Aug 19 23:29:20 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Mon, 19 Aug 2013 19:29:20 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> Message-ID: > Can't agree more here. Desktop firmwares are designed to try harder to > never return an error. While this is what home users want, it's an > anathema to large configurations with redundancy where you would prefer to > just get the error so you can handle it - usually by doing the op on > another drive. > So the big difference is mostly firmware ?? Can you flash a new firmware??? Why do I think we are being ripped off?? -steve > Use enterprise drives if you love your data and your ability to access it. > > Sent from my iPhone > > On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: > >> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>> 4T SATA here >>> >>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>> >>> are $179 >> >> That's a desktop drive, not nearline enterprise, so it's >> apples-oranges. You get what you pay for. Desktop drives can take >> much, much longer to respond to commands, leading the HBA/expander to >> declare them dead and reset them, and the rest is history. >> >> Avoid desktop parts. :) >> >> Eric >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > From heinz at licenser.net Tue Aug 20 06:59:01 2013 From: heinz at licenser.net (Heinz Nikolaus Gies) Date: Tue, 20 Aug 2013 08:59:01 +0200 Subject: [OmniOS-discuss] rsyslogd package on ms.omniti.com Message-ID: Hi a little 'bug' report: the rsyslogd package does not include the config file it uses as a package file: /opt/omni/etc/rsyslogd.conf this leads to the effect that the file gets removed when the last other package using /opt/omni/etc gets removed. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From garrett.damore at dey-sys.com Mon Aug 19 22:25:20 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Mon, 19 Aug 2013 15:25:20 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> Message-ID: <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> Can't agree more here. Desktop firmwares are designed to try harder to never return an error. While this is what home users want, it's an anathema to large configurations with redundancy where you would prefer to just get the error so you can handle it - usually by doing the op on another drive. Use enterprise drives if you love your data and your ability to access it. Sent from my iPhone On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: > On Mon, Aug 19, 2013 at 5:35 PM, wrote: >> 4T SATA here >> >> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >> >> are $179 > > That's a desktop drive, not nearline enterprise, so it's > apples-oranges. You get what you pay for. Desktop drives can take > much, much longer to respond to commands, leading the HBA/expander to > declare them dead and reset them, and the rest is history. > > Avoid desktop parts. :) > > Eric > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From garrett.damore at dey-sys.com Tue Aug 20 01:33:11 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Mon, 19 Aug 2013 18:33:11 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> Message-ID: SATA drives behind SAS expanders have a pathological error case. If a drive encounters errors and it needs to be reset then the entire set of drives will take the reset. If any io is in flight this will error as a result often causing another reset. The result can best be described as a cascade failure. There may be things software can do to mitigate this but experience is that it doesn't. So just because the array is working fine now does not mean that it won't fail tragically when the first problems occur. I have personal experience with this failure mode. As a result I strongly discourage the use of SATA drives unless they are directly connected to the hba without any expanders or port multipliers. You ignore this advice at your own risk. Don't penny pinch on the drives It ALWAYS costs you in the long run. Sent from my iPhone On Aug 19, 2013, at 4:25 PM, steve at linuxsuite.org wrote: >> Can't agree more here. Desktop firmwares are designed to try harder to >> never return an error. While this is what home users want, it's an >> anathema to large configurations with redundancy where you would prefer to >> just get the error so you can handle it - usually by doing the op on >> another drive. >> >> Use enterprise drives if you love your data and your ability to access it. > > Yes I know, I know.... Desktop hardware sucks. Didn't know there was > Enterprise (ie none Desktop) SATA. Perhaps the solution is to partition > the data and do "deep" storage that never gets read on large bulk SATA and > then have a smaller SAS pool for the often read/written stuff. > > Anyway, large storage rollouts will always be looking for > the cheapest solution. Even if it isn't perfect. I do backups > to a SATA pool (behind SAS expanders) with zfs send,.. no problems errors > whatever YET!! > (cross fingers) > > -steve > >> >> Sent from my iPhone >> >> On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: >> >>> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>>> 4T SATA here >>>> >>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>>> >>>> are $179 >>> >>> That's a desktop drive, not nearline enterprise, so it's >>> apples-oranges. You get what you pay for. Desktop drives can take >>> much, much longer to respond to commands, leading the HBA/expander to >>> declare them dead and reset them, and the rest is history. >>> >>> Avoid desktop parts. :) >>> >>> Eric >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From garrett.damore at dey-sys.com Tue Aug 20 01:34:24 2013 From: garrett.damore at dey-sys.com (Garrett D'Amore) Date: Mon, 19 Aug 2013 18:34:24 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> Message-ID: <2D96DC6C-736E-418E-89B9-D4FF281479EF@dey-sys.com> Not just firmware but level if test, overprovisioning to cope with errors etc. Don't be cheap. Trust me. You will regret it. Sent from my iPhone On Aug 19, 2013, at 4:29 PM, steve at linuxsuite.org wrote: >> Can't agree more here. Desktop firmwares are designed to try harder to >> never return an error. While this is what home users want, it's an >> anathema to large configurations with redundancy where you would prefer to >> just get the error so you can handle it - usually by doing the op on >> another drive. > > So the big difference is mostly firmware ?? Can you flash a new > firmware??? > > Why do I think we are being ripped off?? > > -steve > >> Use enterprise drives if you love your data and your ability to access it. >> >> Sent from my iPhone >> >> On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: >> >>> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>>> 4T SATA here >>>> >>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>>> >>>> are $179 >>> >>> That's a desktop drive, not nearline enterprise, so it's >>> apples-oranges. You get what you pay for. Desktop drives can take >>> much, much longer to respond to commands, leading the HBA/expander to >>> declare them dead and reset them, and the rest is history. >>> >>> Avoid desktop parts. :) >>> >>> Eric >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From esproul at omniti.com Tue Aug 20 15:12:03 2013 From: esproul at omniti.com (Eric Sproul) Date: Tue, 20 Aug 2013 11:12:03 -0400 Subject: [OmniOS-discuss] rsyslogd package on ms.omniti.com In-Reply-To: References: Message-ID: On Tue, Aug 20, 2013 at 2:59 AM, Heinz Nikolaus Gies wrote: > Hi a little 'bug' report: > the rsyslogd package does not include the config file it uses as a package file: > > /opt/omni/etc/rsyslogd.conf > > this leads to the effect that the file gets removed when the last other package using /opt/omni/etc gets removed. We could ship a default config file with the package, and mark it as editable. That way pkg would be aware of it and would preserve local content. What would be sensible default content for rsyslogd.conf? I don't know that product well. Eric From heinz at licenser.net Tue Aug 20 15:15:29 2013 From: heinz at licenser.net (Heinz Nikolaus Gies) Date: Tue, 20 Aug 2013 17:15:29 +0200 Subject: [OmniOS-discuss] rsyslogd package on ms.omniti.com In-Reply-To: References: Message-ID: <883BB479-B825-4A26-B4CD-34FCCAD8FC7A@licenser.net> Hi Eric, thanks for the reply :). To be honest I'm not a expert on rsyslog either I just want to include it to ship data to papertrais so SmartOS includes the following default configu: https://gist.github.com/Licenser/6282774 which looks rather sensible to me. Cheers, Heinz On Aug 20, 2013, at 17:12, Eric Sproul wrote: > On Tue, Aug 20, 2013 at 2:59 AM, Heinz Nikolaus Gies wrote: >> Hi a little 'bug' report: >> the rsyslogd package does not include the config file it uses as a package file: >> >> /opt/omni/etc/rsyslogd.conf >> >> this leads to the effect that the file gets removed when the last other package using /opt/omni/etc gets removed. > > We could ship a default config file with the package, and mark it as > editable. That way pkg would be aware of it and would preserve local > content. What would be sensible default content for rsyslogd.conf? I > don't know that product well. > > Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From heinz at licenser.net Tue Aug 20 15:39:49 2013 From: heinz at licenser.net (Heinz Nikolaus Gies) Date: Tue, 20 Aug 2013 17:39:49 +0200 Subject: [OmniOS-discuss] Installing OmniOS Message-ID: <14632A0B-D4BE-44EE-A0BF-F8AE5CC9CC74@licenser.net> Hi, I've been trying (and finally succeeded after about 4 days) in installing OmniOS and wanted to give a little feedback. Since it was a very frustrating experience let me start with saying that, despite that I really enjoy OmniOS now that it is on the system ;) and that your guys work on the system is hugely appriciated. But I think that the install process can be improved, a lot. From what I understand there are basically two ways to go, kayak, and the installer both have their problems, lets start with kayak sice I don't have much to say to that it's the path I did not choose. Kayak might work well to add additional systems to a existing OmniOS installation but when starting off w/o a existing OmniOS system poses some problems, mostly the 'install wit hkayak from a non OmniOS OS' tels you to install OmniOS to copy the required files. So as a suggestion of improvment: package the required files somewhere easiely downlaodable and put a link in the manual. (If that link had existed I propably had went with kayak and you missed all the nice bugs I found in the isntaller ;) The isntaller - it crashes, a lot, a lot lot. It crashes when: - using 3TB disks - entering partition sizes that do not fit (even so are not obviouysly wrong). i.e. if it tells you 110.8 GB are available you enter 110.8 -> crash (I guess it was something like 110.79). - you select a disk with UEFI labels. - randomly when cycling between parition types. - basically any other place where you can enter data and it is not a perfectly valid value. The problem with the crashing is that you need to start form scratch, and that can be painful. In addition to that: It does not allow you to select slices unless you manaually created more then one before hand. This can be very annoying since you need to drop out of the instaler to do that. It can get into a state during slice selection where you can't change the type of slices any more, none could be marked as rpool Generally unless you want a 1 full disk spanning rpool it will be a very painful experience requiring a lot of work in the console beforehand. So now the 'complaining' is over on the installer and here a suggestion how to improve. Since it will propably be very hard to make a good graphical installer and get rid of all the bugs (even so the crashes might be worth fixing) it would be very cool if there was a way to drop in the console, set up the disks/pools/etc and just tell the system: please skip the graphical thing, i created the rpool for you, now install the system in it and put grub onto disk Cheers, Heinz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From lists at mcintyreweb.com Tue Aug 20 18:09:18 2013 From: lists at mcintyreweb.com (Hugh McIntyre) Date: Tue, 20 Aug 2013 11:09:18 -0700 Subject: [OmniOS-discuss] Installing OmniOS In-Reply-To: <14632A0B-D4BE-44EE-A0BF-F8AE5CC9CC74@licenser.net> References: <14632A0B-D4BE-44EE-A0BF-F8AE5CC9CC74@licenser.net> Message-ID: <5044bc9bf7f65e1fe1c835edb1beaf21.squirrel@www.mcintyreweb.com> On Tue, August 20, 2013 8:39 am, Heinz Nikolaus Gies wrote: > Since it will propably be very hard to make a good graphical installer and > get rid of all the bugs (even so the crashes might be worth fixing) it > would be very cool if there was a way to drop in the console, set up the > disks/pools/etc and just tell the system: please skip the graphical thing, > i created the rpool for you, now install the system in it and put grub > onto disk There's already an option to drop into a shell when you boot from install media. At which point you can create disk partitions or anything else you need. Then quit the shell, run the installer and install into that disk. I think when I did this last time I just partitioned the disk in the shell (i.e. no creating rpool), then the menu-based installer created rpool as part of the install. But you should be able to do what you want already. Hugh. From mir at miras.org Tue Aug 20 20:15:20 2013 From: mir at miras.org (Michael Rasmussen) Date: Tue, 20 Aug 2013 22:15:20 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= Message-ID: Hi list, Hos does omnios handle 4k sectored also known as advanced format? Will it be 1MB aligned automaticaly or should I tale any speciale concerns? Regards, Michael From basil.crow at delphix.com Tue Aug 20 22:58:01 2013 From: basil.crow at delphix.com (Basil Crow) Date: Tue, 20 Aug 2013 15:58:01 -0700 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: References: Message-ID: See this page for more information about 4K sectors and ZFS: http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks On Tue, Aug 20, 2013 at 1:15 PM, Michael Rasmussen wrote: > Hi list, > Hos does omnios handle 4k sectored also known as advanced format? Will it be 1MB aligned automaticaly or should I tale any speciale concerns? > > Regards, > Michael From mattfrazer at gmail.com Wed Aug 21 01:40:31 2013 From: mattfrazer at gmail.com (Matthew Frazer) Date: Tue, 20 Aug 2013 21:40:31 -0400 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: References: Message-ID: This is a question I've been meaning to ask as well. Using the information at http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks I've edited the sd.conf on my systems to include the drives I wish to have 4k sector sizes on. On Solaris 11.1 this achieves the proper ashift on 4k drives whether they report 4k or 512e to the system. The same sd.conf does not, at least in my experience, result in an ashift of 12 on OmniOS. My workaround has been to create my pools in 11.1 with version flags set compatible to what omnios supports, then import them into OmniOS and then zfs upgrade them. I'd love to find out I've been doing it wrong as it would simplify pool creation greatly for me. -mjf -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.elling at richardelling.com Wed Aug 21 04:17:36 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Tue, 20 Aug 2013 21:17:36 -0700 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: References: Message-ID: <35B6A902-0DAD-43E8-A83D-8D8B4CE12FCE@RichardElling.com> On Aug 20, 2013, at 6:40 PM, Matthew Frazer wrote: > This is a question I've been meaning to ask as well. > > Using the information at http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks I've edited the sd.conf on my systems to include the drives I wish to have 4k sector sizes on. On Solaris 11.1 this achieves the proper ashift on 4k drives whether they report 4k or 512e to the system. The same sd.conf does not, at least in my experience, result in an ashift of 12 on OmniOS. My workaround has been to create my pools in 11.1 with version flags set compatible to what omnios supports, then import them into OmniOS and then zfs upgrade them. I'd love to find out I've been doing it wrong as it would simplify pool creation greatly for me. > The file is very sensitive to spaces and format, including separators. Make sure you have the file exactly as needed. -- richard -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aney at ivision.fr Wed Aug 21 13:43:23 2013 From: aney at ivision.fr (Alexandre NEY) Date: Wed, 21 Aug 2013 13:43:23 +0000 Subject: [OmniOS-discuss] Using zpool iostat Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> Hello, I have a weird issue and i'm wondering if you guys experience the same behavior. I'm trying to get some simple IOPS statistics of my pool?: SunOS ivi-zfs01 5.11 omnios-b281e50 i86pc i386 i86pc root at ivi-zfs01:~# zpool iostat capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- POOL01 25.8T 32.2T 292 567 32.1M 25.8M rpool 9.02G 102G 3 6 20.2K 41.4K ---------- ----- ----- ----- ----- ----- ----- All looks good except that there is no activity on this pool. If i run the same command over and over again the exact same statistics will appear. If i run root at ivi-zfs01:~# zpool iostat 1 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- POOL01 25.8T 32.2T 292 567 32.1M 25.8M rpool 9.02G 102G 3 6 20.2K 41.4K ---------- ----- ----- ----- ----- ----- ----- POOL01 25.8T 32.2T 0 0 0 0 rpool 9.02G 102G 0 0 0 0 ---------- ----- ----- ----- ----- ----- ----- The second output reflects reality (no activity) Any thoughts ? Thanks ! Alexandre NEY From esproul at omniti.com Wed Aug 21 14:09:28 2013 From: esproul at omniti.com (Eric Sproul) Date: Wed, 21 Aug 2013 10:09:28 -0400 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> Message-ID: On Wed, Aug 21, 2013 at 9:43 AM, Alexandre NEY wrote: > All looks good except that there is no activity on this pool. If i run the same command over and over again the exact same statistics will appear. > > If i run > > root at ivi-zfs01:~# zpool iostat 1 > capacity operations bandwidth > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > POOL01 25.8T 32.2T 292 567 32.1M 25.8M > rpool 9.02G 102G 3 6 20.2K 41.4K > ---------- ----- ----- ----- ----- ----- ----- > POOL01 25.8T 32.2T 0 0 0 0 > rpool 9.02G 102G 0 0 0 0 > ---------- ----- ----- ----- ----- ----- ----- > > The second output reflects reality (no activity) Like iostat and similar commands, the first line or set of output represents an average since boot, with the following lines/sets showing the numbers for that interval. Obviously the first question is, *should* there be activity to the pool? ZFS bundles up async writes and flushes them every 5 seconds, or more often if there is heavy write traffic. If you watch the output for 10-15 seconds you should see activity, assuming something is reading and/or writing on that pool. Eric From aney at ivision.fr Wed Aug 21 19:00:20 2013 From: aney at ivision.fr (Alexandre NEY) Date: Wed, 21 Aug 2013 19:00:20 +0000 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local> Alright then ! Thank you for the information, I wasn't aware of this first output thing and couldn't find anything in the man page referring to that. So everything is working as intended. Do you know if there is a way to bypass that first output to directly get the *actual* statistics and not *since boot* ? Thanks ! -----Message d'origine----- De?: Eric Sproul [mailto:esproul at omniti.com] Envoy??: mercredi 21 ao?t 2013 16:09 ??: Alexandre NEY Cc?: omnios-discuss at lists.omniti.com Objet?: Re: [OmniOS-discuss] Using zpool iostat On Wed, Aug 21, 2013 at 9:43 AM, Alexandre NEY wrote: > All looks good except that there is no activity on this pool. If i run the same command over and over again the exact same statistics will appear. > > If i run > > root at ivi-zfs01:~# zpool iostat 1 > capacity operations bandwidth > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > POOL01 25.8T 32.2T 292 567 32.1M 25.8M > rpool 9.02G 102G 3 6 20.2K 41.4K > ---------- ----- ----- ----- ----- ----- ----- > POOL01 25.8T 32.2T 0 0 0 0 > rpool 9.02G 102G 0 0 0 0 > ---------- ----- ----- ----- ----- ----- ----- > > The second output reflects reality (no activity) Like iostat and similar commands, the first line or set of output represents an average since boot, with the following lines/sets showing the numbers for that interval. Obviously the first question is, *should* there be activity to the pool? ZFS bundles up async writes and flushes them every 5 seconds, or more often if there is heavy write traffic. If you watch the output for 10-15 seconds you should see activity, assuming something is reading and/or writing on that pool. Eric From esproul at omniti.com Wed Aug 21 19:23:55 2013 From: esproul at omniti.com (Eric Sproul) Date: Wed, 21 Aug 2013 15:23:55 -0400 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local> Message-ID: On Wed, Aug 21, 2013 at 3:00 PM, Alexandre NEY wrote: > Alright then ! > > Thank you for the information, I wasn't aware of this first output thing and couldn't find anything in the man page referring to that. So everything is working as intended. Do you know if there is a way to bypass that first output to directly get the *actual* statistics and not *since boot* ? > > Thanks ! For what purpose? These are meant to be human-readable, but it sounds like you want to scrape the output for monitoring purposes. There's a better way, and it's called kstat(1). :) # kstat -p 'zfs:0:rpool:reads' zfs:0:rpool:reads 7719 # kstat -p 'zfs:0:rpool:writes' zfs:0:rpool:writes 22348 These are counters maintained by the kernel, so you can derive over time to graph I/O activity, for example. There are programmatic interfaces to kstat as well, if that interests you. See libkstat(3LIB), Kstat(3PERL). Eric From aney at ivision.fr Wed Aug 21 19:53:39 2013 From: aney at ivision.fr (Alexandre NEY) Date: Wed, 21 Aug 2013 19:53:39 +0000 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local>, Message-ID: <6sk12dcxujv3ek8yg0st6jdo.1377114860917@email.android.com> You are the man. Thanks! Eric Sproul a ?crit : On Wed, Aug 21, 2013 at 3:00 PM, Alexandre NEY wrote: > Alright then ! > > Thank you for the information, I wasn't aware of this first output thing and couldn't find anything in the man page referring to that. So everything is working as intended. Do you know if there is a way to bypass that first output to directly get the *actual* statistics and not *since boot* ? > > Thanks ! For what purpose? These are meant to be human-readable, but it sounds like you want to scrape the output for monitoring purposes. There's a better way, and it's called kstat(1). :) # kstat -p 'zfs:0:rpool:reads' zfs:0:rpool:reads 7719 # kstat -p 'zfs:0:rpool:writes' zfs:0:rpool:writes 22348 These are counters maintained by the kernel, so you can derive over time to graph I/O activity, for example. There are programmatic interfaces to kstat as well, if that interests you. See libkstat(3LIB), Kstat(3PERL). Eric From esproul at omniti.com Wed Aug 21 20:06:59 2013 From: esproul at omniti.com (Eric Sproul) Date: Wed, 21 Aug 2013 16:06:59 -0400 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: <6sk12dcxujv3ek8yg0st6jdo.1377114860917@email.android.com> References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local> <6sk12dcxujv3ek8yg0st6jdo.1377114860917@email.android.com> Message-ID: kstat is awesome, and has tens of thousands of statistics from all over the kernel. On one of my r151006 global zones: $ kstat -l | wc -l 39754 From valrhona at gmail.com Wed Aug 21 21:33:53 2013 From: valrhona at gmail.com (Valrhona) Date: Wed, 21 Aug 2013 17:33:53 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> Message-ID: I have a Dell tower server with a 16-bay 2.5" SAS hotswap backplane. I have been using small SAS drives (36 GB) for the mirrored rootpool, but these are tending to fail (because they are so old). I tried one of the Intel SLC 20 GB SSD drives (311), and that seemed to work in the backplane (some other SATA drives screwed up everything, so I pulled them out). No harm done in testing, but is this a bad idea in general? Will this screw up the rest of the SAS drives in the same 8-drive channel? I would love to get a slow, small (<32 GB) SAS SSD for the rpool, but no one seems to sell one. Just needs to be reliable, and with the rpool being overwhelmingly read-dominated, this doesn's seem too hard. Even Intel's "enterprise" SSDs (DC3500, 3700) are SATA, not SAS. And obviously it's silly to spend $300+ each on two 100 GB SAS drives from Seagate (the cheapest i could find). I don't want to run off of USB sticks, either, for reliability reasons. Any ideas? Thanks! Peter On Mon, Aug 19, 2013 at 9:33 PM, Garrett D'Amore wrote: > SATA drives behind SAS expanders have a pathological error case. If a drive encounters errors and it needs to be reset then the entire set of drives will take the reset. If any io is in flight this will error as a result often causing another reset. The result can best be described as a cascade failure. > > There may be things software can do to mitigate this but experience is that it doesn't. > > So just because the array is working fine now does not mean that it won't fail tragically when the first problems occur. > > I have personal experience with this failure mode. > > As a result I strongly discourage the use of SATA drives unless they are directly connected to the hba without any expanders or port multipliers. > > You ignore this advice at your own risk. Don't penny pinch on the drives It ALWAYS costs you in the long run. > > Sent from my iPhone > > On Aug 19, 2013, at 4:25 PM, steve at linuxsuite.org wrote: > >>> Can't agree more here. Desktop firmwares are designed to try harder to >>> never return an error. While this is what home users want, it's an >>> anathema to large configurations with redundancy where you would prefer to >>> just get the error so you can handle it - usually by doing the op on >>> another drive. >>> >>> Use enterprise drives if you love your data and your ability to access it. >> >> Yes I know, I know.... Desktop hardware sucks. Didn't know there was >> Enterprise (ie none Desktop) SATA. Perhaps the solution is to partition >> the data and do "deep" storage that never gets read on large bulk SATA and >> then have a smaller SAS pool for the often read/written stuff. >> >> Anyway, large storage rollouts will always be looking for >> the cheapest solution. Even if it isn't perfect. I do backups >> to a SATA pool (behind SAS expanders) with zfs send,.. no problems errors >> whatever YET!! >> (cross fingers) >> >> -steve >> >>> >>> Sent from my iPhone >>> >>> On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: >>> >>>> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>>>> 4T SATA here >>>>> >>>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>>>> >>>>> are $179 >>>> >>>> That's a desktop drive, not nearline enterprise, so it's >>>> apples-oranges. You get what you pay for. Desktop drives can take >>>> much, much longer to respond to commands, leading the HBA/expander to >>>> declare them dead and reset them, and the rest is history. >>>> >>>> Avoid desktop parts. :) >>>> >>>> Eric >>>> _______________________________________________ >>>> OmniOS-discuss mailing list >>>> OmniOS-discuss at lists.omniti.com >>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From richard.elling at richardelling.com Wed Aug 21 22:48:31 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 21 Aug 2013 15:48:31 -0700 Subject: [OmniOS-discuss] Using zpool iostat In-Reply-To: References: <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FD484@ivision-mailbox.Exchange.local> <8FB4CA43FEE0C5468AC9AD5F74F1BD43129FE07B@ivision-mailbox.Exchange.local> <6sk12dcxujv3ek8yg0st6jdo.1377114860917@email.android.com> Message-ID: <890B6744-3B7E-4D25-AB72-2D711FCEDF0B@RichardElling.com> FYI, iostat in illumos shows pool stats too :-) Personally, I don't find "zpool iostat" to be very useful, especially now that iostat knows about pools. -- richard On Aug 21, 2013, at 1:06 PM, Eric Sproul wrote: > kstat is awesome, and has tens of thousands of statistics from all > over the kernel. On one of my r151006 global zones: > > $ kstat -l | wc -l > 39754 > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at linuxsuite.org Wed Aug 21 22:53:03 2013 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Wed, 21 Aug 2013 18:53:03 -0400 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> Message-ID: <0782b346b45c0ed0abc557c483abaf5f.squirrel@emailmg.netfirms.com> > SATA drives behind SAS expanders have a pathological error case. If a > drive encounters errors and it needs to be reset then the entire set of > drives will take the reset. If any io is in flight this will error as a > result often causing another reset. The result can best be described as a > cascade failure. > > There may be things software can do to mitigate this but experience is > that it doesn't. Hmmm..... I have put. allow-bus-device-reset=0; In sd.conf as it seemed to mitigate that problem? > > So just because the array is working fine now does not mean that it won't > fail tragically when the first problems occur. > > I have personal experience with this failure mode. > > As a result I strongly discourage the use of SATA drives unless they are > directly connected to the hba without any expanders or port multipliers. > > You ignore this advice at your own risk. Don't penny pinch on the drives > It ALWAYS costs you in the long run. > The advice in this thread, wrt SATA vs SAS, has been my thoughts and position for a while based on my initial research. I was required to explore and verify this option simply because of the significant economic benefit if it was possible. Thanks for your input. I will use this to make my case So now I just need some cheap SAS drives ;-) thanx - steve > Sent from my iPhone > > On Aug 19, 2013, at 4:25 PM, steve at linuxsuite.org wrote: > >>> Can't agree more here. Desktop firmwares are designed to try harder to >>> never return an error. While this is what home users want, it's an >>> anathema to large configurations with redundancy where you would prefer >>> to >>> just get the error so you can handle it - usually by doing the op on >>> another drive. >>> >>> Use enterprise drives if you love your data and your ability to access >>> it. >> >> Yes I know, I know.... Desktop hardware sucks. Didn't know there >> was >> Enterprise (ie none Desktop) SATA. Perhaps the solution is to partition >> the data and do "deep" storage that never gets read on large bulk SATA >> and >> then have a smaller SAS pool for the often read/written stuff. >> >> Anyway, large storage rollouts will always be looking for >> the cheapest solution. Even if it isn't perfect. I do backups >> to a SATA pool (behind SAS expanders) with zfs send,.. no problems >> errors >> whatever YET!! >> (cross fingers) >> >> -steve >> >>> >>> Sent from my iPhone >>> >>> On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: >>> >>>> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>>>> 4T SATA here >>>>> >>>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>>>> >>>>> are $179 >>>> >>>> That's a desktop drive, not nearline enterprise, so it's >>>> apples-oranges. You get what you pay for. Desktop drives can take >>>> much, much longer to respond to commands, leading the HBA/expander to >>>> declare them dead and reset them, and the rest is history. >>>> >>>> Avoid desktop parts. :) >>>> >>>> Eric >>>> _______________________________________________ >>>> OmniOS-discuss mailing list >>>> OmniOS-discuss at lists.omniti.com >>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > From mir at miras.org Wed Aug 21 23:13:37 2013 From: mir at miras.org (Michael Rasmussen) Date: Thu, 22 Aug 2013 01:13:37 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: References: Message-ID: <20130822011337.3b08d758@sleipner.datanom.net> On Tue, 20 Aug 2013 15:58:01 -0700 Basil Crow wrote: > See this page for more information about 4K sectors and ZFS: > > http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks Have tried everything from this wiki but no matter what I do I am not able to have it aligned 4k. filebench given this stats (a mirrored vdev): 150541 ops, 2492.698 ops/s, (226/453 r/w), 59.4mb/s, 898us cpu/op, 71.1ms latency Another mirrored vdev consisting of 512B disks gives this stats: 1684377 ops, 27581.072 ops/s, (2508/5016 r/w), 668.3mb/s, 249us cpu/op, 3.9ms latency Both vdevs are connect via an LSI 1068E HBA -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- "I'd love to go out with you, but the man on television told me to stay tuned." -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From richard.elling at richardelling.com Wed Aug 21 23:30:56 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 21 Aug 2013 16:30:56 -0700 Subject: [OmniOS-discuss] How bad are these controller / io errors?? In-Reply-To: <0782b346b45c0ed0abc557c483abaf5f.squirrel@emailmg.netfirms.com> References: <214ba2cd7ed4cb241e3a4610a78db4df.squirrel@emailmg.netfirms.com> <0db5a36c12a4acfcae804a2389ad5051.squirrel@emailmg.netfirms.com> <6CB50FF3-586B-4AFF-95F3-0C570CC5B58A@RichardElling.com> <9c89f21353b8385e37a7c96f2dd13962.squirrel@emailmg.netfirms.com> <201308192051.r7JKpqVQ026587@elvis.arl.psu.edu> <35aecb5d907b25ea5f7c9052a21eaa5d.squirrel@emailmg.netfirms.com> <29CAC6D6-D96C-4224-AA47-15D6A94FEC63@dey-sys.com> <4e719ab741c958617ddee6c0ccc76059.squirrel@emailmg.netfirms.com> <0782b346b45c0ed0abc557c483abaf5f.squirrel@emailmg.netfirms.com> Message-ID: <9182A83F-082F-4932-9280-D41A139F6E29@RichardElling.com> On Aug 21, 2013, at 3:53 PM, steve at linuxsuite.org wrote: >> SATA drives behind SAS expanders have a pathological error case. If a >> drive encounters errors and it needs to be reset then the entire set of >> drives will take the reset. If any io is in flight this will error as a >> result often causing another reset. The result can best be described as a >> cascade failure. >> >> There may be things software can do to mitigate this but experience is >> that it doesn't. > > Hmmm..... I have put. > > allow-bus-device-reset=0; > > In sd.conf > > as it seemed to mitigate that problem? IIRC, this is a nop for mpt and mptsas drivers. -- richard > > >> >> So just because the array is working fine now does not mean that it won't >> fail tragically when the first problems occur. >> >> I have personal experience with this failure mode. >> >> As a result I strongly discourage the use of SATA drives unless they are >> directly connected to the hba without any expanders or port multipliers. >> >> You ignore this advice at your own risk. Don't penny pinch on the drives >> It ALWAYS costs you in the long run. >> > > The advice in this thread, wrt SATA vs SAS, has been my thoughts > and position > for a while based on my initial research. I was required to explore and > verify this option > simply because of the significant economic benefit if it was possible. > Thanks for your > input. I will use this to make my case > > So now I just need some cheap SAS drives ;-) > > thanx - steve > >> Sent from my iPhone >> >> On Aug 19, 2013, at 4:25 PM, steve at linuxsuite.org wrote: >> >>>> Can't agree more here. Desktop firmwares are designed to try harder to >>>> never return an error. While this is what home users want, it's an >>>> anathema to large configurations with redundancy where you would prefer >>>> to >>>> just get the error so you can handle it - usually by doing the op on >>>> another drive. >>>> >>>> Use enterprise drives if you love your data and your ability to access >>>> it. >>> >>> Yes I know, I know.... Desktop hardware sucks. Didn't know there >>> was >>> Enterprise (ie none Desktop) SATA. Perhaps the solution is to partition >>> the data and do "deep" storage that never gets read on large bulk SATA >>> and >>> then have a smaller SAS pool for the often read/written stuff. >>> >>> Anyway, large storage rollouts will always be looking for >>> the cheapest solution. Even if it isn't perfect. I do backups >>> to a SATA pool (behind SAS expanders) with zfs send,.. no problems >>> errors >>> whatever YET!! >>> (cross fingers) >>> >>> -steve >>> >>>> >>>> Sent from my iPhone >>>> >>>> On Aug 19, 2013, at 2:47 PM, Eric Sproul wrote: >>>> >>>>> On Mon, Aug 19, 2013 at 5:35 PM, wrote: >>>>>> 4T SATA here >>>>>> >>>>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822178338 >>>>>> >>>>>> are $179 >>>>> >>>>> That's a desktop drive, not nearline enterprise, so it's >>>>> apples-oranges. You get what you pay for. Desktop drives can take >>>>> much, much longer to respond to commands, leading the HBA/expander to >>>>> declare them dead and reset them, and the rest is history. >>>>> >>>>> Avoid desktop parts. :) >>>>> >>>>> Eric >>>>> _______________________________________________ >>>>> OmniOS-discuss mailing list >>>>> OmniOS-discuss at lists.omniti.com >>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.elling at richardelling.com Wed Aug 21 23:37:59 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 21 Aug 2013 16:37:59 -0700 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: <20130822011337.3b08d758@sleipner.datanom.net> References: <20130822011337.3b08d758@sleipner.datanom.net> Message-ID: <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> On Aug 21, 2013, at 4:13 PM, Michael Rasmussen wrote: > On Tue, 20 Aug 2013 15:58:01 -0700 > Basil Crow wrote: > >> See this page for more information about 4K sectors and ZFS: >> >> http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks > Have tried everything from this wiki but no matter what I do I am not > able to have it aligned 4k. > > filebench given this stats (a mirrored vdev): > 150541 ops, 2492.698 ops/s, (226/453 r/w), 59.4mb/s, 898us cpu/op, > 71.1ms latency Is this a root disk? Or a non-root disk with EFI label? -- richard > > Another mirrored vdev consisting of 512B disks gives this stats: > 1684377 ops, 27581.072 ops/s, (2508/5016 r/w), 668.3mb/s, > 249us cpu/op, 3.9ms latency > > Both vdevs are connect via an LSI 1068E HBA > > -- > Hilsen/Regards > Michael Rasmussen > > Get my public GnuPG keys: > michael rasmussen cc > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E > mir datanom net > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C > mir miras org > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 > -------------------------------------------------------------- > "I'd love to go out with you, but the man on television told me to stay > tuned." > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mir at miras.org Wed Aug 21 23:50:48 2013 From: mir at miras.org (Michael Rasmussen) Date: Thu, 22 Aug 2013 01:50:48 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> Message-ID: <20130822015048.5db41494@sleipner.datanom.net> On Wed, 21 Aug 2013 16:37:59 -0700 Richard Elling wrote: > > Is this a root disk? Or a non-root disk with EFI label? parted /dev/rdsk/c7t2d0p0 GNU Parted 1.8.8 Using /dev/rdsk/c7t2d0p0 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p p Model: Generic Ide (ide) Disk /dev/rdsk/c7t2d0p0: 1000GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 131kB 1000GB 1000GB zfs 9 1000GB 1000GB 8389kB smartctl -i /dev/rdsk/c7t2d0 smartctl 6.0 2012-10-10 r3643 [i386-pc-solaris2.11] (local build) Copyright (C) 2002-12, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD10EZEX-00RKKA0 Serial Number: WD-WCC1S3904963 LU WWN Device Id: 5 0014ee 2b319a9c6 Firmware Version: 80.00A80 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s) Local Time is: Thu Aug 22 01:49:59 2013 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- The bigger they are, the harder they hit. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From richard.elling at richardelling.com Thu Aug 22 00:17:56 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 21 Aug 2013 17:17:56 -0700 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: <20130822015048.5db41494@sleipner.datanom.net> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> <20130822015048.5db41494@sleipner.datanom.net> Message-ID: <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> On Aug 21, 2013, at 4:50 PM, Michael Rasmussen wrote: > On Wed, 21 Aug 2013 16:37:59 -0700 > Richard Elling wrote: > >> >> Is this a root disk? Or a non-root disk with EFI label? > parted /dev/rdsk/c7t2d0p0 > GNU Parted 1.8.8 > Using /dev/rdsk/c7t2d0p0 > Welcome to GNU Parted! Type 'help' to view a list of commands. > (parted) > p p > Model: Generic Ide (ide) > Disk /dev/rdsk/c7t2d0p0: 1000GB > Sector size (logical/physical): 512B/512B Broken parted? :-P > Partition Table: gpt > > Number Start End Size File system Name Flags > 1 131kB 1000GB 1000GB zfs > 9 1000GB 1000GB 8389kB The OmniOS equivalent "prtvtoc /dev/rdsk/c7t2d0p0" should should show something like: prtvtoc /dev/rdsk/c1t8d0p0 * /dev/rdsk/c1t8d0p0 partition map * * Dimensions: * 512 bytes/sector * 27262976 sectors * 27262909 accessible sectors * * Flags: * 1: unmountable * 10: read-only * * Unallocated space: * First Sector Last * Sector Count Sector * 34 222 255 * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 4 00 256 27246303 27246558 8 11 00 27246559 16384 27262942 Which is how the default ZFS partition is set. 256 sectors is a multiple of 1 and 8. Methinks something else is going on here. -- richard > > smartctl -i /dev/rdsk/c7t2d0 > smartctl 6.0 2012-10-10 r3643 [i386-pc-solaris2.11] (local build) > Copyright (C) 2002-12, Bruce Allen, Christian Franke, > www.smartmontools.org > > === START OF INFORMATION SECTION === > Device Model: WDC WD10EZEX-00RKKA0 > Serial Number: WD-WCC1S3904963 > LU WWN Device Id: 5 0014ee 2b319a9c6 > Firmware Version: 80.00A80 > User Capacity: 1,000,204,886,016 bytes [1.00 TB] > Sector Sizes: 512 bytes logical, 4096 bytes physical > Device is: Not in smartctl database [for details use: -P showall] > ATA Version is: ATA8-ACS (minor revision not indicated) > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s) > Local Time is: Thu Aug 22 01:49:59 2013 CEST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > > > -- > Hilsen/Regards > Michael Rasmussen > > Get my public GnuPG keys: > michael rasmussen cc > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E > mir datanom net > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C > mir miras org > http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 > -------------------------------------------------------------- > The bigger they are, the harder they hit. > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mir at miras.org Thu Aug 22 00:26:40 2013 From: mir at miras.org (Michael Rasmussen) Date: Thu, 22 Aug 2013 02:26:40 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> <20130822015048.5db41494@sleipner.datanom.net> <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> Message-ID: <20130822022640.232e3048@sleipner.datanom.net> On Wed, 21 Aug 2013 17:17:56 -0700 Richard Elling wrote: > > Broken parted? :-P > jep. Generic IDE from a SATA 3 disk looks weird;-) > > Partition Table: gpt > > > > Number Start End Size File system Name Flags > > 1 131kB 1000GB 1000GB zfs > > 9 1000GB 1000GB 8389kB > > The OmniOS equivalent "prtvtoc /dev/rdsk/c7t2d0p0" should should show > something like: > prtvtoc /dev/rdsk/c1t8d0p0 > * /dev/rdsk/c1t8d0p0 partition map > * > * Dimensions: > * 512 bytes/sector > * 27262976 sectors > * 27262909 accessible sectors > * > * Flags: > * 1: unmountable > * 10: read-only > * > * Unallocated space: > * First Sector Last > * Sector Count Sector > * 34 222 255 > * > * First Sector Last > * Partition Tag Flags Sector Count Sector Mount Directory > 0 4 00 256 27246303 27246558 > 8 11 00 27246559 16384 27262942 > > Which is how the default ZFS partition is set. 256 sectors is a multiple of 1 and 8. > Methinks something else is going on here. > -- richard > prtvtoc /dev/rdsk/c7t2d0p0 * /dev/rdsk/c7t2d0p0 partition map * * Dimensions: * 512 bytes/sector * 1953525168 sectors * 1953525101 accessible sectors * * Flags: * 1: unmountable * 10: read-only * * Unallocated space: * First Sector Last * Sector Count Sector * 34 222 255 * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 4 00 256 1953508495 1953508750 8 11 00 1953508751 16384 1953525134 -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- I used to be Snow White, but I drifted. -- Mae West -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From richard.elling at richardelling.com Thu Aug 22 00:30:02 2013 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 21 Aug 2013 17:30:02 -0700 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: <20130822022640.232e3048@sleipner.datanom.net> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> <20130822015048.5db41494@sleipner.datanom.net> <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> <20130822022640.232e3048@sleipner.datanom.net> Message-ID: <25E886CE-8647-4935-B1C9-1FC1510A9959@richardelling.com> On Aug 21, 2013, at 5:26 PM, Michael Rasmussen wrote: > On Wed, 21 Aug 2013 17:17:56 -0700 > Richard Elling wrote: > >> >> Broken parted? :-P >> > jep. Generic IDE from a SATA 3 disk looks weird;-) >>> Partition Table: gpt >>> >>> Number Start End Size File system Name Flags >>> 1 131kB 1000GB 1000GB zfs >>> 9 1000GB 1000GB 8389kB >> >> The OmniOS equivalent "prtvtoc /dev/rdsk/c7t2d0p0" should should show >> something like: >> prtvtoc /dev/rdsk/c1t8d0p0 >> * /dev/rdsk/c1t8d0p0 partition map >> * >> * Dimensions: >> * 512 bytes/sector >> * 27262976 sectors >> * 27262909 accessible sectors >> * >> * Flags: >> * 1: unmountable >> * 10: read-only >> * >> * Unallocated space: >> * First Sector Last >> * Sector Count Sector >> * 34 222 255 >> * >> * First Sector Last >> * Partition Tag Flags Sector Count Sector Mount Directory >> 0 4 00 256 27246303 27246558 >> 8 11 00 27246559 16384 27262942 >> >> Which is how the default ZFS partition is set. 256 sectors is a multiple of 1 and 8. >> Methinks something else is going on here. >> -- richard >> > prtvtoc /dev/rdsk/c7t2d0p0 > * /dev/rdsk/c7t2d0p0 partition map > * > * Dimensions: > * 512 bytes/sector > * 1953525168 sectors > * 1953525101 accessible sectors > * > * Flags: > * 1: unmountable > * 10: read-only > * > * Unallocated space: > * First Sector Last > * Sector Count Sector > * 34 222 255 > * > * First Sector Last > * Partition Tag Flags Sector Count Sector Mount Directory > 0 4 00 256 1953508495 1953508750 > 8 11 00 1953508751 16384 1953525134 ok, so you're aligned :-) Problem is something else :-( Since this is a mirror, use "iostat -zxn 1" to see if the two sides of the mirror have different service times (asvc_t). -- richard -- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mir at miras.org Thu Aug 22 00:51:13 2013 From: mir at miras.org (Michael Rasmussen) Date: Thu, 22 Aug 2013 02:51:13 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: <25E886CE-8647-4935-B1C9-1FC1510A9959@richardelling.com> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> <20130822015048.5db41494@sleipner.datanom.net> <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> <20130822022640.232e3048@sleipner.datanom.net> <25E886CE-8647-4935-B1C9-1FC1510A9959@richardelling.com> Message-ID: <20130822025113.30e0333d@sleipner.datanom.net> On Wed, 21 Aug 2013 17:30:02 -0700 Richard Elling wrote: > > ok, so you're aligned :-) Problem is something else :-( > Since this is a mirror, use "iostat -zxn 1" to see if the two sides of the mirror have > different service times (asvc_t). r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 9.4 4.3 24.0 35.7 0.0 0.0 0.0 1.2 0 1 c7t0d0 9.4 4.3 24.0 35.7 0.0 0.0 0.0 1.2 0 1 c7t1d0 7.0 10.2 10.4 771.1 0.0 0.1 0.0 5.2 0 1 c7t2d0 6.6 10.2 9.3 771.1 0.0 0.1 0.0 5.1 0 1 c7t3d0 c7t0d0 c7t1d0 is another mirror consisting of two: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.12 Device Model: ST31000524AS Serial Number: 9VPG8NC3 LU WWN Device Id: 5 000c50 04d6f1cc8 Firmware Version: JC4B User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Aug 22 02:46:13 2013 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled Something just popped up. Could the bad stats be influenced by that this mirror is just created and therefore running in degraded mode? So therefore I should wait some hours and then try the benchmark once more? PS. The intension is to extend the existing pool with this new vdev to form a raid10 but I will wait with this extending until I am sure the performs as expected. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- Sigmund's wife wore Freudian slips. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From mir at miras.org Thu Aug 22 01:30:13 2013 From: mir at miras.org (Michael Rasmussen) Date: Thu, 22 Aug 2013 03:30:13 +0200 Subject: [OmniOS-discuss] =?utf-8?q?Advanced_format_disks_=E2=80=93_512E?= In-Reply-To: <20130822025113.30e0333d@sleipner.datanom.net> References: <20130822011337.3b08d758@sleipner.datanom.net> <941A4EEE-F3C1-4AFE-8BC8-A77A8B5E4A84@RichardElling.com> <20130822015048.5db41494@sleipner.datanom.net> <19BF6A13-5B23-4525-979A-041FB68DB371@RichardElling.com> <20130822022640.232e3048@sleipner.datanom.net> <25E886CE-8647-4935-B1C9-1FC1510A9959@richardelling.com> <20130822025113.30e0333d@sleipner.datanom.net> Message-ID: <20130822033013.44d591cb@sleipner.datanom.net> On Thu, 22 Aug 2013 02:51:13 +0200 Michael Rasmussen wrote: > > Something just popped up. Could the bad stats be influenced by that > this mirror is just created and therefore running in degraded mode? > Case closed. Waiting half an hour and the remember to activate the same features and settings as used on the other vdev - lz4 and disable sync. 1857659 ops, 30602.580 ops/s, (2782/5565 r/w), 740.1mb/s, 223us cpu/op, 3.5ms latency -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- He hated being thought of as one of those people that wore stupid ornamental armour. It was gilt by association. -- Terry Pratchett, "Night Watch" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From valrhona at gmail.com Thu Aug 22 05:59:48 2013 From: valrhona at gmail.com (Valrhona) Date: Thu, 22 Aug 2013 06:59:48 +0100 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? Message-ID: Thanks to all the hard work put in to getting the pkgsrc programs running. I followed Jonathan Perkin's instructions, and got gnome running on bloody: http://www.perkin.org.uk/posts/whats-new-in-pkgsrc-2013Q2.html I couldn't get any desktop to start running X on the stable version (151006); is this expected? That is, do we need bloody to get any X Windows system going? I also got xfce started, but could not get Thunar to run (just said it core-dumped). Seems like this is a very promising start, but it will be a while before a full-fledged desktop runs on top of OmniOS, though I certainly would be delighted if this could be done: all I need to nautilus, browser, maybe openoffice, and, crucially, virtualbox, maybe VLC or a media player, PDF viewer and some small utilities, and would be very, very happy (I know there are other desktop distributions, but none of them seem to be nearly as reliable as OmniOS, in terms of updates). But shy of that, it would also be extremely useful to me to be able to run a file manager like Nautilus from a remote desktop. The data on the file server is shared over NFS, but moving huge numbers of files over this NFS mapping can be pretty slow. I would like to be able to ssh -X into the server, and then run nautilus to move large quantities of files around more quickly. I tried this, by installing nautilus from pkgsrc, 'pkgin in nautilus'. Then I log in via ssh -X, and at the command prompt just type 'nautilus'. I get the following error: Could not parse arguments: Cannot open display: I always get this error, no matter what nautilus options I run from the command line. Am I doing something stupid, if I just want nautilus to run on my local desktop, while moving files around directly on the server? Or is there another way to accomplish the same end goal? Thanks! Peter From daniel at dgnetwork.com.br Thu Aug 22 19:00:26 2013 From: daniel at dgnetwork.com.br (=?ISO-8859-1?Q?=22Daniel_D=2E_Gon=E7alves=22?=) Date: Thu, 22 Aug 2013 16:00:26 -0300 Subject: [OmniOS-discuss] ZFS Questions Message-ID: <52165FCA.8070803@dgnetwork.com.br> Hi all, I have many questions about the ZFS on OmniOS: 1) It is possible to stop the RESILVER? 2) I can not remove the device FAULTED: " c17t22d1 FAULTED 0 0 0 corrupted data" root at storage01:~# zpool offline STORAGE01 c17t22d1 cannot offline c17t22d1: no valid replicas root at storage01:~# zpool detach STORAGE01 c17t22d1 cannot detach c17t22d1: no valid replicas root at storage01:~# zpool remove STORAGE01 c17t22d1 cannot remove c17t22d1: only inactive hot spares, cache, top-level, or log devices can be removed root at storage01:~# 3) The STATUS "Replacing-1" never changes, how do? Below, the status of my POOL: root at storage01:~# zpool status pool: STORAGE01 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu Aug 22 14:28:34 2013 696G scanned out of 18.3T at 113M/s, 45h31m to go 166G resilvered, 3.71% done config: NAME STATE READ WRITE CKSUM STORAGE01 DEGRADED 0 0 0 mirror-0 ONLINE 0 0 0 c17t36d1 ONLINE 0 0 0 c17t19d1 ONLINE 0 0 0 mirror-1 DEGRADED 0 0 0 c17t24d1 ONLINE 0 0 0 (resilvering) replacing-1 DEGRADED 0 0 0 c17t22d1 FAULTED 0 0 0 corrupted data c17t33d1 ONLINE 0 0 0 (resilvering) c17t21d1 ONLINE 0 0 0 (resilvering) mirror-2 ONLINE 0 0 0 c17t18d1 ONLINE 0 0 0 c17t17d1 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 c17t20d1 ONLINE 0 0 0 c17t22d1 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 c17t25d1 ONLINE 0 0 0 c17t27d1 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 c17t26d1 ONLINE 0 0 0 c17t28d1 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 c17t29d1 ONLINE 0 0 0 c17t31d1 ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 c17t32d1 ONLINE 0 0 0 c17t30d1 ONLINE 0 0 0 mirror-9 ONLINE 0 0 0 c17t23d1 ONLINE 0 0 0 c17t34d1 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 c14t1d0 ONLINE 0 0 0 c14t3d0 ONLINE 0 0 0 cache c14t4d0 ONLINE 0 0 0 errors: 11 data errors, use '-v' for a list pool: rpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c14t5d0s0 ONLINE 0 0 0 errors: No known data errors root at storage01:~# Thanks, Daniel From henson at acm.org Thu Aug 22 19:16:27 2013 From: henson at acm.org (Paul B. Henson) Date: Thu, 22 Aug 2013 12:16:27 -0700 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: References: Message-ID: <20130822191627.GE7098@bender.unx.csupomona.edu> On Thu, Aug 22, 2013 at 06:59:48AM +0100, Valrhona wrote: > Then I log in via ssh -X, and at the command prompt just type > 'nautilus'. I get the following error: > > Could not parse arguments: Cannot open display: Do you have x forwarding enabled on the server? Is xauth installed on the server? What does 'echo $DISPLAY' at a shell prompt return? Run "ssh -v -X" and see what it says, perhaps the forwarding is failing. From skiselkov.ml at gmail.com Thu Aug 22 19:17:36 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Thu, 22 Aug 2013 20:17:36 +0100 Subject: [OmniOS-discuss] ZFS Questions In-Reply-To: <52165FCA.8070803@dgnetwork.com.br> References: <52165FCA.8070803@dgnetwork.com.br> Message-ID: <521663D0.8000804@gmail.com> On 8/22/13 8:00 PM, "Daniel D. Gon?alves" wrote: > Hi all, > > I have many questions about the ZFS on OmniOS: Hi Daniel, > 1) It is possible to stop the RESILVER? Offline the device. > 2) I can not remove the device FAULTED: > > " c17t22d1 FAULTED 0 0 0 corrupted data" > > root at storage01:~# zpool offline STORAGE01 c17t22d1 > cannot offline c17t22d1: no valid replicas > root at storage01:~# zpool detach STORAGE01 c17t22d1 > cannot detach c17t22d1: no valid replicas > root at storage01:~# zpool remove STORAGE01 c17t22d1 > cannot remove c17t22d1: only inactive hot spares, cache, top-level, or > log devices can be removed > root at storage01:~# Detach only works on vdevs in a mirror. If your disk is in a raidz or is in no redundancy group at all (aka top-level vdev), you have to replace it. ZFS doesn't at the moment allow for removing drives in ways that reduces the total capacity of the pool. > 3) The STATUS "Replacing-1" never changes, how do? > > Below, the status of my POOL: > > root at storage01:~# zpool status > pool: STORAGE01 > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Thu Aug 22 14:28:34 2013 > 696G scanned out of 18.3T at 113M/s, 45h31m to go > 166G resilvered, 3.71% done > config: > > NAME STATE READ WRITE CKSUM > STORAGE01 DEGRADED 0 0 0 > mirror-0 ONLINE 0 0 0 > c17t36d1 ONLINE 0 0 0 > c17t19d1 ONLINE 0 0 0 > mirror-1 DEGRADED 0 0 0 > c17t24d1 ONLINE 0 0 0 (resilvering) > replacing-1 DEGRADED 0 0 0 > c17t22d1 FAULTED 0 0 0 corrupted data > c17t33d1 ONLINE 0 0 0 (resilvering) > c17t21d1 ONLINE 0 0 0 (resilvering) > mirror-2 ONLINE 0 0 0 > c17t18d1 ONLINE 0 0 0 > c17t17d1 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > c17t20d1 ONLINE 0 0 0 > c17t22d1 ONLINE 0 0 0 > mirror-5 ONLINE 0 0 0 > c17t25d1 ONLINE 0 0 0 > c17t27d1 ONLINE 0 0 0 > mirror-6 ONLINE 0 0 0 > c17t26d1 ONLINE 0 0 0 > c17t28d1 ONLINE 0 0 0 > mirror-7 ONLINE 0 0 0 > c17t29d1 ONLINE 0 0 0 > c17t31d1 ONLINE 0 0 0 > mirror-8 ONLINE 0 0 0 > c17t32d1 ONLINE 0 0 0 > c17t30d1 ONLINE 0 0 0 > mirror-9 ONLINE 0 0 0 > c17t23d1 ONLINE 0 0 0 > c17t34d1 ONLINE 0 0 0 > logs > mirror-4 ONLINE 0 0 0 > c14t1d0 ONLINE 0 0 0 > c14t3d0 ONLINE 0 0 0 > cache > c14t4d0 ONLINE 0 0 0 > > errors: 11 data errors, use '-v' for a list > > pool: rpool > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > rpool ONLINE 0 0 0 > c14t5d0s0 ONLINE 0 0 0 > > errors: No known data errors > root at storage01:~# You somehow managed to damage your pool by shuffling devices around too liberally, leaving you with both halves of the mirror in a bad state. It's quite difficult to tell at this stage what's going on. Let it run for a while and you'll see if its making any progress. By the looks of it it should take another 45 hours to complete (large pools take a long time to resilver, that's just a fact of life). -- Saso From fabio at fabiorabelo.wiki.br Thu Aug 22 19:29:24 2013 From: fabio at fabiorabelo.wiki.br (=?UTF-8?Q?F=C3=A1bio_Rabelo?=) Date: Thu, 22 Aug 2013 16:29:24 -0300 Subject: [OmniOS-discuss] SolarFlare 10GB network card Message-ID: Hi to all Someone knows if the SolarFlare 10GB network cards works with OmniOS ? They are getting realy cheap lately ... http://www.provantage.com/solarflare-communications-sfn5152f~7SOLA004.htm http://www.provantage.com/solarflare-communications-sfn5162f~7SOLA003.htm http://www.provantage.com/solarflare-communications-sfn5161t~7SOLA001.htm F?bio Rabelo -------------- next part -------------- An HTML attachment was scrubbed... URL: From nagele at wildbit.com Thu Aug 22 19:46:54 2013 From: nagele at wildbit.com (Chris Nagele) Date: Thu, 22 Aug 2013 15:46:54 -0400 Subject: [OmniOS-discuss] SolarFlare 10GB network card In-Reply-To: References: Message-ID: I tried them about a year ago without any luck. We are using them on CentOS and stuck with the Intel x520's for OmniOS. Chris On Thu, Aug 22, 2013 at 3:29 PM, F?bio Rabelo wrote: > Hi to all > > Someone knows if the SolarFlare 10GB network cards works with OmniOS ? > > They are getting realy cheap lately ... > > http://www.provantage.com/solarflare-communications-sfn5152f~7SOLA004.htm > > http://www.provantage.com/solarflare-communications-sfn5162f~7SOLA003.htm > > http://www.provantage.com/solarflare-communications-sfn5161t~7SOLA001.htm > > > F?bio Rabelo > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > From daniel at dgnetwork.com.br Thu Aug 22 20:20:02 2013 From: daniel at dgnetwork.com.br (=?ISO-8859-1?Q?=22Daniel_D=2E_Gon=E7alves=22?=) Date: Thu, 22 Aug 2013 17:20:02 -0300 Subject: [OmniOS-discuss] ZFS Questions In-Reply-To: <521663D0.8000804@gmail.com> References: <52165FCA.8070803@dgnetwork.com.br> <521663D0.8000804@gmail.com> Message-ID: <52167272.5010200@dgnetwork.com.br> Thanks Saso, To stop RESILVER, which device I to set to OFFLINE? I do not know how the device "c17t33d1" was placed in the MIRROR-11/REPLACING-1, how do I remove it from there? Daniel Em 22/08/2013 16:17, Saso Kiselkov escreveu: > On 8/22/13 8:00 PM, "Daniel D. Gon?alves" wrote: >> Hi all, >> >> I have many questions about the ZFS on OmniOS: > Hi Daniel, > >> 1) It is possible to stop the RESILVER? > Offline the device. > >> 2) I can not remove the device FAULTED: >> >> " c17t22d1 FAULTED 0 0 0 corrupted data" >> >> root at storage01:~# zpool offline STORAGE01 c17t22d1 >> cannot offline c17t22d1: no valid replicas >> root at storage01:~# zpool detach STORAGE01 c17t22d1 >> cannot detach c17t22d1: no valid replicas >> root at storage01:~# zpool remove STORAGE01 c17t22d1 >> cannot remove c17t22d1: only inactive hot spares, cache, top-level, or >> log devices can be removed >> root at storage01:~# > Detach only works on vdevs in a mirror. If your disk is in a raidz or is > in no redundancy group at all (aka top-level vdev), you have to replace > it. ZFS doesn't at the moment allow for removing drives in ways that > reduces the total capacity of the pool. > >> 3) The STATUS "Replacing-1" never changes, how do? >> >> Below, the status of my POOL: >> >> root at storage01:~# zpool status >> pool: STORAGE01 >> state: DEGRADED >> status: One or more devices is currently being resilvered. The pool will >> continue to function, possibly in a degraded state. >> action: Wait for the resilver to complete. >> scan: resilver in progress since Thu Aug 22 14:28:34 2013 >> 696G scanned out of 18.3T at 113M/s, 45h31m to go >> 166G resilvered, 3.71% done >> config: >> >> NAME STATE READ WRITE CKSUM >> STORAGE01 DEGRADED 0 0 0 >> mirror-0 ONLINE 0 0 0 >> c17t36d1 ONLINE 0 0 0 >> c17t19d1 ONLINE 0 0 0 >> mirror-1 DEGRADED 0 0 0 >> c17t24d1 ONLINE 0 0 0 (resilvering) >> replacing-1 DEGRADED 0 0 0 >> c17t22d1 FAULTED 0 0 0 corrupted data >> c17t33d1 ONLINE 0 0 0 (resilvering) >> c17t21d1 ONLINE 0 0 0 (resilvering) >> mirror-2 ONLINE 0 0 0 >> c17t18d1 ONLINE 0 0 0 >> c17t17d1 ONLINE 0 0 0 >> mirror-3 ONLINE 0 0 0 >> c17t20d1 ONLINE 0 0 0 >> c17t22d1 ONLINE 0 0 0 >> mirror-5 ONLINE 0 0 0 >> c17t25d1 ONLINE 0 0 0 >> c17t27d1 ONLINE 0 0 0 >> mirror-6 ONLINE 0 0 0 >> c17t26d1 ONLINE 0 0 0 >> c17t28d1 ONLINE 0 0 0 >> mirror-7 ONLINE 0 0 0 >> c17t29d1 ONLINE 0 0 0 >> c17t31d1 ONLINE 0 0 0 >> mirror-8 ONLINE 0 0 0 >> c17t32d1 ONLINE 0 0 0 >> c17t30d1 ONLINE 0 0 0 >> mirror-9 ONLINE 0 0 0 >> c17t23d1 ONLINE 0 0 0 >> c17t34d1 ONLINE 0 0 0 >> logs >> mirror-4 ONLINE 0 0 0 >> c14t1d0 ONLINE 0 0 0 >> c14t3d0 ONLINE 0 0 0 >> cache >> c14t4d0 ONLINE 0 0 0 >> >> errors: 11 data errors, use '-v' for a list >> >> pool: rpool >> state: ONLINE >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> rpool ONLINE 0 0 0 >> c14t5d0s0 ONLINE 0 0 0 >> >> errors: No known data errors >> root at storage01:~# > You somehow managed to damage your pool by shuffling devices around too > liberally, leaving you with both halves of the mirror in a bad state. > It's quite difficult to tell at this stage what's going on. Let it run > for a while and you'll see if its making any progress. By the looks of > it it should take another 45 hours to complete (large pools take a long > time to resilver, that's just a fact of life). > From skiselkov.ml at gmail.com Thu Aug 22 20:42:26 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Thu, 22 Aug 2013 21:42:26 +0100 Subject: [OmniOS-discuss] ZFS Questions In-Reply-To: <52167272.5010200@dgnetwork.com.br> References: <52165FCA.8070803@dgnetwork.com.br> <521663D0.8000804@gmail.com> <52167272.5010200@dgnetwork.com.br> Message-ID: <521677B2.8030105@gmail.com> On 8/22/13 9:20 PM, "Daniel D. Gon?alves" wrote: > Thanks Saso, > > To stop RESILVER, which device I to set to OFFLINE? The one that says 'resilvering'. But beware that that means that the pool might not have full fault tolerance. > I do not know how the device "c17t33d1" was placed in the > MIRROR-11/REPLACING-1, how do I remove it from there? If you can, let it run to completion before attempting any further manipulation. The pool seems to be in quite an unhappy state anyway, so better not compound the situation by doing more changes. Let the thing resync back up, find the files that have the data errors in them ("zpool status -v" I think), restore them or delete them and then post a new "zpool status" to the list - then we'll see what can be done. Above all, be patient if you don't want to lose your data. Cheers, -- Saso From aurfalien at gmail.com Thu Aug 22 21:50:56 2013 From: aurfalien at gmail.com (aurfalien) Date: Thu, 22 Aug 2013 14:50:56 -0700 Subject: [OmniOS-discuss] SolarFlare 10GB network card In-Reply-To: References: Message-ID: <88B7887A-A2F7-4D0C-A348-B6E2127FFF65@gmail.com> Thats a real bummer. Such a good card, OEMs to IBM, HP etc... - aurf On Aug 22, 2013, at 12:46 PM, Chris Nagele wrote: > I tried them about a year ago without any luck. We are using them on > CentOS and stuck with the Intel x520's for OmniOS. > > Chris > > On Thu, Aug 22, 2013 at 3:29 PM, F?bio Rabelo wrote: >> Hi to all >> >> Someone knows if the SolarFlare 10GB network cards works with OmniOS ? >> >> They are getting realy cheap lately ... >> >> http://www.provantage.com/solarflare-communications-sfn5152f~7SOLA004.htm >> >> http://www.provantage.com/solarflare-communications-sfn5162f~7SOLA003.htm >> >> http://www.provantage.com/solarflare-communications-sfn5161t~7SOLA001.htm >> >> >> F?bio Rabelo >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From esproul at omniti.com Thu Aug 22 23:31:08 2013 From: esproul at omniti.com (Eric Sproul) Date: Thu, 22 Aug 2013 19:31:08 -0400 Subject: [OmniOS-discuss] SolarFlare 10GB network card In-Reply-To: <88B7887A-A2F7-4D0C-A348-B6E2127FFF65@gmail.com> References: <88B7887A-A2F7-4D0C-A348-B6E2127FFF65@gmail.com> Message-ID: Perhaps a coincidence, perhaps not, but this was just posted to the illumos developer list: http://www.listbox.com/member/archive/182179/2013/08/sort/time_rev/page/1/entry/0:390/20130822190547:5F8A356C-0B7F-11E3-A7EA-DC1FB2D2E1BB/ A webrev adding the sfxge driver to illumos. http://cr.illumos.org/~webrev/rincebrain/illumos-sfxge/ :) From aurfalien at gmail.com Thu Aug 22 23:51:38 2013 From: aurfalien at gmail.com (aurfalien) Date: Thu, 22 Aug 2013 16:51:38 -0700 Subject: [OmniOS-discuss] SolarFlare 10GB network card In-Reply-To: References: <88B7887A-A2F7-4D0C-A348-B6E2127FFF65@gmail.com> Message-ID: Very very cool. Love it. - aurf On Aug 22, 2013, at 4:31 PM, Eric Sproul wrote: > Perhaps a coincidence, perhaps not, but this was just posted to the > illumos developer list: > > http://www.listbox.com/member/archive/182179/2013/08/sort/time_rev/page/1/entry/0:390/20130822190547:5F8A356C-0B7F-11E3-A7EA-DC1FB2D2E1BB/ > > A webrev adding the sfxge driver to illumos. > http://cr.illumos.org/~webrev/rincebrain/illumos-sfxge/ > > :) > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From valrhona at gmail.com Fri Aug 23 09:56:14 2013 From: valrhona at gmail.com (Valrhona) Date: Fri, 23 Aug 2013 10:56:14 +0100 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: <20130822191627.GE7098@bender.unx.csupomona.edu> References: <20130822191627.GE7098@bender.unx.csupomona.edu> Message-ID: Thanks for the note. I have xauth installed on the server, and running 'which xauth' gives: /opt/local/bin/xauth running 'echo $DISPLAY' gives nothing. So something is not configured correctly. In /etc/ssh/sshd_config, I have: # X11 tunneling options X11Forwarding yes X11DisplayOffset 10 X11UseLocalhost yes running ssh -v -X gives: Sun_SSH_1.5, SSH protocols 1.5/2.0, OpenSSL 0x0090818f debug1: Reading configuration data /etc/ssh/ssh_config debug1: Rhosts Authentication disabled, originating port will not be trusted. debug1: ssh_connect: needpriv 0 debug1: Remote protocol version 2.0, remote software version Sun_SSH_1.5 debug1: match: Sun_SSH_1.5 pat Sun_SSH_1.5* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-Sun_SSH_1.5 debug1: use_engine is 'yes' debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers debug1: pkcs11 engine initialization complete debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none ddebug1: We proposed langtags, ctos: en-US debug1: We proposed langtags, stoc: en-US debug1: Negotiated lang: en-US debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: Remote: Negotiated main locale: en_US.UTF-8 debug1: Remote: Negotiated messages locale: en_US.UTF-8 debug1: dh_gen_key: priv key bits set: 126/256 debug1: bits set: 1619/3191 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: bits set: 1564/3191 debug1: ssh_rsa_verify: signature correct debug1: set_newkeys: setting new keys for 'out' mode debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: set_newkeys: setting new keys for 'in' mode debug1: SSH2_MSG_NEWKEYS received debug1: done: ssh_kex2. debug1: send SSH2_MSG_SERVICE_REQUEST debug1: got SSH2_MSG_SERVICE_ACCEPT debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive debug1: Next authentication method: gssapi-keyex debug1: Next authentication method: gssapi-with-mic debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible Password: debug1: Authentication succeeded (keyboard-interactive) debug1: channel 0: new [client-session] debug1: send channel open 0 debug1: Entering interactive session. debug1: ssh_session2_setup: id 0 debug1: channel request 0: env debug1: channel request 0: pty-req debug1: Requesting X11 forwarding with authentication spoofing. debug1: channel request 0: x11-req debug1: channel request 0: shell debug1: fd 4 setting TCP_NODELAY debug1: channel 0: open confirm rwindow 0 rmax 32768 debug1: Remote: Channel 0 set: LANG=en_US.UTF-8 debug1: Remote: No xauth program; cannot forward with spoofing. Last login: Thu Aug 22 01:57:31 2013 from dellt710 OmniOS 5.11 omnios-8d266aa 2013.05.04 So it says that xauth isn't installed, even though it is, and in the PATH. What stupid thing am I still doing that prevents this from working? Thanks! Peter On Thu, Aug 22, 2013 at 8:16 PM, Paul B. Henson wrote: > On Thu, Aug 22, 2013 at 06:59:48AM +0100, Valrhona wrote: > >> Then I log in via ssh -X, and at the command prompt just type >> 'nautilus'. I get the following error: >> >> Could not parse arguments: Cannot open display: > > Do you have x forwarding enabled on the server? Is xauth installed on > the server? What does 'echo $DISPLAY' at a shell prompt return? Run > "ssh -v -X" and see what it says, perhaps the forwarding is failing. > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From jimklimov at cos.ru Fri Aug 23 11:07:07 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 23 Aug 2013 13:07:07 +0200 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: References: <20130822191627.GE7098@bender.unx.csupomona.edu> Message-ID: <5217425B.70306@cos.ru> On 2013-08-23 11:56, Valrhona wrote: > Thanks for the note. I have xauth installed on the server, and running > 'which xauth' gives: > /opt/local/bin/xauth Possibly /opt/local/bin/ is not in your global PATH (i.e. the sshd server does not know it should search there)? As a simple workaround which may work, symlink xauth into a common path like /usr/bin... > running 'echo $DISPLAY' gives nothing. So something is not configured correctly. > See (with "netstat -an | grep 60" for example) if the X11 ports were forwarded and are open? By default, 6000 is DISPLAY=0, plus one for each new display. So if you see say 6002, then your SSH client tried to define a DISPLAY=2. Set and export this value in the resulting shell (with -X forwarding or plain TCP forwarding if permitted) and see if it helps you remote-execute graphics stuff. HTH, //Jim From tim at multitalents.net Fri Aug 23 19:07:42 2013 From: tim at multitalents.net (Tim Rice) Date: Fri, 23 Aug 2013 12:07:42 -0700 (PDT) Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: References: <20130822191627.GE7098@bender.unx.csupomona.edu> Message-ID: On Fri, 23 Aug 2013, Valrhona wrote: > Thanks for the note. I have xauth installed on the server, and running > 'which xauth' gives: > /opt/local/bin/xauth Unless you have built your own sshd you'll need to have xauth (or a symlink) in /usr/X11/bin. -- Tim Rice Multitalents (707) 456-1146 tim at multitalents.net From high at uw.edu Fri Aug 23 21:56:46 2013 From: high at uw.edu (Brian High) Date: Fri, 23 Aug 2013 14:56:46 -0700 Subject: [OmniOS-discuss] ldap auth Message-ID: <5217DA9E.3020909@uw.edu> I would very much like to migrate from OpenIndiana to OmniOS. But I am hung up on one issue. I am having trouble with ldap authentication (pam_ldap) in OmniOS. I am hoping someone on this list will have some experience with this and can enlighten me. On oi_151.1.7 we are able to get LDAP auth working against our 389 Directory (FDS) with TLS, using these commands to set it up: ### Install Mozilla-NSS to get "certutil". # pkg set-publisher -p http://pkg.openindiana.org/dev # pkg install system/mozilla-nss ### Import the CA Cert. # certutil -A -n defaultCert \ -i /path/to/cacert.asc \ -a -t CT -d /var/ldap ### Test with "ldapsearch" # ldapsearch -v -x -ZZ \ -h ldap.example.com -p 636 \ -b OU=People,dc=example,dc=com \ -P /var/ldap/cert8.db \ -D cn=xxxx,ou=DSA,dc=example,dc=com \ -w yyyy \ -LLL "(uid=root)" gecos ### Configure with "ldapclient" # ldapclient manual -vvv \ -a domainName=example.com \ -a serviceSearchDescriptor=passwd:OU=People,dc=example,dc=com \ -a serviceSearchDescriptor=group:OU=Groups,dc=example,dc=com \ -a authenticationMethod=tls:simple \ -a credentialLevel=proxy \ -a certificatePath=/var/ldap \ -a defaultSearchBase=dc=example,dc=com \ -a searchTimeLimit=60 \ -a defaultServerList=ldap.example.com,ldap2.example.com \ -a proxyDN=cn=xxxx,ou=DSA,dc=example,dc=com \ -a proxyPassword=yyyy ### Edit "/etc/nsswitch.conf" to use "files ldap" for "passwd" & "group" ### Edit "/etc/pam.conf" for "auth required pam_ldap.so.1" ... However, in OmniOS r151006 (omnios-b281e50) the ldapsearch test fails when using TLS (-Z or -ZZ switches used) with: ldap_simple_bind: Can't contact LDAP server But does not fail if TLS is not used (-Z or -ZZ omitted). And, similarly, when using ldap auth (PAM), we see in the logs: Aug 23 12:48:00 MYHOST ldap_cachemgr[847]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to MYLDAPSERVER Aug 23 12:48:00 MYHOST ldap_cachemgr[847]: [ID 687686 daemon.warning] libsldap: Falling back to anonymous, non-SSL mode for __ns_ldap_getRootDSE. openConnection: simple bind failed - Can't contact LDAP server Since debug has not been compiled into the ldapsearch client, I can't get any more details from that. But, interestingly, when I compile my own OpenSSL and OpenLDAP and use it's ldapsearch, I can use TLS with it with no problems. # /usr/local/bin/ldapsearch -x -Z -h MYLDAPSERVER -b OU=People,dc=example,dc=com -LLL "(uid=root)" gecos dn: uid=root,ou=People,dc=example,dc=com gecos: Netbios Domain Administrator Any ideas of what might be wrong here with the OmniOS ldap client? -- Brian High From valrhona at gmail.com Fri Aug 23 22:07:15 2013 From: valrhona at gmail.com (Valrhona) Date: Fri, 23 Aug 2013 23:07:15 +0100 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: <5217425B.70306@cos.ru> References: <20130822191627.GE7098@bender.unx.csupomona.edu> <5217425B.70306@cos.ru> Message-ID: > See (with "netstat -an | grep 60" for example) if the X11 ports were > forwarded and are open? By default, 6000 is DISPLAY=0, plus one for > each new display. So if you see say 6002, then your SSH client tried > to define a DISPLAY=2. Set and export this value in the resulting > shell (with -X forwarding or plain TCP forwarding if permitted) and > see if it helps you remote-execute graphics stuff. So neither 6000 or anything 600* pops up when I run this command. > Unless you have built your own sshd you'll need to have > xauth (or a symlink) in /usr/X11/bin There isn't a directory of /usr/X11 So I am assuming there are some packages that I am missing? I basically followed the instructions to get gnome working (which also doesn't start on this machine) http://www.perkin.org.uk/posts/whats-new-in-pkgsrc-2013Q2.html But I am also running this on OmniOS stable 151006. Do I need bloody to get this to run? If so, can I just switch repositories and update the packages, and keep my old stable version in a BE? Thanks! Peter From henson at acm.org Sat Aug 24 01:56:24 2013 From: henson at acm.org (Paul B. Henson) Date: Fri, 23 Aug 2013 18:56:24 -0700 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: References: <20130822191627.GE7098@bender.unx.csupomona.edu> <5217425B.70306@cos.ru> Message-ID: <20130824015624.GH7098@bender.unx.csupomona.edu> On Fri, Aug 23, 2013 at 11:07:15PM +0100, Valrhona wrote: > > Unless you have built your own sshd you'll need to have > > xauth (or a symlink) in /usr/X11/bin > > There isn't a directory of /usr/X11 It looks like the version of ssh in illumos hardcodes the path to xauth: # strings /usr/lib/ssh/sshd | grep xauth /usr/X11/bin/xauth So, either you build your own sshd to look for it elsewhere, or you make a link from where the bundled sshd wants to find it to where you have it. So, # mkdir -p /usr/X11/bin # ln -s /opt/local/bin/xauth /usr/X11/bin Give that a shot and see what happens. From mattfrazer at gmail.com Sat Aug 24 02:01:56 2013 From: mattfrazer at gmail.com (Matthew Frazer) Date: Fri, 23 Aug 2013 22:01:56 -0400 Subject: [OmniOS-discuss] =?windows-1252?q?Advanced_format_disks_=96_512E?= In-Reply-To: <35B6A902-0DAD-43E8-A83D-8D8B4CE12FCE@RichardElling.com> References: <35B6A902-0DAD-43E8-A83D-8D8B4CE12FCE@RichardElling.com> Message-ID: WRT sensitivity is note #3 here inaccurate? https://www.illumos.org/issues/2665#note-3 As an example, here is a device that is particularly problematic... The HDS721010DLE630 which reports itself to the OS as vid = "ATA " and pid = "Hitachi HDS72101" which should be handled by the following in sd.conf sd-config-list = "ATA HITACHI HDS72101", "physical-block-size:4096"; However, it isn't. Nor is it caught by any amount of wildcarding around all or portions of those ID strings. Pools create stubbornly at ashift 9. Any advice? -mjf On Wed, Aug 21, 2013 at 12:17 AM, Richard Elling < richard.elling at richardelling.com> wrote: > > On Aug 20, 2013, at 6:40 PM, Matthew Frazer wrote: > > This is a question I've been meaning to ask as well. > > Using the information at > http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks I've > edited the sd.conf on my systems to include the drives I wish to have 4k > sector sizes on. On Solaris 11.1 this achieves the proper ashift on 4k > drives whether they report 4k or 512e to the system. The same sd.conf does > not, at least in my experience, result in an ashift of 12 on OmniOS. My > workaround has been to create my pools in 11.1 with version flags set > compatible to what omnios supports, then import them into OmniOS and then > zfs upgrade them. I'd love to find out I've been doing it wrong as it would > simplify pool creation greatly for me. > > The file is very sensitive to spaces and format, including separators. > Make sure you > have the file exactly as needed. > -- richard > > -- > > Richard.Elling at RichardElling.com > +1-760-896-4422 > > > > -- It is in love that we are made, in love we disappear. -Leonard Cohen -------------- next part -------------- An HTML attachment was scrubbed... URL: From henson at acm.org Sat Aug 24 02:30:04 2013 From: henson at acm.org (Paul B. Henson) Date: Fri, 23 Aug 2013 19:30:04 -0700 Subject: [OmniOS-discuss] ldap auth In-Reply-To: <5217DA9E.3020909@uw.edu> References: <5217DA9E.3020909@uw.edu> Message-ID: <20130824023004.GI7098@bender.unx.csupomona.edu> On Fri, Aug 23, 2013 at 02:56:46PM -0700, Brian High wrote: > ... However, in OmniOS r151006 (omnios-b281e50) the ldapsearch test > fails when using TLS (-Z or -ZZ switches used) with: > > ldap_simple_bind: Can't contact LDAP server I've got a vanilla omnios test box of the same vintage which seems to work fine against my openldap server: # ldapsearch -ZZ -P /var/ldap/cert8.db -h ldap.csupomona.edu -p 636 -b dc=csupomona,dc=edu uid=astudent version: 1 dn: uid=astudent,ou=user,dc=csupomona,dc=edu [...] I also just configured it to use ldap as a naming service with the following setup (which we use in production on Solaris 10): # cat /var/ldap/ldap_client_file # # Do not edit this file manually; your changes will be lost.Please use # ldapclient (1M) instead. # NS_LDAP_FILE_VERSION= 2.0 NS_LDAP_SERVERS= ldap.csupomona.edu NS_LDAP_SEARCH_BASEDN= dc=csupomona,dc=edu NS_LDAP_CACHETTL= 0 NS_LDAP_SERVICE_SEARCH_DESC= passwd:ou=user,dc=csupomona,dc=edu NS_LDAP_SERVICE_SEARCH_DESC= automount:ou=automount,ou=service,dc=csupomona,dc=edu?sub NS_LDAP_ATTRIBUTEMAP= group:cn=uid NS_LDAP_ATTRIBUTEMAP= passwd:gecos=displayname NS_LDAP_ATTRIBUTEMAP= automount:automountMapName=ou NS_LDAP_ATTRIBUTEMAP= automount:automountKey=uid NS_LDAP_OBJECTCLASSMAP= shadow:shadowAccount=posixAccount NS_LDAP_OBJECTCLASSMAP= automount:automount=* NS_LDAP_AUTH= tls:simple NS_LDAP_CREDENTIAL_LEVEL= proxy and after enabling ldap in nsswitch.conf, it seems to work fine: # getent passwd astudent astudent:x:45355:1012:A. B. Student Esq.:/user/astudent:/bin/bash > Any ideas of what might be wrong here with the OmniOS ldap client? So, I don't think the omnios client is inherently broken ;). If you'd like to contact me off list and provide access to your ldap server I could try and set up an omnios client pointing to it and see if I can figure out what's not working... From richard.palo at free.fr Sat Aug 24 07:48:14 2013 From: richard.palo at free.fr (Richard PALO) Date: Sat, 24 Aug 2013 09:48:14 +0200 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: References: Message-ID: Perhaps two things to consider: 1. makesure xauth from pkgsrc is installed 2. make sure /etc/ssh/sshd_config contains the following line: XAuthLocation /opt/local/bin/xauth make sure, if you bootstrap yourself pkgsrc, that if you use a different PREFIX for LOCALBASE, then adjust appropriately the path to xauth. From valrhona at gmail.com Sat Aug 24 06:58:17 2013 From: valrhona at gmail.com (Valrhona) Date: Sat, 24 Aug 2013 07:58:17 +0100 Subject: [OmniOS-discuss] Possible to run desktop applications via ssh -X or similar? In-Reply-To: <20130824015624.GH7098@bender.unx.csupomona.edu> References: <20130822191627.GE7098@bender.unx.csupomona.edu> <5217425B.70306@cos.ru> <20130824015624.GH7098@bender.unx.csupomona.edu> Message-ID: This works, and is a HUGE help to my work. Thanks so much! When I launch nautilus, though, I do get an error that comes up many, many times. (nautilus:18312): Eel-WARNING **: GConf error: Failed to contact configuration server; the most common cause is a missing or misconfigured D-Bus session bus daemon. See http://projects.gnome.org/gconf/ for information. (Details - 1: Failed to get connection to session: Error spawning command line `dbus-launch --autolaunch=9c0c30846e3e8e82c3e14f6b52185569 --binary-syntax --close-stderr': Child process exited with code 1) One other error crops up once at the beginning: Gtk-Message: Failed to load module "canberra-gtk-module" Does anyone know what this is? Thanks again! Peter On Sat, Aug 24, 2013 at 2:56 AM, Paul B. Henson wrote: > On Fri, Aug 23, 2013 at 11:07:15PM +0100, Valrhona wrote: > >> > Unless you have built your own sshd you'll need to have >> > xauth (or a symlink) in /usr/X11/bin >> >> There isn't a directory of /usr/X11 > > It looks like the version of ssh in illumos hardcodes the path to xauth: > > # strings /usr/lib/ssh/sshd | grep xauth > /usr/X11/bin/xauth > > So, either you build your own sshd to look for it elsewhere, or you make > a link from where the bundled sshd wants to find it to where you have > it. So, > > # mkdir -p /usr/X11/bin > # ln -s /opt/local/bin/xauth /usr/X11/bin > > Give that a shot and see what happens. > From henson at acm.org Sat Aug 24 19:50:38 2013 From: henson at acm.org (Paul B. Henson) Date: Sat, 24 Aug 2013 12:50:38 -0700 Subject: [OmniOS-discuss] ldap auth In-Reply-To: <20130824023004.GI7098@bender.unx.csupomona.edu> References: <5217DA9E.3020909@uw.edu> <20130824023004.GI7098@bender.unx.csupomona.edu> Message-ID: <20130824195038.GK7098@bender.unx.csupomona.edu> On Fri, Aug 23, 2013 at 07:30:04PM -0700, Paul B. Henson wrote: > On Fri, Aug 23, 2013 at 02:56:46PM -0700, Brian High wrote: > > > ... However, in OmniOS r151006 (omnios-b281e50) the ldapsearch test > > fails when using TLS (-Z or -ZZ switches used) with: > > > > ldap_simple_bind: Can't contact LDAP server > > I've got a vanilla omnios test box of the same vintage which seems to > work fine against my openldap server: It looks like Brian's problem might be that he has an MD5 cert on his ldap server, and the latest release of omnios includes nss 3.14.3, which has by default dropped support for md5 certs: https://developer.mozilla.org/en-US/docs/NSS/NSS_3.14_release_notes This might be worth retroactively adding to the release notes as a compatibility change? RHEL6 evidentally has the same issue, which can be worked around by setting the environment variable NSS_HASH_ALG_SUPPORT=+MD5. If the same workaround resolves the issue under omnios, then # svccfg -s network/ldap/client:default setenv NSS_HASH_ALG_SUPPORT +MD5 should make the ldap client work, I believe all ldap connections are routed through the cache manager. I don't have any servers around with an md5 cert to test against, but I'm sure once Brian tests it out he'll report back to the list what happened. From henson at acm.org Sun Aug 25 00:06:04 2013 From: henson at acm.org (Paul B. Henson) Date: Sat, 24 Aug 2013 17:06:04 -0700 Subject: [OmniOS-discuss] swap volume block size Message-ID: <20130825000604.GN7098@bender.unx.csupomona.edu> I think it's been discussed before, but the omnios installer creates the swap volume with a block size of 8k rather than 4k. While 8k is still the current recommendation for SPARC, the x86 recommendation remains 4k. As it seems unlikely that omnios is going to run on SPARC anytime soon ;), maybe the installer default could be changed to 4k? Thanks... From toan at hoanglai.no Sun Aug 25 21:04:15 2013 From: toan at hoanglai.no (Toan Hoang) Date: Sun, 25 Aug 2013 23:04:15 +0200 Subject: [OmniOS-discuss] KVM on a Intel E3-1220 V2 Message-ID: Hello, I have to installed Windows 7 Pro (with virtio net and disk) on OmniOS on an Intel E3-1220 V2 which claims to have 2nd generation VMX support (with EPT) However, The guest dies from time to time and I see these errors in /var/adm/message: Aug 25 22:55:47 test kvm: [ID 420667 kern.info] kvm_lapic_reset: vcpu=ffffff0929cfd000, id=0, base_msr= fee00100 PRIx64 base_address=fee00000 Aug 25 22:55:47 test kvm: [ID 710719 kern.info] vmcs revision_id = 10 Aug 25 22:55:47 test kvm: [ID 391722 kern.info] unhandled wrmsr: 0xff360168 data fffffd7fff180030 Aug 25 22:55:47 test kvm: [ID 391722 kern.info] unhandled wrmsr: 0x909dfe0 data fffffd7fffdfece0 Aug 25 22:55:47 test kvm: [ID 391722 kern.info] unhandled wrmsr: 0xffdff398 data fffffd7fff1dd638 Aug 25 22:55:47 test kvm: [ID 713435 kern.info] unhandled rdmsr: 0x756e6547 Aug 25 22:55:47 test kvm: [ID 391722 kern.info] unhandled wrmsr: 0x0 data 0 Aug 25 22:56:02 test kvm: [ID 420667 kern.info] kvm_lapic_reset: vcpu=ffffff0a2cb9a000, id=0, base_msr= fee00100 PRIx64 base_address=fee00000 Another thing I observed is the eptv when running kvmstat (this is during guest bootup): pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv 6568 0 | 33088 : 67 60 42 3377 0 | 275 574 27487 6568 0 | 3355 : 65 9 4 603 0 | 72 72 2286 6568 0 | 3836 : 75 3 10 886 0 | 106 156 2255 6568 0 | 7687 : 77 1 5 690 0 | 93 116 6511 6568 0 | 16616 : 72 4 3 14493 0 | 85 93 1697 6568 0 | 44634 : 43 80 7 12496 0 | 144 335 23079 6568 0 | 960 : 68 4 1 514 0 | 75 83 60 6568 0 | 1094 : 73 3 1 541 0 | 81 92 145 6568 0 | 2025 : 75 39 1 915 0 | 98 115 348 6568 0 | 50472 : 607 49 21 35868 0 | 696 815 4969 Isn't the unsual high numbers? All the examples I have seen, shows eptv == 0. Anybody have these problems before? Brgds, Toan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2538 bytes Desc: not available URL: From fabio at fabiorabelo.wiki.br Mon Aug 26 17:10:07 2013 From: fabio at fabiorabelo.wiki.br (=?UTF-8?Q?F=C3=A1bio_Rabelo?=) Date: Mon, 26 Aug 2013 14:10:07 -0300 Subject: [OmniOS-discuss] 10GB adapter Message-ID: This Intel card works with OmniOS ?? http://www.provantage.com/intel-x540t1~7ITEN0HQ.htm F?bio Rabelo From esproul at omniti.com Mon Aug 26 17:30:22 2013 From: esproul at omniti.com (Eric Sproul) Date: Mon, 26 Aug 2013 13:30:22 -0400 Subject: [OmniOS-discuss] 10GB adapter In-Reply-To: References: Message-ID: On Mon, Aug 26, 2013 at 1:10 PM, F?bio Rabelo wrote: > This Intel card works with OmniOS ?? > > http://www.provantage.com/intel-x540t1~7ITEN0HQ.htm If it's on http://illumos.org/hcl then, likely, yes. Once you think you know its PCI ID you can also look for it in: grep ixgbe /etc/driver_aliases on an existing OmniOS install. From henson at acm.org Mon Aug 26 18:14:52 2013 From: henson at acm.org (Paul B. Henson) Date: Mon, 26 Aug 2013 11:14:52 -0700 Subject: [OmniOS-discuss] ldap auth In-Reply-To: <20130824195038.GK7098@bender.unx.csupomona.edu> References: <5217DA9E.3020909@uw.edu> <20130824023004.GI7098@bender.unx.csupomona.edu> <20130824195038.GK7098@bender.unx.csupomona.edu> Message-ID: <521B9B1C.5090207@acm.org> On 8/24/2013 12:50 PM, Paul B. Henson wrote: > RHEL6 evidentally has the same issue, which can be worked around by > setting the environment variable NSS_HASH_ALG_SUPPORT=+MD5. > > If the same workaround resolves the issue under omnios, then > > # svccfg -s network/ldap/client:default setenv NSS_HASH_ALG_SUPPORT +MD5 > > should make the ldap client work, I believe all ldap connections are > routed through the cache manager. For the sake of the archives, setting the environment variable just for network/ldap/client did not work. However, setting it globally in /etc/default/init and rebooting did. From hakansom at ohsu.edu Mon Aug 26 18:52:58 2013 From: hakansom at ohsu.edu (Marion Hakanson) Date: Mon, 26 Aug 2013 11:52:58 -0700 Subject: [OmniOS-discuss] 10GB adapter In-Reply-To: Message from Eric Sproul of "Mon, 26 Aug 2013 13:30:22 EDT." Message-ID: <201308261852.r7QIqwvd000823@kyklops.ohsu.edu> On Mon, Aug 26, 2013 at 1:10 PM, F?bio Rabelo wrote: > This Intel card works with OmniOS ?? > > http://www.provantage.com/intel-x540t1~7ITEN0HQ.htm We have the 2-port version (X540-T2) running on OpenIndiana 151a7 with no issues. The 1-port version uses the same driver (we had one running on Solaris-11 awhile back). Since OmniOS has newer Illumos code than oi151a7, it should just work. Regards, Marion From henrikj at henkis.net Mon Aug 26 19:36:07 2013 From: henrikj at henkis.net (Henrik Johansson) Date: Mon, 26 Aug 2013 21:36:07 +0200 Subject: [OmniOS-discuss] non-amd64 code depends on amd64 privileged header! Message-ID: Hello, I am trying OmniOS as a replacement for my OpenIndiana deployment. I did however get into trouble when compiling netatalk, and based on Google the problem is not limited to netatalk. I am uncertain if this is a generic illumos problem introduced after OpenIndiana stagnated or if it is something unique to OmniOS. Here it is: gmake[3]: Entering directory `/zpool01/build/netatalk/netatalk-3.0.5/etc/uams' CC uams_guest.lo In file included from /usr/include/sys/regset.h:420:0, from /usr/include/sys/ucontext.h:36, from /usr/include/sys/signal.h:245, from /usr/include/sys/procset.h:42, from /usr/include/sys/wait.h:43, from /usr/include/stdlib.h:38, from uams_guest.c:11: /usr/include/amd64/sys/privregs.h:42:2: error: #error "non-amd64 code depends on amd64 privileged header!" This was using GCC 4.7 but I get the same error with 4.4. If i remove the check from the privregs.h header everything seems to compile fine. Regards Henrik From jesus at omniti.com Mon Aug 26 19:55:40 2013 From: jesus at omniti.com (Theo Schlossnagle) Date: Tue, 27 Aug 2013 03:55:40 +0800 Subject: [OmniOS-discuss] non-amd64 code depends on amd64 privileged header! In-Reply-To: References: Message-ID: CC uams_guest.lo isn't descriptive enough. Can you please provide the entire verbatim compile command? On Tue, Aug 27, 2013 at 3:36 AM, Henrik Johansson wrote: > Hello, > > I am trying OmniOS as a replacement for my OpenIndiana deployment. I did > however get into trouble when compiling netatalk, and based on Google the > problem is not limited to netatalk. I am uncertain if this is a generic > illumos problem introduced after OpenIndiana stagnated or if it is > something unique to OmniOS. > > Here it is: > gmake[3]: Entering directory > `/zpool01/build/netatalk/netatalk-3.0.5/etc/uams' > CC uams_guest.lo > In file included from /usr/include/sys/regset.h:420:0, > from /usr/include/sys/ucontext.h:36, > from /usr/include/sys/signal.h:245, > from /usr/include/sys/procset.h:42, > from /usr/include/sys/wait.h:43, > from /usr/include/stdlib.h:38, > from uams_guest.c:11: > /usr/include/amd64/sys/privregs.h:42:2: error: #error "non-amd64 code > depends on amd64 privileged header!" > > This was using GCC 4.7 but I get the same error with 4.4. If i remove the > check from the privregs.h header everything seems to compile fine. > > Regards > Henrik > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From esproul at omniti.com Mon Aug 26 19:55:47 2013 From: esproul at omniti.com (Eric Sproul) Date: Mon, 26 Aug 2013 15:55:47 -0400 Subject: [OmniOS-discuss] non-amd64 code depends on amd64 privileged header! In-Reply-To: References: Message-ID: I *think* this may be a general illumos issue, insofar as certain apps may be trying to do something that isn't likely to work properly. https://www.illumos.org/issues/3803 I looked at the commend from regset.h just above line 420 where privregs.h is included: /* * The version of privregs.h that is used on implementations that run on * processors that support the AMD64 instruction set is deliberately not * imported here. * * The amd64 'struct regs' definition may -not- compatible with either * 32-bit or 64-bit core file contents, nor with the ucontext. As a result, * the 'regs' structure cannot be used portably by applications, and should * only be used by the kernel implementation. * * The inclusion of the i386 version of privregs.h allows for some limited * source compatibility with 32-bit applications who expect to use * 'struct regs' to match the context of a 32-bit core file, or a ucontext_t. * * Note that the ucontext_t actually describes the general register in terms * of the gregset_t data type, as described in this file. Note also * that the core file content is defined by core(4) in terms of data types * defined by procfs -- see proc(4). */ #if defined(__i386) && \ (!defined(_KERNEL) && !defined(_XPG4_2) || defined(__EXTENSIONS__)) #include #endif /* __i386 (!_KERNEL && !_XPG4_2 || __EXTENSIONS__) */ It might help if you built netatalk 64-bit (gcc -m64), but I am by no means a kernel hacker, nor am I familiar enough with netatalk to say for sure. Eric On Mon, Aug 26, 2013 at 3:36 PM, Henrik Johansson wrote: > Hello, > > I am trying OmniOS as a replacement for my OpenIndiana deployment. I did however get into trouble when compiling netatalk, and based on Google the problem is not limited to netatalk. I am uncertain if this is a generic illumos problem introduced after OpenIndiana stagnated or if it is something unique to OmniOS. > > Here it is: > gmake[3]: Entering directory `/zpool01/build/netatalk/netatalk-3.0.5/etc/uams' > CC uams_guest.lo > In file included from /usr/include/sys/regset.h:420:0, > from /usr/include/sys/ucontext.h:36, > from /usr/include/sys/signal.h:245, > from /usr/include/sys/procset.h:42, > from /usr/include/sys/wait.h:43, > from /usr/include/stdlib.h:38, > from uams_guest.c:11: > /usr/include/amd64/sys/privregs.h:42:2: error: #error "non-amd64 code depends on amd64 privileged header!" > > This was using GCC 4.7 but I get the same error with 4.4. If i remove the check from the privregs.h header everything seems to compile fine. > > Regards > Henrik > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From henrikj at henkis.net Mon Aug 26 20:50:34 2013 From: henrikj at henkis.net (Henrik Johansson) Date: Mon, 26 Aug 2013 22:50:34 +0200 Subject: [OmniOS-discuss] non-amd64 code depends on amd64 privileged header! In-Reply-To: References: Message-ID: <813FDD78-45A8-4B3F-BB1F-58D7934512DF@henkis.net> It's a bit weird: /usr/include/sys/privregs.h includes /usr/include/amd64/sys/privregs.h when __amd64 is defined but while then it suddenly is no longed defined? First: #if defined(__amd64) #include And right after it bails on: #if !defined(__amd64) #error "non-amd64 code depends on amd64 privileged header!" Oh well, I try to look at it some more if I find time for it, otherwise it was good that an illumos bug have been filed. Regards Henrik On Aug 26, 2013, at 9:55 PM, Eric Sproul wrote: > I *think* this may be a general illumos issue, insofar as certain apps > may be trying to do something that isn't likely to work properly. > > https://www.illumos.org/issues/3803 > > I looked at the commend from regset.h just above line 420 where > privregs.h is included: > > /* > * The version of privregs.h that is used on implementations that run on > * processors that support the AMD64 instruction set is deliberately not > * imported here. > * > * The amd64 'struct regs' definition may -not- compatible with either > * 32-bit or 64-bit core file contents, nor with the ucontext. As a result, > * the 'regs' structure cannot be used portably by applications, and should > * only be used by the kernel implementation. > * > * The inclusion of the i386 version of privregs.h allows for some limited > * source compatibility with 32-bit applications who expect to use > * 'struct regs' to match the context of a 32-bit core file, or a ucontext_t. > * > * Note that the ucontext_t actually describes the general register in terms > * of the gregset_t data type, as described in this file. Note also > * that the core file content is defined by core(4) in terms of data types > * defined by procfs -- see proc(4). > */ > #if defined(__i386) && \ > (!defined(_KERNEL) && !defined(_XPG4_2) || defined(__EXTENSIONS__)) > #include > #endif /* __i386 (!_KERNEL && !_XPG4_2 || __EXTENSIONS__) */ > > It might help if you built netatalk 64-bit (gcc -m64), but I am by no > means a kernel hacker, nor am I familiar enough with netatalk to say > for sure. > > Eric > > On Mon, Aug 26, 2013 at 3:36 PM, Henrik Johansson wrote: >> Hello, >> >> I am trying OmniOS as a replacement for my OpenIndiana deployment. I did however get into trouble when compiling netatalk, and based on Google the problem is not limited to netatalk. I am uncertain if this is a generic illumos problem introduced after OpenIndiana stagnated or if it is something unique to OmniOS. >> >> Here it is: >> gmake[3]: Entering directory `/zpool01/build/netatalk/netatalk-3.0.5/etc/uams' >> CC uams_guest.lo >> In file included from /usr/include/sys/regset.h:420:0, >> from /usr/include/sys/ucontext.h:36, >> from /usr/include/sys/signal.h:245, >> from /usr/include/sys/procset.h:42, >> from /usr/include/sys/wait.h:43, >> from /usr/include/stdlib.h:38, >> from uams_guest.c:11: >> /usr/include/amd64/sys/privregs.h:42:2: error: #error "non-amd64 code depends on amd64 privileged header!" >> >> This was using GCC 4.7 but I get the same error with 4.4. If i remove the check from the privregs.h header everything seems to compile fine. >> >> Regards >> Henrik >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss From aney at ivision.fr Mon Aug 26 23:33:50 2013 From: aney at ivision.fr (Alexandre NEY) Date: Mon, 26 Aug 2013 23:33:50 +0000 Subject: [OmniOS-discuss] Can't delete old LU file Message-ID: <8FB4CA43FEE0C5468AC9AD5F74F1BD4314890920@ivision-mailbox.Exchange.local> Hello list, I'm having an issue deleting a file, i guess it's more of a cosmetic problem because it's not really using any space but I'd like to understand. I have a few LUs based on files that I present with COMSTAR. Somehow during the creation of some LUs I think I created one with the name zfs01cdp02 then deleted it then created it again. What I have now are 2 files : -rwxrwxrwx+ 1 root root 3.1T Aug 27 01:28 zfs01cdp02 -rwxrwxrwx+ 1 root root 4.5K Jul 10 19:06 zfs01cdp02_ One is actively used as a LU, the other one not. This is the content of the file : root at ivi-zfs01:/POOL01/cdp01# cat zfs01cdp02_ ULDBSNUS?f0?1 ??`D?] Q?/POOL01/cdp01/zfs01cdp02_? ???root at ivi-zfs01:/POOL01/cdp01# I'm trying to delete the file but I always get "access denied" even as root. root at ivi-zfs01:/POOL01/cdp01# rm -rf zfs01cdp02_ rm: cannot remove 'zfs01cdp02_': Permission denied It does not show up in the list when running sbdadm list-lu Any hints ? Thanks ! Alexandre NEY From gate03 at landcroft.co.uk Tue Aug 27 06:46:09 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Tue, 27 Aug 2013 16:46:09 +1000 Subject: [OmniOS-discuss] NBD on OmniOS Message-ID: <20130827164609.7c8aa9db@dickless.landy.net> Network Block Device server. Can't find it in any published repo. I'm using a diskless thin client workstation with NFS root and want to experiment with alternatives as it's pretty slow. Michael. From christian.flaig at gmail.com Tue Aug 27 07:53:08 2013 From: christian.flaig at gmail.com (Christian Flaig) Date: Tue, 27 Aug 2013 09:53:08 +0200 Subject: [OmniOS-discuss] SabNZBD - unrar Message-ID: <73405C4B-03E2-445D-A935-1656AC6448D9@gmail.com> Hello all, I'm trying to set up SabNZBD in a zone, has anyone done this and created some instructions? I'm already struggling with the first step, getting unrar. Where can I find unrar in OmniOS? Thanks alot for your help. Regards, Chris From johan.kragsterman at capvert.se Tue Aug 27 07:57:36 2013 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Tue, 27 Aug 2013 09:57:36 +0200 Subject: [OmniOS-discuss] Ang: NBD on OmniOS In-Reply-To: <20130827164609.7c8aa9db@dickless.landy.net> References: <20130827164609.7c8aa9db@dickless.landy.net> Message-ID: An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Tue Aug 27 07:59:59 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Tue, 27 Aug 2013 09:59:59 +0200 Subject: [OmniOS-discuss] NBD on OmniOS In-Reply-To: <20130827164609.7c8aa9db@dickless.landy.net> References: <20130827164609.7c8aa9db@dickless.landy.net> Message-ID: <521C5C7F.9030408@cos.ru> On 2013-08-27 08:46, Michael Mounteney wrote: > Network Block Device server. Can't find it in any published repo. Isn't iSCSI the same - only better? I am not sure there are linux-type NBDs in Solaris family at all. > I'm using a diskless thin client workstation with NFS root and want to > experiment with alternatives as it's pretty slow. Did you analyze why it is slow? Different NFS versions may perform differently (i.e. NFSv4 may be slower than older ones), or network related settings (from low-level up to NFS transfer sizes) may be suboptimal. Also, the ZFS pool backing the NFS server may be slow to write (all NFS I/O is sync, and you may require an SSD log device to speed this up - seek dtrace scripts that would help you analyze beforehand if you have any sync IO that may be the culprit, or temporarily disable sync I/O on the datasets which serve your NFS roots and subjectively "feel" if things improve or not). If the sync IO is the problem, then likely a responsible block-based system (like iSCSI and ZFS) would also do sync updates and suffer the same lags. HTH, //Jim From jimklimov at cos.ru Tue Aug 27 08:02:03 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Tue, 27 Aug 2013 10:02:03 +0200 Subject: [OmniOS-discuss] SabNZBD - unrar In-Reply-To: <73405C4B-03E2-445D-A935-1656AC6448D9@gmail.com> References: <73405C4B-03E2-445D-A935-1656AC6448D9@gmail.com> Message-ID: <521C5CFB.8070706@cos.ru> On 2013-08-27 09:53, Christian Flaig wrote: > Hello all, > > I'm trying to set up SabNZBD in a zone, has anyone done this and created some instructions? I'm already struggling with the first step, getting unrar. Where can I find unrar in OmniOS? Would this help? http://www.rarlab.com/rar_add.htm There are sources as well as packages for many OSes including Solaris - which may work on illumos too. If not, OTOH the 7z program should support work with RAR archives. //Jim From cnehren+omnios-discuss at omniti.com Tue Aug 27 13:09:46 2013 From: cnehren+omnios-discuss at omniti.com (Chris Nehren) Date: Tue, 27 Aug 2013 09:09:46 -0400 Subject: [OmniOS-discuss] SabNZBD - unrar In-Reply-To: <73405C4B-03E2-445D-A935-1656AC6448D9@gmail.com> References: <73405C4B-03E2-445D-A935-1656AC6448D9@gmail.com> Message-ID: <20130827130946.GH24680@eschaton.local> On Tue, Aug 27, 2013 at 09:53:08 +0200, Christian Flaig wrote: > Hello all, > > I'm trying to set up SabNZBD in a zone, has anyone done this and created > some instructions? I'm already struggling with the first step, getting > unrar. Where can I find unrar in OmniOS? > > Thanks alot for your help. http://omnios.omniti.com/wiki.php/Packaging lists third-party, unsupported-by-OmniTI repos. Jorge Schrauwen's repo at http://omnios.blackdot.be/ has sabnzbd, unrar, and probably everything else required. Give that a shot and see how things go? -- Chris Nehren From cks at cs.toronto.edu Tue Aug 27 13:29:53 2013 From: cks at cs.toronto.edu (Chris Siebenmann) Date: Tue, 27 Aug 2013 09:29:53 -0400 Subject: [OmniOS-discuss] NBD on OmniOS In-Reply-To: jimklimov's message of Tue, 27 Aug 2013 09:59:59 +0200. <521C5C7F.9030408@cos.ru> Message-ID: <20130827132953.BA0731A0314@apps0.cs.toronto.edu> | Also, the ZFS pool backing the NFS server may be slow to write (all | NFS I/O is sync, and you may require an SSD log device to speed this | up - seek dtrace scripts that would help you analyze beforehand if you | have any sync IO that may be the culprit, or temporarily disable sync | I/O on the datasets which serve your NFS roots and subjectively "feel" | if things improve or not). [...] These days this is only half true unless you are still using NFS v2 (you shouldn't be, NFS v3 is better). NFS metadata operations such as renaming files are synchronous but writing data is only periodically synchronous. The NFS client normally issues a stream of asynchronous writes and then periodically asks the NFS server to flush all pending writes (this is the NFS v3 'commit' operation, for people using dtrace to watch NFS server activity). How frequently NFS clients issue NFS commit operations depends on how fast they're writing how much (and also how they're configured; the client must keep a copy of all uncommitted data in case the server tells it 'oops, send that again' when the client sends in the commit operation). - cks From thibault.vincent at smartjog.com Tue Aug 27 15:57:00 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Tue, 27 Aug 2013 17:57:00 +0200 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <52108A50.6030103@cos.ru> References: <52108A50.6030103@cos.ru> Message-ID: <521CCC4C.9020400@smartjog.com> On 08/18/2013 10:48 AM, Jim Klimov wrote: > Does programmatic ungraceful reboot work or also hang? > See the "uadmin" command for flags, I think the reboot > would be "uadmin 1 2"... After 7 attempts it seems pretty sure uadmin reboot works everytime :) What could that suggest ? I was thinking about some kind of blocking I/O or device since the server was tested in these situations, but that doesn't make much sense.. Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From skiselkov.ml at gmail.com Tue Aug 27 16:02:56 2013 From: skiselkov.ml at gmail.com (Saso Kiselkov) Date: Tue, 27 Aug 2013 17:02:56 +0100 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <521CCC4C.9020400@smartjog.com> References: <52108A50.6030103@cos.ru> <521CCC4C.9020400@smartjog.com> Message-ID: <521CCDB0.8040506@gmail.com> On 8/27/13 4:57 PM, Thibault VINCENT wrote: > On 08/18/2013 10:48 AM, Jim Klimov wrote: >> Does programmatic ungraceful reboot work or also hang? >> See the "uadmin" command for flags, I think the reboot >> would be "uadmin 1 2"... > > After 7 attempts it seems pretty sure uadmin reboot works > everytime :) What could that suggest ? I was thinking about > some kind of blocking I/O or device since the server was > tested in these situations, but that doesn't make much sense.. You wouldn't by any coincidence happen to be running COMSTAR? :) Cheers, -- Saso From thibault.vincent at smartjog.com Tue Aug 27 20:14:21 2013 From: thibault.vincent at smartjog.com (Thibault VINCENT) Date: Tue, 27 Aug 2013 22:14:21 +0200 Subject: [OmniOS-discuss] R720 reset magic? In-Reply-To: <521CCDB0.8040506@gmail.com> References: <52108A50.6030103@cos.ru> <521CCC4C.9020400@smartjog.com> <521CCDB0.8040506@gmail.com> Message-ID: <521D089D.5010706@smartjog.com> On 08/27/2013 06:02 PM, Saso Kiselkov wrote: >> After 7 attempts it seems pretty sure uadmin reboot works >> everytime :) What could that suggest ? I was thinking about >> some kind of blocking I/O or device since the server was >> tested in these situations, but that doesn't make much sense.. > > You wouldn't by any coincidence happen to be running COMSTAR? :) Not at all.. As said in the ticket it happens with freshly installed OmniOS, and it's quite possible the only times I see a reboot work is when the server was started minutes ago. On the other hand it also stalls when running for some time while didn't even got an ssh client, I don't say I always run stuff before the issue is seen. Uptime just seem to matter as it's easy to get a reboot sooner. Cheers -- Thibault VINCENT - Infrastructure Engineer SmartJog | T: +33 1 5868 6238 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France www.smartjog.com | a TDF Group company From geoffn at gnaa.net Wed Aug 28 05:44:56 2013 From: geoffn at gnaa.net (Geoff Nordli) Date: Tue, 27 Aug 2013 22:44:56 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries Message-ID: <521D8E58.6020703@gnaa.net> I followed the thread to get the ssh -X working properly by using pkgsrc and installing xauth plus adding: XAuthLocation /opt/pkg/bin/xauth to the ssh config. Now I am trying to run VirtualBox gui remotely. cho $LD_LIBRARY_PATH /opt/pkg/lib/:/opt/pkg/lib/X11/ /usr/bin/VirtualBox VirtualBox: supR3HardenedMainGetTrustedMain: dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: VirtualBox: fatal: libXinerama.so.1: open failed: No such file or directory The files are there: ls -al /opt/pkg/lib/ | grep -i libxinerama.so lrwxrwxrwx 1 root root 20 Oct 6 2012 libXinerama.so -> libXinerama.so.1.0.0 lrwxrwxrwx 1 root root 20 Oct 6 2012 libXinerama.so.1 -> libXinerama.so.1.0.0 -rwxr-xr-x 1 root root 14848 Oct 6 2012 libXinerama.so.1.0.0 Any other ideas on how to get it to find the libraries? thanks, Geoff From henson at acm.org Wed Aug 28 06:30:13 2013 From: henson at acm.org (Paul B. Henson) Date: Tue, 27 Aug 2013 23:30:13 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521D8E58.6020703@gnaa.net> References: <521D8E58.6020703@gnaa.net> Message-ID: <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: > /usr/bin/VirtualBox > VirtualBox: supR3HardenedMainGetTrustedMain: dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: VirtualBox: fatal: libXinerama.so.1: open failed: No such file or directory [...] > The files are there: [...] > -rwxr-xr-x 1 root root 14848 Oct 6 2012 libXinerama.so.1.0.0 I'm guessing this is a 32 bit library and hence unusable by the 64 bit virtualbox binary? What does "file libXinerama.so.1.0.0" have to say? From jimklimov at cos.ru Wed Aug 28 08:43:36 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Wed, 28 Aug 2013 10:43:36 +0200 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> Message-ID: <521DB838.2050104@cos.ru> On 2013-08-28 08:30, Paul B. Henson wrote: > On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: > >> /usr/bin/VirtualBox >> VirtualBox: supR3HardenedMainGetTrustedMain: dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: VirtualBox: fatal: libXinerama.so.1: open failed: No such file or directory > [...] >> The files are there: > [...] >> -rwxr-xr-x 1 root root 14848 Oct 6 2012 libXinerama.so.1.0.0 > > I'm guessing this is a 32 bit library and hence unusable by the 64 bit virtualbox binary? > > What does "file libXinerama.so.1.0.0" have to say? That would find the file, but complain about ELFCLASS32 at a later step (i.e. binary not applicable to dynamically link with current code). If a "amd64/libXinerama.so..." (or some other directory euphemism like "64/libX...") exists under the same libdir, it should be picked up automatically. If the library is not available, I wonder if the 32-bit GUI works and if it would be able to launch 64-bit processes (helpers and headless VMs) that don't need GUI? For console access you could then use RDP or VNC (if available in your VirtualBox build). Actually, since the disconnection of ssh (X-server) would be fatal to its GUI client (VM), I'd be in favour of either using headless VMs (see my vboxsvc project on sourceforge, to run VMs as SMF instances and more) or making some VNC desktop on the VM host and using it to manage the VMs. It does not have to be a heavy GNOME environment, I use a bare-bone twm just fine. Anything that can render X11 and has a window-manager to move and overlap the windows ;) This seems more like a problem of needing a fixed LD_LIBRARY_PATH (I see that you seem to have a correct one, though) in the invokation, passed from ssh by command-line, or the remote host's sshd config, or via the user's .profile, or by some system-wide setting (/etc/default/login ? or crle etc.) Though the latter approach has recently become a matter of near-religious flame wars, especially to be portable between distros and packages with non-standard paths - like /opt/pkg/lib - and perhaps there are better solutions... that is, unless they involve full-blown recompilation of third-party programs and libraries ;) HTH, //Jim From gate03 at landcroft.co.uk Wed Aug 28 10:25:26 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Wed, 28 Aug 2013 20:25:26 +1000 Subject: [OmniOS-discuss] Ang: NBD on OmniOS In-Reply-To: References: <20130827164609.7c8aa9db@dickless.landy.net> Message-ID: <20130828202526.69f202e4@dickless.landy.net> On Tue, 27 Aug 2013 09:57:36 +0200 Johan Kragsterman wrote: > Sorry, can't provide you with the info about NBD, but: > > I'd be interested in your setup. I use thin clients a lot, but so far > only on Linux. I'd love to be able to use omnios as a base server for > that! > > What chroot do you use? More than one? DHCP server setup? TFTP server > setup? Following other suggestions, I've set up iSCSI and now intend to perform some performance tests to see if it's worth pursuing. Mine is a very simple setup. The root file system is one ZFS on the server. The thin client has 2 GiB so I don't bother wth TFTP; i just have a small partition of the flash as a boot FS. DHCP, again, no. I only have one client so just assign a unique static address. The Linux kernel is Gentoo with an NFS3 client built in (obviously). I'm not sure what's stopping your moving, if that's what you want to do. Exporting NFS from OmniOS is trivial, if you're familiar with ZFS, and NFS-servers concepts. If you want to set up DHCP and TFTP as well, I can't help you with that (yet). Sorry. Michael. From geoffn at gnaa.net Wed Aug 28 15:48:46 2013 From: geoffn at gnaa.net (Geoff Nordli) Date: Wed, 28 Aug 2013 08:48:46 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> Message-ID: <521E1BDE.80709@gnaa.net> On 13-08-27 11:30 PM, Paul B. Henson wrote: > On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: > >> /usr/bin/VirtualBox >> VirtualBox: supR3HardenedMainGetTrustedMain: dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: VirtualBox: fatal: libXinerama.so.1: open failed: No such file or directory > [...] >> The files are there: > [...] >> -rwxr-xr-x 1 root root 14848 Oct 6 2012 libXinerama.so.1.0.0 > I'm guessing this is a 32 bit library and hence unusable by the 64 bit virtualbox binary? > > What does "file libXinerama.so.1.0.0" have to say? > Thanks Paul. library mismatch explains it....... root at stor1:~# file /opt/VirtualBox/VirtualBox /opt/VirtualBox/VirtualBox: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, not stripped root at stor1:~# file /opt/pkg/lib/libXinerama.so.1.0.0 /opt/pkg/lib/libXinerama.so.1.0.0: ELF 64-bit LSB dynamic lib AMD64 Version 1, dynamically linked, not stripped, no debugging information available I only use the GUI to get the VM running and then I switch to headless, so it doesn't really matter. Geoff From geoffn at gnaa.net Wed Aug 28 16:07:17 2013 From: geoffn at gnaa.net (Geoff Nordli) Date: Wed, 28 Aug 2013 09:07:17 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521DB838.2050104@cos.ru> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> <521DB838.2050104@cos.ru> Message-ID: <521E2035.2020800@gnaa.net> On 13-08-28 01:43 AM, Jim Klimov wrote: > On 2013-08-28 08:30, Paul B. Henson wrote: >> On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: >> >>> /usr/bin/VirtualBox >>> VirtualBox: supR3HardenedMainGetTrustedMain: >>> dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: >>> VirtualBox: fatal: libXinerama.so.1: open failed: No such file or >>> directory >> [...] >>> The files are there: >> [...] >>> -rwxr-xr-x 1 root root 14848 Oct 6 2012 >>> libXinerama.so.1.0.0 >> >> I'm guessing this is a 32 bit library and hence unusable by the 64 >> bit virtualbox binary? >> >> What does "file libXinerama.so.1.0.0" have to say? > > That would find the file, but complain about ELFCLASS32 at a later step > (i.e. binary not applicable to dynamically link with current code). > If a "amd64/libXinerama.so..." (or some other directory euphemism like > "64/libX...") exists under the same libdir, it should be picked up > automatically. > > If the library is not available, I wonder if the 32-bit GUI works and > if it would be able to launch 64-bit processes (helpers and headless > VMs) that don't need GUI? For console access you could then use RDP > or VNC (if available in your VirtualBox build). Actually, since the > disconnection of ssh (X-server) would be fatal to its GUI client (VM), > I'd be in favour of either using headless VMs (see my vboxsvc project > on sourceforge, to run VMs as SMF instances and more) or making some > VNC desktop on the VM host and using it to manage the VMs. It does not > have to be a heavy GNOME environment, I use a bare-bone twm just fine. > Anything that can render X11 and has a window-manager to move and > overlap the windows ;) > > This seems more like a problem of needing a fixed LD_LIBRARY_PATH (I see > that you seem to have a correct one, though) in the invokation, passed > from ssh by command-line, or the remote host's sshd config, or via the > user's .profile, or by some system-wide setting (/etc/default/login ? > or crle etc.) Though the latter approach has recently become a matter > of near-religious flame wars, especially to be portable between distros > and packages with non-standard paths - like /opt/pkg/lib - and perhaps > there are better solutions... that is, unless they involve full-blown > recompilation of third-party programs and libraries ;) > Thanks Jim. I think I will abandon the quest to run VirtualBox gui over ssh. I only use the GUI to configure the VM and get it running. Once it is ready to go, then I switch to vboxsvc, which is awesome :), and use rdp to view the VMs. I have some scripts that I will stick to. Geoff From jimklimov at cos.ru Wed Aug 28 17:22:25 2013 From: jimklimov at cos.ru (Jim Klimov) Date: Wed, 28 Aug 2013 19:22:25 +0200 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521E1BDE.80709@gnaa.net> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> <521E1BDE.80709@gnaa.net> Message-ID: <521E31D1.8030509@cos.ru> On 2013-08-28 17:48, Geoff Nordli wrote: > On 13-08-27 11:30 PM, Paul B. Henson wrote: >> On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: >> >>> /usr/bin/VirtualBox >>> VirtualBox: supR3HardenedMainGetTrustedMain: >>> dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: >>> VirtualBox: fatal: libXinerama.so.1: open failed: No such file or >>> directory > root at stor1:~# file /opt/VirtualBox/VirtualBox > /opt/VirtualBox/VirtualBox: ELF 32-bit LSB executable 80386 Version > 1 [FPU], dynamically linked, not stripped Well, you're comparing fruits to flies here ;) # ls -la /usr/bin/VirtualBox lrwxrwxrwx 1 root root 28 Dec 10 2009 /usr/bin/VirtualBox -> ../../opt/VirtualBox/VBox.sh The script selects your architecture: ... CURRENT_ISA=`isainfo -k` if test "$CURRENT_ISA" = "amd64"; then INSTALL_DIR="/opt/VirtualBox/amd64" else INSTALL_DIR="/opt/VirtualBox/i386" fi APP=`which $0` APP=`basename $APP` case "$APP" in ... And according to the error coming from the attempted dynamic link of dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) - this does run your amd64 binary. I might only guess that your LD_LIBRARY_PATH with /opt/pkg/lib is somehow overridden or ignored (not "export"ed?) by subsequent programs. Try hardcoding it into the VBox.sh script to see if it helps? LD_LIBRARY_PATH=/opt/pkg/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH > > root at stor1:~# file /opt/pkg/lib/libXinerama.so.1.0.0 > /opt/pkg/lib/libXinerama.so.1.0.0: ELF 64-bit LSB dynamic lib AMD64 > Version 1, dynamically linked, not stripped, no debugging information > available Now, against my old Solaris habits, this packaging looks wrong :) The base directory historically contains the 32-bit stuff (and maybe multiarch if implemented), and 64-bit things are in their own subdir. I wonder if an opposite is catered for... I know there is a tendency to eradicate 32-bitness in a number of OSes, but legacy 32-bit binaries won't (shouldn't) link with a 64-bit library and vice-versa. Normally, both should be provided one way or another. From geoffn at gnaa.net Wed Aug 28 19:37:19 2013 From: geoffn at gnaa.net (Geoff Nordli) Date: Wed, 28 Aug 2013 12:37:19 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521E31D1.8030509@cos.ru> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> <521E1BDE.80709@gnaa.net> <521E31D1.8030509@cos.ru> Message-ID: <521E516F.1050503@gnaa.net> On 13-08-28 10:22 AM, Jim Klimov wrote: > On 2013-08-28 17:48, Geoff Nordli wrote: >> On 13-08-27 11:30 PM, Paul B. Henson wrote: >>> On Aug 27, 2013, at 10:44 PM, Geoff Nordli wrote: >>> >>>> /usr/bin/VirtualBox >>>> VirtualBox: supR3HardenedMainGetTrustedMain: >>>> dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) failed: ld.so.1: >>>> VirtualBox: fatal: libXinerama.so.1: open failed: No such file or >>>> directory >> root at stor1:~# file /opt/VirtualBox/VirtualBox >> /opt/VirtualBox/VirtualBox: ELF 32-bit LSB executable 80386 Version >> 1 [FPU], dynamically linked, not stripped > > > Well, you're comparing fruits to flies here ;) > > # ls -la /usr/bin/VirtualBox > lrwxrwxrwx 1 root root 28 Dec 10 2009 > /usr/bin/VirtualBox -> ../../opt/VirtualBox/VBox.sh > > The script selects your architecture: > ... > CURRENT_ISA=`isainfo -k` > if test "$CURRENT_ISA" = "amd64"; then > INSTALL_DIR="/opt/VirtualBox/amd64" > else > INSTALL_DIR="/opt/VirtualBox/i386" > fi > APP=`which $0` > APP=`basename $APP` > case "$APP" in > ... > > And according to the error coming from the attempted dynamic link of > dlopen("/opt/VirtualBox/amd64/VirtualBox.so",) - this does run your > amd64 binary. > > I might only guess that your LD_LIBRARY_PATH with /opt/pkg/lib is > somehow overridden or ignored (not "export"ed?) by subsequent programs. > Try hardcoding it into the VBox.sh script to see if it helps? > > LD_LIBRARY_PATH=/opt/pkg/lib:$LD_LIBRARY_PATH > export LD_LIBRARY_PATH Making some progress, can start the GUI, but it doesn't display the proper characters. The LD path needs to include VirtualBox as well: LD_LIBRARY_PATH=/opt/VirtualBox:/opt/pkg/lib I went through the process to try to resolve all of the libraries which are missing. I installed the following: pkgin install libXrender pkgin install libXfixes pkgin install png pkgin install freetype2 pkgin install fontconfig This is where things get kind of dicey, because there are library mismatches. so I created some links: ln -s libX11.so libX11.so.4 ln -s libXt.so libXt.so.4 ln -s libXfixes.so libXfixes.so.1 ln -s libpng15.so libpng12.so.0 I am not sure if this is the show stopper, but it complains: libpng warning: Application built with libpng-1.2.35 but running with 1.5.12 The GUI displays a bunch of square characters, but it looks like the functionality is there, since I can click on the links where they should be. http://picpaste.com/Selection_199-rtmxZ76O.png Not a high priority, but would be fun to sort out. Geoff From henson at acm.org Wed Aug 28 20:07:41 2013 From: henson at acm.org (Paul B. Henson) Date: Wed, 28 Aug 2013 13:07:41 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521E516F.1050503@gnaa.net> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> <521E1BDE.80709@gnaa.net> <521E31D1.8030509@cos.ru> <521E516F.1050503@gnaa.net> Message-ID: <521E588D.9030702@acm.org> On 8/28/2013 12:37 PM, Geoff Nordli wrote: > ln -s libX11.so libX11.so.4 > ln -s libXt.so libXt.so.4 > ln -s libXfixes.so libXfixes.so.1 > ln -s libpng15.so libpng12.so.0 [...] > libpng warning: Application built with libpng-1.2.35 but running with > 1.5.12 Hmm, it's not generally a good practice to shoehorn the wrong version of a library into an application 8-/. The soname change generally indicates the ABI is no longer compatible. It's almost guaranteed something will be broken, if the application runs at all. To make it work correctly, you'd have to compile the versions of the libraries to match the versions the binary wants. From geoffn at gnaa.net Wed Aug 28 21:15:29 2013 From: geoffn at gnaa.net (Geoff Nordli) Date: Wed, 28 Aug 2013 14:15:29 -0700 Subject: [OmniOS-discuss] running X application over ssh can't find libraries In-Reply-To: <521E588D.9030702@acm.org> References: <521D8E58.6020703@gnaa.net> <3D51AB9A-2EA9-4D86-82A6-450DB77E925B@acm.org> <521E1BDE.80709@gnaa.net> <521E31D1.8030509@cos.ru> <521E516F.1050503@gnaa.net> <521E588D.9030702@acm.org> Message-ID: <521E6871.9070701@gnaa.net> On 13-08-28 01:07 PM, Paul B. Henson wrote: > On 8/28/2013 12:37 PM, Geoff Nordli wrote: > >> ln -s libX11.so libX11.so.4 >> ln -s libXt.so libXt.so.4 >> ln -s libXfixes.so libXfixes.so.1 >> ln -s libpng15.so libpng12.so.0 > [...] >> libpng warning: Application built with libpng-1.2.35 but running with >> 1.5.12 > > Hmm, it's not generally a good practice to shoehorn the wrong version > of a library into an application 8-/. The soname change generally > indicates the ABI is no longer compatible. It's almost guaranteed > something will be broken, if the application runs at all. > > To make it work correctly, you'd have to compile the versions of the > libraries to match the versions the binary wants. Yes, for sure, it is kind of dicey to play with the versions. It isn't an important application for me to run. I was just curious if I could get it running on OmniOS. From gate03 at landcroft.co.uk Thu Aug 29 06:49:46 2013 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Thu, 29 Aug 2013 16:49:46 +1000 Subject: [OmniOS-discuss] NBD on OmniOS In-Reply-To: <521C5C7F.9030408@cos.ru> References: <20130827164609.7c8aa9db@dickless.landy.net> <521C5C7F.9030408@cos.ru> Message-ID: <20130829164946.5aff4269@dickless.landy.net> On Tue, 27 Aug 2013 09:59:59 +0200 Jim Klimov wrote: > Did you analyze why it is slow? Not why, but how much. I ran bonnie and it produced these figures, for 1. Standard NFS root. 2. NFS root with zfs set sync=disabled 3. iSCSI Switch to a fixed-pitch font to read this properly: -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU NFS 2000 9100 88.8 9495 3.7 4664 7.0 9485 83.8 51780 31.6 642.7 10.1 NFSasync 2000 9452 89.9 16085 7.5 12697 24.6 9505 83.7 85361 34.8 5336.4 69.0 iSCSI 2000 9885 97.0 15523 13.5 16650 16.9 11623 99.1 114326 46.4 13092.1 99.5 so iSCSI is faster, but it also sucks CPU, which is not surprising as it is doing the full filesystem thing rather than working at filename level and handing off work to the server. So I reckon that iSCSI won't be faster as a root filesystem, when the CPU is doing other stuff associated with booting and running. These tests were run while the machine was quiescent. Michael. From daniel at dgnetwork.com.br Fri Aug 30 18:26:26 2013 From: daniel at dgnetwork.com.br (=?ISO-8859-1?Q?=22Daniel_D=2E_Gon=E7alves=22?=) Date: Fri, 30 Aug 2013 15:26:26 -0300 Subject: [OmniOS-discuss] ZFS Questions. Is a bug ? In-Reply-To: <521677B2.8030105@gmail.com> References: <52165FCA.8070803@dgnetwork.com.br> <521663D0.8000804@gmail.com> <52167272.5010200@dgnetwork.com.br> <521677B2.8030105@gmail.com> Message-ID: <5220E3D2.70007@dgnetwork.com.br> My ZFS POOL is over a month doing RESILVER in LOOP thus ending one RESILVER after a few minutes, another starts. The replace command is never finished, did three days ago subsitution of a device, and it never ends: mirror-3 DEGRADED 0 0 28 c17t20d1 ONLINE 0 0 28 replacing-1 DEGRADED 28 0 0 c17t22d1 UNAVAIL 0 0 0 cannot open c17t13d1 ONLINE 0 0 28 (resilvering) In the mirror belowI would like to remove all devices with status UNAVAIL and do a replace again for a correct device, but the commands OFFLINE, REMOVE, DETACH not work: mirror-1 DEGRADED 28 0 0 c17t24d1 ONLINE 0 0 28 (resilvering) replacing-1 UNAVAIL 0 0 0 insufficient replicas c17t22d1 UNAVAIL 0 0 0 cannot open c17t12d1 UNAVAIL 0 0 0 cannot open c17t21d1 UNAVAIL 0 0 0 cannot open My entire POOL: pool: STORAGE01 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Aug 30 14:42:42 2013 530G scanned out of 18.4T at 227M/s, 23h1m to go 62.1G resilvered, 2.80% done config: NAME STATE READ WRITE CKSUM STORAGE01 DEGRADED 14 0 16 mirror-0 ONLINE 0 0 0 c17t15d1 ONLINE 0 0 0 c17t19d1 ONLINE 0 0 0 mirror-1 DEGRADED 28 0 0 c17t24d1 ONLINE 0 0 28 (resilvering) replacing-1 UNAVAIL 0 0 0 insufficient replicas c17t22d1 UNAVAIL 0 0 0 cannot open c17t12d1 UNAVAIL 0 0 0 cannot open c17t21d1 UNAVAIL 0 0 0 cannot open mirror-2 ONLINE 0 0 0 c17t18d1 ONLINE 0 0 0 (resilvering) c17t17d1 ONLINE 0 0 0 (resilvering) mirror-3 DEGRADED 0 0 32 c17t20d1 ONLINE 0 0 32 replacing-1 DEGRADED 32 0 0 c17t22d1 UNAVAIL 0 0 0 cannot open c17t13d1 ONLINE 0 0 32 (resilvering) mirror-5 ONLINE 0 0 0 c17t25d1 ONLINE 0 0 0 c17t27d1 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 c17t26d1 ONLINE 0 0 0 c17t28d1 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 c17t29d1 ONLINE 0 0 0 c17t31d1 ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 c17t32d1 ONLINE 0 0 0 c17t30d1 ONLINE 0 0 0 mirror-9 ONLINE 0 0 0 c17t23d1 ONLINE 0 0 0 c17t14d1 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 c14t1d0 ONLINE 0 0 0 c14t3d0 ONLINE 0 0 0 cache c14t4d0 ONLINE 0 0 0 Need urgent help to solve this. I believe it is a bug in ZFS. Thanks, Daniel Em 22/08/2013 17:42, Saso Kiselkov escreveu: > On 8/22/13 9:20 PM, "Daniel D. Gon?alves" wrote: >> Thanks Saso, >> >> To stop RESILVER, which device I to set to OFFLINE? > The one that says 'resilvering'. But beware that that means that the > pool might not have full fault tolerance. > >> I do not know how the device "c17t33d1" was placed in the >> MIRROR-11/REPLACING-1, how do I remove it from there? > If you can, let it run to completion before attempting any further > manipulation. The pool seems to be in quite an unhappy state anyway, so > better not compound the situation by doing more changes. Let the thing > resync back up, find the files that have the data errors in them ("zpool > status -v" I think), restore them or delete them and then post a new > "zpool status" to the list - then we'll see what can be done. > > Above all, be patient if you don't want to lose your data. > > Cheers,