[OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

[Jim Klimov]

On 2013-08-17 05:53, Valrhona wrote:
> Thanks for the tip. If I understand correctly, lzop is basically a
> drop-in replacement for gzip that is much faster, but single-threaded.
> Is there a comparison anywhere between (single-threaded) lzop and
> (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of
> RAM so I expect the compression to be IO-bound rather than CPU-bound,
> but that is just a guess. And for ZFS streams, is there a reason to
> prefer any one of these programs over another? Thanks!

Just in case, you might also want to consider 7zip - I think it is
parallel out of the box, and might offer best compression of them
all (if your backups happen to be more constrained by space than IO)
though not all versions support stdin|stdout compression.

Also note that the common mailing-list wisdom argues against relying
on ZFS-send streams in files as a backup method. Unlike ZFS itself,
the streams offer little to no protection against bitrot, except that
if a stream is corrupt - this is detectable and you can no longer
import it. When the streams are used "live", piped to zfs receive
to save into a target dataset, the transport error can be detected
(at least by the admin or scripts that do the operation) and retried.
When the stream file image is your only medium during restoration
(especially if it is stored not on ZFS which would protect the file's
backend storage) - by Murphy's Law you can bet to find the needed data
un-importable.

*Possibly*, things like ZIP CRC or RAR "1% redundancy" or some similar
ECC-like schemes in other formats (if available) might help you against
occasional bitrot (though likely not loss of larger chunks of data like
a whole sector-size) - I am not ready to elaborate on this further,
maybe some more knowledgeable experts would step in and say if this is
just marketing voodoo or something that really works ;)

HTH,
//Jim