[OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

[Bob Friesenhahn]

On Sat, 17 Aug 2013, Valrhona wrote:

> Thanks for the tip. If I understand correctly, lzop is basically a
> drop-in replacement for gzip that is much faster, but single-threaded.
> Is there a comparison anywhere between (single-threaded) lzop and
> (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of
> RAM so I expect the compression to be IO-bound rather than CPU-bound,
> but that is just a guess. And for ZFS streams, is there a reason to
> prefer any one of these programs over another? Thanks!

Lzop uses a completely different compression format.  Its default 
compression is a bit less compression than 'gzip -3' but it is much 
more CPU efficient so it is able to achieve "wire speed" level 
compression rates on modern CPUs, and without relying on threading. 
There are some other compressors which are even faster than lzop but 
their compression formats are often not stable.

A problem with most threaded compressors is that more effective 
compression algorithms (e.g. lzma as used by 'xz' and 'lzip') require 
very large data chunk sizes so that the compression algorithm works 
effectively.  This means that the input data size needs to be large 
(hundreds of MB or even gigabytes) in order for the multi-thread chunk 
size to be large enough.  Zfs send streams or tar/cpio archives of 
large directory trees are likely to be large enough but many/most 
ordinary files don't qualify.  As a result, using threading may result 
in much less compression.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/