[OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

[Valrhona]

y> I am not sure about repositories, but there are projects such as
> pigz and pbzip2 which are parallelized interfaces to the same
> compression libraries, and easily compilable
Thanks! Looks like pigz and pbzip2 are in the libraries; thanks Eric
for the tip!

> In some versions of pigz there was a problem with compression of
> multiple filename arguments (some state was not cleared, so they
> were processed as if concatenated), so for predictable results
> better script up loops and call each time for one argument :)
At the moment, I am just interested in a single file, piped in from
zfs send. Would the problem you mention affect me?

> You can look into examples on cpio manpage - it takes results of
> the find command as input, so you can "find" piles according to
> the name pattern(s) you need, and as part of their path they would
> include the directories which contain them. If you need the whole
> directory structure, try "find -type d" - though you might need
> to somehow tell cpio to only archive the directories and not their
> contents recursively. Or if you're doing this locally - just create
> the structure at new root node with "mkdir -p". For remote sending,
> you might do the same and then archive this empty structure ;)
So I found an easier solution: rsync

for files with extension .xyz

rsync -avgR --include='*/' --include='*.xyz' --exclude='*'
--remove-source-files /src /dest

This recreates the whole subdirectory structure, including (many)
directories which have no files of type .xyz.

To remove them, in /dest

find . -type d -empty -exec rmdir {} \;

You have to run this a few times because only one level of empty
directory gets removed each time, so if you have empty directories
that are subdirectories that then become empty, etc. etc.

Thanks!

Peter