[OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

[Jim Klimov]

On 2013-08-17 05:47, Valrhona wrote:
> y> I am not sure about repositories, but there are projects such as
>> pigz and pbzip2 which are parallelized interfaces to the same
>> compression libraries, and easily compilable
> Thanks! Looks like pigz and pbzip2 are in the libraries; thanks Eric
> for the tip!

You're welcome =)

>> In some versions of pigz there was a problem with compression of
>> multiple filename arguments (some state was not cleared, so they
>> were processed as if concatenated), so for predictable results
>> better script up loops and call each time for one argument :)
> At the moment, I am just interested in a single file, piped in from
> zfs send. Would the problem you mention affect me?

I guess not. The bug was back with version 1.7.1 IIRC, a few years ago;
it may be corrected altogether by now.

> So I found an easier solution: rsync
> for files with extension .xyz
>
> rsync -avgR --include='*/' --include='*.xyz' --exclude='*'
> --remove-source-files /src /dest
>
> This recreates the whole subdirectory structure, including (many)
> directories which have no files of type .xyz.

For rsync of dir to dir, it is safer to end them both with a slash
like "... src/ dest/". Otherwise it may be too smart and create a
subdir in the specified existing target, in some cases.

>
> To remove them, in /dest

Take a look at "rsync -m" - see if it helps?

  -m, --prune-empty-dirs      prune empty directory chains from the 
file-list


>
> find . -type d -empty -exec rmdir {} \;
>
> You have to run this a few times because only one level of empty
> directory gets removed each time, so if you have empty directories
> that are subdirectories that then become empty, etc. etc.




I think you can do a reverse search; you certainly can pipe the output
to "sort -r" and then to xargs which would call rmdir - to do it all
in one pass, by starting at "leaf" empty directories in the tree.
Though this would probably not hit directories which only contain
directories - they are not "empty"...

I think I did something similar by searching "root" dirs suspected for
no files inside with "find $dir -type f" - if the output was empty,
the $dir were good for "rm -rf".

HTH,
//Jim