[OmniOS-discuss] Clues for tracking down a drastic ZFS fs space difference?

[Dan McDonald]

> On Apr 29, 2015, at 3:21 PM, Chris Siebenmann <cks at cs.toronto.edu> wrote:
> 
> We have a filesystem/dataset with no snapshots,

You're sure about no snapshots?  "zfs list -t snapshot" has surprised me once or twice in the past.  :-/

> no subordinate
> filesystems, nothing complicated (and no compression), that has a
> drastic difference in space used between what df/zfs list/etc report at
> the ZFS level and what du reports at the filesystem level. ZFS says
> 
> NAME              PROPERTY       VALUE   SOURCE
> fs0-core-01/cs/8  used           70.5G   -
> fs0-core-01/cs/8  usedbydataset  70.5G   -
> 
> On the other hand, 'du -h' says 17 GB, which is what we'd expect.
> More alarmingly, this dataset seems to keep steadily growing at the
> ZFS level despite 'du -h' figures staying constant or even going down.
> On April 2nd it was reporting a 'du -h' of 22 GB but 48 GB used at the
> ZFS level; as you can see, it's added 22 GB of ZFS usage in less than
> a month while losing 5 GB at the user level.

This almost sounds like there's a process with an open file, which was removed, but the process in question still has it open.

pfiles(1) on your processes may be very helpful here.

> What sort of things should I be looking at to try to figure out why
> this is happening, including with eg zdb? Are there any obvious reasons
> why this would be happening? Is there any easy way to fix this short of
> 'copy all data to a new dataset, destroy old dataset, put new dataset
> in the place of the old?'

I take it that a reboot of this machine (which would kill any processes with an open-but-deleted file) has already been done?

Dan