[OmniOS-discuss] Good way to debug DTrace invalid address errors?

[Richard Elling]

a few pointers...

> On Mar 23, 2016, at 12:48 PM, Chris Siebenmann <cks at cs.toronto.edu> wrote:
> 
> I have a relatively complicated chunk of dtrace code that reads kernel
> data structures and chases pointers through them. Some of the time it
> spits out 'invalid address' errors during execution, for example:
> 
> 	dtrace: error on enabled probe ID 8 (ID 75313: fbt:nfssrv:nfs3_fhtovp:return): invalid address (0x2e8) in action #6 at DIF offset 40
> 	dtrace: error on enabled probe ID 8 (ID 75313: fbt:nfssrv:nfs3_fhtovp:return): invalid address (0xbaddcafebaddcc7e) in action #6 at DIF offset 60
> 
> I'd like to find out exactly what pointer dereferences or other
> operations are failing here, so that I can figure out how to work
> around the issue. However, I have no solid idea how to map things
> like 'probe ID 8' and 'DIF offset 60' to particular lines in my
> DTrace source code.
> 
> I assume that the answer to this involves reading DIF (the DTrace
> intermediate form). I've looked at 'dtrace -Se' output from this
> DTrace script, but I can't identify the spot I need to look at.
> In particular, as far as I can see nothing in the output has
> instructions with an offset as high as 40 or 60.
> 
> I can flail around sticking guards in and varying how I do stuff
> to make the errors go away, but I'd like to understand how to debug
> this sort of stuff so I can have more confidence in my changes.
> 
> Thanks in advance for any suggestions, and if people want to see
> the actual code involved it is this DTrace script:
> 
> 	https://github.com/siebenmann/cks-dtrace/blob/master/nfs3-long.d

We do something similar, however we differ in the method to arrive at
pool and dataset. Without knowing your conventions, it is not really possible
to make concrete recommendations, but here is how we do it:
+ each share is unique
+ each share has a UUID
+ mount point contains the share UUID
+ each UUID can be mapped to a pool and dataset
+ all permutations of (operation, client, share) are collected, including "all"
+ today, all of this data is pumped into collectd --> influxdb for queries

With newer illumos distros, there are Riemann sum kstats for each NFS version,
operation, by share. This might or might not help. We find the averages to be
less insightful than the distribution.

Also, for client IP address, here's what we do:
self->remote = (xlate <conninfo_t> (args[3])).ci_remote;
 -- richard

> 
> (Look at line 144 for the specific dtrace probe that is probably
> failing, since it's the only probe on fbt:nfssrv:nfs3_fhtovp:return.)
> 
> 	- cks
> PS: it's entirely possible that there's a better way to do what I'm
>    trying here, too. These DTrace scripts were originally written
>    on Solaris 10 update 8 and haven't been drastically revised
>    since.
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss