<div dir="ltr">regarding the possibility of multi-processor shifting and counters, it seems like that would be fairly easy to rule out (however unlikely) by running pbind on the iostat process to keep it pegged to a specific core. </div><div class="gmail_extra"> <div class="gmail_quote">On Fri, Dec 13, 2013 at 10:38 AM, Richard Elling <<a href="mailto:richard.elling@richardelling.com" target="_blank">richard.elling@richardelling.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Dec 13, 2013, at 5:13 AM, Tobias Oetiker <<a href="mailto:tobi@oetiker.ch">tobi@oetiker.ch</a>> wrote: > Hi Paul, > > Today Paul Jochum wrote: > >> Hi Tobias: >> >> Sorry, I don't know why you are having these strang io-patterns, but I was wondering if you could share how you >> record and display this info? > > I created a little plugin for collectd to interface with iostat. I > guess having one for vfsstat and arcstat along the same lines would > give a better picture as to what users actually experience but this > one gives some impression as to what happens deep down. </div>Your script seems to break this out per-device, but that isn’t clear from the graphs you posted. This pattern is consistent with a misbehaving device. — richard <div><div class="h5"> > > #!/usr/bin/perl > my $filter = $ARGV[0] || '.+'; > > my $pid = open my $iostat, "-|", "/usr/bin/iostat","-Tu","-xnr",int($ENV{COLLECTD_INTERVAL}) or die "launching iostat: $!"; > > $SIG{PIPE} = 'ignore'; > $SIG{CHLD} = 'ignore'; > $SIG{TERM} = sub { > kill 9,$pid; > exit 1; > }; > > my %data; > my @cols; > my $round = 0; > while (<$iostat>){ > chomp; > my @input = split /,/; > if ($#input == 0 and $input[0] =~ /^\d+$/){ > publish() if $round++ > 1; > %data = (); > next; > } > if ($input[-1] eq 'device'){ > @cols = @input; > pop @cols; > next; > } > if ($#input == $#cols+1){ > my $dev = pop @input; > my %row; > @row{@cols} = @input; > $data{$dev} = \%row; > }; > } > > sub publish { > $|=0; > my $time = time(); > for my $dev (sort keys %data){ > next unless $dev =~ /^${filter}$/; > my $d = $data{$dev}; > my $prefix = "PUTVAL $ENV{COLLECTD_HOSTNAME}/sol_iostat-$dev/sol_iostat_"; > print $prefix."op $time:$d->{'r/s'}:$d->{'w/s'}\n"; > print $prefix."byte $time:".($d->{'kr/s'}*1024).":".($d->{'kw/s'}*1024)."\n"; > print $prefix."busypct $time:$d->{'%w'}:$d->{'%b'}\n"; > print $prefix."wait $time:$d->{wait}\n"; > print $prefix."actv $time:$d->{actv}\n"; > print $prefix."wsvc_t $time:".($d->{wsvc_t}/1000)."\n"; > print $prefix."asvc_t $time:".($d->{asvc_t}/1000)."\n"; > } > } > > --------- > adding these new types to the types.db > > sol_iostat_op read:GAUGE:0:U, write:GAUGE:0:U > sol_iostat_byte read:GAUGE:0:U, write:GAUGE:0:U > sol_iostat_busypct queue:GAUGE:0:100, disk:GAUGE:0:100 > sol_iostat_wait count:GAUGE:0:U > sol_iostat_actv count:GAUGE:0:U > sol_iostat_wsvc_t second:GAUGE:0:U > sol_iostat_asvc_t second:GAUGE:0:U > > cheers > tobi > > -- > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland </div></div>> <a href="http://it.oetiker.ch" target="_blank">http://it.oetiker.ch</a> <a href="mailto:tobi@oetiker.ch">tobi@oetiker.ch</a> <a href="tel:%2B%2B41%2062%20775%209902" value="+41627759902">++41 62 775 9902</a> / sb: -9900_______________________________________________ <div class="HOEnZb"><div class="h5">> OmniOS-discuss mailing list > <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a> > <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> _______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> </div></div></blockquote></div> </div>