<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>There are quite a bit of things to look at.</p>
<p>One question: do you have deduplication turned on?</p>
<p>I have seen things like this (with or without dedup) as a result
of memory exhaustion. I happen to know that the x4540 can only
hold about 32GB of ram, so this seems like a possibility.</p>
<p>you might want to start collecting vmstat output (e.g. vmstat 5)
and if you can reproduce this easily, watch that. If this is the
same thing that I've experienced, you'll go into desparation free
swapping (DE column > 0 = very bad) and then the machine will
just fall over. This seems to happen in particular with extremely
heavy NFS traffic which uses up a lot of memory buffers that can't
be freed up fast enough to prevent memory exhaustion.</p>
<p>There are a lot of other contributed scripts in the dtrace
toolkit that may be useful too. It's venerable, but still useful.</p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 3/28/2017 5:40 PM, John Barfield
wrote:<br>
</div>
<blockquote
cite="mid:C0848B1F-08DA-441E-8542-192600475690@bissinc.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal" style="text-indent:.5in"><span
style="font-size:11.0pt">Greetings!<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span
style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span
style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span
style="font-size:11.0pt">I have an issue with ZFS/NFS on
OmniOS where copying or deleting large files causes the SAN
to lockup and requires that we reboot it sometimes to
recover. One customer was trying to copy a 400GB file and
another tried to delete a 4TB vmdk from a vmware NFS vstore.
In both situations the issue is reproducible and in both
situations I cannot seem to find anything crazy happening on
either of the boxes. This doesn’t mean anything though
because I’m limited in my kstat/dtrace knowledge for
troubleshooting this type of behavior on illumos.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If there is
an illumOS bug I’m missing or something that jumps out at ya
please let me know…of if you need more detailed information
I can detail both customer locations hardware, software, and
operating conditions upon request. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The VMware
customer is running on r151016 and the other customer is
running on the latest release.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">NFSv3
Only Clients<o:p></o:p></span></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OS:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ESXi 5.5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">All 64bit
Linux Clients<o:p></o:p></span></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">CentOS
5,6,&7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Some Older
Debian 7 Boxes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I should
also note that the VMware customer experiences this same
issue on two different SAN’s that are very different in
hardware configurations.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The one
common thread is that they are all Sun Servers. X4170 M2,
X4270 M2, x4540<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This thread
is being posted to both the illumos email list and the
omnios email list as well.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m looking,
I guess, for a methodology to follow to trace this down and
determine if it’s a bug or a misconfiguration either on the
client or in ZFS or in the NFS server.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I can get
someone access if so desired.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Have a great
day!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Times New
Roman"">John Barfield<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Times New
Roman"">Engineering and Stuff
<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Times New
Roman""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Times New
Roman";color:#767171">M: +1 (214) 425-0783 O: +1 (214)
506-8354 <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New
Roman""><a moz-do-not-send="true"
href="mailto:john.barfield@bissinc.com"><span
style="color:#0563C1">john.barfield@bissinc.com</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Times New
Roman""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Times New
Roman""><img id="_x0000_i1026"
src="cid:part2.1C7DC199.32A8F11C@will.to"
alt="id:image001.png@01D2837C.19B23B20" height="28"
border="0" width="72"></span></b><b><span
style="font-family:"Times New Roman""><o:p></o:p></span></b></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Times New
Roman";color:#767171">4925 Greenville Ave, Ste 900<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Times New
Roman";color:#767171">Dallas, TX 75206<o:p></o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Times New
Roman""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><u><span
style="font-size:11.0pt;font-family:"Times New
Roman";color:#767171">For Support Requests:<o:p></o:p></span></u></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Times New
Roman""><a moz-do-not-send="true"
href="http://support.bissinc.com/"><span
style="color:#0563C1">http://support.bissinc.com</span></a>
or
<a moz-do-not-send="true" href="mailto:support@bissinc.com"><span
style="color:#0563C1">support@bissinc.com</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New
Roman""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New
Roman"">Follow us on Twitter for Network Status &
Updates!
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New
Roman""><o:p> </o:p></span></p>
<p class="MsoNormal"><a moz-do-not-send="true"
href="https://twitter.com/johnbarfield"><span
style="font-family:"Times New
Roman";color:windowtext;text-decoration:none"><img
id="_x0000_i1025"
src="cid:part5.4331F985.22DD58B1@will.to"
alt="id:image002.gif@01D2837C.19B23B20" height="24"
border="0" width="24"></span></a><span
style="font-family:"Times New Roman""> </span><span
style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OmniOS-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a>
<a class="moz-txt-link-freetext" href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>