<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-text-flowed" style="font-family: -moz-fixed;
font-size: 12px;" lang="x-unicode">Hi,
<br>
<br>
My name is Kobi and I work with a company called Evogene.
<br>
<br>
We have a storage server based on a Supermicro X9DRD-7LN4F
(Intel(R) Xeon(R) CPU E5-2609, 64GB RAM) equipped with Intel
X520-DA2 10Gb Ethernet card.
<br>
<br>
Two weeks ago we encountered a problem which regards Omnios
(r15012) and OpenIndiana (151_a8) kernel stability.
<br>
<br>
The system is a clean install of r151012, only configuration is
networking and zpool (raidz3 1 vdev of 8 disks of data and 3
parity).
<br>
<br>
The storage has a single NFS share mounted (nfsvers=3) on an 8
grid nodes (running CentOS6.6) with the following mount options :
<br>
<br>
async,tcp,vers=3,noacl,timeo=20,rw,bg,hard,intr,rsize=65536,wsize=65536
0 0.
<br>
<br>
Under normal conditions (random reads and writes) we get great
performance and stability but if we increase nfs usage the Kernel
memory usage increases until the machine eventually halts.
<br>
<br>
To test it we used on each node iozone ( Version $Revision: 3.394
) which simultaneously write and reads 12 files with a size of
500MB.
<br>
(iozone -r 64k -i 0 -i 1 -s 500m -t 12 -F <i
class="moz-txt-slash"><span class="moz-txt-tag">/</span>evolon2/iozone<span
class="moz-txt-tag">/</span></i>`hostname`.{1..12}).
<br>
<br>
While the iozone's are running the memory of Kernel (as seen by
echo '::memstat'| mdb -k) is steadily increasing until the machine
is dead.
<br>
<br>
For clarification, when the machine is dead/halts:
<br>
There is no crash files.
<br>
There is no new messages in /var/adm/messages.
<br>
It doesn't respond to pings (and obviously other network
services).
<br>
Waiting doesn't help.
<br>
It can only be brought to life by power cycling it.
<br>
<br>
I attached 3 files: memstat, top and arc-kstat which I hope will
assist to understand why the storage died/halted
<br>
<br>
Thank you in advance,
<br>
<br>
Kobi.
<br>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"><legend
class="mimeAttachmentHeaderName">memstat</legend></fieldset>
</body>
</html>