<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">/agreed for internal network transfers, netcat is the way to go. I have used it in the past for ZFS sends between internal machines with SSD backed volumes. All the crypto
work from ssh/scp and other secure copy methods are the bottleneck in these cases.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Obviously if you transfers leave the secure network you’ll want to suck it up and take the CPU hit. ;)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> OmniOS-discuss [mailto:omnios-discuss-bounces@lists.omniti.com]
<b>On Behalf Of </b>Ludovic Orban<br>
<b>Sent:</b> Thursday, September 14, 2017 8:28 AM<br>
<b>To:</b> Dirk Willems <dirk.willems@exitas.be><br>
<b>Cc:</b> omnios-discuss <omnios-discuss@lists.omniti.com><br>
<b>Subject:</b> Re: [OmniOS-discuss] questions<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Here's a quick thought: if you are copying your files with scp, you might be CPU bound because of all the crypto work. You should try to copy with a much more CPU-lightweight tool. I personally usually use netcat for this kind of stuff. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Just my 2 cents.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Ludovic<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Sep 14, 2017 at 2:26 PM, Dirk Willems <<a href="mailto:dirk.willems@exitas.be" target="_blank">dirk.willems@exitas.be</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p>Hello,<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>I'm trying to understand something let me explain.<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Oracle always told to me that if you create a etherstub switch it has infiniband speed 40GB/s.<o:p></o:p></p>
<p>But I have a customer running on Solaris (Yeah I know but let me explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told him to to use etherstub).<o:p></o:p></p>
<p>The copy witch is performed for a Oracle database with sql command, the DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60 % busy the speed is 30 mb/s.<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>So I did some test just to see and understand if it's the database or the system, but with doing my tests I get very confused ???<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>On another Solaris at my work copy over etherstub switch => copy speed is 185MB/s expected much more of infiniband speed ???<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>root@test1:/export/home/Admin# scp test10G <a href="mailto:Admin@192.168.1.2:/export/home/Admin/" target="_blank">
Admin@192.168.1.2:/export/home/Admin/</a><br>
Password:<br>
test10G 100% |****************************************************************| 10240 MB 00:59<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>root@test2:~# dlstat -i 2<o:p></o:p></p>
<p> LINK IPKTS RBYTES OPKTS OBYTES<br>
net1 25.76K <span style="color:red">185.14M</span> 10.08K 2.62M<br>
net1 27.04K 187.16M 11.23K 3.22M<br>
net1 26.97K 186.37M 11.24K 3.23M<br>
net1 26.63K 187.67M 10.82K 2.99M<br>
net1 27.94K 186.65M 12.17K 3.75M<br>
net1 27.45K 187.46M 11.70K 3.47M<br>
net1 26.01K 181.95M 10.63K 2.99M<br>
net1 27.95K 188.19M 12.14K 3.69M<br>
net1 27.91K 188.36M 12.08K 3.64M<o:p></o:p></p>
<p>The disks are all separate luns with all separated pools => disk are 20 - 30% busy<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>On my OmniOSce at my lab over etherstub<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>root@GNUHealth:~# scp test10G <a href="mailto:witte@192.168.20.3:/export/home/witte/" target="_blank">
witte@192.168.20.3:/export/home/witte/</a><br>
Password: <br>
test10G 76% 7853MB
<span style="color:red">116.4MB/s </span> <o:p></o:p></p>
<p><o:p> </o:p></p>
<p>=> copy is 116.4 MB/s => expected much more from infiniband speed is just the same as Lan ???<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Is not that my disk can not follow 17% busy there sleeping ...<o:p></o:p></p>
<p> extended device statistics <br>
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device<br>
0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1<br>
0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => rpool<br>
0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => rpool<br>
0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => data pool<br>
0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => data pool<br>
0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => data pool<br>
0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => data pool<br>
0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool<br>
0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA<o:p></o:p></p>
<p><o:p> </o:p></p>
<p><o:p> </o:p></p>
<p>root@NGINX:/root# dlstat show-link NGINX1 -i 2 <o:p></o:p></p>
<p style="margin-bottom:12.0pt"> LINK TYPE ID INDEX PKTS BYTES<br>
NGINX1 rx bcast -- 0 0<br>
NGINX1 rx sw -- 0 0<br>
NGINX1 tx bcast -- 0 0<br>
NGINX1 tx sw -- 9.26K 692.00K<br>
NGINX1 rx local -- 26.00K <span style="color:red">216.32M</span><br>
NGINX1 rx bcast -- 0 0<br>
NGINX1 rx sw -- 0 0<br>
NGINX1 tx bcast -- 0 0<br>
NGINX1 tx sw -- 7.01K 531.38K<br>
NGINX1 rx local -- 30.65K <span style="color:red">253.73M</span><br>
NGINX1 rx bcast -- 0 0<br>
NGINX1 rx sw -- 0 0<br>
NGINX1 tx bcast -- 0 0<br>
NGINX1 tx sw -- 8.95K 669.32K<br>
NGINX1 rx local -- 29.10K <span style="color:red">241.15M</span><o:p></o:p></p>
<p><o:p> </o:p></p>
<p>On the other NGZ I receive <span style="color:red">250MB/s </span>????<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>- So my question is how comes that the speed is equal to Lan 100MB/s on OmniOSce but i receive 250MB/s ?<o:p></o:p></p>
<p>- Why is etherstub so slow if infiniband speed is 40GB/s ???<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>I'm very confused right now ...<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>And want to know for sure how to understand and see this in the right way, because this customer will be the first customer from my who gonna switch complety over to OmniOSce on production and because this customer is one or the biggest company's in Belgium
I really don't want to mess up !!!<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>So any help and clarification will be highly appreciate !!!<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Thank you very much.<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Kind Regards,<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Dirk<o:p></o:p></p>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="400" style="width:300.0pt;border-collapse:collapse">
<tbody>
<tr>
<td width="150" style="width:112.5pt;padding:0in 0in 0in 0in">
<p class="MsoNormal"><img border="0" width="130" height="130" style="width:1.3541in;height:1.3541in" id="_x0000_i1025" src="http://signatures.sidekick.be/exitas/images/placeholder-exitas.jpg"><o:p></o:p></p>
</td>
<td width="230" valign="top" style="width:172.5pt;padding:0in 0in 0in 0in">
<p class="MsoNormal" style="mso-line-height-alt:11.25pt"><b><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#215973">Dirk Willems</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#215973"><br>
System Engineer <br>
<br>
<br>
<a href="tel:03%20443%2012%2038" target="_blank">+32 (0)3 443 12 38</a><br>
<a href="mailto:Dirk.Willems@exitas.be" target="_blank"><span style="color:#215973">Dirk.Willems@exitas.be</span></a><br>
<br>
Quality. Passion. Personality <o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 0in"></td>
<td style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td colspan="2" style="padding:3.75pt 0in 0in 0in">
<p class="MsoNormal" style="line-height:11.25pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#215973"><a href="http://www.exitas.be/" target="_blank"><span style="color:#215973">www.exitas.be</span></a> | Veldkant 31 | 2550 Kontich<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 0in"></td>
<td style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td style="padding:3.75pt 0in 0in 0in">
<p class="MsoNormal" style="line-height:11.25pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#215973">Illumos OmniOS Installation and Configuration Implementation Specialist.
<br>
Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist.
<o:p></o:p></span></p>
</td>
<td width="400" style="width:300.0pt;padding:0in 0in 0in 0in">
<p class="MsoNormal"><img border="0" width="236" height="126" style="width:2.4583in;height:1.3125in" id="_x0000_i1026" src="cid:image001.png@01D32D3B.E8AE3E10"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>