<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Thank you all, the water is already clearing up :) So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they not take a standaard and set everything in GB/s or MB/s ? A lot of people make a lot of mistakes between them, me too ... If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s throughput. Little difference 40 or 5 :) So Ian you have the full blow with 36Gbps very cool looks more like it :) Did I play with the frame size, not really sure what you mean by that sorry but I think its default on 9000 Backend_Switch0 etherstub 9000 up Do understand that if we use UDP streams from process to process it will be much quicker over the etherstub gonna need more test to do. We used for a customer Mbuffer with zfs send over Lan that is also very quick sometimes I also use it at my home very good prog. But still do not understand how it is that I copy from 1 NGZ with 100MB/s, I receive on the other NGZ 250MB/s very strange ? the command dlstat difference between OmniOSce and Solaris ? RBYTES => receiving OBYTES => sending root@test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M BYTES => receiving and sending ? But then still if the copy is not running I have 0 so doesn't explain why I see 216 MB where come the rest of the 116 MB from is it compression ? root@NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M Thank you all for your feedback much appreciations ! Kind Regards, Dirk <div class="moz-cite-prefix">On 14-09-17 17:07, Ian Kaufman wrote: </div> <blockquote type="cite" cite="mid:CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <div dir="ltr">Some other things you need to take into account: <div> </div> <div>QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. That is also a theoretical maximum throughput, there is some overhead. In reality, you will never see 40Gbps. </div> <div> </div> <div>My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the theoretical max rates to 18Gbps and 36Gbps respectively. </div> <div> </div> <div>If you are getting 185MB/s, you are seeing 1.48Gbps. </div> <div> </div> <div>Keep your B's and b's straight. Did you play with your frame size at all?</div> <div> </div> <div>Ian</div> </div> <div class="gmail_extra"> <div class="gmail_quote">On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <<a href="mailto:jimklimov@cos.ru" target="_blank" moz-do-not-send="true">jimklimov@cos.ru</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class="HOEnZb"> <div class="h5">On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <<a href="mailto:dirk.willems@exitas.be" moz-do-not-send="true">dirk.willems@exitas.be</a>> wrote: >Hello, > > >I'm trying to understand something let me explain. > > >Oracle always told to me that if you create a etherstub switch it has >infiniband speed 40GB/s. > >But I have a customer running on Solaris (Yeah I know but let me >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >(I know told him to to use etherstub). > >The copy witch is performed for a Oracle database with sql command, the > >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >- >60 % busy the speed is 30 mb/s. > > >So I did some test just to see and understand if it's the database or >the system, but with doing my tests I get very confused ??? > > >On another Solaris at my work copy over etherstub switch => copy speed >is 185MB/s expected much more of infiniband speed ??? > > >root@test1:/export/home/Admin# scp test10G ><a class="moz-txt-link-abbreviated" href="mailto:Admin@192.168.1.2:/export/">Admin@192.168.1.2:/export/</a>home/Admin/ >Password: >test10G 100% >|****************************************************************| >10240 >MB 00:59 > > >root@test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M > net1 26.97K 186.37M 11.24K 3.23M > net1 26.63K 187.67M 10.82K 2.99M > net1 27.94K 186.65M 12.17K 3.75M > net1 27.45K 187.46M 11.70K 3.47M > net1 26.01K 181.95M 10.63K 2.99M > net1 27.95K 188.19M 12.14K 3.69M > net1 27.91K 188.36M 12.08K 3.64M > >The disks are all separate luns with all separated pools => disk are 20 > >- 30% busy > > >On my OmniOSce at my lab over etherstub > > >root@GNUHealth:~# scp test10G <a class="moz-txt-link-abbreviated" href="mailto:witte@192.168.20.3:/export/">witte@192.168.20.3:/export/</a>home/witte/ >Password: >test10G 76% 7853MB 116.4MB/s > > >=> copy is 116.4 MB/s => expected much more from infiniband speed is >just the same as Lan ??? > > >Is not that my disk can not follow 17% busy there sleeping ... > > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => >rpool > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => >rpool > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => >data pool > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => >data pool > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => >data pool > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => >data pool > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > >root@NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 7.01K 531.38K > NGINX1 rx local -- 30.65K 253.73M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 8.95K 669.32K > NGINX1 rx local -- 29.10K 241.15M > > >On the other NGZ I receive 250MB/s ???? > > >- So my question is how comes that the speed is equal to Lan 100MB/s on > >OmniOSce but i receive 250MB/s ? > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > >I'm very confused right now ... > > >And want to know for sure how to understand and see this in the right >way, because this customer will be the first customer from my who gonna > >switch complety over to OmniOSce on production and because this >customer >is one or the biggest company's in Belgium I really don't want to mess >up !!! > > >So any help and clarification will be highly appreciate !!! > > >Thank you very much. > > >Kind Regards, > > >Dirk </div> </div> I am not sure where the infiniband claim comes from, but copying data disk to disk, you involve the slow layers like disk, skewed by faster layers like cache of already-read data and delayed writes :) If you have a wide pipe that you may fill, it doesn't mean you do have the means to fill it with a few disks. To estimate the speeds, try pure UDP streams from process to process (no disk), large-packet floodping, etc. I believe etherstub is not constrained artificially, and defaults to jumbo frames. Going to LAN and back can in fact use external hardware (IIRC there may be a system option to disable that, not sure) and so is constrained by that. Jim -- Typos courtesy of K-9 Mail on my Android <div class="HOEnZb"> <div class="h5">_______________________________________________ OmniOS-discuss mailing list <a href="mailto:OmniOS-discuss@lists.omniti.com" moz-do-not-send="true">OmniOS-discuss@lists.omniti.com</a> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> </div> </div> </blockquote> </div> <div> </div> -- <div class="gmail_signature" data-smartmail="gmail_signature">Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu </div> </div> <fieldset class="mimeAttachmentHeader"></fieldset> <pre wrap="">_______________________________________________ OmniOS-discuss mailing list <a class="moz-txt-link-abbreviated" href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a> <a class="moz-txt-link-freetext" href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a> </pre> </blockquote> <div class="moz-signature">-- <table style="width: 400px; border-collapse: collapse;" cellspacing="0" cellpadding="0" width="400"> <tbody> <tr> <td width="150"> <img src="http://signatures.sidekick.be/exitas/images/placeholder-exitas.jpg" style="border: none; border-collapse: collapse; width: 130px; height: 130px;" border="0" width="130" height="130"></td> <td style="width: 230px; font-family: Arial; font-size: 12px; line-height: 15px; border-collapse: collapse; font-weight: 300; color: #215973; vertical-align:top;" width="250"> Dirk Willems System Engineer +32 (0)3 443 12 38 <a href="mailto:Dirk.Willems@exitas.be" style="color: #215973;">Dirk.Willems@exitas.be</a> Quality. Passion. Personality </td> </tr> <tr> <td> </td> </tr> <tr> <td colspan="2" style="font-family: Arial; font-size: 12px; line-height: 15px; border-collapse: collapse; font-weight: 300; color: #215973; padding-top: 5px;"><a href="http://www.exitas.be/" style="color: #215973;">www.exitas.be</a> | Veldkant 31 | 2550 Kontich</td> </tr> <tr> <td> </td> </tr> <tr> <td style="font-family: Arial; font-size: 12px; line-height: 15px; border-collapse: collapse; font-weight: 500; color: #215973; padding-top: 5px;"> Illumos OmniOS Installation and Configuration Implementation Specialist. Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist. </td> <td width="400"><img src="cid:part8.0C6AA6B9.798DC1B2@exitas.be" style="border: none; border-collapse: collapse; width: 236px; height: 126px;" border="0" width="236" height="126"></td> </tr> </tbody> </table> </div> </body> </html>