<div dir="ltr">We're primarily using iSER. We had similar problems using iscsi with tcp over the ipoib. We're using the system as openstack block storage, so iscsi+iSER was the simplest protocol to use.<div><br></div>
<div style>Depending on the depending on the PCIe chipset, we see up to about 3200 MB/s with a connectX2 on gen2 PCIe (on linux). ConnectX3's are gen3 so those wouldn't be bus limited with a single port card on a system with gen3 PCIe.</div>
<div style> -nld</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 19, 2013 at 11:15 PM, David Bomba <span dir="ltr"><<a href="mailto:turbo124@gmail.com" target="_blank">turbo124@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We use both IPoIB and iSER. iSER can flood the HCA. <div><br></div><div>With IPoIB because TCP is done in software we only see around 450MB/s, using connected mode with 65k MTU and some tuning with ndd for larger frame sizes.</div>
<div><br></div><div>With 40Gb/s equipment the bottleneck will be the PCIe 2.0 BUS, so I doubt we will be able to get over 3Gb/s.. </div><div><br></div><div>What protocols are you using for your testing? SRP , iSER or NFSoRDMA?</div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On 20 June 2013 14:09, Narayan Desai <span dir="ltr"><<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm curious to hear what performance you end up seeing. We've got QDR connectX 2 cards, and only seen ~2300 MB/s out of them, compared with ~3100 on linux with the same hardware. (this is even just the completely local bw tests to the cards)<div>
<br></div><div>Our of curiosity, are you using ipoib or iSER? We've seen much better performance with the latter, topping out around 1800 MB/s, compared with ~300 on ipoib without much tuning. (single clients, in both cases) We can't hit the RDMA benchmark bandwidths with a single client, even though we have sufficient spindles on the back end.<br>
<div> -nld</div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 19, 2013 at 10:01 PM, David Bomba <span dir="ltr"><<a href="mailto:turbo124@gmail.com" target="_blank">turbo124@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">We run 10G Infiniband on OmniOS using the ComSTAR iscsi target.<div><br></div><div>We can flood the network easily between two OmniOS machines.</div>
<div><br></div><div>We are about to upgrade to 40Gb/s HCAs, and i would expect similar performance.</div><div><br></div><div>Dave</div><div><div><div><br><div><div>On 20/06/2013, at 12:25 PM, Narayan Desai wrote:</div>
<br><blockquote type="cite"><div dir="ltr">It looks like someone has managed to get things working on OI:<div><a href="http://syoyo.wordpress.com/category/infiniband/" target="_blank">http://syoyo.wordpress.com/category/infiniband/</a><br>
</div><div><br></div><div>
Just fyi, there don't seem to be many people running IB with omniOS or OI, in general. We are, but aren't getting the performance we get with the same hardware on linux. </div><div> -nld</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jun 19, 2013 at 8:29 PM, Moises Medina <span dir="ltr"><<a href="mailto:raenac@gmail.com" target="_blank">raenac@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Michael, thanks for the response. I tried the usual opensm command, returned nothing, and searched for it in pkg, maybe its named something else?<span><font color="#888888"><div><br></div>
<div>Moises</div></font></span></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jun 19, 2013 at 4:07 PM, Michael Palmer <span dir="ltr"><<a href="mailto:palmertime@gmail.com" target="_blank">palmertime@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div>I believe that OpenSM is included in OmniOS but i'm not sure if VMware ESXi 5.x and above supports it. If you are using ESXi 4.0 then some vendors might have drivers for infiniband on ESXi.<br>
<br></div>Thanks,<br></div>Michael<br></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Wed, Jun 19, 2013 at 10:30 AM, Moises Medina <span dir="ltr"><<a href="mailto:raenac@gmail.com" target="_blank">raenac@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi All, new to the list. Question I had is has anyone successfully ran OpenSM on OmniOS? Trying to use this as ZFS server to ESXI but do not have a switch at all.<div>
<br></div><div>Thanks,</div><div>
<br></div><div>Raenac</div></div>
<br></div></div>_______________________________________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.omniti.com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.omniti.com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.omniti.com</a><br><a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>