<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <style type="text/css" id="owaParaStyle" style="display: none;">P {margin-top:0;margin-bottom:0;}</style><style>  </style> </head> <body dir="ltr" fpstyle="1" aria-label="Message body" tabindex="0"> <div name="divtagdefaultwrapper" id="divtagdefaultwrapper" style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000; margin: 0"> Thanks Narayan for the info that will really help! I assumed using IPoIB realistically for all the things i wanted. but i might want to test out RDSH and ISER as well. I just wanted to know what things i should expect using this type of card... also what firmwares are you using on your CX1/CX2s? i think all of mine are programmed to 2.7.0 as that was the most stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager :) <div><br> </div> <div>Matthew the SX6012 Switch that i have has an OpenSM manager on it so i don't have to deal with that specific issue. Here is a link to the switch i got.</div> <div><br> </div> <div>http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf<br> <div><br> </div> <div> <div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0"> <div style="font-size:16px; background-color:rgb(255,255,255); margin-top:14pt; margin-bottom:14pt"> <font face="Times New Roman" size="3"><font face="Arial,Helvetica,sans-serif" size="2" color="#717074"><b>Matt Mabis</b></font><br> </font></div> <font face="Times New Roman" style="font-size:16px; background-color:rgb(255,255,255)"></font></div> </div> <br> <div style="color: rgb(40, 40, 40);"> <hr tabindex="-1" style="display:inline-block; width:98%"> <div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Lagoe <matthew.lagoe@subrigo.net><br> <b>Sent:</b> Monday, September 29, 2014 11:12 AM<br> <b>To:</b> 'Narayan Desai'; Matthew Mabis<br> <b>Cc:</b> omnios-discuss@lists.omniti.com<br> <b>Subject:</b> RE: [OmniOS-discuss] Infiniband In Omni</font> <div> </div> </div> <div> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">Also for infiniband you need a subnet manager, if your switch doesn’t have a built in one you have to run like a linux box on the same network just for traffic to work, as there wasn’t any I was able to find for openindiana back in the day, that’s why we went with 10gbe.</span></p> <p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"> </span></p> <p class="MsoNormal"><b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif""> OmniOS-discuss [mailto:omnios-discuss-bounces@lists.omniti.com] <b>On Behalf Of </b>Narayan Desai<br> <b>Sent:</b> Monday, September 29, 2014 11:03 AM<br> <b>To:</b> Matthew Mabis<br> <b>Cc:</b> omnios-discuss@lists.omniti.com<br> <b>Subject:</b> Re: [OmniOS-discuss] Infiniband In Omni</span></p> <p class="MsoNormal"> </p> <div> <p class="MsoNormal">I hate to sound negative, but you're in for a bad time. The current drivers were written by sun/oracle, and only just happen to still work. Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware support is limited to connectx 1-2. And if anything goes wrong, there isn't anyone that can really help you. </p> <div> <p class="MsoNormal"> </p> </div> <div> <p class="MsoNormal">That all said, we had decent success with iscsi/iSER over connectX 1 cards. Fujita Syoyo forward ported the last version of the OFED stack that Sun/Oracle modified up to more recent illumos. Those bits are here:</p> </div> <div> <p class="MsoNormal"><a href="https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris-infiniband-tools&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05">https://github.com/syoyo/solaris-infiniband-tools</a></p> </div> <div> <p class="MsoNormal"> -nld</p> </div> </div> <div> <p class="MsoNormal"> </p> <div> <p class="MsoNormal">On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <<a href="mailto:mmabis@vmware.com" target="_blank">mmabis@vmware.com</a>> wrote:</p> <div> <div name="divtagdefaultwrapper"> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Hey All</span></p> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> Any Best Practices for using IPoIB with OmniOS?</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> Should i use the Stock drivers build into solaris (are there different ones that work better more stable?)</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> Are there Management tools i can install to run further tests?</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> Any other Guidelines you might recommend.</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers.</span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p> </div> <div> <p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. </span></p> </div> <div> <div name="divtagdefaultwrapper"> <div style="margin-top:14.0pt; margin-bottom:14.0pt"> <p class="MsoNormal" style="background:white"><span style="font-family:"Calibri","sans-serif"; color:black">Thanks</span></p> </div> <div style="margin-top:14.0pt; margin-bottom:14.0pt"> <p class="MsoNormal" style="background:white"><b><span style="font-family:"Arial","sans-serif"; color:#717074">Matt Mabis</span></b><span style="font-family:"Calibri","sans-serif"; color:black"></span></p> </div> </div> </div> </div> </div> <p class="MsoNormal" style="margin-bottom:12.0pt"><br> _______________________________________________<br> OmniOS-discuss mailing list<br> <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br> <a href="https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/listinfo/omnios-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b713aaa" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a></p> </div> <p class="MsoNormal"> </p> </div> </div> </div> </div> </div> </div> </body> </html>