<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Also for infiniband you need a subnet manager, if your switch doesn’t have a built in one you have to run like a linux box on the same network just for traffic to work, as there wasn’t any I was able to find for openindiana back in the day, that’s why we went with 10gbe.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> OmniOS-discuss [mailto:omnios-discuss-bounces@lists.omniti.com] <b>On Behalf Of </b>Narayan Desai<br><b>Sent:</b> Monday, September 29, 2014 11:03 AM<br><b>To:</b> Matthew Mabis<br><b>Cc:</b> omnios-discuss@lists.omniti.com<br><b>Subject:</b> Re: [OmniOS-discuss] Infiniband In Omni<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>I hate to sound negative, but you're in for a bad time. The current drivers were written by sun/oracle, and only just happen to still work. Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware support is limited to connectx 1-2. And if anything goes wrong, there isn't anyone that can really help you. <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>That all said, we had decent success with iscsi/iSER over connectX 1 cards. Fujita Syoyo forward ported the last version of the OFED stack that Sun/Oracle modified up to more recent illumos. Those bits are here:<o:p></o:p></p></div><div><p class=MsoNormal><a href="https://github.com/syoyo/solaris-infiniband-tools">https://github.com/syoyo/solaris-infiniband-tools</a><o:p></o:p></p></div><div><p class=MsoNormal> -nld<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <<a href="mailto:mmabis@vmware.com" target="_blank">mmabis@vmware.com</a>> wrote:<o:p></o:p></p><div><div name=divtagdefaultwrapper><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Hey All<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Any Best Practices for using IPoIB with OmniOS?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Should i use the Stock drivers build into solaris (are there different ones that work better more stable?)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Are there Management tools i can install to run further tests?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Any other Guidelines you might recommend.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. <o:p></o:p></span></p></div><div><div name=divtagdefaultwrapper><div style='margin-top:14.0pt;margin-bottom:14.0pt'><p class=MsoNormal style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Thanks<o:p></o:p></span></p></div><div style='margin-top:14.0pt;margin-bottom:14.0pt'><p class=MsoNormal style='background:white'><b><span style='font-family:"Arial","sans-serif";color:#717074'>Matt Mabis</span></b><span style='font-family:"Calibri","sans-serif";color:black'><o:p></o:p></span></p></div></div></div></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br><a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>