<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri","sans-serif";}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We talked to mellanox when we were buying our gear, iirc there was an upgrade key you needed to go with the base switch for L2 functionality.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If you are dealing with mellanox directly they are good, just if you are buying used or w/e be careful ^.^<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Matthew Mabis [mailto:mmabis@vmware.com] <br><b>Sent:</b> Monday, September 29, 2014 11:32 AM<br><b>To:</b> Matthew Lagoe; 'Narayan Desai'<br><b>Cc:</b> omnios-discuss@lists.omniti.com<br><b>Subject:</b> RE: [OmniOS-discuss] Infiniband In Omni<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div name=divtagdefaultwrapper id=divtagdefaultwrapper><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Thanks Narayan for the info that will really help! I assumed using IPoIB realistically for all the things i wanted. but i might want to test out RDSH and ISER as well. I just wanted to know what things i should expect using this type of card... also what firmwares are you using on your CX1/CX2s? i think all of mine are programmed to 2.7.0 as that was the most stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager :) <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Matthew the SX6012 Switch that i have has an OpenSM manager on it so i don't have to deal with that specific issue. Here is a link to the switch i got.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><a href="http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf">http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf</a><o:p></o:p></span></p><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p></div><div><div name=divtagdefaultwrapper><div style='margin-top:14.0pt;margin-bottom:14.0pt'><p class=MsoNormal style='background:white'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#717074'>Matt Mabis</span></b><span style='font-family:"Calibri","sans-serif";color:black'><o:p></o:p></span></p></div></div></div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><div><div class=MsoNormal align=center style='text-align:center'><span style='font-family:"Calibri","sans-serif";color:#282828'><hr size=2 width="98%" align=center></span></div><div id=divRplyFwdMsg><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'> Matthew Lagoe <<a href="mailto:matthew.lagoe@subrigo.net">matthew.lagoe@subrigo.net</a>><br><b>Sent:</b> Monday, September 29, 2014 11:12 AM<br><b>To:</b> 'Narayan Desai'; Matthew Mabis<br><b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br><b>Subject:</b> RE: [OmniOS-discuss] Infiniband In Omni</span><span style='font-family:"Calibri","sans-serif";color:#282828'> <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#282828'> <o:p></o:p></span></p></div></div><div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Also for infiniband you need a subnet manager, if your switch doesn’t have a built in one you have to run like a linux box on the same network just for traffic to work, as there wasn’t any I was able to find for openindiana back in the day, that’s why we went with 10gbe.</span><span style='color:#282828'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='color:#282828'><o:p></o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:#282828'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:#282828'> OmniOS-discuss [<a href="mailto:omnios-discuss-bounces@lists.omniti.com">mailto:omnios-discuss-bounces@lists.omniti.com</a>] <b>On Behalf Of </b>Narayan Desai<br><b>Sent:</b> Monday, September 29, 2014 11:03 AM<br><b>To:</b> Matthew Mabis<br><b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br><b>Subject:</b> Re: [OmniOS-discuss] Infiniband In Omni</span><span style='color:#282828'><o:p></o:p></span></p><p class=MsoNormal><span style='color:#282828'> <o:p></o:p></span></p><div><p class=MsoNormal><span style='color:#282828'>I hate to sound negative, but you're in for a bad time. The current drivers were written by sun/oracle, and only just happen to still work. Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware support is limited to connectx 1-2. And if anything goes wrong, there isn't anyone that can really help you. <o:p></o:p></span></p><div><p class=MsoNormal><span style='color:#282828'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#282828'>That all said, we had decent success with iscsi/iSER over connectX 1 cards. Fujita Syoyo forward ported the last version of the OFED stack that Sun/Oracle modified up to more recent illumos. Those bits are here:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#282828'><a href="https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris-infiniband-tools&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05">https://github.com/syoyo/solaris-infiniband-tools</a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#282828'> -nld<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='color:#282828'> <o:p></o:p></span></p><div><p class=MsoNormal><span style='color:#282828'>On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <<a href="mailto:mmabis@vmware.com" target="_blank">mmabis@vmware.com</a>> wrote:<o:p></o:p></span></p><div><div name=divtagdefaultwrapper><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Hey All</span><span style='color:#282828'><o:p></o:p></span></p><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> </span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> </span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Any Best Practices for using IPoIB with OmniOS?</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Should i use the Stock drivers build into solaris (are there different ones that work better more stable?)</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Are there Management tools i can install to run further tests?</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> Any other Guidelines you might recommend.</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> </span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers.</span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'> </span><span style='color:#282828'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:black'>Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. </span><span style='color:#282828'><o:p></o:p></span></p></div><div><div name=divtagdefaultwrapper><div style='margin-top:14.0pt;margin-bottom:14.0pt'><p class=MsoNormal style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Thanks</span><span style='color:#282828'><o:p></o:p></span></p></div><div style='margin-top:14.0pt;margin-bottom:14.0pt'><p class=MsoNormal style='background:white'><b><span style='font-family:"Arial","sans-serif";color:#717074'>Matt Mabis</span></b><span style='color:#282828'><o:p></o:p></span></p></div></div></div></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='color:#282828'><br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br><a href="https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/listinfo/omnios-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b713aaa" target="_blank">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><o:p></o:p></span></p></div><p class=MsoNormal><span style='color:#282828'> <o:p></o:p></span></p></div></div></div></div></div></div></div></body></html>