[OmniOS-discuss] multipathing against compellent
Majeed
mabuqu at ilstu.edu
Thu Oct 18 16:35:33 EDT 2012
Hi All, As noted below, we have attempted to test the illumos based
omniOS in our datacenter on a dell R610 connected through dual cisco
fiber channel fabrics to a compellent SAN with dual controllers. We got
pretty far with an out of the box setup until the last multipathing bit.
I was hoping there might be some additional eyes on this list that may
be able to provide some insight.
Thanks in advance for taking the time to read through my following notes
on this topic,
Majeed Qulbain
On 10/18/2012 03:06 PM, Majeed wrote:
> Thanks Theo, that makes sense. I will definitely do that.
>
> Majeed
>
> On 10/18/2012 02:58 PM, Theo Schlossnagle wrote:
>> This is a more general Illumos problem, you will open your
>> troubleshooting to a larger audience by interfacing with the
>> illumos-discuss list. I encourage you to post there! I'm subscribed
>> there as well, but I haven't done multipathing against compellent and
>> it appears no one else on the omnios-discuss list has either.
>> Anything learned in the illumos-discuss group can be applied directly
>> to OmniOS.
>>
>> On Thu, Oct 18, 2012 at 3:51 PM, Majeed<mabuqu at ilstu.edu> wrote:
>>> Hi Again, just wanted to update the list with my latest progress and
>>> thoughts on this topic. I think the issue lies in
>>> libmpscsi_vhci.so's lack
>>> of support for multipathing with compellent.
>>>
>>> When I do "mpathadm show mpath-support libmpscsi_vhci.so" on omniOS
>>> I see
>>> many venders and models listed but not compellent. We are going to
>>> spin up a
>>> solaris 11 machine to see what it looks like. I wonder if this was
>>> forked
>>> just before SUN/oracle added support for compellent (i could be way
>>> off base
>>> here).
>>>
>>> When I do the same command on solaris 10 u10, at the very bottom of
>>> the list
>>> i see.
>>> Vendor: COMPELNT
>>> Product: Compellent Vol
>>> Revision:
>>> Supported Load Balance Types:
>>> round-robin
>>>
>>> I have verified that multipathing is working properly on other
>>> arrays. I
>>> zoned up omniOS with all controllers on a 6140 (CSM200) that sit in
>>> multiple
>>> fabrics, mapped a lun, and it looks wonderful. mpatham reports as
>>> expected,
>>> and all paths are condensed to a single block device in format's
>>> output.
>>>
>>> % mpathadm list LU
>>> /dev/rdsk/c0t600A0B800033816200001391507FF856d0s2
>>> Total Path Count: 4
>>> Operational Path Count: 4
>>>
>>>
>>>
>>> It seems there are some options in /kernel/drv/scsi_vhci.conf to force
>>> scsi_vhci to accept 3rd party devices (hopefully compellent) for
>>> failover,
>>> but this is kinda getting over my head. It seems like if I had a proper
>>> ddi-forceload ''misc/scsi_vhci/scsi_vhci_f_<x>" file for compellent
>>> and I
>>> knew the proper VenderID and ProductID for the compellent devices, i
>>> might
>>> be able to force it. Im not sure if can use the information from the
>>> format
>>> inquiry to populate these values.
>>>
>>> format> inquiry
>>> Vendor: COMPELNT
>>> Product: Compellent Vol
>>> Revision: 0505
>>>
>>>
>>> Any thoughts on this would be excellent.
>>>
>>> Thanks,
>>> Majeed Qulbain
>>>
>>>
>>>
>>> On 10/18/2012 09:30 AM, Majeed wrote:
>>>> I just noticed that we do have access to the mpathadm command by
>>>> installing the "pkg:/storage/mpathadm at 0.5.11-0.151002" package. After
>>>> installing I don't see any paths with "mpathadm list LU" which is
>>>> what im
>>>> used to on solaris 10 (i know I should be comparing to solaris 11,
>>>> just
>>>> havn't played with it much). I wonder if there are any other bits
>>>> that I
>>>> need to install or other configuration tweaks to get this going.
>>>>
>>>> Thanks in advance for any pointers,
>>>> Majeed
>>>>
>>>> Some other related commands that might assist troubleshooting:
>>>>
>>>> % uname -a
>>>> SunOS jhx011 5.11 omnios-eae537b i86pc i386 i86pc
>>>>
>>>> % mpathadm list initiator-port
>>>> Initiator Port: w5d4ae520937bbb00
>>>> Initiator Port:
>>>> iqn.1986-03.com.sun:01:b06a64f606ff.507f35b3,4000002a00ff
>>>> Initiator Port: w5d4ae520937bbb00
>>>> Initiator Port: 2101001b32b40767
>>>> Initiator Port: 2100001b32940767
>>>>
>>>> % luxadm -e port
>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0:devctl
>>>> CONNECTED
>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0,1/fp at 0,0:devctl
>>>> CONNECTED
>>>>
>>>> % luxadm -v probe
>>>>
>>>> Found Fibre Channel device(s):
>>>> Node WWN:5000d310004a0602 Device Type:Disk device
>>>> Logical Path:/dev/rdsk/c16t5000D310004A064Bd1s2
>>>> Node WWN:5000d310004a0602 Device Type:Disk device
>>>> Logical Path:/dev/rdsk/c17t5000D310004A064Cd1s2
>>>>
>>>> % luxadm disp /dev/rdsk/c16t5000D310004A064Bd1s2
>>>> DEVICE PROPERTIES for disk: /dev/rdsk/c16t5000D310004A064Bd1s2
>>>> Vendor: COMPELNT
>>>> Product ID: Compellent Vol
>>>> Revision: 0505
>>>> Serial Num: 00004a06-00000014
>>>> Unformatted capacity: 512000.000 MBytes
>>>> Read Cache: Enabled
>>>> Minimum prefetch: 0x100
>>>> Maximum prefetch: 0xf0
>>>> Device Type: Disk device
>>>> Path(s):
>>>>
>>>> /dev/rdsk/c16t5000D310004A064Bd1s2
>>>>
>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0/disk at w5000d310004a064b,1:c,raw
>>>>
>>>> Controller /dev/cfg/c16
>>>> Device Address 5000d310004a064b,1
>>>> Host controller port WWN 2100001b32940767
>>>> Controller /dev/cfg/c17
>>>> Device Address 5000d310004a064c,1
>>>> Host controller port WWN 2101001b32b40767
>>>>
>>>>
>>>> On 10/17/2012 06:25 PM, Majeed wrote:
>>>>> Hi All, We are doing some testing with omniOS in our datacenter
>>>>> and would
>>>>> love connect it up to our compellent san via fc connectivity. I
>>>>> slapped
>>>>> omniOS onto a dell r610 that has a qlogic dual port hba (each port
>>>>> is pathed
>>>>> to separate fabrics), performed zoning in each fabric, and
>>>>> assigned some
>>>>> luns. Now im not sure if compellent supports Solaris 11 (i can
>>>>> contact them
>>>>> and see about that), but it definately support solaris 10 which is
>>>>> what we
>>>>> use on other production systems.
>>>>>
>>>>> I'v actually made it pretty far through the process which is really
>>>>> exciting, except for the last multipathing bit. After mapping a
>>>>> volume to
>>>>> the host I can't seem to condense all the devices down to a single
>>>>> mulipathed device. It seems multipathing is enabled for the fp
>>>>> driver. I
>>>>> noticed there is not a mpathadm command.
>>>>>
>>>>> Anyone have any ideas? I'll post some output from the commands i
>>>>> think of
>>>>> below. Let me know if you would like to see any other information.
>>>>>
>>>>> Thanks much,
>>>>> Majeed
>>>>>
>>>>>
>>>>> On solaris 10 the main thing we have to do is add the following to
>>>>> /kernel/drv/scsi_vhci.conf (reconfigure reboot)
>>>>> device-type-scsi-options-list =
>>>>> "COMPELNTCompellent Vol", "symmetric-option";
>>>>> symmetric-option = 0x1000000;
>>>>>
>>>>>
>>>>> Here is the format output:
>>>>> AVAILABLE DISK SELECTIONS:
>>>>> 0. c13t5000CCA0213ED1E1d0<HITACHI-HUC106030CSS600-A360 cyl
>>>>> 36470
>>>>> alt 2 hd 255 sec 63>
>>>>>
>>>>> /pci at 0,0/pci8086,2940 at 1c/pci1028,1f1e at 0/iport at 40/disk at w5000cca0213ed1e1,0
>>>>>
>>>>> 1. c14t5000CCA0213EFC7Dd0<HITACHI-HUC106030CSS600-A360 cyl
>>>>> 36469
>>>>> alt 2 hd 255 sec 63>
>>>>>
>>>>> /pci at 0,0/pci8086,2940 at 1c/pci1028,1f1e at 0/iport at 80/disk at w5000cca0213efc7d,0
>>>>>
>>>>> 2. c16t5000D310004A064Bd1<COMPELNT-CompellentVol-0505 cyl
>>>>> 65268
>>>>> alt 2 hd 255 sec 63>
>>>>>
>>>>> /pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0/disk at w5000d310004a064b,1
>>>>> 3. c17t5000D310004A064Cd1<COMPELNT-CompellentVol-0505 cyl
>>>>> 65268
>>>>> alt 2 hd 255 sec 63>
>>>>>
>>>>> /pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0,1/fp at 0,0/disk at w5000d310004a064c,1
>>>>>
>>>>> Specify disk (enter its number): Specify disk (enter its number):
>>>>>
>>>>>
>>>>>
>>>>> % fcinfo hba-port
>>>>> HBA Port WWN: 2100001b32940767
>>>>> Port Mode: Initiator
>>>>> Port ID: 91c00
>>>>> OS Device Name: /dev/cfg/c16
>>>>> Manufacturer: QLogic Corp.
>>>>> Model: 375-3356-02
>>>>> Firmware Version: 05.03.01
>>>>> FCode/BIOS Version: BIOS: 2.02; fcode: 2.01; EFI: 2.00;
>>>>> Serial Number: 0402H00-0941774287
>>>>> Driver Name: qlc
>>>>> Driver Version: 20100408-3.01
>>>>> Type: N-port
>>>>> State: online
>>>>> Supported Speeds: 1Gb 2Gb 4Gb
>>>>> Current Speed: 4Gb
>>>>> Node WWN: 2000001b32940767
>>>>> Max NPIV Ports: 127
>>>>> NPIV port list:
>>>>> HBA Port WWN: 2101001b32b40767
>>>>> Port Mode: Initiator
>>>>> Port ID: a1900
>>>>> OS Device Name: /dev/cfg/c17
>>>>> Manufacturer: QLogic Corp.
>>>>> Model: 375-3356-02
>>>>> Firmware Version: 05.03.01
>>>>> FCode/BIOS Version: BIOS: 2.02; fcode: 2.01; EFI: 2.00;
>>>>> Serial Number: 0402H00-0941774287
>>>>> Driver Name: qlc
>>>>> Driver Version: 20100408-3.01
>>>>> Type: N-port
>>>>> State: online
>>>>> Supported Speeds: 1Gb 2Gb 4Gb
>>>>> Current Speed: 4Gb
>>>>> Node WWN: 2001001b32b40767
>>>>> Max NPIV Ports: 127
>>>>> NPIV port list:
>>>>>
>>>>>
>>>>> % cfgadm -al -o show_SCSI_LUN
>>>>> Ap_Id Type Receptacle Occupant
>>>>> Condition
>>>>> c16 fc-fabric connected configured
>>>>> unknown
>>>>> c16::5000d310004a0649 unknown connected unconfigured
>>>>> unknown
>>>>> c16::5000d310004a064b,1 disk connected configured
>>>>> unknown
>>>>> c17 fc-fabric connected configured
>>>>> unknown
>>>>> c17::5000d310004a064a unknown connected unconfigured
>>>>> unknown
>>>>> c17::5000d310004a064c,1 disk connected configured
>>>>> unknown
>>>>>
>>>>>
>>>>> _______________________________________________
More information about the OmniOS-discuss
mailing list