[OmniOS-discuss] multipathing against compellent
Majeed
mabuqu at ilstu.edu
Thu Oct 18 16:45:15 EDT 2012
Sorry for the spam everyone, wrong list :(
On 10/18/2012 03:35 PM, Majeed wrote:
> Hi All, As noted below, we have attempted to test the illumos based
> omniOS in our datacenter on a dell R610 connected through dual cisco
> fiber channel fabrics to a compellent SAN with dual controllers. We
> got pretty far with an out of the box setup until the last
> multipathing bit. I was hoping there might be some additional eyes on
> this list that may be able to provide some insight.
>
> Thanks in advance for taking the time to read through my following
> notes on this topic,
> Majeed Qulbain
>
>
>
>
> On 10/18/2012 03:06 PM, Majeed wrote:
>> Thanks Theo, that makes sense. I will definitely do that.
>>
>> Majeed
>>
>> On 10/18/2012 02:58 PM, Theo Schlossnagle wrote:
>>> This is a more general Illumos problem, you will open your
>>> troubleshooting to a larger audience by interfacing with the
>>> illumos-discuss list. I encourage you to post there! I'm subscribed
>>> there as well, but I haven't done multipathing against compellent and
>>> it appears no one else on the omnios-discuss list has either.
>>> Anything learned in the illumos-discuss group can be applied directly
>>> to OmniOS.
>>>
>>> On Thu, Oct 18, 2012 at 3:51 PM, Majeed<mabuqu at ilstu.edu> wrote:
>>>> Hi Again, just wanted to update the list with my latest progress and
>>>> thoughts on this topic. I think the issue lies in
>>>> libmpscsi_vhci.so's lack
>>>> of support for multipathing with compellent.
>>>>
>>>> When I do "mpathadm show mpath-support libmpscsi_vhci.so" on omniOS
>>>> I see
>>>> many venders and models listed but not compellent. We are going to
>>>> spin up a
>>>> solaris 11 machine to see what it looks like. I wonder if this was
>>>> forked
>>>> just before SUN/oracle added support for compellent (i could be way
>>>> off base
>>>> here).
>>>>
>>>> When I do the same command on solaris 10 u10, at the very bottom of
>>>> the list
>>>> i see.
>>>> Vendor: COMPELNT
>>>> Product: Compellent Vol
>>>> Revision:
>>>> Supported Load Balance Types:
>>>> round-robin
>>>>
>>>> I have verified that multipathing is working properly on other
>>>> arrays. I
>>>> zoned up omniOS with all controllers on a 6140 (CSM200) that sit in
>>>> multiple
>>>> fabrics, mapped a lun, and it looks wonderful. mpatham reports as
>>>> expected,
>>>> and all paths are condensed to a single block device in format's
>>>> output.
>>>>
>>>> % mpathadm list LU
>>>> /dev/rdsk/c0t600A0B800033816200001391507FF856d0s2
>>>> Total Path Count: 4
>>>> Operational Path Count: 4
>>>>
>>>>
>>>>
>>>> It seems there are some options in /kernel/drv/scsi_vhci.conf to force
>>>> scsi_vhci to accept 3rd party devices (hopefully compellent) for
>>>> failover,
>>>> but this is kinda getting over my head. It seems like if I had a
>>>> proper
>>>> ddi-forceload ''misc/scsi_vhci/scsi_vhci_f_<x>" file for compellent
>>>> and I
>>>> knew the proper VenderID and ProductID for the compellent devices,
>>>> i might
>>>> be able to force it. Im not sure if can use the information from
>>>> the format
>>>> inquiry to populate these values.
>>>>
>>>> format> inquiry
>>>> Vendor: COMPELNT
>>>> Product: Compellent Vol
>>>> Revision: 0505
>>>>
>>>>
>>>> Any thoughts on this would be excellent.
>>>>
>>>> Thanks,
>>>> Majeed Qulbain
>>>>
>>>>
>>>>
>>>> On 10/18/2012 09:30 AM, Majeed wrote:
>>>>> I just noticed that we do have access to the mpathadm command by
>>>>> installing the "pkg:/storage/mpathadm at 0.5.11-0.151002" package. After
>>>>> installing I don't see any paths with "mpathadm list LU" which is
>>>>> what im
>>>>> used to on solaris 10 (i know I should be comparing to solaris 11,
>>>>> just
>>>>> havn't played with it much). I wonder if there are any other bits
>>>>> that I
>>>>> need to install or other configuration tweaks to get this going.
>>>>>
>>>>> Thanks in advance for any pointers,
>>>>> Majeed
>>>>>
>>>>> Some other related commands that might assist troubleshooting:
>>>>>
>>>>> % uname -a
>>>>> SunOS jhx011 5.11 omnios-eae537b i86pc i386 i86pc
>>>>>
>>>>> % mpathadm list initiator-port
>>>>> Initiator Port: w5d4ae520937bbb00
>>>>> Initiator Port:
>>>>> iqn.1986-03.com.sun:01:b06a64f606ff.507f35b3,4000002a00ff
>>>>> Initiator Port: w5d4ae520937bbb00
>>>>> Initiator Port: 2101001b32b40767
>>>>> Initiator Port: 2100001b32940767
>>>>>
>>>>> % luxadm -e port
>>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0:devctl
>>>>> CONNECTED
>>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0,1/fp at 0,0:devctl
>>>>> CONNECTED
>>>>>
>>>>> % luxadm -v probe
>>>>>
>>>>> Found Fibre Channel device(s):
>>>>> Node WWN:5000d310004a0602 Device Type:Disk device
>>>>> Logical Path:/dev/rdsk/c16t5000D310004A064Bd1s2
>>>>> Node WWN:5000d310004a0602 Device Type:Disk device
>>>>> Logical Path:/dev/rdsk/c17t5000D310004A064Cd1s2
>>>>>
>>>>> % luxadm disp /dev/rdsk/c16t5000D310004A064Bd1s2
>>>>> DEVICE PROPERTIES for disk: /dev/rdsk/c16t5000D310004A064Bd1s2
>>>>> Vendor: COMPELNT
>>>>> Product ID: Compellent Vol
>>>>> Revision: 0505
>>>>> Serial Num: 00004a06-00000014
>>>>> Unformatted capacity: 512000.000 MBytes
>>>>> Read Cache: Enabled
>>>>> Minimum prefetch: 0x100
>>>>> Maximum prefetch: 0xf0
>>>>> Device Type: Disk device
>>>>> Path(s):
>>>>>
>>>>> /dev/rdsk/c16t5000D310004A064Bd1s2
>>>>>
>>>>> /devices/pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0/disk at w5000d310004a064b,1:c,raw
>>>>>
>>>>> Controller /dev/cfg/c16
>>>>> Device Address 5000d310004a064b,1
>>>>> Host controller port WWN 2100001b32940767
>>>>> Controller /dev/cfg/c17
>>>>> Device Address 5000d310004a064c,1
>>>>> Host controller port WWN 2101001b32b40767
>>>>>
>>>>>
>>>>> On 10/17/2012 06:25 PM, Majeed wrote:
>>>>>> Hi All, We are doing some testing with omniOS in our datacenter
>>>>>> and would
>>>>>> love connect it up to our compellent san via fc connectivity. I
>>>>>> slapped
>>>>>> omniOS onto a dell r610 that has a qlogic dual port hba (each
>>>>>> port is pathed
>>>>>> to separate fabrics), performed zoning in each fabric, and
>>>>>> assigned some
>>>>>> luns. Now im not sure if compellent supports Solaris 11 (i can
>>>>>> contact them
>>>>>> and see about that), but it definately support solaris 10 which
>>>>>> is what we
>>>>>> use on other production systems.
>>>>>>
>>>>>> I'v actually made it pretty far through the process which is really
>>>>>> exciting, except for the last multipathing bit. After mapping a
>>>>>> volume to
>>>>>> the host I can't seem to condense all the devices down to a single
>>>>>> mulipathed device. It seems multipathing is enabled for the fp
>>>>>> driver. I
>>>>>> noticed there is not a mpathadm command.
>>>>>>
>>>>>> Anyone have any ideas? I'll post some output from the commands i
>>>>>> think of
>>>>>> below. Let me know if you would like to see any other information.
>>>>>>
>>>>>> Thanks much,
>>>>>> Majeed
>>>>>>
>>>>>>
>>>>>> On solaris 10 the main thing we have to do is add the following to
>>>>>> /kernel/drv/scsi_vhci.conf (reconfigure reboot)
>>>>>> device-type-scsi-options-list =
>>>>>> "COMPELNTCompellent Vol", "symmetric-option";
>>>>>> symmetric-option = 0x1000000;
>>>>>>
>>>>>>
>>>>>> Here is the format output:
>>>>>> AVAILABLE DISK SELECTIONS:
>>>>>> 0. c13t5000CCA0213ED1E1d0<HITACHI-HUC106030CSS600-A360
>>>>>> cyl 36470
>>>>>> alt 2 hd 255 sec 63>
>>>>>>
>>>>>> /pci at 0,0/pci8086,2940 at 1c/pci1028,1f1e at 0/iport at 40/disk at w5000cca0213ed1e1,0
>>>>>>
>>>>>> 1. c14t5000CCA0213EFC7Dd0<HITACHI-HUC106030CSS600-A360
>>>>>> cyl 36469
>>>>>> alt 2 hd 255 sec 63>
>>>>>>
>>>>>> /pci at 0,0/pci8086,2940 at 1c/pci1028,1f1e at 0/iport at 80/disk at w5000cca0213efc7d,0
>>>>>>
>>>>>> 2. c16t5000D310004A064Bd1<COMPELNT-CompellentVol-0505 cyl
>>>>>> 65268
>>>>>> alt 2 hd 255 sec 63>
>>>>>>
>>>>>> /pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0/fp at 0,0/disk at w5000d310004a064b,1
>>>>>>
>>>>>> 3. c17t5000D310004A064Cd1<COMPELNT-CompellentVol-0505 cyl
>>>>>> 65268
>>>>>> alt 2 hd 255 sec 63>
>>>>>>
>>>>>> /pci at 0,0/pci8086,3410 at 9/pci1077,143 at 0,1/fp at 0,0/disk at w5000d310004a064c,1
>>>>>>
>>>>>> Specify disk (enter its number): Specify disk (enter its number):
>>>>>>
>>>>>>
>>>>>>
>>>>>> % fcinfo hba-port
>>>>>> HBA Port WWN: 2100001b32940767
>>>>>> Port Mode: Initiator
>>>>>> Port ID: 91c00
>>>>>> OS Device Name: /dev/cfg/c16
>>>>>> Manufacturer: QLogic Corp.
>>>>>> Model: 375-3356-02
>>>>>> Firmware Version: 05.03.01
>>>>>> FCode/BIOS Version: BIOS: 2.02; fcode: 2.01; EFI: 2.00;
>>>>>> Serial Number: 0402H00-0941774287
>>>>>> Driver Name: qlc
>>>>>> Driver Version: 20100408-3.01
>>>>>> Type: N-port
>>>>>> State: online
>>>>>> Supported Speeds: 1Gb 2Gb 4Gb
>>>>>> Current Speed: 4Gb
>>>>>> Node WWN: 2000001b32940767
>>>>>> Max NPIV Ports: 127
>>>>>> NPIV port list:
>>>>>> HBA Port WWN: 2101001b32b40767
>>>>>> Port Mode: Initiator
>>>>>> Port ID: a1900
>>>>>> OS Device Name: /dev/cfg/c17
>>>>>> Manufacturer: QLogic Corp.
>>>>>> Model: 375-3356-02
>>>>>> Firmware Version: 05.03.01
>>>>>> FCode/BIOS Version: BIOS: 2.02; fcode: 2.01; EFI: 2.00;
>>>>>> Serial Number: 0402H00-0941774287
>>>>>> Driver Name: qlc
>>>>>> Driver Version: 20100408-3.01
>>>>>> Type: N-port
>>>>>> State: online
>>>>>> Supported Speeds: 1Gb 2Gb 4Gb
>>>>>> Current Speed: 4Gb
>>>>>> Node WWN: 2001001b32b40767
>>>>>> Max NPIV Ports: 127
>>>>>> NPIV port list:
>>>>>>
>>>>>>
>>>>>> % cfgadm -al -o show_SCSI_LUN
>>>>>> Ap_Id Type Receptacle Occupant
>>>>>> Condition
>>>>>> c16 fc-fabric connected configured
>>>>>> unknown
>>>>>> c16::5000d310004a0649 unknown connected
>>>>>> unconfigured
>>>>>> unknown
>>>>>> c16::5000d310004a064b,1 disk connected configured
>>>>>> unknown
>>>>>> c17 fc-fabric connected configured
>>>>>> unknown
>>>>>> c17::5000d310004a064a unknown connected
>>>>>> unconfigured
>>>>>> unknown
>>>>>> c17::5000d310004a064c,1 disk connected configured
>>>>>> unknown
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
--
Majeed Abu-Qulbain
Lead Unix Admin - CISS
309-438-3455
Illinois State University
More information about the OmniOS-discuss
mailing list