[OmniOS-discuss] [discuss] Trouble getting Supermicro board to see all its PCI-e slots

Wed May 1 09:54:31 EDT 2013

Check the value of pci_bios_maxbus.

I have a sneaking feeling that the problem may be related to acpi enumeration of the bus.  We are limited to seeing the busses that ACPI properly enumerates.  Take a look at the code in pci_boot.c for more details.  You should especially note the call sites for create_pcie_root_bus.

I'd love to help further, but lack of access to hardware that exhibits this problem prevents me from doing any active debug myself.

	- Garrett

On May 1, 2013, at 12:50 AM, Sašo Kiselkov <skiselkov.ml at gmail.com> wrote:

> Just a follow up on this: this appears to be an Illumos bug, though I
> haven't figured out how to resolve it yet. The problem appears to be
> that Illumos doesn't recognize any PCI bridges beyond those attached to
> CPU1. Notably, nothing on CPU2 appears to work. I've tested with Linux
> 3.5 and that works flawless, attached are the lspci -vvvv outputs.
> 
> The problem appears to be that Illumos stops after PCI address 0b:00.0
> (on-board MegaRAID SAS 2208). After that the scan should proceed with
> the QPI inter-processor links, but doesn't.
> 
> The full lspci should look something like this:
> 
> 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI..
> 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 00:04.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.5 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:04.7 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:05.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:05.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..
> 00:11.0 PCI bridge: Intel Corporation C600/X79 series chip..
> 00:16.0 Communication controller: Intel Corporation C600/X..
> 00:16.1 Communication controller: Intel Corporation C600/X..
> 00:1a.0 USB controller: Intel Corporation C600/X79 series ..
> 00:1d.0 USB controller: Intel Corporation C600/X79 series ..
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (re..
> 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chip..
> 00:1f.2 SATA controller: Intel Corporation C600/X79 series..
> 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset S..
> 00:1f.6 Signal processing controller: Intel Corporation C6..
> 02:00.0 Serial Attached SCSI controller: Intel Corporation..
> 03:00.0 SCSI storage controller: LSI Logic / Symbios Logic..
> 05:00.0 Ethernet controller: Intel Corporation 82599EB 10-..
> 05:00.1 Ethernet controller: Intel Corporation 82599EB 10-..
> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113..
> 08:01.0 VGA compatible controller: Matrox Electronics Syst..
> 09:00.0 Ethernet controller: Intel Corporation I350 Gigabi..
> 09:00.1 Ethernet controller: Intel Corporation I350 Gigabi..
> 0b:00.0 RAID bus controller: LSI Logic / Symbios Logic Meg..
>     <<< Illumos gives up here, Linux continues >>>
> 7f:08.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:08.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:08.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:09.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:09.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:09.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0e.1 Performance counters: Intel Corporation Xeon E5/Co..
> 7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.5 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:10.7 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:11.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:13.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 7f:13.1 Performance counters: Intel Corporation Xeon E5/Co..
> 7f:13.4 Performance counters: Intel Corporation Xeon E5/Co..
> 7f:13.5 Performance counters: Intel Corporation Xeon E5/Co..
> 7f:13.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2..
> 80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
> 80:04.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.1 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.3 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.4 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.5 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.6 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:04.7 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:05.0 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:05.2 System peripheral: Intel Corporation Xeon E5/Core ..
> 80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..
> 83:00.0 SCSI storage controller: LSI Logic / Symbios Logic..
> 
> See http://217.75.71.10/MB.png for the system interconnection diagram.
> Can somebody please recommend how to debug this issue?
> 
> Cheers,
> --
> Saso
> 
> 
> -------------------------------------------
> illumos-discuss
> Archives: https://www.listbox.com/member/archive/182180/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/182180/22003744-9012f59c
> Modify Your Subscription: https://www.listbox.com/member/?member_id=22003744&id_secret=22003744-e9cd8436
> Powered by Listbox: http://www.listbox.com