[OmniOS-discuss] Trouble getting Supermicro board to see all its PCI-e slots

[Sašo Kiselkov]

Just a follow up on this: this appears to be an Illumos bug, though I
haven't figured out how to resolve it yet. The problem appears to be
that Illumos doesn't recognize any PCI bridges beyond those attached to
CPU1. Notably, nothing on CPU2 appears to work. I've tested with Linux
3.5 and that works flawless, attached are the lspci -vvvv outputs.

The problem appears to be that Illumos stops after PCI address 0b:00.0
(on-board MegaRAID SAS 2208). After that the scan should proceed with
the QPI inter-processor links, but doesn't.

The full lspci should look something like this:

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI..
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
00:04.0 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.1 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.2 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.3 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.4 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.5 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.6 System peripheral: Intel Corporation Xeon E5/Core ..
00:04.7 System peripheral: Intel Corporation Xeon E5/Core ..
00:05.0 System peripheral: Intel Corporation Xeon E5/Core ..
00:05.2 System peripheral: Intel Corporation Xeon E5/Core ..
00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..
00:11.0 PCI bridge: Intel Corporation C600/X79 series chip..
00:16.0 Communication controller: Intel Corporation C600/X..
00:16.1 Communication controller: Intel Corporation C600/X..
00:1a.0 USB controller: Intel Corporation C600/X79 series ..
00:1d.0 USB controller: Intel Corporation C600/X79 series ..
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (re..
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chip..
00:1f.2 SATA controller: Intel Corporation C600/X79 series..
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset S..
00:1f.6 Signal processing controller: Intel Corporation C6..
02:00.0 Serial Attached SCSI controller: Intel Corporation..
03:00.0 SCSI storage controller: LSI Logic / Symbios Logic..
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-..
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-..
07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113..
08:01.0 VGA compatible controller: Matrox Electronics Syst..
09:00.0 Ethernet controller: Intel Corporation I350 Gigabi..
09:00.1 Ethernet controller: Intel Corporation I350 Gigabi..
0b:00.0 RAID bus controller: LSI Logic / Symbios Logic Meg..
     <<< Illumos gives up here, Linux continues >>>
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:08.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:08.4 System peripheral: Intel Corporation Xeon E5/Core ..
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:09.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:09.4 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0e.1 Performance counters: Intel Corporation Xeon E5/Co..
7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core ..
7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.1 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.2 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.3 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.4 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.5 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.6 System peripheral: Intel Corporation Xeon E5/Core ..
7f:10.7 System peripheral: Intel Corporation Xeon E5/Core ..
7f:11.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:13.0 System peripheral: Intel Corporation Xeon E5/Core ..
7f:13.1 Performance counters: Intel Corporation Xeon E5/Co..
7f:13.4 Performance counters: Intel Corporation Xeon E5/Co..
7f:13.5 Performance counters: Intel Corporation Xeon E5/Co..
7f:13.6 System peripheral: Intel Corporation Xeon E5/Core ..
80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2..
80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..
80:04.0 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.1 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.2 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.3 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.4 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.5 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.6 System peripheral: Intel Corporation Xeon E5/Core ..
80:04.7 System peripheral: Intel Corporation Xeon E5/Core ..
80:05.0 System peripheral: Intel Corporation Xeon E5/Core ..
80:05.2 System peripheral: Intel Corporation Xeon E5/Core ..
80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..
83:00.0 SCSI storage controller: LSI Logic / Symbios Logic..

See http://217.75.71.10/MB.png for the system interconnection diagram.
Can somebody please recommend how to debug this issue?

Cheers,
--
Saso