Accessing PCI devices

[kakaroto]

Hi, I have a question and maybe it's a dumb one but I can't seem to figure it out. When the JTAG is enabled, it's from the BUP process, which is a user space program, at Ring-3. Is there a way to escalate privilege to kernel mode? I can't seem to be able to set a breakpoint in kernel code, modifying the TSS doesn't work and thread.step() doesn't actually seem to work either (does the same as thread.go())
I'm not sure how we're supposed to get the full access to the ME from Ring-3, I'm trying to just read io port 0x80 for example (or any I/O port for that matter) and it all returns 0xFFFFFFF and that's because the TSS for BUP process has IOPB empty.
Any advice?
Thanks!

[h0t]

Hi @kakaroto. You can get it through several ways:

1. setup a breakpoint in kernel space (cs for kernel is 0x8, like "0x8:");
2. manually change cs register;

You can also use the suffix "P" to access physical memory (like "0xf0080004P"). ME doesn't have I/O ports, if I recall correctly.

[kakaroto]

Hi @h0t, thanks for the quick answer.

I guess my issue was the selector, I was setting the breakpoints on the linear address and that doesn't seem to work. Weirdly though even if I break inside the interrupt now (int 0x80) the "flags" register still show ring-3 and yes, it seems you're right, ME has no I/O ports according to datasheet vol.2, I had missed that it seems. I'm still unclear then on how we can access any device, without I/O ports, I can't access PCI devices without using MMIO and I can't figure out what the MMIO addresses for PCI is (or for IOSF-SB or for anything pretty much). in your README.md, you talk about HROMMB at offset 0xe20 of the MISA MMIO, and I can see in datasheet vol3, section 27.13 that HROMMB is at offset 0xe20 of MISA MMIO, but I don't see how you got that MISA MMIO is at 0xf0000000P. Also it doesn't seem to help me figure out how to access the PCI configuration space (or if that's not possible, how to use IOSF-SB to access a PCI device, although vol2 says that CSE has access to all those PCI devices so there must be a way to enumerate them).

FYI: I've been working on reversing the exact details of the exploit used here to write up an analysis/explanation of how it works (since you used syslib_tracer, not sys_shared_mem to achieve it on apollolake, it's technically a different method from what you showed at Blackhat), the only thing missing for me to figure out is what that side_band_mapping in the ROPs does which sets values in segment 0xC7 and why you call it with the values 0x706a8 for DCI and 0x70684 for DfX personality. Any hints would be welcome!

Thanks for the help!

[peterbjornx]

Hi kakaroto,
The base address for MISA MMIO can be glanced from the PCI BARs for it, the ME uses PCIE memory based config space access starting at 0xe000_0000. Device 0:0.0 is the SA. The sideband mapping routine sets up the AddressTranslationTable peripheral to provide a MMIO window into a sideband BAR.

[h0t]

Hi @kakaroto,
In generally ME works with private-PCI devices through MMIO space. Some devices (like SPI) have the
special MMIO space for ME and we didn't see the information about this ranges in documentation. Do you want get access to non-pPCI devices? What kind of pci devices do you need?

[kakaroto]

Thanks for the help.
Two days ago I had dumped all the MMIO ranges from the bup metadata and found 0xe0070000 to be a PCI device and then I used 0xe0000000 as a base address to find the device 0:0.0, 0:1.0, 0:13.2 (SPI) and 0:14.0 (audio). The 13.2 and 14.2 have same device ID as for host, but with 0 instead of 5 in the highest nibble (0x0a96 and 0x0a98 instead of 0x5a96 and 0x5a98).
I've also found that there are other PCI devices at 0xF1000000. Those have IDs like 0xFFF1, 0xFFF2, etc.. so not very useful. I haven't yet looked into them in details to figure out what they are for.
I also found thanks to @peterbjornx's hint about setting up the ATT for the sideband MMIO window that the sideband mapping changes the content of the MMIO at address 0xF6110000 depending on the value in 0xf00a9018. I'm now dumping a bunch of those to see what's in them. I still don't know how the magic number 0x70684 translates to the DFx aggregator at offset 0x8400, especially since the "Inside Intel Management Engine" slides say that the DFx personality register is at offset 0xF5010000 (that MMIO address is not accessible though).

@h0t now that I know how to access the ATT, I'll try and bruteforce my way around it, maybe I'll find something, but with the datasheet saying "The CSE can access IOSF-SB Private CR space from an SB ATT window" and your Visa slides saying "Intel CSME has full access to SB (via its own ATT bridge with a privileged SAI).", I guess I should be able to access the PCI devices with the right DestID sent to the right IOSF CR in the right ATT window... The problem is knowing what values are needed.

I'd like to be able to access PCIe, USB or SATA PCI devices so yes, I want the non-private PCI devices which according to your presentations should be doable from the ME but I can't find information on how to achieve it..
Thanks!

[kakaroto]

By the way, I still don't know how you got the 0x706a8 (for DCI) and 0x70684 for (DfX aggregator) values. I assume those are the Dest ID to the IOSF SB through ATT window, but I don't see where you found them (I did find 0x10070684 in TXE code but not 0x706a8) or how you got that full list of SB ports for LBG in your bonus slides. (I did find the 0x54 IR command for DFX Aggregator from the TDef files, so at least there's that answer).

[peterbjornx]

I'm not entirely sure about the highest nybble of the sideband address, but the breakdown of that bitfield is:

• Bits 7 downto 0 : Sideband port/endpoint ID
• Bits 15 downto 8 : Read opcode
• Bits 23 downto 16 : Write opcode
• Bits 26 downto 24 : BAR number
• Bit 27: Unknown, possibly also part of BAR
• Bit 28: Flag, unknown
• Bit 29: Flag, unknown
• Bit 30: Not used by bup on SPT
• Bit 31: Not used by bup on SPT

Which for the examples yields:

• 0x10070684 : Endpoint 0x84:BAR0, readop=6, writeop=7, flag28 set
• 0x00070684 : Endpoint 0x84:BAR0, readop=6, writeop=7

And to find those numbers, there are tables in bup and busdrv which contain all devices that are available at runtime, which have base addresses, SB endpoints, PCI BDFs and so on.

Source: Own research and https://twitter.com/_markel___/status/1121131631245443074

[kakaroto]

Thank you Peter, that was very helpful. I facepalmed when I realized the sideband port is basically the PCR port id. I just made a loop to dump all of their bars and found quite a few. I'm not sure about those readop/writeop or flag28, it didn't seem to make any difference to me what I set in bits 24 and up.
I found the bridge host PCI device (id 8086:5AF0) at 0x050441, a PCI device with id 8086:5A8E at 0x050488 and id 8086:0008 at 0x050432 (+ the audio PCI device found previously). When I saw the host bridge, I was hoping that the read opcode 4 was for the PCI config space for all the pci devices, but no luck.

I did manage to get the MODPHY (0xa5) and USB Host (0xa2) BARs, just not the PCI config space.
I'm still searching for those, but for now I have enough to attempt my port to skylake!

Unfortunately, I still can't get the main CPU to boot (seems to block in an infinite loop at 0x35644), if I can't figure it out soon, I'll be opening another issue (or re-open #11).

[h0t]

Unfortunately, I still can't get the main CPU to boot (seems to block in an infinite loop at 0x35644), if I can't figure it out soon, I'll be opening another issue (or re-open #11).

Did you try activate HAP-mode? #11 (comment)

[kakaroto]

Did you try activate HAP-mode? #11 (comment)

Yes of course. I've tried the F5 image as is and the Brix boots, I tried with the exploit only, and I get that expected behavior of it shutting down continuously, I enabled HAP bit and I get that infinite loop issue in bup where it doesn't seem to finish that initiatlization (it was at index (ebx) 0xC and byte_97258 has 0x2 and byte_97259 has 0x3). I used JTAG to make it unlock it (changed byte_97258 to 0x3) and BUP did finish but then the kernel went into the hang function. I have the v2 hardware and so far, I haven't been able to debug what's the issue, especially with you saying it works on your side.

[h0t]

Hi @ValZapod. We aren't going to publish your collection of visa.xml files. You can get it from Intel System Studio, installers is available on Intel's website. We don't use USB-C. Unfortunately, I have no idea about DCI throgh USB-C. What is your hardware?

Please create separate issues for different topics!

[kakaroto]

Hi again, I'm still struggling to get access to PCI devices here.
On Apollolake I was able to access some device BARs through the sideband channel, but not the PCI devices themselves (xHCI controller, and PHY, but not SATA for some reason), but on Skylake and Kabylake, I can't get any of those devices. The only devices I'm able to read through the sideband channel (for KBP) are :
Read opcode 0:

• 0x96, 0xb6: unknown devices

Read opcode 4:

• 0x91, 0x92, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0xdb : This gives us the PCI devices accessible via 0xe0000000
  Read opcode 6:
• 0x89, 0x93: unknown PCI device
• 0x8f, 0x9c, 0x9e, 0x9f, : unknown devices
• 0xac, 0xad, 0xae, 0xaf: GPIO Communities
• 0xb7: DfX Aggregator
• 0xb8: DCI
• 0xb9: unknown PCI device
• 0xba, 0xbb, 0xbc, 0xbd: PSF devices
• 0xd2, 0xd5: Unknown devices
• 0xdc: ICC clock controller

There's nothing else when I try the other read opcodes. As you can see, there's no LPC, USB, HDA, SATA, PCIe, xHCI or SPI device that the sideband channel grants access to.
At least, with apollolake we could read the xHCI controller MMIO, but I haven't been able to find a way of accessing the non-private PCI devices or Sideband channels from within the CSME.

I haven't looked at the private PCI devices yet, but without a valid device ID, I can't really figure out what they are for, and their class IDs are all zero apart from the Host bridge. I also found one PCI-to-PCI bridge device, but even after enabling MMIO access and bus master enable (PCI command 0x6) and setting the primary and secondary bus channels, it didn't seem to give me access to an additional PCI devices/buses.

From your presentation, you said that the ME had access to every device on the machine, was there anything that needed to be done in order to enable that access ?

If there's anything you can help with, I'd appreciate it.

Thanks!

[kakaroto]

By the way, I thought I should clarify something.

I've tried using ipc.stateport.spt_tpsb0.sbreg(0, 0, 0x8004, 0xe6, 1, 4, 0) to read from port 0xe6 (xHCI), and it says "Endpoint returned powered down status" if the main CPU is off, but after powering it up, it does return the "USB " string as expected.
What I'm talking about however is trying to access it by changing the SB ATT window in address 0xf00a9000.
I set 0xf00a9018 to the destination id, 0xf00a901c to 0x101 and I read the mmio from the address in 0xf00a9000 (up to the size in 0xf00a9004)
See this screenshot for clarification :

and :

This shows that xHCI is indeed available on port 0xe6 with opcode 0 and bar 0, it's readable via JTAG but not by the ME core itself.
I'm trying to see if I can reset the core and break as early as possible (following on hints given here : #14 (comment)) to see if the ME itself is locking access to those devices early on, but I'm unable to get it to reset and halt.
You've said in the past that the ME has access to all the devices the main CPU has + extra private ones, and in your presentation you said the ME has a special SAI that gives it full access to the SB, but I'm unable to reproduce that from my side. Any ideas of what I'm doing wrong here ?

[markel777]

Hi @kakaroto, the next are some explanations regarding your problem and ME ability to access platform devices (IPs):

1. The modern PCHs are built on IOSF architecture. There are to two buses used by IPs to interact: IOSF Primary and IOSF Side Band. You are only trying to leverage IOSF SB to access those IPs
2. Both buses implement some abstraction called Root Space: we can consider it as an isolation mechanism for address spaces designed for different consumers. There are three Root Spaces: Host, IE and CSME
3. As a general rule, IPs implement different interfaces in distinct Root Spaces
4. IOSF Side Band doesn't require devices connected by it to implement all opcodes. Some devices can implement only Private Configuration Registers (MSG addr space, opcode 6), some - only BARs (opcode 0), rarely - all standard opcodes (cfg, mem, msg)
5. No all IPs are connected by IOSF SB (almost though). IOSF allows only one bus (Primary or SB) to be implemented by an IP.
6. ATT SB bridge in CSE Gasket allows specifying Root Space for an address range at 0x1c offset of corresponding slot (bits 0-7 - function id, 8-15 - root space).
7. Because you reused the ATT SB slot which was previously used to access an device in CSME root space, it still contains at 0x1c offset 0x0101 (fid 1, rs 1)
8. The devices you are attempting to access expose their regular interfaces in Host Root Space, so you need to change the rs in att slot to 0 for host rs. Also, change fid to 0, because it's very probably you are intending to access zero function of the devices

[kakaroto]

Hi @markel777 and thanks a lot for taking the time to explain that. It was very helpful in better understanding how the IOSF SB works.
Unfortunately, I had already tried setting that offset 0x1c to values 0, 0x1, 0x100 and 0x101. Although I didn't know what exactly it was for, I had tried all these values to see if it made a difference but it didn't.

As you can see from that sbreg command I had used root space 1 for the request to tap2iosb and it worked fine, so I don't think the root space is the problem here. Either way, I've tried setting it to 0 with no success.
It was very helpful though to know that the offset 0x1c held the rootspace and function id as I didn't know what that value was. It also led me to figure out that the 'fid' also has the pci device in it, so I could do a read of 0xe6 with opcode 4 (PCI Config) and fid 0xa0 (D20:F0) and it gave me the ability to dump the PCI configuration space of the xHCI device. Unfortunately, it still only works when using TAP2IOSB, not when using the SB channel in the ATT Gasket.

Are you able to reproduce this problem from your side? Do you have any other ideas on what could be preventing the ME thread from accessing the SB channel ?

Thank you!

[kakaroto]

@markel777 sorry for bumping this thread.

It's been a month and I'm wondering if you have any updates relating to this or any ideas for me to explore ? I don't know what is different from using the ipc.stateport.sbreg or using the SB window directly.

I think the issue might somehow be related to the EPMASK register, but when I checked the P2SB device's PCI config, the EPMASK isn't blocking access to those ports. I assume that this is because that configuration is specific to the host, and there must be an equivalent one for the CSME. Any ideas where to find it or access it (none of the CSME PCI devices seem to match)?

I'd appreciate if you have any other thoughts/ideas that I could experiment with.
Thanks.

[sladen]

+1 year later, and @kakaroto's blog post gives some more information about why the XHCI registers are write-only on Skylake—because of the setting of the WP/RAC/WAC registers.

[sladen]

And a link to the blog post itself…:

• https://kakaroto.ca/2020/08/exploiting-intels-management-engine-part-3-usb-hijacking-intel-sa-00086/