feat: pv-iommu support and fixes for amd IOMMU on GCE - #34
Open
azenla wants to merge 27 commits into
Open
Conversation
Port of Teddy Astie's RFC v7 series, which reworks the IOMMU subsystem
around per-domain IOMMU contexts and exposes them to a guest through a new
PV-IOMMU hypercall interface. Squashed into one commit because the series
only builds as a whole -- the intermediate patches leave the VT-d and
AMD-Vi drivers half-converted.
https://patchwork.kernel.org/project/xen-devel/list/?series=1025765
Changes needed against this tree:
- iommu_iotlb_flush() grew a flags argument; the two callers in
xenmem_add_to_physmap() sit on locally modified lines and had to be
updated by hand.
- iommu_flush_iotlb() takes the context explicitly now instead of
deriving the default one, and reads the DID out of the context's didmap
rather than get_iommu_did(), which the series deletes.
- The series drops iommu_enable_device(); pci_add_device()'s
already-owned-by-d path attaches the context instead.
- pci_assign_device() and _assign_hwdom_pci_devices() called the old
assign_device(), replaced here by device_assigned() plus
pci_reassign_device().
- vtd's extern.h still declared domain_context_mapping_one() and
domain_context_unmap_one() with the pre-series signatures.
The interface is Dom0-only as posted: the hypercall identifies devices by
machine BDF, which a guest does not see.
PV-IOMMU was Dom0-only for one reason: IOMMU_reattach_device looks the device up with pci_get_pdev(), which is keyed on the machine SBDF. Dom0 sees machine addresses so that works there, but any other domain sees whatever vPCI handed it -- our PVH guests get the GPU at 0000:00:01.0 -- and has no way of knowing where the device really lives. Translate instead of rejecting. vPCI already records the virtual SBDF it assigned in pdev->vpci->guest_sbdf and walks it in translate_virtual_device(); split that lookup out as vpci_get_pdev_by_guest_sbdf() and use it for non-hardware domains. This covers domains whose devices come through vPCI, which on x86 means PVH. A PV guest driving pcifront still cannot name its devices: the virtual BDF there is invented by pciback in Dom0 and never reaches the hypervisor, so there is nothing to translate against. Also fix IOMMU_init falling through into IOMMU_alloc_context, which clobbered init's return value with a context number and leaked a context on every init. Dom0 got away with it; a guest has to call IOMMU_init to get any contexts at all, so it does not.
intel_iommu_get_max_iova() derives the aperture from the domain's agaw, which iommu_domain_init() sets to DEFAULT_DOMAIN_ADDRESS_WIDTH (48) for every domain regardless of what the hardware can address. Dom0 never noticed: its IOVAs come from Xen and never exceed physical memory. A guest allocates from the top of whatever aperture it is told about, so it hands the device addresses the IOMMU refuses: xen-iommu: max_iova_addr=ffffffffffff nvidia 0000:00:01.0: Using 47-bit DMA addresses [VT-D]DMAR:[DMA Read] Request device [0000:01:00.0] fault addr 7fff7fac000 [VT-D]DMAR: reason 04 - Access beyond MGAW Clamping to min_pt_levels is not enough. SAGAW only says which page table depths a unit implements -- on this machine it selects 4 levels, so that bound is 48 bits and changes nothing. How wide an address the unit will actually accept is MGAW, which is 39 bits here and which this tree never read. Track the narrowest MGAW in the system next to min_pt_levels and clamp the reported aperture to it as well.
A guest whose devices come through vPCI names them by the virtual SBDF Xen assigned, which the previous patch translates. A PV guest driving pcifront has no such mapping: its virtual BDF is invented by pciback in the hardware domain and the hypervisor never sees it, so there is nothing to translate against. Let those name the device by machine BDF instead. pci_get_pdev() already constrains the search to devices the domain owns, so a guest can only reach its own, and pcifront publishes the machine BDF in xenstore for the guest to find. Both guest kinds now end up at the same pdev; vPCI is tried first so a virtual SBDF that happens to collide with a machine BDF still resolves the way that domain sees the world.
apply_context_single() flushes the context cache and IOTLB for prev_did, which clears what the device's former owner left behind. Nothing flushes the did being installed. That matters because the redesign hands out dids from a map and recycles them between domains, and VT-d is free to cache non-present translations. A domain that gets a recycled did inherits whatever the previous owner of that did left cached, so a device reads through a stale not-present entry even though its page tables have the address mapped: [VT-D]DMAR:[DMA Read] Request device [0000:01:00.0] fault addr 11e590000 [VT-D]DMAR: reason 06 - PTE Read access is not set print_vtd_entries: iommu #1 dev 0000:01:00.0 gmfn 11e590 context[00] = b02_18b88cd001 l4[000] = b800001233916107 rw l3[004] = b8000008000003b7 rw The first domain to be handed a device gets a did nothing has cached against and works; every domain after it does not. It looks like the device is wedged -- it survives FLR, a secondary bus reset and a fresh guest kernel -- because the device was never the problem. Flush the new did too when it differs from the previous one.
My previous commit blamed did recycling and flushed the did being
installed. That was wrong, and it could not have worked: a device-selective
context flush names the did of the entry it is evicting, so flushing the
new did never touches an entry cached under the old one. Reverted here.
The real problem is that prev_dom is declared NULL in apply_context_single()
and never assigned, so !prev_dom is always true and both flushes are always
issued as non-present-entry flushes. vtd_flush_iotlb_reg() and the context
equivalent skip those entirely on hardware without caching mode, which is
any normal VT-d implementation:
if ( flush_non_present_entry && !cap_caching_mode(iommu->cap) )
/* nothing to do */
So a device's context cache and IOTLB were never invalidated when its
context changed. It kept translating through the dom_io quarantine context
it was in before assignment, which maps nothing, and every DMA faulted
while a walk of the new page tables showed the address mapped read/write:
[VT-D]DMAR:[DMA Read] Request device [0000:01:00.0] fault addr f7fab000
[VT-D]DMAR: reason 06 - PTE Read access is not set
print_vtd_entries: iommu #1 dev 0000:01:00.0 gmfn f7fab
context[00] = 902_f47000001
l4[000] = f47001003 rw
l3[003] = f47002003 rw
l2[1bf] = f47005003 rw
l1[1ab] = 82388b003 rw
overwrite_entry already records whether the entry being replaced was
present, which is exactly what the flush wants to know. Use it, and drop
prev_dom.
amd_iommu_detect_one_acpi() validated the block against sizeof(struct acpi_ivrs_hardware), which covers the efr_image and reserved fields that only types 0x11 and 0x40 carry. A type 0x10 block whose fixed 24-byte portion is followed by fewer than 16 bytes of device entries was rejected as malformed, disabling the IOMMU entirely. iommu_acpi.c already had get_ivhd_header_size() for exactly this; share it rather than leaving the two paths to disagree.
flush_command_buffer() logged a warning on timeout, cleared its deadline and then kept polling, so an IOMMU that never stores the completion word hangs the boot outright. Linux's wait_on_sem() gives up after its own timeout and returns an error instead. Bail out on timeout and report the command buffer head, tail and status along with it, which distinguishes an IOMMU that never consumed the command from one that consumed it without storing the result.
Two paths assumed the command buffer always drains. send_iommu_command() polled for a free slot forever once the ring filled, and flush_command_buffer() waited on a completion word that is never stored. An IOMMU that does not advance its head pointer hangs both. Bound each wait and latch cmd_buffer_dead when the head is still zero, after which invalidation is skipped instead of attempted. Translation still works; caches, if any, do not get invalidated, so this is only sound on an IOMMU that does not cache. Also hoist the find_iommu_for_device() lookup in attach, detach and reattach above get_dma_requestor_id(). All three already return early for a device with no IOMMU, but computing the requestor id first hit BUG_ON(bdf >= ivrs_bdf_entries) before that check, so a device beyond the highest bdf named by the IVRS panicked instead of being skipped.
The device table, command buffer, event log and PPR log base registers are each architecturally 64 bits, holding an address plus a length field in the upper half. Xen programmed them as two 32-bit writes, low half first, so the register transiently holds a base address with a zero length. Linux instead builds the full value and stores it with a single 8-byte memcpy_toio(). An emulated IOMMU that only decodes 8-byte accesses to these registers therefore never sees a valid configuration from Xen. Observed on a Google Cloud instance's virtual AMD-Vi, where the control register -- which Xen does write with writeq() -- takes effect, while the command buffer never starts (CMD_BUFFER_RUN clear, head pinned at 0) and the device table is never consulted.
An IOMMU that stops consuming commands records why in its event log -- ILLEGAL_COMMAND_ERROR or COMMAND_HARDWARE_ERROR name the offending command and address. Report the log pointers and parse any entries when the completion wait times out, rather than leaving the reason unread.
allocate_cmd_buffer() asked allocate_ring_buffer() not to clear, unlike the event and PPR logs beside it, leaving the ring full of whatever was in the page. Linux's allocator forces __GFP_ZERO, so its ring is always zeroed. An IOMMU that reads a slot Xen has not written therefore decodes stale data as a command. Observed on an emulated AMD-Vi reporting ILLEGAL_COMMAND_ERROR against a ring offset well past the tail, after which it stops consuming commands entirely. Also decode that event: report the offending command's ring offset and contents rather than four raw dwords.
iommu_init_domid(DOMID_INVALID) reserves nothing, and iommu_alloc_domid() scans from bit 0, so the first context Xen sets up gets DomainID 0 -- carried in both its device table entry and every invalidation naming it. Linux allocates AMD IOMMU domain IDs from 1 (ida_alloc_range(&pdom_ids, 1, MAX_DOMAIN_ID - 1)) and has long treated 0 as its unallocated/error placeholder, so no Linux guest ever emits a command naming it. An emulated IOMMU written against Linux may reject one: a Google Cloud instance's virtual AMD-Vi answers the first such invalidation with ILLEGAL_COMMAND_ERROR and halts its command processor. Reserve it for AMD only; the shared allocator is also used by VT-d.
Report a mismatch between what is programmed into the device table, command buffer and log base registers and what reads back, which distinguishes an IOMMU that rejects the configuration from one that accepted it and is failing later.
The IOMMU halts at the command it objects to, so with the head pointer still at zero the offending command is the first in the ring. Print the first two entries so the rejected command can be compared against what Linux emits.
An ILLEGAL_COMMAND_ERROR event stores the offending command's address as addr[63:4], so reading the entry as a plain 64-bit value reports it shifted down by four and makes it look like the IOMMU fetched from outside the ring. Shift it back. Also issue a lone completion wait when enabling an IOMMU, before anything else is queued. A ring that completes it but chokes on the first real command is a very different problem from one that never consumes anything, and the two were indistinguishable.
An entry whose TV bit is clear translates nothing, so there is no cached translation for it and the invalidation achieves nothing. It also matters on an emulated AMD-Vi. Linux prefills V and TV together (init_device_table_dma) and so never names a V-only entry in INVALIDATE_DEVTAB_ENTRY; Xen prefills V alone, deliberately, so that unconfigured requests abort. One emulation rejects the command for such an entry with ILLEGAL_COMMAND_ERROR and halts its command processor, which starves every later invalidation -- including the device table updates a guest's assigned device depends on. Skipping it keeps the boot-time flush of a prefilled entry off the ring while still invalidating real entries.
Print the control register, status and device table base once an IOMMU is enabled, and the raw device table entry whenever one is set up. Both are needed to compare a configuration an IOMMU accepts against one it rejects, which is otherwise guesswork.
The control register's InvTimeout field is left at zero, which means the IOMMU waits indefinitely for an invalidation to complete. Linux always programs one second (CTRL_INV_TO_1S), so no Linux guest leaves it clear. On a Google Cloud instance's emulated AMD-Vi this is the sole difference between the control register Xen programs (0x140d) and the one Linux programs for the same IVHD (0x148d), and that IOMMU rejects INVALIDATE_DEVTAB_ENTRY from Xen while accepting the byte-identical command from Linux.
Detection read the control register only to report whether firmware had already enabled the IOMMU, then wrote zero, discarding every other bit. Linux never does this: iommu_feature_enable() and iommu_feature_set() read-modify-write throughout, so bits a driver knows nothing about are carried across its whole init. Such bits exist. A Google Cloud instance's emulated AMD-Vi reports control 0x2000000148d under Linux, where bit 41 corresponds to no CONTROL_* the driver defines and is therefore never written by it -- the IOMMU sets it for itself and Linux leaves it alone. Xen clears it and that IOMMU then answers INVALIDATE_DEVTAB_ENTRY with ILLEGAL_COMMAND_ERROR, while accepting the byte-identical command from Linux. Clear only what Xen goes on to configure.
Xen derives a domain's IOMMU page table depth from the address width and lands on five levels where Linux programs three. An IOMMU that has only ever seen Linux may reject a command naming a deeper entry; an emulated one has been seen to answer INVALIDATE_DEVTAB_ENTRY with ILLEGAL_COMMAND_ERROR and halt its command processor, starving every later invalidation. Add amd-iommu-guest-pt-levels to cap the depth. It applies per context, so the hardware domain's own identity map keeps the depth its address width requires. Skip invalidating an entry deeper than the cap for the same reason.
A rejected command halts the command processor: the head pointer stops and nothing further is consumed. Xen had no recovery, so one bad command at boot starved every invalidation afterwards. Linux restarts the ring from its event handler on ILLEGAL_COMMAND_ERROR, discarding whatever was queued, and carries on. Do the same on a completion wait timeout. Only give up on the ring after several consecutive failures, so a genuinely dead command buffer still cannot cost a timeout per flush, and clear the count once a wait completes. The previous condition for giving up -- head still at zero -- stopped working once a probe command was queued ahead of the failing one, since the head then advances past the probe. Every later flush paid a full timeout with a doubling threshold, which is indistinguishable from a hang.
Add amd-iommu-dump-bdf. For the first few mappings of the named device, report the dfn and mfn and then walk the entry back from the device table the way the IOMMU does. Whether the tables a domain builds actually resolve is otherwise only visible from a fault, and a device that is not being translated at all never raises one.
Xen skips the flush after installing a mapping over a non-present entry, on the grounds that an IOMMU only caches entries it found present. An IOMMU that shadows the page tables rather than walking them never sees such a mapping, and DMA through it fails silently. The extended feature register says whether the hardware caches non-present entries, so read it: flush when it does, and when there is no register to read assume it does, since nothing says otherwise. amd-iommu-flush-on-map overrides the decision either way.
Every mapped page was looked up before being mapped, purely to refuse one that is already mapped, which doubles the page table walks. A guest mapping a large region a page at a time -- which a PV guest must, its machine frames not being contiguous -- pays that for every page, and a 4G mapping then takes longer than its caller is willing to wait. Skip the check by default, keeping it available as pv-iommu-check-conflicts.
azenla
requested review from
alexandermerritt,
bleggett,
kaniini and
tycho
as code owners
September 3, 2026 18:49
map_pages_op() flushes the IOTLB once per call, and a guest issues one call per run of frames contiguous in both dfn and gfn. Those runs are short -- a PV domain's gfns are machine frames, so a scattered buffer breaks into runs of a few pages -- and mapping 4G that way costs on the order of a million flushes. Where the IOMMU is emulated each one is a command and a completion wait to a device that is not there, which swamps the cost of the mappings themselves. Add IOMMU_MAP_no_flush to suppress the per-call flush and IOMMU_flush_pages to perform it over a range, so a caller can map a whole buffer and pay one flush for it. Advertised as IOMMUCAP_deferred_flush; a guest that does not ask for it is unaffected. The flush names both flush flags because it cannot know what the calls it covers did.
Where an IOMMU is emulated, a completion wait can cost far more than the invalidation it confirms, and nothing outside the hypervisor can tell how much of a slow page table teardown is spent inside it. Count the waits and time them, and time do_mmu_update() alongside, on the 'y' key. Four rounds of reasoning from guest stack traces about the cost of PV page table teardown were each wrong, and the first dump of these two numbers falsified all of them: the waits were 66ms of a hundred second stall, and the time inside do_mmu_update() is lower when the teardown is slow than when it is fast. Cheap enough to leave in, and it settles in one line what argument could not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While developing TPU support in PV, we needed a PV-IOMMU. This implements that and adds options and support for GCE's IOMMU intricacies.
This is based on https://patchew.org/Xen/cover.1763569135.git.teddy.astie@vates.tech/ but it has been modified to work in domU, as well as significantly changed for the IOMMU on GCE, which is particularly... particular.
The main goal was to support libtpu-based workloads natively on Edera on top of PV.
The Linux-side patches are viewable in stub-PR form here: edera-dev/linux#1
Some follow-up work that I think should be done is to speed up page table updates on PV. libtpu loves to allocate significant amounts of pages, and the async page table updates cause domU to interrupt frequently. There were some allocation improvements done but the unmapping of pages is a bit wonky still. The added tracing in this PR was added to make that clear.
After some Claude analysis of the numbers, the unmapping latency came down to scheduler noise due to the async page table updates. Interestingly, the IOMMU was not the problem here.