Skip to content

Fix fork reset mmap source#88

Open
perbu wants to merge 2 commits into
masterfrom
fix-fork-reset-mmap-source
Open

Fix fork reset mmap source#88
perbu wants to merge 2 commits into
masterfrom
fix-fork-reset-mmap-source

Conversation

@perbu

@perbu perbu commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

I ran into an issue with Python 3.14. First attempt at a fix had some potential performance on a throw, that is addressed in the second commit.

The reset_keep_all_work_memory=true fast recycle path in vMemory::fork_reset
restores each dirty bank page with:

page_duplicate((uint64_t*)our_page,
    (const uint64_t*)main_vm.main_memory().safely_at(addr, page_size));

safely_at(virtual_addr) treats the recorded guest-virtual address as a
physical one. That only holds for identity-mapped main memory. Pages backed by
the mmap physical region (MMAP_PHYS_BASE) are not identity-mapped, so for
every recorded page living there the copy sourced an unpopulated main-memory
page — silently writing zeros into the fork's bank page on every reset.

The 2MB-leaf case was doubly wrong: the lambda parameter shadowed the loop
variable, so the copy source was the bank's physical block address rather
than any guest virtual address.

Observed impact

CPython 3.14 places PyInterpreterState on mmap-region pages. A recycled fork
reads interp->obmalloc as NULL on its next turn and faults in obmalloc's
arena_map_get — deterministically, every second call (fresh mint OK →
recycled fork faults → retire/remint repeats). CPython 3.12 survives only
because nothing it reads back post-reset happens to live on mmap-backed pages;
the copy-back corrupts them there too.
page-table walk and from the vCPU (not stale TLB); the master's true page
(PTE → mmap region) holds the pointer while the identity read returns zeros;
and reset_keep_all_work_memory=false makes the failures vanish.

Fix

Commit 1 — resolve each 4K source page through the master's page tables
with readable_page_at, which handles identity-mapped and mmap-backed
physical pages alike and returns the correct 4K segment when the master holds
a 2MB huge page. A 2MB fork leaf is restored one 4K page at a time, covering
masters with either page size underneath. The loop is arch-neutral (the arm64
backend has its own readable_page_at); the PRESENT/VALID flag ifdef follows
the existing pattern in this file.

Commit 2readable_page_at throws when the master's entry is unpresent,
and unpresent-but-cloneable entries are real (anonymous memory the master
mapped but never touched). Before this commit, one such recorded page made
every fork_reset throw mid-loop into the full bank-reset fallback — silently
and permanently demoting that fork off the fast recycle path. The master is
frozen after fork, so an entry unpresent at reset time was unpresent at write
time — exactly the unpresent-CoW case in writable_page_at, where a fresh
fork gets a zeroed page (zero_and_update_entry, no DIRTY bit). Catching per
page and page_memzero-ing the destination reproduces fresh-fork semantics
bit-for-bit; the happy path pays nothing and the outer catch remains the last
resort for genuinely unexpected failures.

perbu and others added 2 commits July 19, 2026 13:47
…e tables

The reset_keep_all_work_memory fast path restored each dirty bank page
with main_memory().safely_at(virtual_addr), treating the recorded
virtual address as physical. That only holds for identity-mapped main
memory: pages backed by the mmap physical region (MMAP_PHYS_BASE) are
not identity-mapped, so every recorded page living there was silently
"restored" from an unpopulated main-memory page — writing zeros into
the fork's bank page. The 2MB-leaf case was doubly wrong: the shadowed
lambda parameter made the copy source the bank's physical block
address rather than any guest virtual address.

CPython 3.14 places PyInterpreterState on mmap-region pages; a
recycled fork then read interp->obmalloc as NULL and faulted in
obmalloc's arena_map_get on its next turn — deterministically, every
second call. 3.12 survived only because nothing it reads back
post-reset happens to live on mmap-backed pages.

Resolve each 4K source page through the master's page tables with
readable_page_at, which handles both identity-mapped and mmap-backed
physical pages, and returns the correct 4K segment when the master
holds a 2MB huge page. A 2MB fork leaf is restored one 4K page at a
time, covering masters with either page size underneath. The loop is
arch-neutral (the arm64 backend has its own readable_page_at); the
PRESENT/VALID flag ifdef follows the existing pattern in this file.

Repro: fast-agent's fa-exec-bench with CPython 3.14 failed exactly
half its calls (fresh mint OK, first recycle faults); with this fix
3.14 runs 400/400 clean at threads=4, 3.12 unchanged, reset p50 still
~15-20us.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
readable_page_at throws when the master's entry is not present, and
unpresent-but-cloneable master entries are real: anonymous memory the
master mapped but never touched keeps an unpresent CoW entry, and a
fork writing to such a page records it in cow_written_pages like any
other. Before this change, one such recorded page made every
fork_reset throw mid-loop and land in the full bank-reset fallback --
silently and permanently demoting that fork off the fast recycle path.

The master is frozen after fork, so an entry unpresent at reset time
was unpresent when the fork wrote, i.e. exactly the unpresent
copy-on-write case in writable_page_at. A fresh fork writing to such a
page goes through zero_and_update_entry (an unpresent entry carries no
DIRTY bit) and observes zeros, so zeroing the recycled fork's page
reproduces fresh-fork semantics bit-for-bit. Catch per page and
page_memzero the destination; the happy path pays nothing, and the
outer catch remains the last resort for genuinely unexpected failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant