Skip to content

Release the page cache of rank cores sent to peer ranks - #303

Closed
original-el8 wants to merge 1 commit into
mainfrom
fix/release-distributed-core-page-cache
Closed

original-el8 wants to merge 1 commit into
mainfrom
fix/release-distributed-core-page-cache

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

When flare run starts a multi-host core, rank zero reads every peer's rank core through the page cache and sends it over TCP. On a host whose GPU allocates from system memory, such as a DGX Spark (GB10), those cached pages are not free memory for CUDA, so the pages of several rank cores can keep rank zero from creating its CUDA context. Helios already releases the page cache of a rank's own core files before it creates the context (Files::release_page_cache in helios/src/runtime_core/model.rs), and a receiving rank syncs the files it writes; the files rank zero reads on behalf of other ranks were never released. Releasing them where the distribution source stops closes that gap with the same eos operation and needs no change to loading or to the transfer protocol.

Before this change the distribution source in flare-cli/src/distribution.rs returned when its serving threads had joined and left the pages it had read in the cache. Now release_served_pages calls flare_eos::release_page_cache (POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one and above when the source thread finishes, whether distribution succeeded or failed. Rank zero's own files are left to the existing release in Helios. flare-cli gains a path dependency on the in-tree eos crate, which the lock file records.

The result is that a rank-zero host returns the memory used to read peer cores before GPU initialization, without a manual page-cache drop.

Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host) serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank zero distributed three rank cores, its host showed 114 GiB of page cache and 2 GiB free, and rank zero failed at "Initialize GPU" with CudaError { operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The failure was intermittent: the first launch of a smaller core went through the same distribution without it. After, with rebuilt binaries on all four hosts, the peers' rank cores removed and no manual cache drop: three rank cores were sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded. The Linux test source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a 4 MiB on-disk file to one peer and checks with mincore that the file has resident pages after the transfer and none after the source finishes. cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30 passed, 0 failed, 2 ignored (they need external fixtures). Not run: other models, two- and three-rank cores, and hosts with discrete GPUs, where the release changes only the page cache.

When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.

Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.

The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.

Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.

Design-Signed-Off-By: flare-02; design-review=ed22
@original-el8

Copy link
Copy Markdown
Collaborator Author

Folded into #304 as its own commit, so the Inkling 2 DGX Spark enablement is one review. The commit and its description are unchanged there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant