Repository navigation
Release the page cache of rank cores sent to peer ranks - #303
Closed
original-el8 wants to merge 1 commit into
Closed
original-el8 wants to merge 1 commit into
original-el8 wants to merge 1 commit into
Conversation
When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.
Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.
The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.
Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.
Design-Signed-Off-By: flare-02; design-review=ed22
Collaborator
Author
|
Folded into #304 as its own commit, so the Inkling 2 DGX Spark enablement is one review. The commit and its description are unchanged there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
flare runstarts a multi-host core, rank zero reads every peer's rank core through the page cache and sends it over TCP. On a host whose GPU allocates from system memory, such as a DGX Spark (GB10), those cached pages are not free memory for CUDA, so the pages of several rank cores can keep rank zero from creating its CUDA context. Helios already releases the page cache of a rank's own core files before it creates the context (Files::release_page_cache in helios/src/runtime_core/model.rs), and a receiving rank syncs the files it writes; the files rank zero reads on behalf of other ranks were never released. Releasing them where the distribution source stops closes that gap with the same eos operation and needs no change to loading or to the transfer protocol.Before this change the distribution source in flare-cli/src/distribution.rs returned when its serving threads had joined and left the pages it had read in the cache. Now release_served_pages calls flare_eos::release_page_cache (POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one and above when the source thread finishes, whether distribution succeeded or failed. Rank zero's own files are left to the existing release in Helios. flare-cli gains a path dependency on the in-tree eos crate, which the lock file records.
The result is that a rank-zero host returns the memory used to read peer cores before GPU initialization, without a manual page-cache drop.
Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host) serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank zero distributed three rank cores, its host showed 114 GiB of page cache and 2 GiB free, and rank zero failed at "Initialize GPU" with CudaError { operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The failure was intermittent: the first launch of a smaller core went through the same distribution without it. After, with rebuilt binaries on all four hosts, the peers' rank cores removed and no manual cache drop: three rank cores were sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded. The Linux test source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a 4 MiB on-disk file to one peer and checks with mincore that the file has resident pages after the transfer and none after the source finishes. cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30 passed, 0 failed, 2 ignored (they need external fixtures). Not run: other models, two- and three-rank cores, and hosts with discrete GPUs, where the release changes only the page cache.