Repository navigation
Track FlashInfer main in the preparation dependency instead of a pinned commit - #215
brandonmmusic-max wants to merge 5 commits into
Conversation
…ed commit The preparation dependency named one exact `local-inference-lab/flashinfer` commit in `pyproject.toml` and `uv.lock`; updating FlashInfer required changing the revision in `pyproject.toml` and refreshing `uv.lock` before preparation could use it. `scripts/setup_preparation.py` ran `uv sync --upgrade-package flashinfer-python`, but that command could not move past an exact pin. The documented setup commands used `--locked`, which installed the recorded commit and did not resolve main. The dependency is declared as `flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main`, with `[tool.uv] upgrade-package = ["flashinfer-python"]`. Plain `uv lock`, `uv sync`, and `uv run` resolve main's head commit and rewrite the FlashInfer entry in uv.lock; scripts/setup_preparation.py does the same and rebuilds FlashInfer and Heliopause. `.venv/bin/ignition`, an activated environment, `--frozen`, and `--no-sync` use the last synced commit. `--offline` cannot fetch main, so the reproducible offline form is `uv sync --frozen --offline --extra prepare --extra test` with a populated uv cache. The docs state which commands track main, when to commit the rewritten lock, and how peers confirm the installed commit from direct_url.json. .github/workflows/flashinfer-main.yml runs on every pull request without a path filter, on pushes to main, daily, and by manual dispatch. It reads the exact refs/heads/main hash, runs plain `uv lock`, checks that the requirement is `@main` with `upgrade-package`, fails unless the resolved FlashInfer commit equals that hash, installs that commit in a standalone venv, checks the installed distribution's direct_url.json and __git_commit__, and imports flashinfer, b12x, and the b12x startup-tuning APIs with CUDA_VISIBLE_DEVICES empty and torch.cuda.is_available() false. The setup unit test asserted `uv sync --locked`; the script runs `uv sync --upgrade-package flashinfer-python`. A static declaration test rejects a hand-edited revision; it is a declaration check, not runtime validation. The workflow is the behavior proof. The Muse Spark section's dependency wording states that the commit recorded in uv.lock lacks the required operators and that `uv sync --frozen --extra prepare --extra test` restores the recorded dependency. Muse Spark was not exercised. Resolution and CI validation (uv 0.11.30, Python 3.12.3, no GPU): - `git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main` returned 1f8ed8a3c46e27d03cff54e08f23e9bc50476d57. - In a scratch copy, plain `uv lock` resolved flashinfer-python to that commit with source `?rev=main#1f8ed8a3...`; the resolved commit equaled the remote hash and `uv lock --check` passed. - A lock recording ddbe93fc moved to 1f8ed8a3 under plain `uv lock`, plain `uv sync --extra api-test`, and plain `uv run`; `uv run --no-sync` left the lock unchanged; `uv sync --frozen --extra prepare --extra test --dry-run` reproduced the recorded commit; `uv sync --frozen --offline --extra prepare --extra test --dry-run` did too. Plain `uv sync --offline --extra api-test` failed to fetch main, which is why the offline form uses `--frozen`. - An exact ddbe93fc pin without upgrade-package stayed at ddbe93fc under plain `uv lock`, and the workflow's comparison failed as intended. - A real `uv pip install "flashinfer-python[b12x] @ git+...@1f8ed8a3..."` in a fresh venv built and installed; direct_url.json and flashinfer.__git_commit__ reported 1f8ed8a3; `import flashinfer`, `import b12x`, and the b12x.comm.pcie startup-tuning imports succeeded with CUDA_VISIBLE_DEVICES empty and torch.cuda.is_available() false. The cold install took 4m22s and the venv was 6.4 GB; the warm install took 2s. - `pytest tests/preparation/test_packaging.py tests/preparation/test_setup_preparation.py` reports 8 passed and 1 failed. The failure is test_server_cli_help_does_not_prepare_model, which fails identically on 0009b89 and is unrelated to this change. The static declaration test passes, and the setup test that failed on 0009b89 passes. - The workflow YAML parses with PyYAML, and the extracted run scripts pass `bash -n`. actionlint was not available. - End-to-end check of `83c49b02`, which differs from this change only in comments and documentation: on one RTX PRO 6000 (SM120), documented `uv sync --extra prepare --extra test` installed FlashInfer `1f8ed8a3c46e27d03cff54e08f23e9bc50476d57`; Qwen3.8-27B NVFP4 was exported (rc 0) and served with `flare run`; the functional checks passed 6/6; the benchmark script returned rc 0 with 0 device errors; the NVIDIA#116 reproduction had no failure in 143 prompts and 0 device errors. Limitations: - The CI job uses FlashInfer's own resolved dependencies, not Flare's pinned torch `2.14.1+cu132`. It sets FLASHINFER_BUILD_NO_PIP=1 and BUILD_NVEP=0, matching scripts/setup_preparation.py, and does not build Heliopause or run GPU preparation. It proves the FlashInfer commit builds, installs, and imports, not the complete preparation environment. - The end-to-end check covers one model on one RTX PRO 6000 at the FlashInfer commit that was head at build time; it does not establish GPU correctness for every later FlashInfer main commit. - The CI install is about 6.4 GB and took 4m22s cold locally. The FlashInfer main job passed in 3m28s. A hosted runner that runs out of disk or time fails the job closed. - Existing exported packages keep their prepared kernels; a FlashInfer kernel change requires re-export. The Rust flare binary does not link FlashInfer and does not need rebuilding for a FlashInfer commit change. - Plain uv commands rewrite uv.lock. Commit the rewritten FlashInfer entry only with a change that needs that commit. A lock that records a working commit, with `uv sync --frozen`, reproduces that environment. - Offline preparation was validated with `--dry-run` only; a full offline prepare install needs a populated uv cache. - A FlashInfer main change can break Flare. The workflow catches resolution, build, install, and import failures, not GPU kernel correctness. - The Muse Spark section's dependency wording states that the commit recorded in uv.lock lacks the required operators and that `uv sync --frozen --extra prepare --extra test` restores the recorded dependency. Muse Spark was not exercised. - The `Native serving` job on windows-2022 fails eight hub and load-package Rust tests and reports 149 passed and 8 failed. The same job on NVIDIA main at 0009b89 fails the same eight tests, and the two lists are identical. At base 78b18b1 the same eight tests fail. This change touches no Rust and does not address them. Signed-off-by: Brandon M. Music <266531604+brandonmmusic-max@users.noreply.github.com> Design-Signed-Off-By: Brandon M. Music; design-review=a38f
|
This changes preparation's dependency policy: ordinary The added CI checks dependency resolution, standalone installation, and imports. It does not qualify Flare's complete preparation environment or GPU behavior against each selected commit; the reported SM120 export qualifies the recorded combination only. The decision should cover reproducible bug reports, using one lock across TP peers, and recovery from an incompatible dependency update. This PR is distinct from an ordinary pin update and does not by itself resolve reports requiring particular kernel fixes and re-export. |
|
Thanks. On the three points:
Agreed that CI proves resolution, build, install and imports only, and that kernel fixes still need a re-export. If a pinned default is preferred, I can keep the pin and put main tracking behind an explicit command instead. |
|
@brandonmmusic-max GitHub reports merge conflicts with
Please reconcile these files with main and rerun the checks affected by the resolution. |
I’m finally past the Kiddo’s birthday party this weekend. I’ll get this done tonight |
Commit 3ce4caf (NVIDIA#215) makes the preparation dependency follow the head commit of local-inference-lab/flashinfer main instead of a pinned commit. Main at d2b5c5f conflicts with it in six files: docs/cli-design.md, docs/model-preparation.md, ignition/patches/__init__.py, pyproject.toml, tests/preparation/test_setup_preparation.py, and uv.lock. Three main changes cause the conflicts: `ignition.runtime_patches` replaces the source patches in `ignition.patches`, scripts/setup_preparation.py no longer runs `python -m ignition.patches` after its sync, and the FlashInfer requirement is pinned at e0f105b3. The resolution keeps main's runtime-patch design and setup script and keeps 3ce4caf's dependency policy, so the pull request applies to main without reverting either. Resolution, by file, from main's version to the merged version: - pyproject.toml: main's exact `@e0f105b3...` requirement is replaced by `flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main` with `[tool.uv] upgrade-package = ["flashinfer-python"]`, as in 3ce4caf. - uv.lock: main's lock, re-resolved with plain `uv lock`. Two lines differ: the flashinfer-python source changes from `?rev=e0f105b3...#e0f105b3...` to `?rev=main#ea74ea610ce89d4499ad4b1ecb6c9b525fc2a795`, the head of FlashInfer main at resolution time, and the project's recorded FlashInfer requirement changes from `?rev=e0f105b3...` to `?rev=main`. - ignition/patches/__init__.py: main's deletion stands. 3ce4caf changed only that module's error messages, from `uv sync --locked` to `uv sync`; the `ignition.runtime_patches` error messages name no uv command. - tests/preparation/test_setup_preparation.py: main's tests for the single-call setup script are kept. Main's `test_setup_updates_preparation_dependencies` checked the first four command elements; the merged test asserts the complete command that main's unchanged scripts/setup_preparation.py runs and asserts that the command contains neither `--locked` nor `--frozen`, as 3ce4caf's test did. - docs/cli-design.md: 3ce4caf's install command without `--locked` and its FlashInfer paragraph, with main's runtime-patch sentence; that sentence's example `uv run --locked --extra prepare ignition ...` becomes `uv run --extra prepare ignition ...`. - docs/model-preparation.md: main's `ignition.runtime_patches` paragraphs and `runtime_patches.apply()` example, with 3ce4caf's FlashInfer paragraph, FlashInfer dependency section, and Muse Spark dependency wording. Main's setup-script paragraph says the script resolves the revision declared in pyproject.toml; the merged paragraph says it updates FlashInfer to the head commit of main, rebuilds FlashInfer and Heliopause, and rewrites the FlashInfer entry in uv.lock. Result: the pull request branch contains main at d2b5c5f and has no conflicts with it. The merged tree differs from d2b5c5f only in the ten files that 3ce4caf changes: .github/workflows/flashinfer-main.yml, AGENTS.md, docs/api.md, docs/cli-design.md, docs/model-preparation.md, docs/roce-ring.md, pyproject.toml, tests/preparation/test_packaging.py, tests/preparation/test_setup_preparation.py, and uv.lock. Of the files in 3ce4caf's diff against its base 0009b89, only ignition/patches/__init__.py is no longer changed. The locked FlashInfer commit ea74ea61 contains main's pinned fix: FlashInfer main holds e0f105b3's change ("Preserve NaN signs in coarse radix selection") as 07132dcab (local-inference-lab/flashinfer#82), whose tree is identical to e0f105b3's. ea74ea61 adds three commits after 07132dcab: c08707c2b (Kimi K3.1 MoE benchmarks), 47946d885 (small-batch native MXFP4 MoE execution), and ea74ea610 (CSF scale expansion fused with small-route preparation). Validation (uv 0.11.30, Python 3.12.3, no GPU): - `git diff d2b5c5f` on the merged tree names exactly the ten files above; `git diff 0009b89 3ce4caf` names the same ten and ignition/patches/__init__.py. - In a FlashInfer clone, `git diff e0f105b3 07132dcab` is empty, 07132dcab is an ancestor of ea74ea61, and `git rev-list 07132dcab..ea74ea61` lists only c08707c2b, 47946d885, and ea74ea610. - `git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main` returned ea74ea610ce89d4499ad4b1ecb6c9b525fc2a795. Plain `uv lock` recorded that commit, `uv lock --check` exited 0, and the declaration and lock assertions from the FlashInfer main workflow passed with the locked commit equal to that hash. - The workflow's install-and-import step, run locally in a fresh Python 3.12 venv with FLASHINFER_BUILD_NO_PIP=1 and BUILD_NVEP=0, installed FlashInfer 0.7.3 at ea74ea61. direct_url.json and flashinfer.__git_commit__ reported that commit, and flashinfer, b12x, and the b12x.comm.pcie startup-tuning APIs imported with CUDA_VISIBLE_DEVICES empty and torch.cuda.is_available() false. The local run used PyPI plus the PyTorch cu132 index and resolved torch 2.15.0+cu132; the workflow uses PyPI only. - `pytest tests/preparation/test_setup_preparation.py tests/preparation/test_packaging.py` reports 6 passed and 1 failed. The failure, test_server_cli_help_does_not_prepare_model, fails identically on main d2b5c5f: `python -m ignition --help` prints `debug-serve` on a line by itself with its description on the following line, so the expected text "\n debug-serve " is absent. The merge does not change the CLI. - `git diff --check` reports no whitespace errors. No `ignition.patches` reference remains, and no uv command uses `--locked`; the remaining `--locked` flags are Cargo's. Limitations: - No GPU run covers this merge or FlashInfer ea74ea61, including the three commits after 07132dcab. GitHub Actions has not run on this merge commit. - The end-to-end GPU check recorded in 3ce4caf used FlashInfer 1f8ed8a3, which FlashInfer main's history does not contain: GitHub's comparison of ea74ea61 with 1f8ed8a3 reports diverged histories, with 2,408 commits only in ea74ea61 and 247 only in 1f8ed8a3. That check does not carry over to the commit this lock records. Main's pin e0f105b3 branches from FlashInfer main's history: its parent, 07c41f329, is a FlashInfer main commit. GitHub still serves 1f8ed8a3 by hash; a lock that records a commit no longer on main depends on GitHub retaining it. Signed-off-by: Brandon M. Music <266531604+brandonmmusic-max@users.noreply.github.com> Design-Signed-Off-By: Claude Opus 5.5; design-review=3ad6
The FlashInfer main workflow failed on 36fd8d1 because the hosted runner ran out of disk in the install-and-import step: extracting torch 2.15.0 into the uv cache stopped with "No space left on device" (GitHub Actions run 42553335278 in brandonmmusic-max/flare-review). FlashInfer main at ea74ea61 has a 1.01 GB source tree in 19,374 files; at 1f8ed8a3, where the same job passed in 3m28s, the tree was 0.45 GB in 9,281 files. The lock step fetched FlashInfer into the runner's default uv cache, and the install step fetched it again into a second cache before installing an environment that measures 6.9 GB locally. Removing toolchains the job does not use and fetching FlashInfer once frees space without changing what the job installs or checks. The workflow gains a step, after Python setup, that prints the root filesystem's free space, removes the runner's preinstalled Android SDK, .NET SDK, GHC, and CodeQL bundle, prints the free space again, and sets UV_CACHE_DIR to one cache under the runner's temporary directory for every later step. The install step no longer sets its own cache directory and prints the free space after the import check. Resolution, installation, and every assertion are unchanged. Expected result: FlashInfer is fetched once, and the install completes within the runner's disk. This is an expectation; the free-space lines in the job log measure the headroom. Validation: PyYAML parses the workflow, and bash -n accepts every run script. The workflow has not run on this commit. Signed-off-by: Brandon M. Music <266531604+brandonmmusic-max@users.noreply.github.com> Design-Signed-Off-By: Claude Opus 5.5; design-review=3ad6
|
Merged main at d2b5c5f, which contains 91cfb36, in 36fd8d1. Main replaced
At ea74ea61, the FlashInfer main workflow ran out of disk on the hosted runner (run 42553335278 in brandonmmusic-max/flare-review, on 36fd8d1): the FlashInfer source tree is 1.01 GB, against 0.45 GB at 1f8ed8a3, and the lock and install steps each fetched it into a separate uv cache. a79c43c removes the runner's unused Android, .NET, GHC, and CodeQL toolchains first and points every uv command at one cache. Resolution, installation, and the assertions are unchanged. Checks rerun:
No GPU run covers ea74ea61. The end-to-end check in the description used FlashInfer 1f8ed8a3, which FlashInfer main's history no longer contains, so that result does not cover the commit this lock records. The description is updated with these results and that limitation. |
|
@brandonmmusic-max GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
Why the merge is needed and why it is resolved this way The branch replaces the exact FlashInfer commit in the preparation dependency with the `main` requirement and `[tool.uv] upgrade-package`. Main advanced from d2b5c5f to 7006928, 80 commits, and changed the same six paths: the FlashInfer pin from e0f105b3576611b26145ed15f3080d47a7c3bb44 to aa6c4221586f7607c2bca64abbcdc0699845cf3b, `protobuf>=7.36.2,<8`, two test renames and one key in `tests/preparation/test_packaging.py`, and rewrites of `AGENTS.md`, `docs/model-preparation.md` and `docs/roce-ring.md`. GitHub reports the branch as conflicting, so it cannot merge until the six paths are resolved. Main's text and build-path changes are kept, and the branch's dependency change is applied on top. Main's pin is the behavior the branch exists to remove, so keeping it would leave the pull request with no effect on the dependency. The resolution is content-based in each path, not a union of both sides. What exactly changed in each conflicting path - `AGENTS.md`: main's `cargo build --release --manifest-path flare-cli/Cargo.toml` is kept, and main's `uv sync --locked` is replaced by plain `uv sync`. The paragraph on which commands resolve main and on committing the rewritten lock is kept. - `docs/model-preparation.md`: main's environment-setup text and `flare-cli` build path are kept, and `--locked` is replaced by plain `uv sync`. Main's paragraph describing the pin is replaced by the branch's "FlashInfer dependency" section, which defines the commands that track main and those that reproduce a recorded commit. The branch's setup-script paragraph is kept. Main removed the Muse Spark paragraph that the branch had edited, so it stays removed. - `docs/roce-ring.md`: main's `flare-cli` build path is kept, with the branch's first-host and peer commands and the instruction to copy `uv.lock` and sync with `--frozen`. - `pyproject.toml`: main's `protobuf>=7.36.2,<8` is kept, main's `aa6c4221` pin is replaced by `@main`, and the branch's `[tool.uv] upgrade-package` is kept. - `tests/preparation/test_packaging.py`: main's renamed `test_core_import_has_no_runtime_or_cuda_side_effects` and `cores` key are kept, and the branch's `test_flashinfer_declaration_tracks_main_statically` is added. - `uv.lock`: main's lock outside the FlashInfer entries, with the FlashInfer source changed to `?rev=main#aa6c4221586f7607c2bca64abbcdc0699845cf3b`. Two paths outside the six carried statements that the branch makes false. Main's `docs/core-design.md` said `pyproject.toml` pins FlashInfer to a commit; the sentence now states that `pyproject.toml` tracks main and `uv.lock` records the resolved commit. Main's `docs/TODO.txt` said "The pinned FlashInfer proxy" and "prepared from the pin"; both now refer to the proxy at the commit recorded in `uv.lock`. Result Every main change is kept. The preparation dependency is `flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main` with `[tool.uv] upgrade-package = ["flashinfer-python"]`, and `uv.lock` records aa6c4221586f7607c2bca64abbcdc0699845cf3b, the head of local-inference-lab/flashinfer main at merge time. No exact FlashInfer pin remains in `pyproject.toml`, and `uv.lock` equals main's lock outside the FlashInfer entries. Evidence - `git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main` returned aa6c4221586f7607c2bca64abbcdc0699845cf3b. - Plain `uv lock` resolved 104 packages and updated only the FlashInfer entries, from ea74ea610ce89d4499ad4b1ecb6c9b525fc2a795 to aa6c4221586f7607c2bca64abbcdc0699845cf3b. - `uv lock --check` exited 0. - The declaration and lock assertions in `.github/workflows/flashinfer-main.yml`, run locally, pass with the locked commit equal to aa6c4221586f7607c2bca64abbcdc0699845cf3b. - `pytest tests/preparation/test_packaging.py tests/preparation/test_setup_preparation.py` on the branch reports 2 failed and 5 passed. On main at 7006928 it reports 2 failed and 4 passed. The two failures are the same on both: `test_pyproject_declares_preparation_entrypoint_and_cores` raises `KeyError: 'cores'` because main's `pyproject.toml` defines `[tool.setuptools.packages.find]`, and `test_server_cli_help_does_not_prepare_model` expects `"\n debug-serve "` while the help prints `debug-serve` on its own line. The resolution adds no failure. The checks do not cover GPU execution. The merged tree has not run in GitHub Actions. The workflow's install-and-import step was not rerun at aa6c4221; the last run used ea74ea61 on a79c43c. The full preparation environment with torch 2.14.1+cu132 and Heliopause was not built, and native serving tests were not run. Signed-off-by: Brandon M. Music <266531604+brandonmmusic-max@users.noreply.github.com> Design-Signed-Off-By: Claude Fable 5.2; design-review=ed22
|
Merged main at
Checks rerun:
GitHub Actions in |
Why the merge is needed and why it is resolved this way The branch replaces the exact FlashInfer commit in the preparation dependency with the `main` requirement and `[tool.uv] upgrade-package`. Main advanced from 7006928, the commit the branch last merged, to 5a4e867, 13 commits. In `pyproject.toml` and `uv.lock` main moved two preparation requirements: Transformers from Git commit 1d9654e78a23fb167ef539cbd699dfbd3e4830cf to the release `transformers[torch,vision,chat_template]==5.23.0`, and the FlashInfer pin from aa6c4221586f7607c2bca64abbcdc0699845cf3b to 4fd836dcda0998b875129aad76774ca073df10c8. GitHub reports the branch as conflicting in those two paths, so it cannot merge until they are resolved. Main's requirement changes are kept, and the branch's dependency change is applied on top. Main's FlashInfer pin is the behavior the branch exists to remove, so keeping it would leave the pull request with no effect on the dependency. What exactly changed in each conflicting path - `pyproject.toml`: main's `transformers[torch,vision,chat_template]==5.23.0` is kept, main's 4fd836dc pin is replaced by `@main`, and the branch's `[tool.uv] upgrade-package = ["flashinfer-python"]` is kept. - `uv.lock`: main's lock, with only the FlashInfer source changed from `?rev=4fd836dcda0998b875129aad76774ca073df10c8#4fd836dcda0998b875129aad76774ca073df10c8` to `?rev=main#4fd836dcda0998b875129aad76774ca073df10c8`. The head of local-inference-lab/flashinfer main at merge time is the same commit main pins, so the locked FlashInfer code equals main's. One path outside the two carried a statement that the branch makes false. Main added an Inkling 2 Small section to `docs/model-preparation.md` that says "The pinned FlashInfer dependency provides relative bias, private paged-attention tails, and explicit expert activation selection." With the branch the requirement is not pinned, so the sentence now begins "The FlashInfer commit recorded in `uv.lock` provides". The other paths that both sides changed (`docs/TODO.txt`, `docs/cli-design.md`, `docs/core-design.md` and the rest of `docs/model-preparation.md`) merged without conflict. Result Every main change is kept. The preparation dependency is `flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main` with `[tool.uv] upgrade-package = ["flashinfer-python"]`, and `uv.lock` records 4fd836dcda0998b875129aad76774ca073df10c8. No exact FlashInfer pin remains in `pyproject.toml`, and `uv.lock` equals main's lock outside the FlashInfer entries. Evidence - `git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main` returned 4fd836dcda0998b875129aad76774ca073df10c8. - `uv lock --upgrade-package flashinfer-python`, run on main's lock with the resolved `pyproject.toml`, resolved 104 packages and changed only the two FlashInfer source lines. A following plain `uv lock` left the lock byte-identical, and `uv lock --check` exited 0. - The declaration and lock assertions of `.github/workflows/flashinfer-main.yml`, run locally in the merged tree, pass with the locked commit equal to 4fd836dcda0998b875129aad76774ca073df10c8. - `pytest tests/preparation/test_packaging.py tests/preparation/test_setup_preparation.py` reports 2 failed and 5 passed on the branch and 2 failed and 4 passed on main at 5a4e867. The two failures are the same on both: `test_pyproject_declares_preparation_entrypoint_and_cores` raises `KeyError: 'cores'` because main's `pyproject.toml` defines `[tool.setuptools.packages.find]`, and `test_server_cli_help_does_not_prepare_model` expects `"\n debug-serve "` while the help prints `debug-serve` on its own line. The resolution adds no failure. The checks do not cover GPU execution. The merged tree has not run in GitHub Actions. The workflow's install-and-import step was not rerun locally at 4fd836dc. The full preparation environment with torch 2.14.1+cu132 and Heliopause was not built, and native serving tests were not run. Signed-off-by: Brandon M. Music <266531604+brandonmmusic-max@users.noreply.github.com> Design-Signed-Off-By: Claude Opus 5.5; design-review=ed22
The preparation dependency named one exact
local-inference-lab/flashinfercommit in
pyproject.tomlanduv.lock; updating FlashInfer required changingthe revision in
pyproject.tomland refreshinguv.lockbefore preparationcould use it.
scripts/setup_preparation.pyranuv sync --upgrade-package flashinfer-python, but that command could not movepast an exact pin. The documented setup commands used
--locked, whichinstalled the recorded commit and did not resolve main.
The dependency is declared as
flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main,and
[tool.uv] upgrade-package = ["flashinfer-python"]makes plainuv lock,uv sync, anduv runresolve main's head commit and record it inuv.lock..github/workflows/flashinfer-main.ymlproves that resolution, the build, theinstall, and the imports on a GPU-less runner. No person or bot edits a pin. The
Rust Flare binary does not link FlashInfer, so a FlashInfer commit change does
not require rebuilding it; a kernel change requires a re-export.
What changed, path by path
uv sync --extra prepare --extra test)pyproject.tomlanduv.lock.uv.lock.uv syncoruv runuv.lock.scripts/setup_preparation.pyuv sync --upgrade-package flashinfer-pythoncould not move past the exact revision.uv.lock, and rebuilds FlashInfer and Heliopause..venv/bin/ignitionor an activatedignitionuv syncorscripts/setup_preparation.pyfirst when a preparation must use main's head.uv sync --frozenuv sync --frozen --offlineuv sync --offlinewithout--frozenmainand fails when the network is disabled. It is not a tracking path.uv.lock; peers copy that file and runuv sync --frozen. Every host can confirm its installed commit fromdirect_url.json..github/workflows/flashinfer-main.ymlruns on every pull request without a path filter, on pushes to main, daily, and by manual dispatch. It first removes the runner's preinstalled Android, .NET, GHC, and CodeQL toolchains, which the job does not use, and points every uv command at one cache, so the FlashInfer checkout (about 1 GB of source atea74ea61) is fetched once. It then reads the exactrefs/heads/mainhash, runs plainuv lock, checks that the requirement is@mainwithupgrade-package, fails unless the resolved commit equals that hash, installs that commit in a standalone venv, checksdirect_url.jsonandflashinfer.__git_commit__, and importsflashinfer,b12x, and theb12x.comm.pciestartup-tuning APIs with no visible GPU.flare runwith an exported packageThe docs state which commands track main (
uv lock,uv sync,uv run,scripts/setup_preparation.py) and which use the last synced commit(
.venv/bin/ignition, an activated environment,--frozen,--no-sync). Theyalso state that plain uv commands rewrite
uv.lock, that the rewrittenFlashInfer entry should be committed only with a change that needs that commit,
and that
--offlinemust be paired with--frozen. Main removed the Muse Sparkparagraph that stated the recorded dependency lacked the required operators, so
the merged documentation makes no such statement. Muse Spark itself was not
exercised.
What this does not do
.venv/bin/ignitionor an activated environment re-resolvemain. Those paths use the environment that was last synced. The update step
is
uv syncorscripts/setup_preparation.py.flare runor an exported package update. Serving uses thegraphs and kernels captured at export.
installs FlashInfer with FlashInfer's own resolved dependencies. It proves
the FlashInfer commit builds, installs, and imports, not that the complete
CUDA 13.2, torch 2.14.1, Heliopause preparation environment builds.
correctness. The workflow catches resolution, build, install, and import
failures.
at
5a4e86795459has.prepared-graph boundary.
Validation
Environment for the resolution and CI checks: uv 0.11.30, Python 3.12.3,
Linux, no GPU work.
On this branch merged with main at
5a4e86795459, with FlashInfer main at4fd836dcda0998b875129aad76774ca073df10c8:git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/mainreturned
4fd836dcda0998b875129aad76774ca073df10c8, the same commit mainpins.
uv lock --upgrade-package flashinfer-pythonon main's lock recordedthat commit and changed only the FlashInfer source lines, a following plain
uv lockleft the lock byte-identical,uv lock --checkexited 0, and theworkflow's declaration and lock assertions passed with the locked commit
equal to that hash.
venv with
FLASHINFER_BUILD_NO_PIP=1andBUILD_NVEP=0, installed FlashInfer0.7.3 at
ea74ea61.direct_url.jsonandflashinfer.__git_commit__reported that commit, and
flashinfer,b12x, and theb12x.comm.pciestartup-tuning APIs imported with
CUDA_VISIBLE_DEVICESempty andtorch.cuda.is_available()false. The local run used PyPI plus the PyTorchcu132 index and resolved torch 2.15.0+cu132; the workflow uses PyPI only. That
run used
ea74ea61; the local install-and-import step was not rerun ataa6c4221or4fd836dc; the GitHub Actions runs on5eff6460and93798dca, described below, installed and imported FlashInfer ataa6c4221and
4fd836dc.pytest tests/preparation/test_setup_preparation.py tests/preparation/test_packaging.pyreports 5 passed and 2 failed. The static declaration test
(
test_flashinfer_declaration_tracks_main_statically) passes; it is adeclaration check, and the workflow is the behavior proof. The two failures,
test_pyproject_declares_preparation_entrypoint_and_coresandtest_server_cli_help_does_not_prepare_model, fail identically on main at5a4e86795459. The first reads[tool.setuptools.cores], which main'spyproject.tomldoes not define (it defines[tool.setuptools.packages.find]),so it raises
KeyError: 'cores'. In the second,python -m ignition --helpprints
debug-serveon a line by itself, so the expected"\n debug-serve "text is absent.93798dcainbrandonmmusic-max/flare-review, the mergecommit of main
5a4e86795459into5eff6460: the FlashInfer main job passedin 6m33s (run
43730555592). Plainuv lockresolved 104 packages; FlashInfermain and the locked commit were both
4fd836dcda0998b875129aad76774ca073df10c8; and the install-and-import stepinstalled FlashInfer 0.7.3 at that commit and passed every assertion. In
Native serving (run
43730555565), the rdma job passed. The Ubuntu job failedrecord::tests::persisted_header_matches_the_descriptor_layoutin the step"Test Selene and its manager on Linux" (124 passed, 1 failed), and
orbit::agent::tests::groups_run_together_on_disjoint_gpus_and_status_lists_themand
orbit::agent::tests::running_groups_are_stopped_and_outlive_their_controller_sessionin the step "Test native CLI" (179 passed, 2 failed). The Windows job failed
the step "Test host OS helpers", where the
flare-eosunit-test binary exitedwith
STATUS_DLL_NOT_FOUND(0xc0000135), and sevencache_commandstestsin the step "Test native CLI" (0 passed, 7 failed). Native serving on main
at
5a4e86795459(run43732636120inbrandonmmusic-max/flare-review,started on that commit by manual dispatch) fails the same jobs at the same
steps with the same ten tests and the same
STATUS_DLL_NOT_FOUNDexit, andits rdma job passes. The
branch differs from main
5a4e86795459in the same twelve files listedbelow, none of them Rust source.
5eff6460inbrandonmmusic-max/flare-review, the mergecommit of main
70069281cb3aintoa79c43cc: the FlashInfer main job passedin 7m04s (run
43603519723). Plainuv lockresolved 104 packages; FlashInfermain and the locked commit were both
aa6c4221586f7607c2bca64abbcdc0699845cf3b; and the install-and-import stepinstalled FlashInfer 0.7.3 at that commit and passed every assertion. In
Native serving (run
43603519699), the rdma job passed; the Windows job failedthe step "Build native serving" with
error[E0432]: unresolved import super::Accessathelios/src/runtime_core/host.rs:45:21; and the Ubuntu job failed threetests:
record::tests::persisted_header_matches_the_descriptor_layoutin thestep "Test Selene and its manager on Linux" (124 passed, 1 failed), and
orbit::agent::tests::groups_run_together_on_disjoint_gpus_and_status_lists_themand
orbit::agent::tests::running_groups_are_stopped_and_outlive_their_controller_sessionin the step "Test native CLI" (179 passed, 2 failed). Native serving on main
at
70069281cb3a(run43551918627in NVIDIA/hyperflare-inference) fails thesame Windows compile error at the same file and line and the same three Ubuntu
tests with the same pass counts, and its rdma job passes. The branch differs
from main
70069281cb3ain twelve files, none of them Rust source:.github/workflows/flashinfer-main.yml,AGENTS.md,docs/TODO.txt,docs/api.md,docs/cli-design.md,docs/core-design.md,docs/model-preparation.md,docs/roce-ring.md,pyproject.toml,tests/preparation/test_packaging.py,tests/preparation/test_setup_preparation.py, anduv.lock.a79c43ccinbrandonmmusic-max/flare-review, which addsthe workflow's disk and cache step to
36fd8d19: the FlashInfer main jobpassed in 5m16s (run
42553853087). It lockedea74ea61, installedFlashInfer 0.7.3 at that commit, and passed every assertion. That run does not
include main's
70069281cb3a. In Native serving (run42553853071), the rdmajob passed, the Ubuntu job failed one test
(
legacy_templates_receive_image_blocks_and_return_original_sources), and theWindows job failed eight hub and load-package tests with 149 passed. Native
serving on main at
d2b5c5f3(run42544646868inNVIDIA/spark-local-inference) fails the same Ubuntu test and the same eight
Windows tests, at the same steps, and its rdma job passes.
On commits
3ce4caf3and83c49b02, which differ only in comments anddocumentation, without main's
d2b5c5f3changes:git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/mainreturned
1f8ed8a3c46e27d03cff54e08f23e9bc50476d57.uv lockresolvedflashinfer-pythonto that commit with source
?rev=main#1f8ed8a3.... The resolved commitequaled the remote hash, and
uv lock --checkpassed.ddbe93fcmoved to1f8ed8a3under plainuv lock, plainuv sync --extra api-test, and plainuv run.uv run --no-syncleft thelock unchanged.
uv sync --frozen --extra prepare --extra test --dry-runreproduced the recorded commit, and
uv sync --frozen --offline --extra prepare --extra test --dry-rundid too. Plainuv sync --offline --extra api-testfailed to fetchmain, which is why the offline form uses--frozen.ddbe93fcpin withoutupgrade-packagestayed atddbe93fcunderplain
uv lock, and the workflow's comparison failed as intended. Theworkflow does not pass with a stale exact pin.
uv pip install "flashinfer-python[b12x] @ git+...@1f8ed8a3..."in afresh venv built and installed.
direct_url.jsonandflashinfer.__git_commit__reported1f8ed8a3.import flashinfer,import b12x, and theb12x.comm.pciestartup-tuning imports succeeded withCUDA_VISIBLE_DEVICESempty andtorch.cuda.is_available()false. The coldinstall took 4m22s and the venv was 6.4 GB. A warm install took 2s.
runscripts passbash -n.actionlintwas not available.83c49b02. On one RTX PRO 6000 (SM120), the tree wasbuilt with the documented
uv sync --extra prepare --extra test. Thatinstalled FlashInfer
1f8ed8a3c46e27d03cff54e08f23e9bc50476d57, the head oflocal-inference-lab/flashinfer main at build time, with no pin edited.
Qwen3.8-27B NVFP4 with the DFlash2 drafter was exported (rc 0) and served
with
flare run. A prefill-only benchmark cell on a fresh server returnedrc 0 with 0 device errors. The functional checks passed 6/6. The benchmark
script (decode and three prefill cells) returned rc 0 with 0 device errors.
The Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure) #116 reproduction at
--memory-limit 50Ghadno failure in 143 prompts and 0 device errors. This exercises the documented
build path once, at the main commit that was head at build time. A build of
this tree with plain
uv sync --extra prepare --extra testresolvedlocal-inference-lab/flashinfermain at42b221db437e80fc66e41f5e630da88aefc387e7, rewrote theuv.lockFlashInferentry from
1f8ed8a3c46e27d03cff54e08f23e9bc50476d57to that commit, builtFlashInfer, and installed it; the distribution's
direct_url.jsonreported42b221db437e80fc66e41f5e630da88aefc387e7. No pin was edited.Limitations
Flare's pinned torch
2.14.1+cu132. It setsFLASHINFER_BUILD_NO_PIP=1andBUILD_NVEP=0, matchingscripts/setup_preparation.py, so the build hookdoes not run nested installs or build the NIXL and NCCL EP backends. It does
not build Heliopause or run GPU preparation. The full
uv sync --extra prepare --extra testwas not run in CI. It was run once on one RTX PRO 6000for the end-to-end check above.
ea74ea61the standalone installmeasures 6.9 GB locally, and the FlashInfer source tree is 1.01 GB in 19,374
files (0.45 GB in 9,281 files at
1f8ed8a3). Without the disk step, thehosted runner ran out of space extracting torch (GitHub Actions run
42553335278inbrandonmmusic-max/flare-review, on36fd8d19). With thedisk step, the runner had 14 GB free before the removal and 32 GB after it,
the job used about 12 GB, and it passed in 5m16s (run
42553853087, commita79c43cc). At1f8ed8a3the install was about 6.4 GB and took 4m22s cold locally, andthe job passed in 3m28s (run
42056263565, commit83c49b02). On the mergecommit
5eff6460, with FlashInferaa6c4221, the runner had 13 GB freebefore the disk step and 32 GB after it, and the job passed in 7m04s (run
43603519723). On the merge commit93798dca, with FlashInfer4fd836dc,the runner had 13 GB free before the disk step, 32 GB after it and 20 GB
after the install, and the job passed in 6m33s (run
43730555592). Theinstall size at
aa6c4221and4fd836dcwas not measured. If the hostedrunner runs out of disk or time, the job fails closed.
kernel change requires an export. The Rust
flarebinary does not linkFlashInfer, so it does not need rebuilding for a FlashInfer commit change.
uv.lock. Commit the rewritten entry only with a change that needs thatFlashInfer commit. A
uv.lockthat records a commit, withuv sync --frozen,reproduces the environment it recorded while that commit remains fetchable
(see FlashInfer main history below).
--dry-runonly. A fulloffline prepare install needs a populated uv cache and was not run.
b12x reaches the next build. The workflow catches resolution, build, install,
and import failures, not GPU kernel correctness. A lock that records a
working commit, with
--frozen, reproduces that environment while thatcommit remains fetchable.
main at
4fd836dccontains neither1f8ed8a3, the commit the end-to-endcheck used, nor
42b221db, the commit the later build resolved.4fd836dcdescends from
aa6c4221andea74ea61, and GitHub's comparison ofea74ea61with1f8ed8a3reports diverged histories, with 2,408 commits only inea74ea61and 247 only in
1f8ed8a3. GitHub still serves1f8ed8a3by hash.Reproducing a report from a
uv.lockthat records a commit later removed frommain depends on GitHub retaining that commit.
recorded in
uv.locklacks the required operators and thatuv sync --frozen --extra prepare --extra testrestores the recordeddependency. The merged documentation makes no such statement. Muse Spark
itself was not exercised.
Native servingworkflow fails on main itself. On mainat
d2b5c5f3(run42544646868), the Ubuntu job failslegacy_templates_receive_image_blocks_and_return_original_sourcesand theWindows job fails eight hub and load-package Rust tests. On
a79c43cc(run42553853071) the two jobs fail exactly the same tests at the same steps,and the rdma job passes on both. On main at
70069281cb3a(run43551918627in NVIDIA/hyperflare-inference), the rdma job passes, theWindows job fails the step "Build native serving" with
error[E0432]: unresolved import super::Accessathelios/src/runtime_core/host.rs:45:21, and the Ubuntu job fails threetests:
record::tests::persisted_header_matches_the_descriptor_layoutin thestep "Test Selene and its manager on Linux" (124 passed, 1 failed), and
orbit::agent::tests::groups_run_together_on_disjoint_gpus_and_status_lists_themand
orbit::agent::tests::running_groups_are_stopped_and_outlive_their_controller_sessionin the step "Test native CLI" (179 passed, 2 failed). On the merge commit
5eff6460(run43603519699) the Windows and Ubuntu jobs fail the same errorand the same three tests, and the rdma job passes. On main at
5a4e86795459(run
43732636120inbrandonmmusic-max/flare-review) and on the mergecommit
93798dca(run43730555565), the rdma job passes and the other twojobs fail the same steps and tests: the three Ubuntu tests above, and on
Windows the
flare-eosunit-test binary exiting withSTATUS_DLL_NOT_FOUNDin "Test host OS helpers" and seven
cache_commandstests in "Test nativeCLI". The eight Windows hub and
load-package tests also failed on
83c49b02(run42056263704) and on mainat
0009b89d(run41941475176) and78b18b1c(run41885630157). Thischange touches no Rust and does not address them.
check above covers one model on one RTX PRO 6000 at FlashInfer
1f8ed8a3,plus a build at
42b221db. No GPU run covers4fd836dc, the commituv.lockrecords. This change does not modify kernels, graphs, or serving,and the dependency resolution behavior is covered by the CPU checks above.