Skip to content

E2E self-hosted: downloads that bypass the DevLab Dispatch pool's mirror #846

Open

Description

@dhekimian

Hi, I run the DevLab Dispatch pool that the devlab-dispatch Strix Halo lanes use. In the 7 days to 2026-11-02 those lanes ran 1,629 jobs on the pool. The pool shares one internet link, and these downloads are the largest load on it.

The TheRock wheel indexes are now handled on our side: the runner sets ROCM_CLI_THEROCK_*, and the WSL2 forwarding is in PR #845. The items below need changes in rocm-cli itself. Each one is independent.

1. Hugging Face hosts other than huggingface.co

Every pool job gets HF_ENDPOINT, which points at the pool's Hugging Face mirror. Two checks accept only huggingface.co and hf.co hosts:

  • apps/rocmd/src/sandbox.rs:608 (is_huggingface_url)
  • crates/rocm-core/src/model_catalog.rs:627 (recipe_artifact_uri_is_huggingface)

When HF_ENDPOINT is set, a model URL on that host isn't recognised as Hugging Face. Suggestion: also accept the host of HF_ENDPOINT when it is set. Likewise, any code that builds https://huggingface.co/... URLs itself should build them from HF_ENDPOINT when it is set.

Which component fetches model weights in the serve scenarios? Knowing that would let us check it against the mirror. We have seen Lemonade on Windows ignore HF_ENDPOINT.

2. GitHub release downloads

Two places hard-code https://github.com/.../releases/download/...:

  • crates/rocm-deps/src/lib.rs:40 (Lemonade)
  • crates/rocm-core/src/uv.rs:649-651 (uv)

Could these take an opt-in base override, gated by ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE or a sibling flag the same way the TheRock bases are? The pool's mirror can serve <base>/<owner>/<repo>/releases/download/<tag>/<file>, and the pool would then set the base on its runners.

3. Unauthenticated GitHub API call

engines/lemonade/src/backend_alignment.rs:557 (latest_llamacpp_rocm_stable_tag) calls https://api.github.com/repos/.../releases/latest with no token.

Every pool host shares one public address, so these calls share GitHub's limit of 60 anonymous requests an hour with every other job on the pool. Once that's used up, the call fails with "API rate limit exceeded". Suggestion:

  • Send Authorization: Bearer $GITHUB_TOKEN when the variable is set.
  • Add GITHUB_TOKEN: ${{ github.token }} to the env of the three Strix lanes.

4. Per-run re-downloads

  • serve-02 installs a llama.cpp ROCm backend of about 3.3 GB through Lemonade on every run. The lanes already keep $RUNNER_WORKSPACE/e2e-shared between jobs, and the pool keeps it too. Putting Lemonade's backend cache there would keep the backend between runs.
  • The WSL2 lane loses its caches along with the distro. A cache directory on the Windows-side RUNNER_WORKSPACE, mounted into the guest, would survive.

5. Question: merge queue and main push both run the full matrix

In those 7 days the pool ran 77 merge_group runs (298 jobs, about 141 host-hours) and 78 push runs on main (298 jobs, about 119 host-hours). In nearly every case that's the same commit tested twice. We understand why:

  • The push run keeps a record of each landed commit (the per-SHA concurrency key).
  • E2E_MERGE_QUEUE is set only on merge_group, so the @merge-queue scenarios run only there.
  • These lanes aren't required checks, so the queue doesn't wait for them.

We aren't asking you to change this. If you'd consider one of these, it would free roughly 120–140 host-hours a week of Strix Halo capacity for PR runs:

  • Skip the Strix lanes on merge_group, as e2e-gpu (MI300X) already does, and set E2E_MERGE_QUEUE on push to main so the @merge-queue scenarios still run once per landed commit.
  • Or skip the push run when the same SHA already passed in the queue.

6. Release tarball base points at a catalog that stops at 7.13.0

THEROCK_RELEASE_TARBALL_BASE (apps/rocm/src/therock.rs:70) is https://repo.amd.com/rocm/tarball/. That catalog's newest release is 7.13.0. The 7.14.x archives are published only in https://repo.amd.com/rocm/tarball-multi-arch/, so the URL rocm-cli builds for 7.14.1 returns 403:

  • 403: https://repo.amd.com/rocm/tarball/therock-dist-windows-gfx1151-7.14.1.tar.gz (since about 2026-10-18; re-checked 2026-11-02)
  • 200: https://repo.amd.com/rocm/tarball-multi-arch/therock-dist-windows-gfx1151-7.14.1.tar.gz

This broke Windows Lemonade installs on the pool, with 39 failures on 10-26 and 10-27.

The 10.0.0 and 10.1.0 archives are in neither catalog. They are on https://stable.repo.amd.com/rocm/core/tarball/, which THEROCK_NEXT_TARBALL_BASE already uses.

Suggestion: use tarball-multi-arch/ for 7.14 and later, the way the pip base already uses whl-multi-arch.

We're happy to test any branch on the pool's staging hosts and report download numbers before and after.

Activity

  1. dhekimian commented on Nov 2, 2026

    @dhekimian
    Author

    Correction to item 1, after reading the code more closely:

    • is_huggingface_url (apps/rocmd/src/sandbox.rs) and recipe_artifact_uri_is_huggingface (crates/rocm-core/src/model_catalog.rs) only validate recipe artifact URIs. rocm-cli doesn't download model weights itself, so changing them wouldn't move any traffic. Please disregard item 1.
    • Most of the large downloads in item 4 come from inside Lemonade, not from rocm-cli. Its backend installer hard-codes https://github.com/<repo>/releases/download/... for the llama.cpp backend, and https://stable.repo.amd.com/rocm/core/tarball/ and https://stable.repo.amd.com/rocm/whl-next/ for the ROCm runtime it pairs with it. We'll propose opt-in base overrides there in a Lemonade PR and link it here.
    • Lemonade's current main honours HF_ENDPOINT for model downloads. rocm-cli pins Lemonade 11.5.1 (runtime-deps.toml), so a newer pin may already cover the Hugging Face side. We haven't yet confirmed which release added it.
  2. dhekimian commented on Nov 3, 2026

    @dhekimian
    Author

    Update on item 1 (TheRock wheels) and a heads-up.

    From 2026-11-02 19:16Z to 2026-11-03 ~17:10Z our pool's runners set ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE and the release and nightly pip bases to our mirror. That made runtime-09 fail on every Windows E2E job, because the preview's canonical_source was our mirror URL. That was our change, not yours, and it's now reverted. Please re-run any PR whose Windows E2E failed in that window.

    We'd still like the pool's TheRock downloads to go through the mirror (about 1.5 GB per job). Two ways that could work. Which would you accept?

    • (a) runtime-09 drops the ROCM_CLI_THEROCK_* variables for its own dry run, so it always checks the real canonical source. It's a dry run, so no download is involved. Or:
    • (b) when ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE is set, runtime-09 checks canonical_source against the configured base instead of the hard-coded aggregate.

    I see #820 rewrites runtime-09 to the ROCm 10 index (stable.repo.amd.com/rocm/whl-next). If that lands first, we'd mirror that index instead and the question applies to ROCM_CLI_THEROCK_NEXT_PIP_BASE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions