Skip to content

e2e: follow-ups from the shared therock/tool cache work (#355) #359

Open

Description

@juhovainio

Tracked follow-ups from review on #355 ("fix(e2e,lemonade): share the therock/tool cache and fix the shared runtime tree getting wiped per scenario"), per @tomastola's request that these not be discoverable only via a merged PR's description. None of these are regressions from that PR; they're pre-existing or newly-incidental gaps it surfaced.

1. Stale E2E_SHARED_* cache doc in tests/e2e-cucumber/README.md

tests/e2e-cucumber/README.md:100 still lists:

| `ROCM_CLI_CACHE_DIR` | (temp dir) | Isolated cache directory per scenario |

which is no longer true under CI now that #355 routes it through a shared dir when E2E_SHARED_CACHE_DIR is set. The E2E_SHARED_* vars aren't in that table at all (pre-existing gap, not introduced by #355).

2. xtask e2e-prewarm doesn't warm the shared therock/tool cache

xtask/src/e2e_prewarm.rs:723 points ROCM_CLI_CACHE_DIR at prewarm_dir.join("cache"), a different tree than the one isolate_env() now shares across scenarios (E2E_SHARED_CACHE_DIR/rocm-cli subdir). The shared cache is never pre-populated, so the first scenario in a job still pays the cold therock/tool download cost that #355 otherwise eliminates for the rest.

3. rocm uninstall / build_downloads_plan are a wipe hazard against a shared cache dir

  • rocm uninstall without --keep-cache removes paths.cache_dir wholesale (apps/rocm/src/main.rs:17569).
  • build_downloads_plan (apps/rocm/src/storage.rs:715-720) clears cache/therock + cache/tools.

Nothing in the current e2e suite routes either of these at the shared cache dir today — the lifecycle uninstall scenarios correctly use their own isolated smoke_cache dirs (lifecycle_steps.rs:894, :952). But a future scenario that shells out to rocm uninstall (or otherwise exercises these paths) without deliberately avoiding the shared dir would silently wipe it mid-job for every concurrent/subsequent scenario. Worth either a guard, or at minimum a comment at both call sites pointing back here.

4. Concurrency-cap coupling isn't enforced, only documented — done in #355

max_concurrent is now if cap.has_amd_gpu || shared_cache_dir().is_some() { 1 } else { 64 }, deriving the cap from the hazard itself rather than only from the GPU-capability probe. A lane like WSL2 (probe reads false, still exports a shared cache dir) is now genuinely serialized instead of accidentally-safe.

5. Unbounded cache growth (capacity, not correctness)

Noted in review as a non-blocker for #355: none of the shared caches (E2E_SHARED_CACHE_DIR, E2E_SHARED_UV_CACHE_DIR, E2E_SHARED_RUNTIMES_DIR) have pruning. Volume now accumulates across three unbounded caches (plus #352's ~23GB runtimes move). Probably its own ticket when it becomes a real constraint rather than a change to make speculatively now.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions