Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,10 @@ jobs:
# this replaced, and a run took 26m54s against the old 35min cap (run
# 34489547654) versus 18-19m on a warm static host — keep the same headroom
# ratio the 35 was chosen for.
timeout-minutes: 45
# An include_nightly dispatch also runs the AIM serve scenarios (serve-26/27,
# @nightly, up to 1200s readiness each on a cold pull) and the large-model
# serve, which cannot fit the per-PR 45min budget.
timeout-minutes: ${{ inputs.include_nightly && 105 || 45 }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please confirm the first run on this PR accepts the inputs expression in job-level timeout-minutes (per-PR should resolve to 45).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the documentation rather than guessing: GitHub's context-availability table lists inputs for jobs.<job_id>.timeout-minutes, and on pull_request, merge_group and push there are no dispatch inputs, so inputs.include_nightly is empty and the expression resolves to 45 (the same job already uses inputs.include_nightly && '1' || '' at env level on those events). The first run of this PR on the workflow is the final check, and the per-PR Strix Ubuntu job's timeout-minutes should show 45 there.

runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]
needs: [changes]
# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-ubuntu.
Expand Down
17 changes: 16 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -718,7 +718,12 @@ jobs:
# Strix Ubuntu job.
e2e-gpu-nightly-strix:
name: E2E tests (Strix Halo, incl. nightly-only)
timeout-minutes: 90
# 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits when

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment hedges on whether this host has Docker, and the PR body says it's unconfirmed. If it doesn't, serve-26/27 skip every night here and nothing runs AIM serve except a manual include_nightly dispatch. Worth confirming before merge, or filing a follow-up for a pool-based AIM nightly. docs/testing.md and the feature comment could carry the same caveat.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Nothing in nightly.yml or its scripts installs or checks Docker, so it cannot be settled from the repo. I added a step before the E2E run that runs docker info (the harness's own probe) and prints a ::warning:: when Docker is unreachable, since the job is continue-on-error and a silent skip would otherwise never show. The caveat is also in docs/testing.md and the feature comment, and the description lists the follow-up if the warning shows: a pool-based AIM nightly (or moving this lane onto the pool).

# this host has Docker; on a host without it they skip and cost nothing.
# Whether it has Docker is unconfirmed: nothing in this job installs or checks
# it, so the "Report whether Docker is reachable" step logs the answer. If it
# does not, no nightly lane runs AIM serve and a pool-based one is needed.
timeout-minutes: 120
# `native` disambiguates this runner from any host that might carry a bare
# `wsl` label alongside `strix-halo` (the static WSL runner this repo used
# before the WSL lanes moved to the ephemeral pool) -- the paths below
Expand Down Expand Up @@ -770,6 +775,16 @@ jobs:
fi
echo "$CARGO_HOME/bin" >> "$GITHUB_PATH"

# serve-26/27 (AIM) need Docker reachable on this host; without it they skip
# silently. Say which it is, so a nightly that never ran them is visible.
- name: Report whether Docker is reachable (AIM serve scenarios need it)
run: |
if docker info >/dev/null 2>&1; then
echo "docker reachable: serve-26/27 (AIM) can run on this host"
else
echo "::warning::docker is not reachable on this host: serve-26/27 (AIM) will skip"
fi

- name: Run E2E tests on Strix Halo (incl. nightly-only)
run: |
export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
Expand Down
18 changes: 18 additions & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -739,6 +739,24 @@ self-hosted CI dispatch (which forwards `--name`): each bypasses every gate
above, so on WSL2, Windows or a machine with no GPU it would still run the real
install.

`serve-26` and `serve-27` (`features/model_serving.feature`, the AIM serve and
restart canaries) are `@nightly` for cost: each pulls the AIM image and a 9B
checkpoint on a cold runner and may wait up to 1200s for readiness, which on the
per-PR Strix Ubuntu job (about 25 minutes without them, 45 minute cap) used up
the whole budget. They still need `@requires-engine:aim`, i.e. Docker reachable
and a Radeon or Strix Halo GPU, so they run only where `E2E_INCLUDE_NIGHTLY=1`
is set on such a host: the nightly Strix Halo lane, or an `include_nightly`
dispatch of `e2e-selfhosted.yml` (whose Strix Ubuntu job then gets a 105 minute
cap). `aim_serve_scenarios_stay_off_the_per_pr_lanes` pins the tagging and that
they stay reachable when the opt-in is set.

Caveat: it is unconfirmed that the nightly Strix Halo host has Docker (nothing in
`nightly.yml` installs or checks it). Without it these two scenarios skip on
every nightly run, and only a manual `include_nightly` dispatch on a host with
Docker exercises AIM serve. The `e2e-gpu-nightly-strix` job logs whether Docker
is reachable ("Report whether Docker is reachable"); check that line before
relying on this coverage.

## Runtime Selection And Activation

List registered managed or read-only runtimes and their exact side-by-side keys:
Expand Down
12 changes: 10 additions & 2 deletions tests/e2e-cucumber/features/model_serving.feature
Original file line number Diff line number Diff line change
Expand Up @@ -350,12 +350,20 @@ Feature: Model serving
# verified on (see `HostCapability::engine_available`'s "aim" arm — Radeon
# gfx110X/120X and Strix Halo gfx1151, not Instinct); a Strix Halo
# self-hosted runner with Docker is where this actually runs.
# `@nightly` keeps both AIM scenarios off the per-PR lanes: each one pulls the
# image and a 9B checkpoint on a cold runner and may wait out the full
# 1200s readiness cap, so the pair on the per-PR Strix Ubuntu job (~25 min
# without them, 45 min cap) ended every run at the cap. They run only when
# E2E_INCLUDE_NIGHTLY is set (nightly.yml, or an include_nightly dispatch of
# e2e-selfhosted.yml) on a host that has Docker and a supported GPU.
# Unconfirmed: whether the nightly Strix Halo host has Docker; without it both
# skip every night (the nightly job logs whether Docker is reachable).
# `@serve-timeout:1200`
# widens both the harness's readiness poll and the CLI's own
# ROCM_CLI_AIM_READY_TIMEOUT_SECS (see isolate_cmd) — a cold-start HF weight
# download for this 9B model legitimately exceeds the engine's default
# 5-minute readiness cap.
@id:serve-aim-inference @requires-gpu @requires-engine:aim @serve-timeout:1200
@id:serve-aim-inference @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly
Scenario: serve-26 - A model served on AIM responds to inference requests
Given a managed runtime is active
And a model is being served on aim
Expand All @@ -370,7 +378,7 @@ Feature: Model serving
# on (`stop_managed_services`, via the `stop_aim_service` step), and the
# same serve+inference fixture `serve-26` uses, rather than adding a new
# stop/restart mechanism just for AIM.
@id:serve-aim-lifecycle-parity @requires-gpu @requires-engine:aim @serve-timeout:1200
@id:serve-aim-lifecycle-parity @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly
Scenario: serve-27 - A stopped AIM service can be restarted and serves inference again
Given a managed runtime is active
And a model is being served on aim
Expand Down
75 changes: 75 additions & 0 deletions tests/e2e-cucumber/src/expectation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1042,6 +1042,81 @@ serve_timeout_secs = 90
assert!(unmatched_also_run_ids(&[], &known).is_empty());
}

/// The tag line directly above `Scenario: <prefix>` in `model_serving.feature`.
fn model_serving_tags(prefix: &str) -> Vec<String> {
let feature = include_str!("../features/model_serving.feature");
let lines: Vec<&str> = feature.lines().collect();
let at = lines
.iter()
.position(|l| l.trim_start().starts_with(&format!("Scenario: {prefix}")))
.unwrap_or_else(|| panic!("no `Scenario: {prefix}` in model_serving.feature"));
lines[at - 1]
.split_whitespace()
.map(str::to_owned)
.collect()
}

#[test]
fn aim_serve_scenarios_stay_off_the_per_pr_lanes() {
// serve-26/27 pull the AIM image and a 9B checkpoint on a cold runner and
// may each wait out a 1200s readiness cap. Run per-PR on the Strix Ubuntu
// pool they pushed the job (~25 min otherwise) into its 45 min cap on every
// run. They must resolve to Skip unless the nightly opt-in is set, and must
// still be reachable (not Skip for the nightly reason) when it is.
let m = Expectations::default();
for prefix in ["serve-26 -", "serve-27 -"] {
let d = ScenarioDecl::from_tags(&model_serving_tags(prefix));
assert_eq!(d.requires_engine.as_deref(), Some("aim"), "{prefix}");
assert!(d.nightly, "{prefix} must be @nightly");
match resolve(&d, &cap("strix-ubuntu"), &m, Included::default()) {
Expectation::Skip { reason } => {
assert!(reason.contains("E2E_INCLUDE_NIGHTLY"), "{prefix}: {reason}");
}
other => panic!("{prefix} must skip per-PR, got {other:?}"),
}
// Opted in: the nightly gate must not fire. What remains is the engine
// requirement, which `engine_available("aim")` decides from the host's
// engine list, its GFX family and whether Docker answers. Pin both
// sides of it: a host that lists aim runs the scenario exactly when
// Docker is reachable, and one that does not list aim skips on the
// engine, never on the nightly opt-in.
let opted_in = Included {
nightly: true,
..Included::default()
};
let mut with_aim = cap("strix-ubuntu");
with_aim.available_engines.push("aim".to_owned());
match resolve(&d, &with_aim, &m, opted_in) {
Expectation::ExpectPass => assert!(
docker_available(),
"{prefix} ran although Docker is unreachable"
),
Expectation::Skip { reason } => {
assert!(
!docker_available(),
"{prefix} skipped with the opt-in set, aim listed and Docker reachable: {reason}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this message literal contains a long run of spaces between "Docker" and "reachable" (looks like a collapsed \ line continuation). Rejoin with a single space.

);
assert!(
reason.contains("engine 'aim' cannot start"),
"{prefix}: wrong skip reason with the opt-in set: {reason}"
);
}
other @ Expectation::ExpectXfail { .. } => {
panic!("{prefix}: unexpected outcome when opted in: {other:?}")
}
}
match resolve(&d, &cap("strix-ubuntu"), &m, opted_in) {
Expectation::Skip { reason } => assert!(
reason.contains("engine 'aim' cannot start"),
"{prefix}: {reason}"
),
other => {
panic!("{prefix}: host without aim must skip on the engine, got {other:?}")
}
}
}
}

#[test]
fn docker_scenario_skips_unless_included() {
// A container-backed scenario is opt-in, not merely "docker is here".
Expand Down
Loading