Repository navigation
ci(e2e): move the AIM serve scenarios off the per-PR Strix Ubuntu lane #852
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -718,7 +718,12 @@ jobs: | |
| # Strix Ubuntu job. | ||
| e2e-gpu-nightly-strix: | ||
| name: E2E tests (Strix Halo, incl. nightly-only) | ||
| timeout-minutes: 90 | ||
| # 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits when | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The comment hedges on whether this host has Docker, and the PR body says it's unconfirmed. If it doesn't, serve-26/27 skip every night here and nothing runs AIM serve except a manual
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right. Nothing in |
||
| # this host has Docker; on a host without it they skip and cost nothing. | ||
| # Whether it has Docker is unconfirmed: nothing in this job installs or checks | ||
| # it, so the "Report whether Docker is reachable" step logs the answer. If it | ||
| # does not, no nightly lane runs AIM serve and a pool-based one is needed. | ||
| timeout-minutes: 120 | ||
| # `native` disambiguates this runner from any host that might carry a bare | ||
| # `wsl` label alongside `strix-halo` (the static WSL runner this repo used | ||
| # before the WSL lanes moved to the ephemeral pool) -- the paths below | ||
|
|
@@ -770,6 +775,16 @@ jobs: | |
| fi | ||
| echo "$CARGO_HOME/bin" >> "$GITHUB_PATH" | ||
|
|
||
| # serve-26/27 (AIM) need Docker reachable on this host; without it they skip | ||
| # silently. Say which it is, so a nightly that never ran them is visible. | ||
| - name: Report whether Docker is reachable (AIM serve scenarios need it) | ||
| run: | | ||
| if docker info >/dev/null 2>&1; then | ||
| echo "docker reachable: serve-26/27 (AIM) can run on this host" | ||
| else | ||
| echo "::warning::docker is not reachable on this host: serve-26/27 (AIM) will skip" | ||
| fi | ||
|
|
||
| - name: Run E2E tests on Strix Halo (incl. nightly-only) | ||
| run: | | ||
| export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target" | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1042,6 +1042,81 @@ serve_timeout_secs = 90 | |
| assert!(unmatched_also_run_ids(&[], &known).is_empty()); | ||
| } | ||
|
|
||
| /// The tag line directly above `Scenario: <prefix>` in `model_serving.feature`. | ||
| fn model_serving_tags(prefix: &str) -> Vec<String> { | ||
| let feature = include_str!("../features/model_serving.feature"); | ||
| let lines: Vec<&str> = feature.lines().collect(); | ||
| let at = lines | ||
| .iter() | ||
| .position(|l| l.trim_start().starts_with(&format!("Scenario: {prefix}"))) | ||
| .unwrap_or_else(|| panic!("no `Scenario: {prefix}` in model_serving.feature")); | ||
| lines[at - 1] | ||
| .split_whitespace() | ||
| .map(str::to_owned) | ||
| .collect() | ||
| } | ||
|
|
||
| #[test] | ||
| fn aim_serve_scenarios_stay_off_the_per_pr_lanes() { | ||
| // serve-26/27 pull the AIM image and a 9B checkpoint on a cold runner and | ||
| // may each wait out a 1200s readiness cap. Run per-PR on the Strix Ubuntu | ||
| // pool they pushed the job (~25 min otherwise) into its 45 min cap on every | ||
| // run. They must resolve to Skip unless the nightly opt-in is set, and must | ||
| // still be reachable (not Skip for the nightly reason) when it is. | ||
| let m = Expectations::default(); | ||
| for prefix in ["serve-26 -", "serve-27 -"] { | ||
| let d = ScenarioDecl::from_tags(&model_serving_tags(prefix)); | ||
| assert_eq!(d.requires_engine.as_deref(), Some("aim"), "{prefix}"); | ||
| assert!(d.nightly, "{prefix} must be @nightly"); | ||
| match resolve(&d, &cap("strix-ubuntu"), &m, Included::default()) { | ||
| Expectation::Skip { reason } => { | ||
| assert!(reason.contains("E2E_INCLUDE_NIGHTLY"), "{prefix}: {reason}"); | ||
| } | ||
| other => panic!("{prefix} must skip per-PR, got {other:?}"), | ||
| } | ||
| // Opted in: the nightly gate must not fire. What remains is the engine | ||
| // requirement, which `engine_available("aim")` decides from the host's | ||
| // engine list, its GFX family and whether Docker answers. Pin both | ||
| // sides of it: a host that lists aim runs the scenario exactly when | ||
| // Docker is reachable, and one that does not list aim skips on the | ||
| // engine, never on the nightly opt-in. | ||
| let opted_in = Included { | ||
| nightly: true, | ||
| ..Included::default() | ||
| }; | ||
| let mut with_aim = cap("strix-ubuntu"); | ||
| with_aim.available_engines.push("aim".to_owned()); | ||
| match resolve(&d, &with_aim, &m, opted_in) { | ||
| Expectation::ExpectPass => assert!( | ||
| docker_available(), | ||
| "{prefix} ran although Docker is unreachable" | ||
| ), | ||
| Expectation::Skip { reason } => { | ||
| assert!( | ||
| !docker_available(), | ||
| "{prefix} skipped with the opt-in set, aim listed and Docker reachable: {reason}" | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: this message literal contains a long run of spaces between "Docker" and "reachable" (looks like a collapsed |
||
| ); | ||
| assert!( | ||
| reason.contains("engine 'aim' cannot start"), | ||
| "{prefix}: wrong skip reason with the opt-in set: {reason}" | ||
| ); | ||
| } | ||
| other @ Expectation::ExpectXfail { .. } => { | ||
| panic!("{prefix}: unexpected outcome when opted in: {other:?}") | ||
| } | ||
| } | ||
| match resolve(&d, &cap("strix-ubuntu"), &m, opted_in) { | ||
| Expectation::Skip { reason } => assert!( | ||
| reason.contains("engine 'aim' cannot start"), | ||
| "{prefix}: {reason}" | ||
| ), | ||
| other => { | ||
| panic!("{prefix}: host without aim must skip on the engine, got {other:?}") | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn docker_scenario_skips_unless_included() { | ||
| // A container-backed scenario is opt-in, not merely "docker is here". | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please confirm the first run on this PR accepts the
inputsexpression in job-leveltimeout-minutes(per-PR should resolve to 45).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checked the documentation rather than guessing: GitHub's context-availability table lists
inputsforjobs.<job_id>.timeout-minutes, and onpull_request,merge_groupandpushthere are no dispatch inputs, soinputs.include_nightlyis empty and the expression resolves to 45 (the same job already usesinputs.include_nightly && '1' || ''at env level on those events). The first run of this PR on the workflow is the final check, and the per-PR Strix Ubuntu job'stimeout-minutesshould show 45 there.