Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1632,12 +1632,103 @@ jobs:
pattern: '*-report'
path: e2e-artifacts

# The shared GPU queue's nightly (docs/nightly-on-the-queue.md) runs R9700,
# MI350P, W7900 and the Instinct lanes of the queue's fleet on its own
# schedule, and its results are published as one `queue-run-results`
# artifact of that run, not as `*-report` artifacts of this one. So fetch
# the newest such artifact of a run on main that finished, and started, in
# the last 24 hours (a run still going, or one from an earlier night, is
# not today's), and the job log of that run, which is the only place the
# queue prints which branch it tested and which lanes it blocks on. The
# artifact is found by its name, not through the workflow that makes it:
# that workflow runs on a runner no other workflow may name or depend on.
# A conclusion of `failure` is no reason to skip it: a red lane fails the
# queue's run and the artifact is still uploaded. This step never fails the
# job and never stops the report: when there is nothing to import it
# records why, and the next step says so at the top of the page instead of
# leaving the queue's GPUs out without a word.
- name: Fetch the latest queue results
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
set -uo pipefail
mkdir -p queue-results
absent() {
echo "$1" > queue-results/absent.txt
echo "::warning::No queue lanes in this report: $1"
exit 0
}
repo_id=$(gh api "repos/${GITHUB_REPOSITORY}" --jq .id) ||
absent "the repository could not be read"
since=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)
found=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=queue-run-results&per_page=50" |
jq -r --arg since "$since" --argjson repo "$repo_id" '
[.artifacts[]
| select(.expired == false)
| select(.workflow_run.head_branch == "main" and .workflow_run.head_repository_id == $repo)
| select(.created_at >= $since)]
| sort_by(.created_at) | last // empty | .workflow_run.id') ||
absent "the queue results could not be listed"
[ -n "$found" ] ||
absent "no queue run on main finished in the last 24 hours (a run still going is not imported)"
# The run of main's workflow can have been asked to test another branch,
# and a run that started a day ago is not tonight's.
started=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${found}" --jq .run_started_at) ||
absent "when the newest queue run started could not be read"
started_at=$(date -u -d "$started" +%s 2>/dev/null) &&
[ "$started_at" -ge "$(date -u -d '24 hours ago' +%s)" ] ||
absent "the newest queue run started more than 24 hours ago"
echo "$started" > queue-results/started.txt
gh run download "$found" --repo "$GITHUB_REPOSITORY" --name queue-run-results --dir queue-results/results ||
absent "the queue results of run $found could not be downloaded"
# The log names the branch the run tested and which lanes the queue
# blocks on. Without it nothing is imported.
job=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${found}/jobs" --jq '.jobs[0].id') &&
gh api "repos/${GITHUB_REPOSITORY}/actions/jobs/${job}/logs" > queue-results/job.log ||
rm -f queue-results/job.log
echo "$found" > queue-results/run-id.txt

# Turn the run's lanes into reports of this job's own kind, and write the
# status section for the top of the matrix (`cargo xtask e2e-queue-import`).
# A lane the queue marks advisory is labelled so and does not count towards
# the release-readiness verdict; a blocking lane with no result does.
- name: Add the queue lanes to the report
continue-on-error: true
run: |
set -uo pipefail
mkdir -p queue-results
if [ -s queue-results/run-id.txt ] && [ -d queue-results/results ]; then
run_id=$(cat queue-results/run-id.txt)
args=(--queue-dir queue-results/results
--run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${run_id}"
--run-started "$(cat queue-results/started.txt 2>/dev/null)")
[ -f queue-results/job.log ] && args+=(--log queue-results/job.log)
else
args=(--absent "$(cat queue-results/absent.txt 2>/dev/null || echo 'fetching the queue results did not complete')")
fi
cargo xtask e2e-queue-import "${args[@]}" \
--artifacts-dir e2e-artifacts \
--section-out queue-results/queue-lanes.md || {
# Half an import is worse than none, and none is said at the top of the page.
rm -rf e2e-artifacts/e2e-queue-* queue-results/queue-lanes.md
echo "::warning::The queue lanes could not be added to the report."
cargo xtask e2e-queue-import --absent "the queue lanes could not be imported" \
--artifacts-dir e2e-artifacts --section-out queue-results/queue-lanes.md || true
}

# The queue lanes' status leads the page. The matrix keeps its own first
# table, the one `release-readiness` reads its verdict from.
- name: Build consolidated report + step summary
run: |
mkdir -p consolidated
cargo xtask e2e-report \
--artifacts-dir e2e-artifacts \
--html-out consolidated/index.html > consolidated/support-matrix.md
if [ -f queue-results/queue-lanes.md ]; then
cat queue-results/queue-lanes.md consolidated/support-matrix.md > "$RUNNER_TEMP/with-queue.md"
mv "$RUNNER_TEMP/with-queue.md" consolidated/support-matrix.md
fi
cat consolidated/support-matrix.md >> "$GITHUB_STEP_SUMMARY"

# A lane that dies before writing report.json (preflight, build, a crash
Expand Down
94 changes: 94 additions & 0 deletions crates/e2e-report/src/consolidated/descriptor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ pub(super) fn parse_descriptor(name: &str) -> Descriptor {
},
};

// A lane of the shared GPU queue (see `queue_descriptor`).
if let Some(lane) = core.strip_prefix("queue-") {
return queue_descriptor(lane, channel_from_name);
}

// Legacy `-known-bugs` suffix (retained only as a stable secondary sort key).
let (core, known_bugs) = match core.strip_suffix("known-bugs") {
Some(rest) => (rest.trim_end_matches('-'), true),
Expand Down Expand Up @@ -86,6 +91,68 @@ pub(super) fn parse_descriptor(name: &str) -> Descriptor {
}
}

/// The marker an advisory queue lane carries in its platform label.
///
/// A reader of the rendered report, and `xtask release-readiness`, which reads
/// that rendering, tell a lane the queue lets fail from one it does not by it.
pub const ADVISORY_QUEUE_MARKER: &str = "(queue, advisory)";

/// The GPU each shared-queue lane stands for, by the lane's name. The queue
/// discovers its lanes from its fleet, so a lane missing from this table is not
/// an error: it renders under its own name in capitals.
const QUEUE_LANE_GPUS: &[(&str, &str)] = &[
("mi300x", "MI300X"),
("mi325x", "MI325X"),
("mi350p", "MI350P"),
("mi350x", "MI350X"),
("mi355x", "MI355X"),
("r9700s", "R9700"),
("w7900-dual", "W7900"),
];

/// The GPU a shared-queue lane stands for (`w7900-dual` is a W7900), or the
/// lane's own name in capitals for a lane this table does not know yet.
pub fn queue_lane_gpu(lane: &str) -> String {
QUEUE_LANE_GPUS
.iter()
.find(|(name, _)| *name == lane)
.map_or_else(|| lane.to_uppercase(), |(_, gpu)| (*gpu).to_owned())
}

/// The artifact name `xtask e2e-queue-import` gives a queue lane's report.
///
/// For one channel, or none for a lane that reported no channel at all. The lane
/// is named, not the `platform_slug` in its `platform.json`: queue lanes of
/// different models report the same slug (an MI325X lane probes as `mi300x`, an
/// MI355X as `mi350p`), so the slug cannot say which lane a report came from.
/// A lane the queue does not block on carries the `-advisory` suffix.
pub fn queue_artifact_name(lane: &str, advisory: bool, channel: Option<&str>) -> String {
let advisory = if advisory { "-advisory" } else { "" };
let channel = channel.map_or_else(String::new, |c| format!("-{c}"));
format!("e2e-queue-{lane}{advisory}{channel}-report")
}

/// Parse the core of a queue lane's artifact name (`mi325x`, `mi325x-advisory`;
/// see [`queue_artifact_name`]). Always Linux: the queue runs Linux pods.
fn queue_descriptor(lane: &str, channel_from_name: Option<&'static str>) -> Descriptor {
let (lane, advisory) = match lane.strip_suffix("-advisory") {
Some(rest) => (rest, true),
None => (lane, false),
};
let gpu = queue_lane_gpu(lane);
let queue = if advisory {
ADVISORY_QUEUE_MARKER
} else {
"(queue)"
};
Descriptor {
platform: format!("{gpu} {queue}"),
os: "Linux".to_owned(),
known_bugs: false,
channel_from_name,
}
}

/// The identity-bearing core of an artifact name: `e2e-` prefix and `-report`
/// suffix stripped, each relative to the prior result (not `name`), so
/// `e2e-report` correctly reduces to `report`, not back to itself.
Expand Down Expand Up @@ -328,6 +395,33 @@ mod tests {
}
}

#[test]
fn parse_descriptor_names_each_queue_lane_and_its_gate() {
// Every lane name is a path segment of a queue artifact, so the table is
// what turns `w7900-dual` into the GPU a reader knows. A lane the table
// lacks keeps its own name rather than being guessed at.
for (lane, gpu) in QUEUE_LANE_GPUS {
let blocking = parse_descriptor(&queue_artifact_name(lane, false, Some("release")));
assert_eq!(
(blocking.platform.as_str(), blocking.os.as_str()),
(format!("{gpu} (queue)").as_str(), "Linux"),
"{lane}"
);
assert_eq!(blocking.channel_from_name, Some("release"));
let advisory = parse_descriptor(&queue_artifact_name(lane, true, Some("nightly")));
assert_eq!(advisory.platform, format!("{gpu} {ADVISORY_QUEUE_MARKER}"));
assert_eq!(advisory.channel_from_name, Some("nightly"));
}
let new_lane = parse_descriptor(&queue_artifact_name("mi400x", true, None));
assert_eq!(new_lane.platform, "MI400X (queue, advisory)");
assert_eq!(new_lane.channel_from_name, None);
// A queue MI300X lane is not the self-hosted MI300X lane.
assert_ne!(
parse_descriptor("e2e-queue-mi300x-report").platform,
parse_descriptor("e2e-gpu-report").platform
);
}

#[test]
fn parse_descriptor_distinguishes_sdk_version_matrix_legs() {
// The sdk_version matrix (ROCMAI-430) uploads non-`current` legs under a
Expand Down
4 changes: 3 additions & 1 deletion crates/e2e-report/src/consolidated/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ mod sidecars;
#[cfg(test)]
mod test_support;

pub use descriptor::artifact_name_for_platform_slug;
pub use descriptor::{
ADVISORY_QUEUE_MARKER, artifact_name_for_platform_slug, queue_artifact_name, queue_lane_gpu,
};
use descriptor::{Descriptor, full_platform_label, parse_descriptor, with_paren_suffix};
use sidecars::{
CommandRecord, ManifestExpectation, PlatformManifest, PlatformVersions, parse_commands,
Expand Down
5 changes: 4 additions & 1 deletion crates/e2e-report/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ mod single_report;

pub use consolidated::html::{RunMeta, generate_consolidated};
pub use consolidated::markdown::consolidated_summary_markdown;
pub use consolidated::{PlatformReports, artifact_name_for_platform_slug, load_platform_reports};
pub use consolidated::{
ADVISORY_QUEUE_MARKER, PlatformReports, artifact_name_for_platform_slug, load_platform_reports,
queue_artifact_name, queue_lane_gpu,
};
pub use parse::{XfailReport, evaluate_xfail, scenario_results_by_id};
pub use single_report::generate;

Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -738,6 +738,7 @@ Repository tasks, run through the workspace alias `cargo xtask <command>`: relea
| [`demos.rs`](../xtask/src/demos.rs) | `cargo xtask demos`: builds the demo binaries, starts the mock environment before VHS runs, and renders the documentation GIFs. | flat |
| [`e2e.rs`](../xtask/src/e2e.rs) | `cargo xtask e2e`: builds release `rocm` and `rocmd` with the E2E test features and runs the cucumber suite, forwarding extra arguments to cucumber. | flat |
| [`e2e_prewarm.rs`](../xtask/src/e2e_prewarm.rs) | `cargo xtask e2e-prewarm`: prepares the shared managed runtime the GPU E2E lanes serve against, refreshing it when the channel index publishes something newer or the required packages differ. | flat |
| [`e2e_queue_import.rs`](../xtask/src/e2e_queue_import.rs) | `cargo xtask e2e-queue-import`: puts a finished shared-queue run's per-lane reports where `e2e-report` reads them, named for the lane and the queue's gate, and writes the queue lanes' status section for the top of the support matrix. | flat |
| [`e2e_report.rs`](../xtask/src/e2e_report.rs) | `cargo xtask e2e-report`: merges per-platform `report.json` files into one HTML report and a step-summary matrix, one platform per artifact subdirectory. | flat |
| [`engine_stdio_client_contract.rs`](../xtask/src/engine_stdio_client_contract.rs) | Source-scan test that only the shared stdio client in [`stdio_client.rs`](../crates/rocm-engine-protocol/src/stdio_client.rs) puts a request on an engine child's stdin: a file that both pipes a child's stdin and names an engine envelope is reported unless it is a reviewed `ALLOWED` entry, and a stale entry fails too. | flat |
| [`env_mutation_contract.rs`](../xtask/src/env_mutation_contract.rs) | Source-scan test that every test which mutates the process environment serializes itself. | flat |
Expand Down
12 changes: 12 additions & 0 deletions docs/ci-hardware-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,18 @@ and GitHub step summary. Since ROCMAI-429 the report keys each column by
release/nightly channel matrix renders two columns per platform instead of
one channel's result silently overwriting the other.

`e2e-report-nightly` also reads the shared queue's nightly (below), which runs
R9700, MI350P, W7900 and the Instinct lanes of the queue's fleet. It fetches the
`queue-run-results` artifact of the newest queue run that tested `main` and
finished, and started, in the last 24 hours, and `cargo xtask e2e-queue-import`
puts each lane's reports among the others, as `MI325X (queue) Linux (release)`
and the like. A lane the queue does not block on is labelled `(queue, advisory)`
and does not count towards the release-readiness verdict; a blocking lane that
came back with no result is shown as `EMPTY` and does. The page opens with a
`Queue lanes` table. A missing or late queue run never fails the report: the
page says no queue lanes are in it. See
[The support matrix and the wiki](nightly-on-the-queue.md#the-support-matrix-and-the-wiki).

On `main`, the `publish-e2e-wiki` job in `nightly.yml` also copies the
consolidated nightly matrix to the `E2E-Support-Matrix` page of the repository
wiki, overwriting it each night (the page header names the run, commit and
Expand Down
Loading
Loading