Skip to content

ROCMAI-955: describe each exit code of rocm fix --json in the manifest - #1067

Merged
volen-silo merged 1 commit into
mainfrom
rocmai-955-exit-codes
Nov 16, 2026
Merged

volen-silo merged 1 commit into
mainfrom
rocmai-955-exit-codes

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm fix --json publishes the exit codes it can return (exit_codes: ok 0, internal 1, unknown_id 2, not_applicable 3, failed 4, declined 5), but only as names and numbers. A consumer such as the rocm-doctor agent skill has to guess what each one means, and one of them is ambiguous: exit 2 is unknown_id (no catalog entry has that fix-id), and it is also what clap returns for any malformed command line, such as an unknown option or a missing value. The code alone cannot tell "no such fix" from "bad command line".

What changes

  • The manifest gains exit_code_descriptions, a sibling of exit_codes with the same names mapped to text saying what a caller should conclude. exit_codes is untouched, so existing readers see no difference.
  • Each description was checked against every return path of its code, across fix.rs, fix/runners.rs, fix/consent.rs, main.rs and diagnose_cmd.rs. Several paths are not obvious: a --dry-run preview can exit 3 or 4 (a missing tool, or the Windows User PATH read failing before any plan), not_applicable also covers --device-index=-1, and failed also covers a rc-file write or a state read that failed, not only a command.
  • Exit 2 stays shared and is documented, not split: giving usage errors a code of their own would change what existing readers of exit 2 receive. The unknown_id text says how to tell the two causes apart: an unknown fix-id prints a stderr line starting Unknown fix-id:, a usage error prints a stderr line starting error: and never says that. It says "a stderr line starting" because a one-time rocm: notice (for example about the uv cache moving) can print first. It does not promise a Usage: line, because a missing option value prints none.
  • A negative --device-index depends on how it is written. --device-index=-1 reaches the fix, which refuses it with exit 3. --device-index -1 (with a space) is a usage error and exits 2, because clap reads the -1 as an option. The descriptions, README and docs say so, and a test pins both spellings through the real parser and exit path.
  • Both published maps are built from one table, exit::MEANINGS in fix/options.rs, so a code cannot gain a name without a meaning. A test reads options.rs and checks that the exit:: constants and the MEANINGS codes match one to one, so a new constant fails until it is described. The refusal that apply() prints and the description's claim about it share one constant, exit::UNKNOWN_ID_PREFIX.
  • CATALOG_CONTRACT_VERSION stays at 1. The rule in fix/manifest.rs is that an added field keeps the version and a removal or type change raises it, and exit_codes keeps its type. The new field is #[serde(default)], so a manifest written before it still parses.
  • doctor-catalog.json and the 7 recorded rocm fix --json replies under skills/rocm-doctor/evals/fixtures/ are regenerated (cargo xtask catalog, cargo xtask eval-fixtures). The only change in them is the added exit_code_descriptions object.
  • README.md, --help for --json, docs/testing.md and docs/manual-testing.md describe the field and the shared exit 2. rocm fix --no-such-option is added to the deliberate exceptions of every_advised_command_parses, with its reason, because the new manual-testing step runs a command that is meant to be rejected.

Verification

What each description is checked against:

  • unknown_id and the usage-error exit are checked against the running CLI. Scenario diagnose-34 (title now "A tool can read the catalog without parsing prose, and the shared exit code 2 can be told apart"; @id and number unchanged) runs rocm fix with an id the catalog does not have and with --no-such-option. Both must exit with the published code, the first must print a line starting Unknown fix-id:, the second a line starting error: and no unknown-id text anywhere in stderr. A unit test runs three malformed command lines through the real parser and exit path, and a second pins both spellings of a negative --device-index (with a mutation check: adding allow_negative_numbers to the argument makes it fail).
  • ok, internal, not_applicable, failed and declined are tied to unit tests named in a comment above each MEANINGS entry (for example declining_consent_returns_5_and_writes_nothing, windows_path_read_failure_aborts_before_plan_or_write, a_recipe_the_catalog_says_is_applied_with_no_runner_exits_internal). The same comments name the branches no test drives: for not_applicable, the missing-tool, missing-install and missing-home-directory branches; for failed, a command (usermod, setx, PowerShell) that exits non-zero, because those runners start the real program. The descriptions still cover those branches because the code returns the code there.
  • The diagnose-34 step requires a non-empty description for every name in exit_codes.
Platform Run Result
Windows cargo test --workspace --all-targets --no-fail-fast (AGENTS.md §8) 46 test binaries, 6598 pass, 0 fail, 17 ignored
Windows cargo xtask smoke (AGENTS.md §8) smoke: ok
Windows cargo nextest run --workspace (cargo xtask affected selects the workspace) 6594 pass, 17 skipped
Windows cargo xtask e2e for diagnose-06, -34, -35 3 scenarios, 11 steps pass
WSL2 (Ubuntu, Strix Halo) same nextest set 7068 run, 7066 pass; the 2 failures are below
WSL2 cargo xtask e2e for diagnose-06, -34, -35 3 scenarios, 11 steps pass
Windows cargo test -p xtask, cargo xtask catalog --check, cargo xtask eval-fixtures --check, cargo clippy --workspace --all-targets -D warnings, cargo clippy -p e2e-cucumber --test e2e -D warnings, cargo fmt --check, feature_naming, skill_reference pass

The last commit changes only comments, so the full-gate rows were taken one commit earlier. The pre-push hook (which runs the workspace tests, both clippy runs, the signature check and the architecture-doc check) ran on the final commit and passed.

Failures and instability outside this change:

  • WSL2: build_metadata::build_script::tests::git_watch_paths_* fail because that account's git config signs commits with an SSH key that is absent there.
  • Intermittent under load on this machine: the first pre-push hook run failed once on service_launch::spawn::tests::a_failed_spawn_past_the_first_write_terminates_the_child_and_keeps_an_abandoned_record, in code this change does not touch. It passed three times alone and in a second full hook run.

Scope

  • skills/rocm-doctor/SKILL.md and reference.md are not changed here. The skill rewrite in fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 edits them. reference.md row 3 lists "negative --device-index" under exit 3 without the = qualifier above, and row 2 does not mention the spaced spelling; both are corrected in the follow-up that makes the skill read exit_code_descriptions.
  • The seven regenerated replies are also regenerated by other open changes. Whichever merges second reruns cargo xtask eval-fixtures.
  • Four MEANINGS clauses have no test driving them, as listed above.

Part of ROCMAI-955.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no bug fix.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: a field on an existing manifest.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, per AGENTS.md §3. The unknown_id description and the usage-error exit are asserted against the running CLI (diagnose-34, and the parser and exit-path tests). The other five descriptions are each tied to a named unit test, and the branches no test drives are listed rather than claimed.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo
volen-silo force-pushed the rocmai-955-exit-codes branch from 3ab6131 to e9372cd Compare November 13, 2026 10:40
@volen-silo
volen-silo marked this pull request as ready for review November 13, 2026 11:30
@volen-silo
volen-silo requested a review from a team as a code owner November 13, 2026 11:30

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. No blocking issues.

  • exit::MEANINGS is the single source for both published maps, and the test tying it to the exit:: constants by code and name keeps them from drifting.
  • The exit-2 claim (unknown fix id and clap usage error share a code) is proved against the real parser and exit path, and diagnose-34 covers it end to end.
  • README, --help, docs/testing.md, docs/manual-testing.md, the catalog JSON and eval fixtures are in sync; the change is additive and the contract version is unchanged.

Non-blocking:

  • skills/rocm-doctor/reference.md still lists a negative --device-index under exit 3 without the =-1 vs -1 distinction the new descriptions make. Fine to leave for the follow-up the PR body mentions.
  • One cosmetic nit inline.

Comment thread crates/rocm-core/src/fix/options.rs Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head e9372cd

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (single-commit PR, fetched read-only via git fetch origin +pull/1067/head:refs/remotes/pr/1067, diffed against merge-base 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe).
Implements one line item of ticket ROCMAI-955 (documenting what each rocm fix --json exit code means, including the shared exit-2 ambiguity) — not the ticket's other, unrelated CLI follow-ups; it does not claim to close the ticket.

Blocking

None

Non-blocking

None

Decisions for the author

  • Exit code 2 stays shared between unknown_id and a clap usage error, disambiguated only by stderr text rather than a dedicated exit code — tradeoff. Both independent review passes flagged this as worth a maintainer decision: the feature's own stated purpose is to let a tool read the catalog "without parsing prose," yet this one case still requires parsing an English stderr line. The PR's description already argues the other side of the tradeoff (a dedicated code would change what existing readers of exit 2 receive), and the parent ticket posed it as an open "say so or separate them" choice that this PR answers by documenting rather than splitting. Flagging so the choice is confirmed deliberate, not re-litigating it.
  • exit_codes and exit_code_descriptions are two independently-built maps kept in sync only by tests, not by the type system — non-blocking-improvement. every_published_exit_code_is_described_once and every_exit_constant_is_published_and_nothing_else_is (the latter parsing options.rs with syn to tie MEANINGS to the exit:: constants) are both real, non-decorative guards, so nothing is currently broken. A single combined struct serialized into both public shapes would make the two maps structurally incapable of diverging instead of relying on tests to catch it — worth considering if this contract grows further, not required now.

Positive signals

  • every_exit_constant_is_published_and_nothing_else_is (crates/rocm-core/src/fix/manifest.rs:225-277) parses the actual source of options.rs via syn and fails both when a new exit:: constant has no MEANINGS entry and when MEANINGS publishes a code with no backing constant — a genuine mechanism for the "a code cannot gain a name without a meaning" claim, not just an assertion of it.
  • The new exit-code description text makes dozens of falsifiable, fix-specific claims (e.g. exactly which fixes can return not_applicable from a missing home directory). Two independent review passes traced a large sample of these against the real branches in fix/runners.rs and fix.rs and found none inaccurate — unusually high documentation-to-code fidelity for a change of this size.

Deployment notes

  • CatalogManifest::exit_code_descriptions is additive and #[serde(default)], so CATALOG_CONTRACT_VERSION correctly stays at 1 and a manifest written before this field existed still deserializes (verified: a_manifest_without_descriptions_still_parses strips the field from a serialized value and re-parses it). No consumer migration required.
  • One CI lane, "E2E tests (MI300X, current)", was still QUEUED (not failed) at review time; every other check — 60+ lanes including the Windows/WSL2/Strix-Halo E2E lanes, clippy, module-size and architecture-doc gates — reports SUCCESS. Worth confirming that lane completes green before merging, though reviewDecision is already APPROVED.

What this covered

Every file changed in the PR (README.md, apps/rocm/src/{advised_commands.rs,cli.rs,main.rs}, crates/rocm-core/src/fix.rs and fix/{manifest,options,runners}.rs, docs/{manual-testing,testing}.md, doctor-catalog.json, all 7 skills/rocm-doctor/evals/fixtures/*/replies.json, and the Cucumber feature/step files), read completely at the PR head, plus the surrounding code each change touches (apply()'s callers, run_hip_visible_devices's branches, clap_exit_code/exit_code_for, the Cli arg definitions, and cross-repo references to exit_codes/exit_code_descriptions including skills/rocm-doctor/SKILL.md/reference.md, which already hand-document the same exit-2 ambiguity and are unchanged by this PR — pre-existing and unworsened, not a finding here).

Two independent fan-out passes ran: core Rust logic, and docs/e2e-tests/fixtures. Both traced claims against actual code/runtime behavior rather than trusting the prose; nothing either pass found reached the 70-confidence bar for a defect. History (git log) was read for the touched files. Review-thread discussion was not read and is not reconciled against here. Agent-instruction adherence (AGENTS.md) was checked throughout — module size, docs/architecture.md coverage (no new source file), markdown link conventions, the message+state co-assertion rule, and test coverage for introduced behavior — no violation found.

CI: all checks report SUCCESS except the one still-QUEUED lane noted above; no check failed.

Ticket: ROCMAI-955 is named in the PR title and was retrieved directly. It is a multi-item follow-up ticket from an earlier skill review; this PR implements exactly one of its line items and does not claim to close it.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

`rocm fix --json` named its exit codes but said nothing about what each
one means, so a caller holding a 3 or a 5 had to guess. Exit 2 was worse:
it is both "no such fix-id" and what the argument parser returns for a
malformed command line, and nothing told a caller that, or how to tell
the two apart.

The manifest now carries `exit_code_descriptions`, keyed by the same
names as `exit_codes`. Both maps are built from one table next to the
exit-code constants, and a test holds that table to the set of constants,
so a code cannot be published without a meaning or added without one.
`exit_codes` keeps its shape and the field is an addition, so the
contract version stays at 1, per the rule in the manifest module.

Usage errors keep exit 2 rather than gaining a code of their own, which
would change what existing readers of 2 receive. The `unknown_id`
description says so instead, and names the message that separates them:
a stderr line starting `Unknown fix-id:` against one starting `error:`.
It promises a line rather than the start of stderr, because a one-time
`rocm:` startup notice can come first.

Each description is written to be true on every path that returns its
code: `not_applicable` covers an argument the fix cannot use (a negative
device index written `--device-index=-1`, which the parser accepts;
written `--device-index -1` the parser reads `-1` as an unknown option,
a usage error that exits 2, and both descriptions say so), `failed`
covers a failed file write and a failed read of the state a fix needs
(which also happens under `--dry-run`), and `ok` says a dry-run preview
can still exit 3 or 4. The table names, above each entry, the tests that prove it
and the branches no test drives; new tests cover the missing-runner exit
1 and the negative device index, the latter in both spellings through
the real parser.

diagnose-34 now runs the binary to check that an unknown fix-id and an
unknown option both exit with the code the catalog describes and print
the message it names, and that every published code has a description.
doctor-catalog.json and the skill eval replays are regenerated by their
xtask commands.

ROCMAI-955

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-955-exit-codes branch from e9372cd to 67541c9 Compare November 16, 2026 08:12
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the approval. The reflow nit is fixed in 67541c9 (see the inline reply). On skills/rocm-doctor/reference.md: agreed, row 3 still lists a negative --device-index under exit 3 without the =-1 versus -1 distinction, and row 2 does not mention the spaced spelling. That file belongs to the skill rewrite, so the correction goes with the follow-up that makes the skill read exit_code_descriptions. It is listed in this PR's Scope section.

@volen-silo
volen-silo enabled auto-merge November 16, 2026 08:25
@volen-silo
volen-silo added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 20914c7 Nov 16, 2026
45 of 46 checks passed
@volen-silo
volen-silo deleted the rocmai-955-exit-codes branch November 16, 2026 08:55
r0x0r added a commit that referenced this pull request Nov 16, 2026
One conflict: `apps/rocm/src/advised_commands.rs`. Both sides appended
entries to the same `NOT_INVOCATIONS` table, so the hunks collided on the
trailing `),` they share rather than on any disagreement.

#1067 added the `rocm fix --no-such-option; echo $?` entry, which belongs
to the "Deliberate negative examples" group already above it; this branch
added the ROCMAI-550 `device:`-value entries for README.md and
docs/manual-testing.md under their own comment. Both kept, with #1067's
entry restored to its group and this branch's block following it. All
three entries still name text that exists in those docs.

#1066's removal of the W7900 lane from e2e-selfhosted.yml is untouched by
this branch and carried through as main left it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
One conflict: `apps/rocm/src/advised_commands.rs`. Both sides appended
entries to the same `NOT_INVOCATIONS` table, so the hunks collided on the
trailing `),` they share rather than on any disagreement.

#1067 added the `rocm fix --no-such-option; echo $?` entry, which belongs
to the "Deliberate negative examples" group already above it; this branch
added the ROCMAI-550 `device:`-value entries for README.md and
docs/manual-testing.md under their own comment. Both kept, with #1067's
entry restored to its group and this branch's block following it. All
three entries still name text that exists in those docs.

#1066's removal of the W7900 lane from e2e-selfhosted.yml is untouched by
this branch and carried through as main left it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants