[tzhouam]

Why

The vLLM v0.33.0 rebase campaign (run-20261028-152707-a60091) cannot finish cleanly for three engine reasons:

1. imx-omni-pytest added only a hardware marker, so a broad module run collected advanced_model / full_model items and ran them at the default --run-level core_model (dummy weights). online_serving failed with 60 errors (duplex fixture requires real weights) and 17 dummy-audio failures. Buildkite always pairs -m <level> with --run-level <level>.
2. Ready's CPU "Simple" steps budget 20 minutes for CI agents. On the shared rebase host three of them passed 53-88% of their items with zero failures and were killed, which records structural infra failures the push gate blocks on.
3. A resume cannot re-execute a completed step, so failed wave-1 modules and recorded timeouts stay terminal even after the cause is fixed. The only recovery was hand-editing progress.json / substate.json.

What

• module_pytest: with no caller -m, inject --run-level core_model and (<platform> or neutral) and core_model; an explicit --run-level selects that level instead. Explicit -m is still authoritative.
• rebase.testing.local_timeout_multiplier (adapter data, default 1, must be >= 1): scales TestJob.timeout_sec in the local test loop only. The built manifest keeps the CI value (shell golden unchanged). vllm-omni sets 3.
• --resume RUN --rerun-from STEP_ID: the executor drops that step and every later one from the checkpoint and records a rerun_from trace event; an unknown id blocks without touching the checkpoint. --rerun-from without --resume is a CLI error.
• run_test_loop(retry_infra=...): the v3 test loop passes True only under --rerun-from, so recorded timeouts run again while passed jobs are not rerun. A plain resume keeps infra sticky.

SPEC pages updated: executor, rebase_v3, rebase_engine, app/core, app/workflow_execution, cli.

Tests

• New: executor rerun-from (re-executes later steps; unknown id blocks), CLI forwarding and --resume requirement, test loop retry_infra, timeout multiplier validation, wrapper run-level pairing.
• Full suite pytest test/ green; check_spec_freshness --strict, check_doc_links, check_doc_citations clean.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-29T09:41:03.213451Z	03c978b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03c978b82b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Scale embedded shell timeouts in the local execution copy

For the vllm-omni CPU Simple jobs, increasing only TestJob.timeout_sec leaves the effective timeout unchanged: adapters/vllm_omni/rebase/shell_golden.json records simple_diffusion_model_executor_test and simple_other_test commands beginning with timeout 20m pytest, and _extract_steps preserves those wrappers. As timeout --help states, it will “kill it if still running after DURATION,” regardless of the larger outer runner budget. These jobs therefore still terminate after 20 minutes with the new multiplier set to 3; a scaled-down runner reproduction returned rc 124 with timed_out=False well before the enlarged budget. Scale or remove embedded timeout wrappers in the local execution copy as well, while preserving the original manifest for CI parity.

Useful? React with 👍 / 👎.