Repository navigation
rebase: make reports and metrics truthful - #131
Conversation
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e27ff92523
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| llm_requests = sum(1 for ev in events | ||
| if ev.get("kind") == "rebase_llm_request") | ||
| usage_missing = span_cost is None and llm_requests > 0 \ | ||
| and tokens_in == 0 and tokens_out == 0 |
There was a problem hiding this comment.
When a repo-rebase local test triggers a Tier-2 watchdog review, _watchdog_collaborators.review_fn calls client.messages.create directly without producing either an llm span or a rebase_llm_request event. Consequently this check remains false, and the paid request is reported as zero token/USD cost with cost_partial: false, biasing experiment CATQ upward. Instrument those requests or include a durable watchdog-review signal in the partial-cost test.
Useful? React with 👍 / 👎.
| manifest_jobs = int(ctx.state.get("manifest_jobs") or 0) | ||
| total = manifest_jobs or passed + failed + skipped |
There was a problem hiding this comment.
In the locked workflow, phase5_report runs only in full mode, while the only producer of manifest_jobs is rebase.v3_scan, which runs only in report-only mode. Full runs therefore always take this fallback; when the manifest contains nightly jobs, rebase.v3_test_loop excludes them, so the report labels only the local pipeline total as “manifest jobs” and omits those unrun jobs. Publish len(built.jobs) from the full test-manifest construction or derive the count from a persisted artifact.
Useful? React with 👍 / 👎.
| f"- raw provider state: {round_.get('build_state') or 'unknown'} " | ||
| f"(build {build}; {round_.get('passed', 0)} passed, " | ||
| f"{len(round_.get('failed') or [])} unresolved, " | ||
| f"{round_.get('ignored_baseline', 0)} baseline, " |
There was a problem hiding this comment.
When monitoring ends with scheduled/running jobs or per-job budget timeouts, _record_outcome stores them in incomplete and budget_timeouts, not in failed. This report therefore prints 0 unresolved for rounds that still have unresolved jobs, contradicting the raw provider state and potentially misleading operators reviewing a failed/no-signal run. Count the unresolved union rather than only failed.
Useful? React with 👍 / 👎.
Summary
Tests
pytest -qsuitegit diff --check