Skip CLI E2E on draft PRs - #4700
Draft
mokagio wants to merge 1 commit into
Draft
Conversation
mokagio
commented
Aug 28, 2026
mokagio
commented
Aug 28, 2026
mokagio
commented
Aug 28, 2026
mokagio
commented
Aug 28, 2026
mokagio
commented
Aug 28, 2026
| if [[ "${BUILDKITE_PULL_REQUEST_DRAFT:-false}" == "true" ]]; then | ||
| message="Draft PR detected. Skipping CLI E2E tests. They will run in full once PR is ready for review." | ||
| if command -v buildkite-agent &> /dev/null; then | ||
| echo "$message" | buildkite-agent annotate --style "info" --context "skip-cli-e2e-draft" |
Contributor
Author
There was a problem hiding this comment.
The LLM suggested to || true here, but I didn't like that. I'd rather know it ASAP if buildkite-agent annotate failed for some reason.
`E2E Tests` already skips drafts through an `if:` guard on the group. `CLI E2E Tests` cannot use one — a required status check that never reports leaves the PR waiting on it — so it runs on 99% of builds, for 859 agent-hours a month across mac, windows and linux. The repo already solved this the other way: `run-cli-e2e-tests.sh` exits 0 early for doc-only PRs and the check still reports green in seconds. This extends that early exit to drafts, reclaiming roughly a third of those hours and up to 20 minutes of wait on every draft PR. `BUILDKITE_PULL_REQUEST_DRAFT` is exported only when the PR is a draft, so the `:-false` default is load bearing. Verified against the job environments of studio builds 21213 (draft, set to `true`) and 21397 (not a draft, absent). The trade: a regression in a draft surfaces when the PR is marked ready rather than on the draft's own build. `E2E Tests` already makes exactly that trade, and CLI E2E still runs in full on the ready PR and on trunk. AINFRA-3003. --- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Code Opus 5 <noreply@anthropic.com>
mokagio
force-pushed
the
ainfra-3003-skip-studio-cli-e2e-on-draft-prs-keeping-the-check-reporting
branch
from
August 28, 2026 03:51
27aa8fe to
4a3b8a8
Compare
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude Code wrote the change and ran the verification below. I reviewed the diff.
Claude Code also noticed the issue this PR addresses, when prompted by me to research opportunities to improve the DX of the tools Apps Infra provides to our "users", CI being a core one of them.
Proposed Changes
E2E Testsskips draft PRs through anif:guard on the group.CLI E2E Testscannot use one — a required status check that never reports leaves the PR waiting on it — so it runs on 99% of builds, costing 859 agent-hours a month across mac, windows and linux.The repo already solved this the other way round:
run-cli-e2e-tests.shexits 0 early for doc-only PRs, and the check still reports green in seconds. This extends that early exit to drafts. Expected saving is roughly a third of those hours, and up to 20 minutes of wait on every draft PR (CLI E2E Tests on windowsis 20.0 min p50).Testing Instructions
Steps the AI took
BUILDKITE_PULL_REQUEST_DRAFTis exported only when the PR is a draft, which is what makes the:-falsedefault load-bearing. Confirmed against the real job environments of two studio builds:BUILDKITE_PULL_REQUEST_DRAFT'true'I also checked that the
if:guard genuinely suppresses reporting, rather than reporting a skip: on build 21213 theE2E Testsgroup was skipped and its commit status is missing entirely from the head SHA's statuses, while the four groups that ran all reported. The comment inpipeline.ymlis right, and the guard has to stay in the script.Locally, the four branches of the new guard, with
should-skip-job.shstubbed so the case is isolated:BUILDKITE_PULL_REQUEST_DRAFTtrueshould-skip-job.shnever reachedshould-skip-job.shfalseTRUEThis PR is opened as a draft on purpose — CLI E2E should skip on it with the annotation
Skipping CLI E2E - draft PR, distinct from the doc-only message. Note the diff is.buildkite/**-only, whichshould-skip-job.shalready treats as non-code, so marking it ready will swap one skip message for the other rather than running the suite in full. That still demonstrates the load-bearing half: the draft guard stops firing once the PR is ready.Pre-merge Checklist