acc: deflake dashboard read-after-deploy retry#5895
Merged
Conversation
The bundle/resources/dashboards acceptance test flaked on cloud: after deploy, reads of a just-renamed/updated dashboard hit eventual consistency and retry.py's default window (5 attempts x 500ms ~= 2s) wasn't enough. retry.py: correct condition handling on exhaustion. The old loop always fell through to writing the last attempt's stdout to stdout, even when a --until/--until-not condition was never satisfied, letting stale output pass through as if valid and masking the flake as a downstream output diff. Now: on success, write the successful attempt's output to stdout and exit; if a content condition was requested but never met, write the stale output to stderr (never stdout) and exit non-zero; with no content condition, pass the final attempt's output and exit code through unchanged. test.toml: widen the retry window to RETRY_MAX_ATTEMPTS=20 and RETRY_INTERVAL_MS=1000 (~20s). retry.py's loop isn't scaled by TimeoutCIMultiplier (that only scales the overall test timeout), and real cloud rename/update eventual consistency regularly exceeds the ~2s default. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 1b1dbbf
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 10 slowest tests (at least 2 minutes):
|
janniklasrose
approved these changes
Jul 13, 2026
Collaborator
Integration test reportCommit: 9b498de
36 interesting tests: 17 FAIL, 14 flaky, 2 RECOVERED, 2 SKIP, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
bundle/resources/dashboardsacceptance test flaked on cloud: after deploy, reads of a just-renamed/updated dashboard hit eventual consistency, andretry.py's default ~2s window (5 × 500ms) wasn't enough.retry.py: On exhaustion, the old loop passed the last attempt's stdout through as if valid even when a--until/--until-notcondition never held — masking the flake as a downstream output diff. Now, an unmet content condition writes stale output to stderr and exits non-zero (retry: condition not met after N attempts); success and the no-condition case pass output through on stdout unchanged.test.toml: Widen the window toRETRY_MAX_ATTEMPTS=20/RETRY_INTERVAL_MS=1000(~20s).retry.py's loop isn't scaled byTimeoutCIMultiplier, and real cloud eventual consistency regularly exceeds the ~2s default.This pull request and its description were written by Isaac.