Skip to content

fix: deflake load-sensitive tests and retry Devin OAuth callback reads - #7482

Open
jiroamato wants to merge 3 commits into
router-for-me:devfrom
jiroamato:test/deflake-load-sensitive-tests
Open

jiroamato wants to merge 3 commits into
router-for-me:devfrom
jiroamato:test/deflake-load-sensitive-tests

Conversation

@jiroamato

Copy link
Copy Markdown

This PR fixes five tests that fail intermittently when go test ./... runs on a loaded Windows host. Four fixes change test files only. One changes production code. The Devin OAuth flow now retries a failed read of its callback file instead of ending the login.

1. Incomplete request body test

TestServerRejectsIncompleteUnauthenticatedRequestBodiesPromptly in internal/api fails on dev with

protocol_multiplexer_test.go:328: server did not close connection promptly (err=read tcp 127.0.0.1:39241->127.0.0.1:39238: i/o timeout, partial="")
protocol_multiplexer_test.go:360: authenticated request failed: Post "http://127.0.0.1:39238/v1/chat/completions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

A goroutine dump at the failure showed the handler blocked in the CreateFile syscall in logging.writeRequestInfoWithBody, before it wrote any response bytes. Under heavy parallel I/O, Windows stalls that file open for several seconds. The test allowed 500ms on the raw connection and 2s on the follow-up HTTP client.

Both bounds are now 15s. The server sets no body read timeout, so a server that waits for the missing body still fails the test.

2. Antigravity blocked persistence test

TestAntigravityBlockedHintPersistenceDoesNotBlockPublicationOrOtherAccounts in sdk/cliproxy fails on dev with

panic: close of closed channel [recovered, repanicked]
	sdk/cliproxy/antigravity_models_registration_test.go:868

The last timeout branch closes store.releaseSave, which the test already closed 22 lines earlier. A slow run panics there and the panic hides the real failure message.

I removed the second close and raised the six 1s time.After bounds in this test to 15s. Each of those selects already waits on a channel, so the bound only applies when the test fails. #6677 edits a different part of this test, and git merge-tree of this branch with the #6677 head reports no conflict.

3. OAuth status poll helper

pollOAuthStatusUntilDone in internal/api/handlers/management fails on dev with

--- FAIL: TestTargetedOAuthReloginClaudeXAIMetaDevinAndAuthIndexVerify (6.66s)
    auth_files_targeted_oauth_test.go:664: timed out waiting for oauth session 8OK-wGp8pOkxoHIvv8hffcANo5pI0uUr7_y2b1jel4g

Five tests use this helper. It requested the status endpoint in a loop with no pause and gave up after 5s. The OAuth flow it waits for checks its callback file every 500ms, and the busy loop took CPU from that goroutine.

The helper now waits on a 10ms ticker between requests and uses a 15s deadline timer, as TestDevinRemoteOAuthFlow already does. The session store has no completion signal a test could wait on.

4. Devin OAuth callback read

TestDevinRemoteOAuthFlow fails with

auth_files_devin_oauth_test.go:131: token exchange did not start

and the Devin step of TestTargetedOAuthReloginClaudeXAIMetaDevinAndAuthIndexVerify fails with

auth_files_targeted_oauth_test.go:738: devin match status = map[string]interface {}{"error":"Failed to read OAuth callback", "status":"error"}, want ok & adopted

This one is a production bug. The callback handler writes a temp file and renames it into place. On Windows, a read of the destination path that lands during the rename fails with a sharing violation. waitDevinOAuthCallback treated every read error except not-exist as fatal and set the session to Failed to read OAuth callback, so the login failed although the callback had arrived. A standalone program that reads a path while another goroutine renames a file onto it got 6775 of these errors in 3000 renames.

waitDevinOAuthCallback now retries on the next 500ms tick for any read error. The Claude, Codex and Antigravity flows already do this. The context deadline and the session state still end the wait. TestWaitDevinOAuthCallbackRetriesTransientReadError covers it and fails on the old code with transient read error ended the login: Failed to read OAuth callback. TestDevinRemoteOAuthFlow now prints the session status when the exchange does not start, and its two 5s bounds are 15s.

5. Watcher config reload test

TestHandleEventConfigChangeSchedulesReload in internal/watcher failed in an earlier full run with

expected config change to trigger reload once, got 0

followed by a TempDir RemoveAll cleanup error on config.yaml. The test slept 400ms and then read the counter, which races the 150ms configReloadDebounce timer. When the timer fired late, the reload was still reading config.yaml during cleanup.

reloadCallback now signals a channel and the test waits on it. The test then takes configApplyMu, which the reload holds until it returns, so the reload has finished before the assertion and the cleanup. The sleep is gone.

Verification

Windows 11, Go 1.27.2, 16 logical CPUs.

  • One go test ./... on unmodified dev at 05fd819 reproduced failures 1, 2 and 3 with the output quoted above.
  • Before the Devin fix, two loop runs of the management tests below each had 1 failure in 150 executions. Those are the two errors quoted in section 4.
  • go build ./cmd/server and go vet on the four touched packages pass.
  • Each touched test ran 150 times, -count=50 -cpu=1,2,8, with the four packages running at the same time. All passed.
    • internal/api, 1 test, 21.6s
    • sdk/cliproxy, 1 test, 36.7s
    • internal/watcher, 1 test, 25.8s
    • internal/api/handlers/management, 9 tests, 474.5s. These are TestDevinRemoteOAuthFlow, both TestWaitDevinOAuthCallback* tests, TestCompleteDevinOAuthFailures, the four TestTargetedOAuthRelogin* tests and TestListAuthFiles_V8CodexTerminalRenewalRecoveryWorkflow_Issue5783.
  • go test ./... ran twice on this branch. No test from this PR failed in either run.

Both full runs still exit 1 on this host because of tests this PR does not touch. These six packages fail on unmodified dev too, in every run:

  • cmd/server, TestTUIClientFailureExitsWithFailure and TestTUIStandaloneFailureExitsWithFailure
  • internal/home, internal/kvstore and internal/pluginhost, four tests with TempDir RemoveAll cleanup errors
  • internal/translator/claude/openai/responses, TestToolSearchDisabledMatchesDevGolden
  • internal/util, TestInPlaceByteWritesAreReviewed, which Review weighted selector member copies in the no-copy guard #7467 covers

Run 1 also had TestServerResponsesWebsocketReauthorizesEachTurn fail on a TempDir RemoveAll cleanup error, and two management tests fail in waitForOAuthStatus, a second status helper with a 5s deadline that this PR leaves alone.

- Incomplete-body test: widen the 500ms connection deadline to a 15s failure
  bound. Under full-suite load the handler stalls opening the request log
  temp file before it writes any response.
- Antigravity blocked-persistence test: stop closing releaseSave a second
  time in the final timeout branch, where the panic replaced the real failure
  message, and widen the 1s failure bounds.
- pollOAuthStatusUntilDone: pace the status polls with a ticker instead of a
  busy loop that starves the OAuth flow goroutine, and widen the deadline.
- Devin remote OAuth test: widen the 5s failure bounds.
- Watcher config reload test: wait for the reload callback and for the reload
  to return instead of sleeping 400ms against the 150ms debounce timer.
waitDevinOAuthCallback ended the login with "Failed to read OAuth callback"
on any read error other than not-exist. On Windows a read that lands while the
callback handler renames the file into place fails with a sharing violation,
so a login could fail although the callback arrived. Retry on the next tick,
as the other OAuth flows already do; the context deadline and session state
still bound the wait.

This was the cause of the intermittent "token exchange did not start" in
TestDevinRemoteOAuthFlow. The test now reports the session status on that
failure.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T06:36:35.354330Z a50f54e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a50f54eb68

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// into place fails with a sharing violation. The next read succeeds.
reads := 0
originalRead := readDevinOAuthCallbackFile
readDevinOAuthCallbackFile = func(string) ([]byte, error) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Scope the callback-reader override to a single waiter

In a shuffled package run, TestStartOAuthV8TargetedSessionRegistrationAcrossBuiltinProviders can run immediately before this test. Its Devin subtest only calls CancelOAuthSession, which deletes the session without joining completeDevinOAuth, so that goroutine may still be reading the global hook at auth_files_devin_oauth.go:152 when it is replaced or restored here. This introduces a data race, and a leftover waiter reaching the stub can also consume its first read and break the reads == 2 assertion. Inject the reader per waiter rather than replacing a package-wide function.

Useful? React with 👍 / 👎.

jiroamato added a commit to jiroamato/CLIProxyAPI that referenced this pull request Nov 12, 2026
waitDevinOAuthCallback ended the login with "Failed to read OAuth callback"
on any read error other than not-exist. On Windows a read that lands while the
callback handler renames the file into place fails with a sharing violation,
so a login could fail although the callback arrived. Retry on the next tick,
as the other OAuth flows already do. The context deadline and session state
still bound the wait.

This was the cause of the intermittent "token exchange did not start" in
TestDevinRemoteOAuthFlow. The test now reports the session status on that
failure, and its 5s failure bounds are 15s.

Backport of upstream router-for-me#7482 (a50f54e), with the Devin
test bound change from 97b8a03 that the regression test depends on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant