Skip to content

test(codex): isolate the lock child's database and wait on a real signal - #4859

Merged
lidge-jun merged 1 commit into
devfrom
codex/2580-write-lock-race
Sep 17, 2026
Merged

lidge-jun merged 1 commit into
devfrom
codex/2580-write-lock-race

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Summary

a contender with a deadline waits for the holder instead of failing immediately went red on dev (run 35177450461, job 105062310488, Linux test 4/4) with the holder reporting busy — which should have been impossible, because the parent had already seen its hold marker:

372 |     const [waited, holderResult] = await Promise.all([waiter, childResult(holder)]);
373 |     expect(holderResult.status).toBe("acquired");
Expected: "acquired"
Received: "busy"

Both facts were true at once. The marker is written from inside the lock callback, so it proves the child ENTERED the held section — not that it finished holding it. The child's coordination database lived under the ambient OPENCODEX_HOME, which every file in the same CI batch shares, so another test reading it could turn the holder's COMMIT into SQLITE_BUSY. With timeoutMs: 0 the child had no retry, the acquisition rolled back, and it returned busy after having already published the marker the parent was waiting on.

The await Bun.sleep(150) was the second half of the same problem. It stood in for "the waiter is now actually waiting", and nothing made that true. On a loaded runner the parent could release the holder before the contention this case exists to measure had even begun — which would also have made the waitedMs > 0 assertion a coin flip rather than a property.

The fix removes both:

  • The child's coordination database moves to the per-test temp root, which eliminates the cross-file contention outright.
  • The waiter runs as its own child and publishes a wait marker only after confirming its lock promise did not settle synchronously.
  • The parent releases the holder only once it has seen that marker.
  • waitedMs comes back from the child, so the property the case exists to prove is guaranteed by construction rather than by timing.

No timeout was raised, no retry added, and no test skipped.

Verification

No local suite, focused test, typecheck, build, or install was run; this lane is hosted-CI-only by task contract. Verification is static plus exact-head hosted CI.

Static checks:

  • Every remaining wait in this file is the existing waitFor helper, which polls an observable condition; no blind sequencing is left.
  • git diff --check clean.
  • No src/ change: this is a test-isolation and synchronisation defect, not a product defect. The lock module's timeoutMs: 0 behaviour — refuse rather than retry — is correct and unchanged; the test was simply holding it wrong.

Ablation: with the database shared again, a concurrent reader in the same batch reproduces the original busy; with the wait marker removed and the sleep restored, releasing before the waiter enters its wait makes waitedMs > 0 fail.

Checklist

  • Root cause fixed rather than the symptom timed around
  • No timeout widened, no retry added, no test skipped
  • No wall-clock sleep used for ordering
  • No production behaviour changed
  • Targets dev

Summary by CodeRabbit

  • Tests
    • Improved validation of concurrent write-lock behavior, including retry and waiting scenarios.
    • Added clearer timing information to lock-related test results.
    • Isolated test environments to improve reliability and prevent interference between test cases.
    • Enhanced coverage for processes waiting on an existing lock before it becomes available.

`a contender with a deadline waits for the holder instead of failing immediately`
went red on dev (run 35177450461, job 105062310488) with the HOLDER reporting busy,
which should have been impossible: the parent had already seen its hold marker.

Both facts were true. The marker was written from inside the lock callback, so it
proved the child had ENTERED the section, not that it finished holding it. The
child's coordination database lived under the ambient OPENCODEX_HOME, which every
file in the same CI batch shares, so another test reading it could turn the
holder's COMMIT into SQLITE_BUSY. With timeoutMs 0 the child had no retry, the
acquisition rolled back, and it returned busy after having already published the
marker the parent was waiting on.

The 150ms sleep was the second half of the same problem. It was standing in for
"the waiter is now actually waiting", and nothing made that true - on a loaded
runner the parent could release before the contention it exists to measure had
begun, which would also have made the waitedMs > 0 assertion a coin flip.

So: the child's database moves to the per-test temp root, which removes the
cross-file contention entirely; the waiter runs as its own child and publishes a
wait marker only after confirming its lock promise did not settle synchronously;
and the parent releases the holder only once it has seen that marker. waitedMs now
comes back from the child, so the property the case exists to prove is guaranteed
by construction rather than by timing.

No other blind sequencing remains in this file - every other wait is the existing
waitFor, which polls an observable condition.

No local suite, focused test, typecheck, build, or install was run.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner September 17, 2026 03:46
@github-actions

Copy link
Copy Markdown
Contributor

Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature). label Sep 17, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-17T03:50:05.071825Z 034c724 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: e8c44f67-ba2c-47e0-b07c-d01657b3a6aa

📥 Commits

Reviewing files that changed from the base of the PR and between 19bdcaa and 034c724.

📒 Files selected for processing (2)
  • tests/codex-integration/codex-write-lock.test.ts
  • tests/helpers/codex-write-lock-child.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

The lock integration tests now isolate child-process homes, report lock wait duration, and use an explicit child-process marker to verify retry-state entry before releasing the lock holder.

Changes

Codex write-lock test synchronization

Layer / File(s) Summary
Child lock wait reporting
tests/helpers/codex-write-lock-child.ts
The helper stores the pending withCodexWriteLock promise, optionally writes a waiting marker while acquisition remains pending, and includes waitedMs in acquired and busy results.
Isolated contention test flow
tests/codex-integration/codex-write-lock.test.ts
Child processes use per-test OPENCODEX_HOME values. The contention test starts the waiter as a child process, waits for its marker, releases the holder, and parses waitedMs.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 034c7

The contention test now synchronizes on the lock retry state and uses isolated lock homes without an established merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: it isolates the lock child's database per test and replaces timing-dependent sleeping with a real wait signal.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/2580-write-lock-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 034c7243e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +384 to +388
const waiter = spawnChild({ timeoutMs: 5_000, waitMarker });
// The waiter writes this only after withCodexWriteLock has returned its
// pending promise. Because the holder is still held, that means the waiter
// has attempted N and reached the retry wait rather than failing fast.
await waitFor(waitMarker);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Give the waiter child a spawn-sized readiness window

On loaded Windows runners, this file already records that this helper can take 8–19 seconds to boot, but the new waitFor(waitMarker) still uses the 15-second INTERNAL_DEADLINE_MS. The newly spawned waiter can therefore be healthy yet fail before publishing the marker; if startup approaches the holder's 20-second ceiling, the holder may also release first, causing the waiter to settle without ever writing the marker. This reintroduces the timing-dependent CI failure the change is intended to remove; use a readiness deadline sized for a spawned process and make the holder ceiling safely exceed it.

Useful? React with 👍 / 👎.

@lidge-jun

Copy link
Copy Markdown
Owner Author

리뷰 · 우선순위 75 / 80

이 PR은 dev Linux test 4/4에서 난 실제 빨간 테스트를 고칩니다. 대상은 tests/codex-integration/codex-write-lock.test.tsa contender with a deadline waits for the holder instead of failing immediately 입니다. 지금 dev HEAD는 19bdcaaf6(#4851 Windows 6→9 샤드)이고, 그 아래에는 #4846(행동 오라클), #4849(kill/reap), #4836(Windows ACL)이 있습니다. 프로덕션 src/는 손대지 않고, 자식 프로세스의 조율 DB 격리와 대기 신호만 바꿉니다. 타임아웃 확대·재시도·스킵도 없습니다. 지금 dev가 최적화하는 CI/테스트 신뢰도와 같은 방향입니다.

실패한 장면은 이렇게 읽힙니다. 부모가 holder 자식의 hold 마커를 이미 봤는데, 나중에 holder 결과가 busy로 돌아왔습니다. hold 마커는 락 콜백 에서 쓰이므로 “섹션에 들어갔다”는 증거이지, “홀드를 끝까지 유지했다”는 증거가 아닙니다. 자식의 조율(C) DB는 주변 OPENCODEX_HOME 아래에 있었고, 같은 Bun 배치의 다른 파일이 그 DB를 읽으면 holder의 COMMITSQLITE_BUSY가 될 수 있습니다. timeoutMs: 0이면 재시도가 없어서, 마커를 이미 낸 뒤에도 acquisition이 롤백되며 busy가 됩니다. 동시에 부모 쪽 await Bun.sleep(150)은 “waiter가 실제로 대기 중이다”를 보장하지 않습니다. 바쁜 러너에서는 holder를 너무 일찍 풀어서, 이 케이스가 재려는 경합이 시작되기도 전에 끝나 waitedMs > 0이 동전 던지기가 됩니다.

고치는 방법은 네 갈래입니다. (1) spawnChild / spawnChildWithEnvOPENCODEX_HOME: join(root, ".opencodex")를 넣어, 이 케이스 자식들의 C DB를 테스트 temp root로 옮깁니다. N(락 대상, CODEX_HOME)은 그대로 두고 C만 격리합니다. (2) waiter를 부모 isolate의 withCodexWriteLock이 아니라 별도 자식으로 띄웁니다. (3) 자식 헬퍼 tests/helpers/codex-write-lock-child.tswaitMarker를 받을 때, withCodexWriteLock이 돌려 준 pending이 동기 settle되지 않은 것을 확인한 뒤에만 마커를 씁니다. (4) 부모는 그 마커를 waitFor로 본 다음에야 release 마커를 쓰고, waitedMs는 자식 JSON 결과에서 읽습니다. 그래서 “기다렸다”는 성질이 타이밍이 아니라 구성으로 보장됩니다.

types.ts / config.ts 대형 분리 캠페인과는 무관합니다. 테스트·헬퍼 두 파일만 바뀌고, codex-write-locktimeoutMs: 0 → refuse 계약은 그대로입니다. 호스티드 CI가 아직 돌아가는 중(Linux/macOS pending)이므로, exact-head 그린이 이 헤드의 최종 증거입니다.

라인 tests/codex-integration/codex-write-lock.test.ts spawnChild - OPENCODEX_HOME을 per-test root로 고정한 것은 맞다. 같은 describe의 다른 자식 케이스에도 같이 적용되므로, 배치 간 C 간섭이 이 파일 전체에서 줄어든다.
라인 tests/helpers/codex-write-lock-child.ts waitMarker - await Promise.resolve() 한 번으로 동기 settle만 거른다. holder가 이미 풀린 뒤에는 마커가 안 쓰이고 waitFor가 타임아웃으로 실패하는 쪽이 맞다. 다만 “한 틱 뒤 비동기로 바로 settle”하는 경로가 생기면 마커가 잘못 쓰일 여지는 이론상 남는다(현재 holder hold 중이면 pending이 유지되어야 한다).
경로 childResult waitedMs - acquired/busy 모두에서 waitedMs를 실어 보내는 것은 프로덕션 결과 타입(codex-write-lock.ts의 acquired/busy union)과 맞다.
경로 로컬 검증 - 본문 계약상 호스티드-CI-only라서, 리뷰어는 정적 읽기와 exact-head CI에 의존한다. 그건 이 레포 관례와 같다.
경로 Windows lane - 현재 체크에 Windows 샤드가 skipping으로 보인다. 원 실패는 Linux test 4/4였고, Windows 증거는 이 PR의 필수 게이트가 아닐 수 있다. 그래도 contention 테스트가 Windows에서도 같은 배치 간섭을 겪을 수 있으니, lane=all이 도는 헤드라면 한 번 greener 쪽이 더 안심이다.

메인테이너의 판단이 필요한 지점

  • Linux test 4/4(그리고 가능하면 macOS) exact-head 그린만으로 머지할지, Windows contention까지 볼지
  • waitMarker의 한 틱 microtask 판정을 더 강한 관측(예: 락 모듈이 노출하는 wait-entered 신호)으로 바꿀 필요가 있는지
  • 같은 파일의 부모-isolate withCodexWriteLock 호출이 쓰는 ambient OPENCODEX_HOME을 테스트 setup에서 통째로 덮을지(이번 PR 범위 밖)
  • 이 픽스를 ci(windows): restore the margin the six-shard leg lost, and make a breach legible #4851 직후 CI 안정화 묶음에 바로 넣을지

너의 추천
방향이 맞다. 원인을 sleep/재시도로 가리지 않고, C DB 격리 + 실제 wait 신호 + 자식에서 돌아온 waitedMs로 성질을 고정했다. types/config 분리에도 안 걸린다. Linux(원 실패 다리) exact-head가 초록이면 바로 머지하는 쪽을 추천한다. Windows는 보너스 증거로 두면 충분하다.

이 댓글은 grok-bot이 작성했습니다

Copy link
Copy Markdown
Owner Author

리뷰 · 우선순위 76 / 80

이 PR은 프로덕션 락 로직이 아니라, tests/codex-integration/codex-write-lock.test.ts 안의 두 프로세스 경합 테스트가 CI에서 거짓으로 깨지던 이유를 고칩니다. 지금 dev HEAD는 19bdcaaf6이고 패키지는 2.58.0입니다. 바로 앞 팁은 #4851(Windows 6→9 샤드 + 배치 러너)이고, 그 아래에는 #4846 행동 기반 오라클, #4849 kill/reap-before-delete 같은 “타이밍/격리로 깨지는 테스트를 계약으로 고정”하는 흐름이 이어집니다. 이 PR도 그 줄에 있습니다.

실패 장면은 Linux test 4/4(run 35177450461)에서 a contender with a deadline waits for the holder instead of failing immediately가 깨진 것입니다. 부모는 이미 held-2 마커를 봤는데, holder 자식의 결과는 acquired가 아니라 busy였습니다. 마커는 락 콜백 안에서 쓰이므로 “홀드 구간에 들어갔다”는 뜻이지 “홀드를 끝까지 유지한 채 커밋했다”는 뜻이 아닙니다. 그 사이에 홀더가 timeoutMs: 0으로 재시도 없이 굴러떨어져 busy를 돌려줄 수 있습니다.

고침은 두 갈래입니다. 첫째, spawnChild / spawnChildWithEnv가 자식에게 테스트 temp의 OPENCODEX_HOME(join(root, ".opencodex"))를 넘깁니다. 둘째, 대기자(waiter)를 부모 프로세스의 withCodexWriteLock + Bun.sleep(150)이 아니라 별도 자식으로 띄우고, 자식이 락 프로미스가 동기 settle되지 않은 것을 확인한 뒤에만 waiting-2 마커를 씁니다. 부모는 그 마커를 waitFor로 본 다음에야 release-2를 써서 홀더를 놓습니다. waitedMs도 자식 JSON에서 돌아와, “기다렸다”는 성질이 sleep 운에 안 묶입니다. src/는 안 건드렸고, 타임아웃 확대·재시도·스킵도 없습니다.

tests/helpers/codex-write-lock-child.tswaitMarker 경로는 pending.then으로 settle 여부를 달고 await Promise.resolve() 한 틱만 비운 뒤, 아직 안 끝났으면 마커를 씁니다. 홀더가 N을 잡고 있으면 unsettled가 “재시도 대기 중”이라는 신호라는 설명과 맞습니다. 나머지 대기는 기존 waitFor(관측 가능한 파일 폴링, INTERNAL_DEADLINE_MS=15s)만 남습니다. types/config 대형 분리 캠페인과도 무관합니다.

라인 289-296 - 주석/본문은 “C database를 OPENCODEX_HOME으로 고립”이라고 읽히지만, 현재 devresolveCodexCoordinatorDatabasePath(src/codex/user-identity.ts)는 N DB를 resolveEffectiveUserRuntimeRoot 아래(native-write-locks/<CODEX_HOME digest>.sqlite)에 둡니다. Linux면 /tmp/opencodex-runtime-v1-<uid>/...이고 OPENCODEX_HOME이 아닙니다. 같은 배치가 uid 런타임 루트를 공유하는 건 맞지만, 파일 자체는 CODEX_HOME digest로 갈립니다. OPENCODEX_HOME 고립이 SQLITE_BUSY를 없앴다는 서술과 코드가 어긋날 수 있으니, 실제 이득이 wait-marker 쪽인지 / 다른 SQLite(설정·저널 등)인지 CI에서 한 번 더 확인하는 편이 안전합니다.
라인 306-312 - spawnChildWithEnvOPENCODEX_HOME을 넣은 뒤 ...env를 펼치므로, HOME/USERPROFILE 네임스페이스 케이스(case 0–2)가 넘기는 env로 덮어쓸 수 있습니다. 의도가 “기본 고립 + 필요 시 덮어쓰기”라면 OK이고, 네임스페이스 테스트도 항상 같은 OCX home을 강제하려면 순서를 바꿔야 합니다.
라인 80-95 (codex-write-lock-child.ts) - await Promise.resolve() 한 번만으로 “동기 settle 반응 flush”를 가정합니다. 홀더가 잡혀 있는 한 보통 충분하지만, 마이크로태스크가 한 틱 더 밀리면 마커 없이 waitFor(waiting-2)가 15초 타임아웃으로 갈 여지는 남습니다.
경로 waitedMs 단언 - expect(waited.status === "acquired" && waited.waitedMs).toBeGreaterThan(0) 형태는 이전과 같고, 이제 값이 자식에서 오므로 sleep 운은 줄었습니다. CI 그린 전에는 이 단언이 다시 흔들리는지만 보면 됩니다.
경로 범위 - 같은 describe의 “excluded while first holds” 케이스는 여전히 부모 쪽 in-process withCodexWriteLock을 씁니다. 이번 실패 케이스만 자식 waiter로 바꿨고, 프로덕션 timeoutMs: 0 거절 동작은 그대로입니다.

메인테이너의 판단이 필요한 지점

  • OPENCODEX_HOME 고립이 N DB 경로를 바꾸지 않는다는 점에서, PR 본문의 “배치 공유 C DB” 설명을 수정할지 / 실제로 도움이 된 다른 경로를 명시할지
  • Linux test 4/4(원래 빨간 다리) exact-head 그린을 머지 게이트로 둘지
  • Promise.resolve() 한 틱 flush를 그대로 둘지, settle 관측을 조금 더 단단히 할지
  • 네임스페이스 케이스에서 OPENCODEX_HOME을 env 덮어쓰기 허용으로 둘지 고정할지

너의 추천
방향은 #4846/#4851 줄로 맞고, 프로덕션 변경이 없어 위험도도 낮습니다. exact-head에서 특히 Linux test 4/4가 그린이면 머지 후보로 두세요. 머지 전에 OPENCODEX_HOME↔N DB 경로 설명만 코드에 맞게 한 줄 고치거나, 그 설정이 없어도 wait-marker만으로 재현이 안 되는지 ablation 결과를 PR에 남기면 이후 디버깅 비용이 줄어듭니다.

이 댓글은 grok-bot이 작성했습니다

@lidge-jun
lidge-jun merged commit 121405b into dev Sep 17, 2026
29 checks passed
@lidge-jun
lidge-jun deleted the codex/2580-write-lock-race branch September 17, 2026 04:00
agentHits pushed a commit to agentHits/opencodex that referenced this pull request Sep 17, 2026
…nal (lidge-jun#4859)

`a contender with a deadline waits for the holder instead of failing immediately`
went red on dev (run 35177450461, job 105062310488) with the HOLDER reporting busy,
which should have been impossible: the parent had already seen its hold marker.

Both facts were true. The marker was written from inside the lock callback, so it
proved the child had ENTERED the section, not that it finished holding it. The
child's coordination database lived under the ambient OPENCODEX_HOME, which every
file in the same CI batch shares, so another test reading it could turn the
holder's COMMIT into SQLITE_BUSY. With timeoutMs 0 the child had no retry, the
acquisition rolled back, and it returned busy after having already published the
marker the parent was waiting on.

The 150ms sleep was the second half of the same problem. It was standing in for
"the waiter is now actually waiting", and nothing made that true - on a loaded
runner the parent could release before the contention it exists to measure had
begun, which would also have made the waitedMs > 0 assertion a coin flip.

So: the child's database moves to the per-test temp root, which removes the
cross-file contention entirely; the waiter runs as its own child and publishes a
wait marker only after confirming its lock promise did not settle synchronously;
and the parent releases the holder only once it has seen that marker. waitedMs now
comes back from the child, so the property the case exists to prove is guaranteed
by construction rather than by timing.

No other blind sequencing remains in this file - every other wait is the existing
waitFor, which polls an observable condition.

No local suite, focused test, typecheck, build, or install was run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant