[khaliqgant]

Problem

In production, relayfile-mount --push-local-once seeds a 13,068-file (41 MB) tree in 206–218s. The server sees about 100s with no requests, then roughly 6 /fs/bulk POSTs, one at a time.

The local backlog benchmark below shows the same pattern: 6 POSTs, the first one at 61–66s, and 119 GB allocated. Almost all of that is chunkOutboxRecords re-marshalling the whole pending chunk each time it adds one record. WriteFilesBulk does the same thing again for each request. On an Apple M3 the baseline drain overruns the default 60s outbox flush budget (context deadline exceeded).

Changes

1. Linear batch sizing. chunkOutboxRecords, WriteFilesBulk (via bulkWriteChunkBounds) and chunkPendingBulkWrites now encode each file once. They track the size as {"files":[ + items + commas + ]}, and bulkWriteRequestSizer copies the old "0 on encode error" result. The old functions are kept in the tests as reference implementations. Over a mixed corpus (empty, HTML-escaped <&>, U+2028, multibyte, base64, symlinks, draft content identities, a near-cap file and an over-cap file), with caps from 0 to 8 MiB, the tests assert that files per request and bytes per request are identical. A mutation check (an off-by-one in the comma count) fails these tests.
2. New files are read once. When pushLocal scans, it reads small untracked regular files whole and reuses that content instead of hashing first and reading again. Limits: 1 MiB per file, 64 MiB per scan, never in low-memory mode, never for lazy-skipped files. The scan map keeps its hash-only shape, built from the same read, so the hash and the content always match. Tracked files still go through hash first, then read. Outbox writes and durability are unchanged.
3. Bounded concurrent POSTs. flushOutboxChunks keeps up to RELAYFILE_MOUNT_BULK_UPLOAD_CONCURRENCY POSTs in flight (default 2, 1 = serial, clamped to 8). The default is 2 because Cloud's per-workspace in-flight body budget is 16 MiB, so a third 8 MiB request would only buy workspace_busy retries. flushOutboxRecordChunk is split into three steps: prepare, upload, apply.
   • Only the upload of the immutable request runs on a goroutine.
   • Prepare and apply run on the caller, which holds s.mu the whole time, in chunk order.
   • Acks come only from server responses.
   • A chunk that shares a path with an in-flight chunk waits for it.
   • After an error, no new chunks are dispatched (same as before). Uploads already in flight are still applied, so writes the server accepted get acked and are not re-sent.
   • Serial behaviour is unchanged for single chunks and for the watcher path (unlockDuringUpload).
   • 429 handling is unchanged: doJSON retries with Retry-After, and once retries run out, incrementOutboxAttempt applies backoff to that chunk only.
4. No third tree hash. PushLocalAndFlushOnce builds the public state from pushLocal's scan instead of calling saveState(), which re-hashes every file.

Not changed

• pushLocal flushes every 256 files, and each of those flushes is a single chunk of about 768 KB. Concurrency only helps when one flush drains several chunks: the outbox backlog drain used by --push-local-once and --flush-outbox-once, and the drain during a --notify-flush reconcile. Overlapping pushLocal's own batches would mean applying batch results later while pushLocal keeps mutating state. That is a bigger semantic change, so I left it as a possible follow-up.
• Each record still causes two outbox file writes (pending, then acked). ensureOutboxDirs also re-reads capabilities.json on every ack. Both are now the largest local costs, and I didn't touch them because they are part of how durability works.

Benchmarks (Apple M3, real httpapi handler + relayfile.Store, 13,000 files × 3 KiB, -benchtime 1x)

Benchmark	main	this PR
FlushOutboxOnceLargeBacklog wall	109.1s / 112.7s	11.0–12.0s
— first /fs/bulk after	60.8s / 66.2s	0.6–0.8s
— allocated	118.9–120.1 GB	1.88 GB
— POSTs / max in flight	6 / 1	6 / 2
PushLocalAndFlushOnceLargeTree wall	27.6s / 28.3s / 31.1s	14.3s / 14.5s / 17.7s
— allocated	15.8 GB	2.05 GB
— allocs	11.8M	7.65M

The backlog benchmark sets outboxFlushTimeout = 1h so that main can finish; with the default 60s budget main fails.

Commands

GOMODCACHE=/tmp/relayfile-gomod go test ./internal/mountsync -run '^$' -bench 'LargeTree|LargeBacklog' -benchtime 1x -count 2 -benchmem   # on main (benchmark file copied in) and on this branch
go vet ./...                                   # clean
gofmt -l internal/ cmd/                        # only internal/writeback/dead-letter.go, which is unformatted on main too
go test ./... -count=1                         # all packages ok (mountsync 95s)
go test ./internal/mountsync -race -count=1 -run 'FlushOutboxChunks|PushLocal|MatchesLegacy|Sizer|ReadsNewFileOnce|Capture'   # ok

New tests:

• TestChunkOutboxRecordsMatchesLegacyBoundaries, TestBulkWriteChunkBoundsMatchesLegacyBoundaries, TestChunkPendingBulkWritesMatchesLegacyBoundaries, TestBulkWriteRequestSizerMatchesMarshal
• TestFlushOutboxChunksBoundsInFlightAndAcksEachRecordOnce: exactly 3 in flight, every path uploaded and acked once.
• TestFlushOutboxChunksWorkspaceBusyKeepsAccountingExact: a 429 workspace_busy chunk stays pending with backoff, the in-flight chunk next to it is acked, later chunks are not dispatched, and recovery uploads each path exactly once.
• TestFlushOutboxChunksSerializesSamePathAcrossChunks
• TestFlushOutboxChunksSerialWhenConcurrencyIsOne
• TestPushLocalAndFlushOnceReadsNewFileOnce: before, a new file was read 3 times; now once. An oversized file is still reported as writeback-skipped.
• TestPushLocalCapturesOnlyUntrackedFiles, TestPushLocalLowMemoryDoesNotCaptureContent

Mutation checks: removing the in-flight drain after an error fails the 429 test, and removing the same-path gate fails the same-path test.

Reviewer notes / risks

• Receipt timing against servers that return opIds. The in-process Go store gives every write an opId and dispatches it asynchronously. With 2 POSTs in flight, the receipt GET sometimes runs before the store has marked a few ops as done (14–25 of 13k in some runs). FlushOutboxOnce then returns outbox pending remains. Those records keep their opId and have zero failed attempts, and polling again 2–5s later settles all of them with no re-upload. The same race was possible with serial uploads, just less likely. Cloud's bulkWrite results carry no opId (they are acked the legacy way, with no GETs), so production should not see this. Self-hosted Go servers may hit it more often; setting RELAYFILE_MOUNT_BULK_UPLOAD_CONCURRENCY=1 restores the old timing. The benchmark tolerates only this one outcome (settleLaggingReceipts).
• With the Go store, receipt settlement does one serial GET per file (about 13k GETs). This is not a factor on Cloud.
• The public state written at the end of PushLocalAndFlushOnce now reflects the pushLocal scan, not a scan taken after the flush. This only affects observability.
• Helpers newMountsyncAPIHandler, mustMountsyncTestJWT and ensureMountsyncJWTVerifier now take testing.TB so the benchmarks can use them.

No version bump or release.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Note

Medium Risk
Concurrent outbox flushing and reused push scans change timing and observability while preserving ack ordering in tests; mis-handling could duplicate uploads or miss errors on large drains.

Overview
Speeds up mount writeback for large --push-local-once / outbox drains by cutting redundant JSON sizing work, overlapping /fs/bulk uploads, and avoiding extra local file reads.

Bulk batch sizing encodes each file once and tracks request size incrementally (bulkWriteRequestSizer, bulkWriteChunkBounds) in chunkOutboxRecords, chunkPendingBulkWrites, and HTTPClient.WriteFilesBulk, replacing repeated full-chunk marshaling that was O(n²) on big backlogs.

Concurrent outbox flush adds flushOutboxChunks with a configurable cap via RELAYFILE_MOUNT_BULK_UPLOAD_CONCURRENCY (default 2, 1 = serial, max 8). Prepare/apply stay ordered under s.mu; only the POST runs on workers, with same-path serialization and stop-on-first-error semantics matching the old serial loop. Chunk flush is split into prepareOutboxChunkUpload / applyOutboxChunkResponse.

Local push captures small untracked files during scanLocalFilesCapturing (with fresh() revalidation) so new files are read once; PushLocalAndFlushOnce reuses that scan for public state instead of a third tree hash, dropping entries removed during flush.

Docs add the new env var; tests/benchmarks assert legacy-identical chunk boundaries and concurrency/error behavior.

^{Reviewed by Cursor Bugbot for commit 8d12343. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Agent Relay sessions

• claude session a26b88d9 · opened via gh pr create · last active 2026-10-10
• claude session 5d87b716 · contributor · ran gh pr commands · last active 2026-10-10

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-10T22:57:49.441103Z	f31608e	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0d4199f9-8436-405b-8f6b-433aac1248ed

---

📥 Commits

Reviewing files that changed from the base of the PR and between f31608e and 8d12343.

---

📒 Files selected for processing (3)

• internal/mountsync/bulk_upload_concurrency_test.go
• internal/mountsync/push_local_read_once_test.go
• internal/mountsync/syncer.go

---

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: cc6bca6e-2208-4816-a367-cf6d1367ee22

---

---

📥 Commits

Reviewing files that changed from the base of the PR and between 4c2b2dd and f31608e.

---

---

📒 Files selected for processing (8)

• docs/environment-variables.md
• internal/mountsync/bulk_batch_sizing_test.go
• internal/mountsync/bulk_upload_concurrency_test.go
• internal/mountsync/push_local_read_once_test.go
• internal/mountsync/push_perf_test.go
• internal/mountsync/stream_write.go
• internal/mountsync/syncer.go
• internal/mountsync/syncer_test.go

---

---

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

---

---

📝 Walkthrough

📝 Walkthrough

Walkthrough

Mount sync now calculates bulk request sizes incrementally, supports configurable concurrent outbox uploads, and reuses eligible file snapshots from local scans. Tests and benchmarks cover chunk sizing, upload behavior, local reads, and large workloads.

Changes

Mount sync upload and scan

Layer / File(s)	Summary
Incremental bulk request sizing internal/mountsync/syncer.go, internal/mountsync/stream_write.go, internal/mountsync/bulk_batch_sizing_test.go	Bulk request chunks use incremental encoded-size accounting. Tests compare request sizes and chunk boundaries with marshal-based reference implementations.
Configurable outbox upload concurrency internal/mountsync/syncer.go, internal/mountsync/bulk_upload_concurrency_test.go, docs/environment-variables.md	Outbox chunks can upload concurrently, up to the configured limit. Results are applied in order, uploads for the same path do not overlap, and the documented setting defaults to 2 and is capped at 8.
Local scan snapshot capture internal/mountsync/syncer.go, internal/mountsync/push_local_read_once_test.go	Eligible untracked files are captured during scanning and reused for upload and public-state generation. Tests cover read counts, oversized files, tracked changes, and low-memory scans.
Push and backlog benchmarks internal/mountsync/push_perf_test.go, internal/mountsync/syncer_test.go	Benchmarks measure large local-tree pushes and large outbox drains. Test helpers now accept testing.TB.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Syncer
  participant ChunkFlusher as flushOutboxChunks
  participant BulkClient
  Syncer->>ChunkFlusher: Submit selected outbox chunks
  ChunkFlusher->>BulkClient: Upload prepared chunk
  BulkClient-->>ChunkFlusher: Return upload response
  ChunkFlusher->>Syncer: Apply chunk results in order

Loading

Suggested reviewers: miyaontherelay

---

---

Merge Risk: ⚪ Minimal · up to f3160

No actionable merge-blocking issue is established; the change is mergeable after normal checks.

Pre-merge checks | 4 | 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 45.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 40 functions across 6 files. (2 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly summarizes the PR’s three primary changes: linear bulk batch sizing, single reads for new files, and bounded concurrent uploads.
Description check		The description is directly related to the changeset and explains the performance problems, implementation changes, tests, benchmarks, and known risks.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

---

Full details: Docstring Coverage

Explanation

Docstring coverage is 45.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 40 functions across 6 files. (2 skipped: 1 unsupported, 1 too large.)

---

---

---

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

---

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

---

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{A rabbit checks the upload queue,
And counts the chunks that travel through.
Small files are read and held with care,
While tests watch every path laid bare.
The backlog drains; the benchmarks cheer,
Then carrots close the reviewer's year.}

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Relayfile Eval Review

Run: .relayfile/evals/runs/2026-10-10T23-36-05-374Z-HEAD-provider
Mode: provider
Git SHA: ec1bda2

Passed: 4 | Needs human: 0 | Reviewable: 0 | Missing output: 0 | Failed: 0 | Skipped: 0

Human Review Cases

No reviewable human-review cases captured Relayfile output.

[devin-ai-integration]

Devin Review found 3 potential issues.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f31608e786

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".