Skip to content

fix(loadtest): bound parallel nonce fetch by --concurrency - #998

Merged
minhd-vu merged 3 commits into
mainfrom
fix/bounded-parallel-nonce-fetch
Aug 26, 2026
Merged

fix(loadtest): bound parallel nonce fetch by --concurrency#998
minhd-vu merged 3 commits into
mainfrom
fix/bounded-parallel-nonce-fetch

Conversation

@minhd-vu

Copy link
Copy Markdown
Contributor

Problem

FetchNoncesInParallel spawned one goroutine per account and fired every eth_getTransactionCount simultaneously, so --concurrency (and --rate-limit) did not apply to the heaviest read burst the tool produces. With a 10,000-account --sending-accounts-file, the RPC endpoint receives ~10k concurrent requests in the first seconds of a run.

Observed against a load-balanced Amoy gateway: the sweep drew HTTP 500s from the GCP frontend and the run aborted before sending anything. A local bor node absorbs the same sweep fine, so this mainly breaks runs pointed at load-balanced/managed endpoints — exactly the ones a --send-rpc-url split is meant to enable. The existing workaround, --sequential-nonce-fetch, is one-at-a-time and impractical for 10k accounts over a WAN.

Change

  • Acquire a semaphore slot before spawning each fetch, so both in-flight requests and live goroutines stay capped at min(--concurrency, len(accountsToFetch)). Values <= 0 clamp to 1.
  • New fetchNonceWithRetry: up to 5 attempts with exponential backoff (250ms → 4s), so transient 429/5xx load shedding no longer aborts the run.
  • Cancellation: both the acquire loop and the backoff wait select on ctx.Done(), and the backoff uses time.NewTimer + Stop() rather than time.After, so Ctrl+C returns promptly without leaking timers.
  • --sequential-nonce-fetch behavior is unchanged; its help text now describes what the two paths actually do ("one at a time through the rate limiter" vs "in parallel bounded by --concurrency").

Testing

New loadtest/account_test.go uses an httptest JSON-RPC server that records peak concurrent nonce calls and can inject failures. It covers: concurrency bound respected, 0/negative clamping to 1, retry-then-succeed, failure after max attempts, cancellation during backoff, and the no-work case issuing zero RPCs. go test -race passes; go vet, gofmt, and golangci-lint are clean.

Manually verified against a public Amoy endpoint with a counting HTTP proxy in between to measure peak concurrent eth_getTransactionCount (2,000-account file):

binary --concurrency peak in-flight nonce calls
main 100 200+ (proxy-limited; effectively unbounded — flag ignored)
this PR 100 100
this PR 25 25

Peak matches --concurrency exactly. main's 200 is a floor rather than its real ceiling — the proxy could not accept faster, so the remainder sat in the socket backlog uncounted.

Direct against the endpoint, no proxy:

  • 10,000 accounts at -c 200: sweep completed in 4s, zero errors, zero retries. main unbounded on the same file: 3s — so the bound costs roughly 1s at 10k scale.
  • 2,000 accounts at -c 100: 2s. -c 1 on 50 accounts: 2s, log confirms concurrency: 1.
  • --sequential-nonce-fetch still reaches "All accounts are ready" as before.

The retry path also exercised itself for real: while proxied, the proxy intermittently 500'd under its own load and the sweep absorbed it — 2,101 nonce calls for 2,000 accounts at -c 100 (101 retries) and 2,027 at -c 25, both finishing successfully.

Caveat: I could not reproduce the original failure on the public endpoint I tested — it absorbed the full unbounded 10k sweep without a single error, so main never broke there. The reported 500s came from a private gateway's GCP frontend; confirming the fix against that specific endpoint is still worth doing.

🤖 Generated with Claude Code

minhd-vu and others added 2 commits August 26, 2026 11:39
FetchNoncesInParallel spawned one goroutine per account and issued every
eth_getTransactionCount at once, so --concurrency did not apply to the
heaviest read burst the tool produces. With a 10,000-account
--sending-accounts-file the endpoint received ~10k concurrent requests in
the first seconds of a run, which load-balanced and managed endpoints
answer with 429s / 5xx, aborting the run before anything is sent.

Acquire a semaphore slot before spawning each fetch so both in-flight
requests and live goroutines stay capped at min(--concurrency, N); values
<= 0 clamp to 1. Each fetch now retries up to 5 times with exponential
backoff (250ms -> 4s) so transient load shedding no longer fails the run,
and both the acquire loop and the backoff select on ctx.Done() so Ctrl+C
returns promptly.

--sequential-nonce-fetch is unchanged; its help text now describes what
the two paths actually do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Satisfies errcheck.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@minhd-vu
minhd-vu merged commit f276d04 into main Aug 26, 2026
15 checks passed
@minhd-vu
minhd-vu deleted the fix/bounded-parallel-nonce-fetch branch August 26, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants