Skip to content

perf(control): keep 5,000-claim churn on pace by committing only changed rows, and seat up to 256 JVMs per release - #346

Draft
oglassdev wants to merge 3 commits into
mainfrom
perf/control-incremental-commits
Draft

oglassdev wants to merge 3 commits into
mainfrom
perf/control-incremental-commits

Conversation

@oglassdev

Copy link
Copy Markdown
Contributor

Refs #49, #155. A 5,000-player environment had two control-side blockers:

  • Commit cost grew with total state. Each commit cloned control's whole state, diffed every row, and scanned every claim and move to stamp generations. At 4,000 claims that was about 89% of commit time, and churn fell behind.
  • One release seated at most 4,096 players: 32 processes of 128.

Commits now scale with the rows they change

  • Persistent maps: every map in control's State is an imbl::OrdMap (MPL-2.0, allowed by deny.toml). Copying the state at the start of an update shares structure, so it costs almost nothing.
  • Diffing: the store diff uses OrdMap::diff, which skips shared subtrees.
  • Stamping: only the commit's pending rows get the commit's generation, plus method_sequence. Committed state never holds Generation::PENDING, so every row that names it differs from the previous state and goes through Writes::pending.
  • Call sites: the places that edited every row in place now use state::modify, which changes only the rows it selects.

Up to 256 processes per release

  • Shared limit: chunk_control::MAX_PROCESSES is 256. Release::validate and managed environments both use it, so a release can seat up to 32,768 players. chunk-build's local.max_processes allows the same.
  • Unchanged: sessions and hosts still seat at most 128 players.
  • Bench fixture: it now allows 64 processes.

Benchmarks (4 pinned CPUs, two runs per point, main 7a1f97a vs this branch)

At 4,000 claims main branch
Churn completed/s at 20/s offered 9.8 and 18.3, with failures and drops 20.0 and 20.0, none
Churn p99 4.3–5.0 s 0.14–0.62 s
Churn peak RSS 315–398 MiB 54–56 MiB
Population p99 (100/s) 263–986 ms 14–22 ms
Seeding 4,000 players ~290–330 s ~105–130 s

At 5,000 claims the branch still sustains 20/s churn and 100/s population, with no failures. The machine was shared, so the first pair of runs is the noisiest.

Breaking: existing deployments recorded max_processes = 32. If core has to rebuild one of them after this upgrade, for example a replacement core restoring its database without the archive, control rejects it as release configuration changed. The fix is to deploy again. Pre-release, so there's no migration.

Next limits for #155, not changed here

  • Commits are no longer the bottleneck: at 5,000 claims they take about 0.01% of control's CPU. About 40% goes to jvm::topic::deliveries, which scans every claim on each host-topic update, so CPU per operation still grows with population (about 54 → 65 ms).
  • Other limits: at most 256 retained sessions across the environment, and a default of 1,024 gateway connections.

Checks

  • cargo test -p chunk-control (92), cargo test -p chunk-environment managed:: (43), chunk-build project tests (9), chunk-bench config.
  • Clippy with warnings as errors, and fmt, on all four crates.
  • cargo deny check licenses.
  • Merged with feat(cli): list a node's sessions in the chunk dev JVM tab and filter logs by session #344, which also touches chunk-control: it type-checks and its operator tests pass.
  • One review round: the breaking change above, plus a stale comment, now fixed.

🤖 Generated with Claude Code

…processes per release

Control state's maps are now imbl OrdMaps. Cloning the published state shares
its structure, and the commit diff skips shared subtrees, so a commit costs in
proportion to the rows it touches rather than to the whole state. Stamping
visits only the rows written with a pending generation, since committed state
never names one. Code that changed many rows in place now changes only the rows
it selects.

A release may run up to 256 processes, so one release seats up to 32,768
players at 128 per process. The bench fixture allows 64 processes.

Refs #49, #155

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant