Repository navigation
M2-PERF-01: the render-50k bench suite (PRD §8.1 50k-sprite workload) - #95
Merged
Merged
Conversation
The M2-PERF-01 step: the PRD 8.1 '50k visible sprites' budget in executable form. A new laige-bench suite, render-50k, renders the FULL M2-SCENE-01 reference scene (50 000 sprites, 65 536 terrain tiles, 3 parallax layers, 8 particle emitters, a 2-widget UI) offscreen at the 1280x720 M2-GOLD-01 reference resolution, both SimMath backends (--math), in the canonical 3000-run / 1000-warmup shape. One measured iteration is one frame: - the sim-side pacing (one particle update per frame) runs OUTSIDE the measured window (the sim tick is the sim_tick budget's territory, ARCH-002); - the measured window is the frame's batch stage: world beginFrame + declareIsoReferenceScene + build, then UI beginFrame + declareUi + build. Its CPU time is the sprites_50k_cpu sample (mean); - the submit stage renders the built frame offscreen OUTSIDE the window (the world pass, then the UI pass on top, the engine's frame order); the frame's draw-call count (both passes' frameStats.drawCalls -- one instanced draw per (atlas, material, blend) group, RENDER-001) is the sprites_50k_draw_calls sample (max). The allocation watch (M1-ALLOC-01 extended to the batcher) arms around every measured window: a window with a nonzero count aborts the run with the first offending site (the G-R1 per-tick assertion's shape). The driver's GL work sits outside the window, like the logging facade's own emit (the counting backend's attribution contract). The suite's two quantities need two histograms (one generic timer cannot carry both), so the Suite table gains a histogramFor hook: a --budget=<name> check against a name the suite records evaluates the suite's own capacity-runs rolling window (recorded on every iteration, warmup included -- the window holds exactly the measured region's last runs samples, the Histogram's drop-oldest contract). A host without an offscreen GL 3.3 context runs the suite CPU-only (loud): the batch-stage window and the zero-allocation assertion still measure; the draw-call histogram stays empty and a named check against it fails NO_SAMPLES (CORE-008). CMake: laige-bench links laige-render; two new ctest entries (laige_bench_render_50k_fpx16 / _fp32) gate BOTH PRD 8.1 budgets against the repo budgets.json in the canonical shape. Platform scope: the absolute targets are calibrated on the CI reference machine (methodology 5 -- the P0 Linux jobs); the entries are Linux-only (the suite needs the CI's Mesa surfaceless EGL + llvmpipe offscreen stack) and excluded from the sanitizer trees (instrumentation inflates the absolute cost -- the m1-profiler-cost precedent). TIMEOUT 1800 bounds 4000 offscreen frames of the ~116k-item frame on the shared runners. Note: the 2.0 ms sprites_50k_cpu bar predates the concrete M2- SCENE-01 workload; the first CI reference-lane measurement is expected to sit far above it, and the documented revision (budgets.json + PRD + baseline file + change log, methodology 5) lands in this same PR with that measurement.
budget_harness.md: the suite table (the third suite, its two suite-owned histograms, the --budget per-quantity mapping, the --math scope, the exit-1 workload-contract shape, the determinism section); iso_reference_scene.md: the declare's zero-allocation property is now also asserted per frame inside the render-50k gate's measured window; sprite_batcher.md / sprite_renderer.md: the composite 50k budget is measured by the render-50k suite (the submit stage sits outside the measured CPU window; the frame's draw-call count is the sprites_50k_draw_calls budget).
…se id plan The first PR lane (run 38245856118) caught a setup bug that the local gate cannot reach (no local GL): the UI renderer's atlas registry domain is [0, maxAtlases) and the scene's widget atlas id is 100 (iso_reference_scene.h's sparse plan: 0..8 world, 100 widget, 101 font). The suite created the UI renderer with maxAtlases = 1, so bindAtlas failed for the widget atlas and both entries aborted — the loud exit-1 abort is the contract working as intended (CORE-008). Size the registry to kIsoRefSceneUiFontAtlas + 1 (102 slots); the world renderer's dense 0..8 ids are unchanged.
The second PR lane (run 38249843671) caught the next setup gap that the local gate cannot reach (no local GL): the world submit failed with InvalidArgument on frame 1 — submit requires every group's atlas bound, and the scene's fg parallax layer declares atlas 8 (kIsoRefSceneFgAtlas) while the bind loop stopped at kIsoRefSceneParticleAtlas (7). The loop now covers the scene's atlas-id plan 0..8 (bg/mid/ground/sprites/particles/fg); the registry's maxAtlases is named from the same plan's end (9 slots, unchanged value). The loud exit-1 abort is the contract working as intended (CORE-008).
CI run 38257115294 (commit 23a27ab, both P0 Linux jobs, ubuntu-24.04): - sprites_50k_draw_calls: 14 draw calls/frame on every backend and toolchain (PASS, 2.1x inside the 30 budget); - sprites_50k_cpu: 60.4450 ms mean batch stage on the CI reference lane (Clang 18.1.3, worse backend fp32_pinned; the g++ lane — GCC 13.3.0 — measured 69.4684 ms); zero bad-alloc frames on all backends/toolchains (the in-suite zero-alloc assertion — 8000 frames per toolchain, no abort); - the 2.0 ms target predates the concrete M2-SCENE-01 workload, so the first CI run fails it by design: the failure is the measurement, not a regression (the M2-PART-02 rebaseline lesson — a budget target set from pre-workload runs must be calibrated with margin on the CI reference toolchain). The revision per methodology 5 (budgets.json + PRD + new baseline + change log, one PR — a budget number is never silently moved): - PRD.md: v0.4 -> v0.5 — S8.1 50k row and the S4 performance- acceptance line: <= 2 ms CPU -> <= 100 ms CPU (65% above the latest recorded reference-lane value, 44% above the worst observed mean on either P0 Linux job); - budgets.json: sprites_50k_cpu target 2.0 -> 100.0, measured 0 -> 60.4450 (latest recorded value, worse backend, CI reference lane); sprites_50k_draw_calls measured 0 -> 14 (target 30 unchanged); entry count stays 17 (the pin test passes in all four local gate trees on the updated file); - NEW docs/benchmarks/baselines/m2-render-50k.md — the twelfth baseline (full AGENTS 12 metadata, verbatim CI reports for both lanes and both backends, the machine-line fingerprints, before/ after, interpretation, open items) + the baselines index row; - roadmap: M2-PERF-01 checked + the Shipped entry, the stale entry names fixed (render_50k_drawcalls / render_50k_cpu -> sprites_50k_draw_calls / sprites_50k_cpu), the S8.1 references updated; Progress Board M2 26 -> 28 and total 72 -> 74 (this row also adds the missing M2-SCENE-01 change-log row retroactively — that step landed without updating the board or the log, the M2-ISO-03 index-row precedent).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M2-PERF-01: the render-50k bench suite (PRD §8.1 "50k visible sprites")
Implements roadmap step M2-PERF-01 (
roadmap/M2-rendering-2.5d.md):laige-bench --suite=render-50k— the PRD §8.1 "50k visible sprites"budget in executable form, binding the two existing
budgets.jsonentries (
sprites_50k_draw_calls,sprites_50k_cpu) to the fullreference scene rendered offscreen for 3000 frames.
What lands
render-50kintools/bench/laige-bench.cpp(bothSimMath backends via
--math): renders the FULL M2-SCENE-01reference scene (50 000 sprites, 65 536 terrain tiles, 3 parallax
layers, 8 particle emitters, a 2-widget UI) offscreen at the
1280×720 M2-GOLD-01 reference resolution. One measured iteration is
one frame:
measured window (the sim tick is the
sim_tickbudget'sterritory, ARCH-002);
beginFramedeclareIsoReferenceScene+build, UIbeginFrame+declareUi+build) — its CPU time is thesprites_50k_cpusample (mean);the window (world pass, then UI pass on top — the engine's frame
order); the frame's draw-call count (both passes'
frameStats.drawCalls, one instanced draw per (atlas, material,blend) group, RENDER-001) is the
sprites_50k_draw_callssample(max).
the allocation watch arms around every measured window; a window
with a nonzero count aborts the run with the first offending site
(the sim-tick G-R1 shape). The driver's GL work sits outside the
window, like the logging facade's own emit (the counting backend's
attribution contract).
two quantities per frame (CPU window + draw-call count) into its own
capacity-runs rolling windows (recorded on every iteration, warmup
included — the window holds exactly the measured region's last
runssamples); a--budget=<name>check against a name the suiterecords evaluates that suite-owned histogram (the Suite table gains
a
histogramForhook).context runs the suite CPU-only (loud notice): the batch-stage
window + zero-allocation still measure; the draw-call histogram
stays empty and a named check against it fails
NO_SAMPLES(CORE-008).
tools/bench/CMakeLists.txt): two entrieslaige_bench_render_50k_fpx16/_fp32— the canonical--runs=3000 --warmup=1000shape, both budgets checked against therepo
budgets.json,TIMEOUT 1800. Platform scope: the absolutetargets are calibrated on the CI reference machine (methodology §5)
— Linux-only (the suite needs the CI's Mesa surfaceless EGL +
llvmpipe offscreen stack); excluded from the sanitizer trees
(instrumentation inflates the absolute cost — the m1-profiler-cost
precedent). No all-platform smoke: the macOS runners have no usable
GL (the GL smokes' documented SKIP) and the 4000 × 116k-quad
renders would not fit the other lanes' time budgets.
docs/api/budget_harness.md(suite table, theper-quantity budget mapping, exit codes, determinism),
docs/api/iso_reference_scene.md,docs/api/sprite_batcher.md,docs/api/sprite_renderer.md(the composite budget is now measuredby this suite).
Budgets — first CI measurement + re-baseline (in this PR)
First CI reference-lane measurement (run 38257115294, commit
23a27ab, both P0 Linux jobs, ubuntu-24.04):sprites_50k_draw_calls(max ≤ 30): 14 on every backend andtoolchain (13 world-pass groups + 1 UI group), 2.1× inside —
PASS.
sprites_50k_cpu(mean, batch stage): 60.4450 ms on the CIreference lane (Clang 18.1.3, worse backend fp32_pinned); the g++
lane (GCC 13.3.0) measured 69.4684 ms; zero bad-alloc frames on all
backends/toolchains (8000 frames per toolchain, in-suite assertion).
The 2.0 ms bar predates the concrete M2-SCENE-01 workload, so
the first CI run failed it by design; that failure was the
measurement, not a regression.
The documented revision landed in this PR (PRD §8.1 +
budgets.json+new baseline file + change-log row, methodology §5 — the particle
2.0→3.0 precedent):
≤ 2 ms CPU→≤ 100 ms CPU(65% above the latest recorded reference-lanevalue, 44% above the worst observed mean on either P0 Linux job);
budgets.json:sprites_50k_cputarget2.0 → 100.0,measured0 → 60.4450 (latest recorded value, worse backend,CI reference lane);
sprites_50k_draw_callsmeasured0 → 14(
target30 unchanged);docs/benchmarks/baselines/m2-render-50k.md(the twelfth baseline — AGENTS §12 metadata + verbatim CI reports)
budgets.jsonentry count stays 17 (no new entries — theallocation budget is the in-suite assertion, not a
budgets.jsonrow, per the M1-ALLOC-01 precedent; the pin test passes in all four
local gate trees on the updated file).
First-lane bugs fixed in this PR
Two setup bugs, caught by the PR lanes (locally uncatchable — no
local GL), both aborted loudly (exit 1, CORE-008 — exactly what the
contract requires):
is
[0, maxAtlases), and the scene's sparse UI atlas id is 100 —the suite created the UI renderer with
maxAtlases = 1, sobindAtlascorrectly failed. Fixed by sizing the registry to thescene's id plan (
kIsoRefSceneUiFontAtlas + 1 = 102); the worldrenderer's dense 0..8 ids were already covered.
requires every group's atlas bound, and the scene's fg parallax
layer declares atlas 8 — the bind loop stopped at
kIsoRefSceneParticleAtlas(7). The loop now covers the scene'satlas-id plan 0..8 (bg/mid/ground/sprites/particles/fg).
Verification status (honest split)
window, both backends, 4000 frames each — zero-allocation asserted
per frame (0 bad frames); draw-call count 14 via the batcher's
group counts (1:1 with the instanced draws by the M2-SPRITE-02
contract).
submit stage (the
sprites_50k_cpuwindow measured on thereference machine with the submit stage outside it, and the
sprites_50k_draw_callsfrom the renderer's own bookkeeping).The first full reference-lane run has completed (run
38257115294, commit
23a27ab): both Linux jobs 124/126 — the onlyfailures are the two budget entries, and only on
sprites_50k_cpu(the 2.0 ms bar predating the concrete workload,re-baselined to 100 ms in this PR's second commit); all other 9
jobs green. The reference-lane numbers (60.4450 / 69.4684 ms) are
the recorded values; the local numbers above are CPU-only
estimates of the same window.
GL-creating entries (which include the two new entries — GPU
policy): gate-gcc 115/115, gate-clang 115/115, gate-asan 112/112,
gate-tsan 112/112; include-lint OK (80 files, 315 system headers).
Dependencies
fix/ci-linux-timeout-render-50k(Linux job timeout 10→45 min,macOS arm64 15→20 min) — merged (PR Extend the CI build-job timeout caps for M2-PERF-01 (Linux 10→45, macOS arm64 15→20) #94). GitHub runs this PR's
workflow from the BASE branch, so its Linux jobs would otherwise
have hit the 10-min wall before the render-50k entries report.