Repository navigation
[Benchmark] Preserve complete serving latency evidence and offline summaries #130
Description
Extend the existing benchmarks/ runner so a saved serving round can be recomputed with its full planned, attempted, successful and failed request counts. This is the initial text contribution to the performance scorecard in #125; implementation: PR #131.
The measured scope is Open-Jev-9B and Open-Jev-27B-v1.1 public231 text serving on one H800, concurrency 1. Each deployment has three rounds of 231 measured requests and five separate client warmups per round. Image/VL, cache sequences and higher concurrency remain follow-ups.
| Deployment | Round | Successful / planned | p50 ms | Nearest-rank p95 ms |
|---|---|---|---|---|
| Open-Jev-9B | 1 | 231 / 231 | 43.461 | 326.968 |
| Open-Jev-9B | 2 | 231 / 231 | 43.051 | 326.572 |
| Open-Jev-9B | 3 | 231 / 231 | 42.899 | 327.221 |
| Open-Jev-27B-v1.1 | 1 | 231 / 231 | 69.429 | 1069.310 |
| Open-Jev-27B-v1.1 | 2 | 231 / 231 | 69.459 | 1063.843 |
| Open-Jev-27B-v1.1 | 3 | 231 / 231 | 69.581 | 1066.037 |
Pooled p50/p95 are 43.082/326.968 ms (9B) and 69.459/1066.037 ms (27B). Each deployment has 693 planned, attempted and successful measured requests, zero failures, and 231 unique tasks. The clock includes full response-body decoding, JSON parsing and schema validation. Startup, first POST and warmups are separate. Different deployments do not establish same-model speedup or numerical parity.
Recompute with python benchmarks/bench.py summary <saved-round-directory>; all six saved-round summaries exited 0. The contribution has 22 passing CPU unit tests, passing core Ruff rules, and PASS from both independent static source reviews. Default Ruff still reports nine diagnostics; the MkDocs build is unverified because MkDocs was unavailable. Full Rust/GPU project validation is outside this tooling check.
Obtain public231 from JevBench commit 7ce310c7262ed49cc85853339a8a42459298e3f3; expected input SHA-256: 00d1f10ca1142499a598b5f1c6602d329cce1cef04c961c46fc0318daa2c67d3. Collection and recomputation use the repository benchmark documentation. Original and text-derived raw artifacts are not attached pending redistribution permission. Companion quality work reports 179/231 and 198/231 correct each round.
Acceptance for this contribution: preserve failed and unattempted items, recompute saved rounds without new inference, report per-round and pooled successful-request quantiles, and keep startup/warmup and task execution timing distinct.
Companion decision-quality benchmark: System1-Agents #68, implemented in PR #69.
Activity