docs: steady-state metrics reporting design proposal - #447
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| a run killed by `SIGKILL`/OOM before the log is flushed yields a truncated log; | ||
| the step must degrade to a best-effort result with a status flag rather than | ||
| fail (§5.6). |
There was a problem hiding this comment.
Maybe I am not reading it in the right way - but I thought it should fail if the original run fails?
| # 1 Objective {#1-objective} | ||
|
|
||
| Add a post-processing step that runs after a benchmark completes and reports the | ||
| metrics the system _sustains in steady state_, rather than the whole-run average, |
There was a problem hiding this comment.
reword the metrics the system _sustains in steady state_ - one suggestion: reports the sustained steady state metrics
| in the run report, reported as the **official result** with the whole-run | ||
| (`total`) metrics retained as supplementary context. | ||
| - Detect _when there is no steady state_ (a progressively degrading run) and say | ||
| so, instead of reporting a false point estimate. |
There was a problem hiding this comment.
suggestion - "false point estimate" -> "unstable/unsteady result"?
| - Detect _when there is no steady state_ (a progressively degrading run) and say | ||
| so, instead of reporting a false point estimate. | ||
| - Provide a single implementation reachable two ways: automatically at the end of | ||
| a run, and manually as a CLI over any recorded `events.jsonl`. |
There was a problem hiding this comment.
suggestion - manually as a CLI -> manually as a post-processing step
| - **Relative active concurrency.** In-flight count as a percentage of the | ||
| max-concurrency budget (100% at saturation). | ||
| - **Issue-time window.** A contiguous range of super-passes `[start, end)`. Its | ||
| measured set is every request _issued_ in that range. Throughput uses the |
There was a problem hiding this comment.
This might requires some clarification - as far as i understand from below a sample issued but whose response is not received has no logged TTFT and is naturally excluded. So either it is issue only or issue with at least one event to be included in a window.
| client. The window measures the sustained behavior of a healthy server; a | ||
| genuinely unhealthy server produces bad-but-real numbers, which the drift | ||
| detector (§5.5) distinguishes from transient pollution. | ||
| - **Super-pass.** One full **dataset pass** — `dataset_size` consecutive requests |
There was a problem hiding this comment.
minor - would it be easier to just call it a pass instead of a super-pass since it is a single pass over the entire dataset?
| (everything issued at `t=0`), so there is no client-side issue-time window and no | ||
| client-side drain boundary. The step detects the mode and finds the steady region |
There was a problem hiding this comment.
for TPOT calculations, we could potentially use the completion-first_token time, but that would require the server to not evict requests in decode phase.
| ## 5.7 Complementary: staggered ("feathered") issuance | ||
|
|
||
| The ramp exists because the target concurrency `C` is filled as a burst. A | ||
| staggered fill flattens the ramp-up spike: issue in steps of `ceil(C/k)` and, between |
There was a problem hiding this comment.
add a small definition for k.
| - **ISL / OSL** are reported for analytics, not as validation numbers: once the | ||
| window is allowed to drop samples (not enforcing full dataset-pass boundaries), | ||
| the input/output-length distribution is skewed relative to the constructed | ||
| dataset and is no longer a meaningful validation quantity, though it remains | ||
| useful for analysis. |
There was a problem hiding this comment.
If the window is allowed to drop samples, should the ISL/OSL statistics be computed for the samples included in the measurement?
|
|
||
| # 6 Open questions {#6-open-questions} | ||
|
|
||
| - **Default parameters.** The warmup band, trailing-window length, per-percentile |
There was a problem hiding this comment.
Can we ignore the first pass/super-pass over the dataset as a warmup pass and the other parameters can be benchmark specific decided by TF?
| - **Guard distance measure.** The exact two-sample statistic and acceptance bound | ||
| for the per-token-invariance guard (§5.4) need to be fixed. | ||
| - **No-steady-state runs.** When no steady state is found (a tracked metric is | ||
| Drifting Up), should the run be reported _invalid_ — analogous to legacy |
There was a problem hiding this comment.
We should declare the gates for each benchmark and also define whether a submission should be invalid if it fails a certain gate. It should be a benchmark TF decision on what gates to include.
Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
102896c to
5e936f6
Compare
3 participants
Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling.
What does this PR do?
Adds a design doc for a new steady-state window detection and metrics reporting feature
Type of change
Related issues
cc @arekay-nv to tag any related issues
Testing
Checklist