From 8bedd680b45923c969d72eb3acfb6f4bb0a132f0 Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Sat, 14 Nov 2026 15:38:13 -0500 Subject: [PATCH] docs: file the performance inventory, and close the PLE read-path knobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things from one session: a page that records what has been tried (so the next session stops re-deriving it), and the first result from it. docs/PERF-NEXT.md: - kept levers, each with the number that earned it; - closed levers, each with the number that killed it (batch/slots, kv-grow, suffix-draft, prefill 16k/32k, 4-bit KV, PLE in RAM, expert-cache size, spec_min_p 0.7, spec_window, DFlash2 on SYCL); - the remaining cost shape: prefill 336 ms fixed + 0.67 ms/token over 18 fresh legs; - the untried list, ranked, each with evidence, cost and a first test — batch_after 1-2 first (already measured, not shipped), then attributing the 330 ms, then the SYCL residency router, prefill-chunk CIs, pool affinity, dispatch overhead, DFlash2. The PLE sweep, six card windows on the 0.2.13 tree, closes two knobs: - row cache 4 M rows and --ple-inflight 1024 are flat against a drift band of 0.2-3.6 %; - row cache 16 M rows (1.44 GB) costs 8-17 % prefill (fresh 4.2k, 8.8k, a 4.4k follow-up, 30k). So the 94 MB default and inflight 256 stand, and the fixed per-request cost is not PLE row-read starvation. Artifacts, configs, per-leg JSON, log and script are in dirigo-results/0.2.13-port/ple-sweep/. Production stayed on 0.1.58 throughout: each window's restore line shows :8095 http=200 with the engine 0.1.58, and the service was verified after the last one. --- .../0.2.13-port/ple-sweep/bench-ple-b.json | 263 ++++++++++++++++++ .../0.2.13-port/ple-sweep/bench-ple-b2.json | 263 ++++++++++++++++++ .../0.2.13-port/ple-sweep/bench-ple-c16.json | 263 ++++++++++++++++++ .../ple-sweep/bench-ple-c16i1k.json | 263 ++++++++++++++++++ .../0.2.13-port/ple-sweep/bench-ple-c4.json | 263 ++++++++++++++++++ .../0.2.13-port/ple-sweep/bench-ple-i1k.json | 263 ++++++++++++++++++ .../0.2.13-port/ple-sweep/conf-b.json | 63 +++++ .../0.2.13-port/ple-sweep/conf-b2.json | 63 +++++ .../0.2.13-port/ple-sweep/conf-c16.json | 65 +++++ .../0.2.13-port/ple-sweep/conf-c16i1k.json | 67 +++++ .../0.2.13-port/ple-sweep/conf-c4.json | 65 +++++ .../0.2.13-port/ple-sweep/conf-i1k.json | 65 +++++ .../ple-sweep/longprobe-ple-b.json | 87 ++++++ .../ple-sweep/longprobe-ple-b2.json | 87 ++++++ .../ple-sweep/longprobe-ple-c16.json | 87 ++++++ .../ple-sweep/longprobe-ple-c16i1k.json | 87 ++++++ .../ple-sweep/longprobe-ple-c4.json | 87 ++++++ .../ple-sweep/longprobe-ple-i1k.json | 87 ++++++ .../0.2.13-port/ple-sweep/strata-ple-sweep.sh | 69 +++++ .../0.2.13-port/ple-sweep/summary.json | 226 +++++++++++++++ docs/PERF-NEXT.md | 90 ++++++ docs/README.md | 1 + 22 files changed, 2874 insertions(+) create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-b.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-b2.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16i1k.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-c4.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/bench-ple-i1k.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-b.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-b2.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-c16.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-c16i1k.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-c4.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/conf-i1k.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b2.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16i1k.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c4.json create mode 100644 dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-i1k.json create mode 100755 dirigo-results/0.2.13-port/ple-sweep/strata-ple-sweep.sh create mode 100644 dirigo-results/0.2.13-port/ple-sweep/summary.json create mode 100644 docs/PERF-NEXT.md diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b.json new file mode 100644 index 00000000000..a0324dd86b8 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-b", + "kind": "warmup", + "trial": 0, + "wall_s": 0.3168691350147128, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 209.0, + "prefill_tps": 71.7, + "generated": 2.0, + "decode_ms": 82.0, + "decode_tps": 24.4 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-b", + "kind": "fresh", + "trial": 1, + "wall_s": 4.852005727007054, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3214.0, + "prefill_tps": 1310.1, + "generated": 128.0, + "decode_ms": 1633.0, + "decode_tps": 78.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-b", + "kind": "followup", + "trial": 1, + "wall_s": 2.1206404130207375, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 429.0, + "prefill_tps": 265.7, + "generated": 128.0, + "decode_ms": 1687.0, + "decode_tps": 75.9 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-b", + "kind": "fresh", + "trial": 2, + "wall_s": 8.365401799965184, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6873.0, + "prefill_tps": 1284.8, + "generated": 128.0, + "decode_ms": 1486.0, + "decode_tps": 86.1 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-b", + "kind": "followup", + "trial": 2, + "wall_s": 1.9242842710227706, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 418.0, + "prefill_tps": 270.6, + "generated": 128.0, + "decode_ms": 1503.0, + "decode_tps": 85.2 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-b", + "kind": "fresh", + "trial": 3, + "wall_s": 4.735613274038769, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3212.0, + "prefill_tps": 1310.8, + "generated": 128.0, + "decode_ms": 1520.0, + "decode_tps": 84.2 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-b", + "kind": "followup", + "trial": 3, + "wall_s": 2.334400308027398, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 646.0, + "prefill_tps": 383.8, + "generated": 128.0, + "decode_ms": 1685.0, + "decode_tps": 76.0 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-b", + "kind": "fresh", + "trial": 4, + "wall_s": 7.797196410014294, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6292.0, + "prefill_tps": 1403.4, + "generated": 128.0, + "decode_ms": 1500.0, + "decode_tps": 85.4 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-b", + "kind": "followup", + "trial": 4, + "wall_s": 1.8936358939972706, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 417.0, + "prefill_tps": 271.0, + "generated": 128.0, + "decode_ms": 1473.0, + "decode_tps": 86.9 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b2.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b2.json new file mode 100644 index 00000000000..31789d2d2cc --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-b2.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-b2", + "kind": "warmup", + "trial": 0, + "wall_s": 0.31629365496337414, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 219.0, + "prefill_tps": 68.4, + "generated": 2.0, + "decode_ms": 82.0, + "decode_tps": 24.3 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-b2", + "kind": "fresh", + "trial": 1, + "wall_s": 4.808019958960358, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3169.0, + "prefill_tps": 1328.6, + "generated": 128.0, + "decode_ms": 1634.0, + "decode_tps": 78.3 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-b2", + "kind": "followup", + "trial": 1, + "wall_s": 2.130483146000188, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 432.0, + "prefill_tps": 263.9, + "generated": 128.0, + "decode_ms": 1694.0, + "decode_tps": 75.6 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-b2", + "kind": "fresh", + "trial": 2, + "wall_s": 7.844577223993838, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6345.0, + "prefill_tps": 1391.6, + "generated": 128.0, + "decode_ms": 1493.0, + "decode_tps": 85.7 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-b2", + "kind": "followup", + "trial": 2, + "wall_s": 1.9407666479819454, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 420.0, + "prefill_tps": 269.2, + "generated": 128.0, + "decode_ms": 1517.0, + "decode_tps": 84.4 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-b2", + "kind": "fresh", + "trial": 3, + "wall_s": 4.7884508820134215, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3249.0, + "prefill_tps": 1295.7, + "generated": 128.0, + "decode_ms": 1534.0, + "decode_tps": 83.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-b2", + "kind": "followup", + "trial": 3, + "wall_s": 2.354951841000002, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 657.0, + "prefill_tps": 377.3, + "generated": 128.0, + "decode_ms": 1694.0, + "decode_tps": 75.6 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-b2", + "kind": "fresh", + "trial": 4, + "wall_s": 7.855977691011503, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6344.0, + "prefill_tps": 1391.8, + "generated": 128.0, + "decode_ms": 1506.0, + "decode_tps": 85.0 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-b2", + "kind": "followup", + "trial": 4, + "wall_s": 1.9022006320301443, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 420.0, + "prefill_tps": 269.3, + "generated": 128.0, + "decode_ms": 1479.0, + "decode_tps": 86.6 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16.json new file mode 100644 index 00000000000..f817aacfcc9 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-c16", + "kind": "warmup", + "trial": 0, + "wall_s": 0.3325591920292936, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 235.0, + "prefill_tps": 63.8, + "generated": 2.0, + "decode_ms": 84.0, + "decode_tps": 23.7 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-c16", + "kind": "fresh", + "trial": 1, + "wall_s": 4.818653429974802, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3181.0, + "prefill_tps": 1323.4, + "generated": 128.0, + "decode_ms": 1632.0, + "decode_tps": 78.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-c16", + "kind": "followup", + "trial": 1, + "wall_s": 2.128351060964633, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 433.0, + "prefill_tps": 263.4, + "generated": 128.0, + "decode_ms": 1691.0, + "decode_tps": 75.7 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-c16", + "kind": "fresh", + "trial": 2, + "wall_s": 7.8270649870391935, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6328.0, + "prefill_tps": 1395.3, + "generated": 128.0, + "decode_ms": 1493.0, + "decode_tps": 85.8 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c16", + "kind": "followup", + "trial": 2, + "wall_s": 1.928398194955662, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 421.0, + "prefill_tps": 268.5, + "generated": 128.0, + "decode_ms": 1504.0, + "decode_tps": 85.1 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-c16", + "kind": "fresh", + "trial": 3, + "wall_s": 5.405857170000672, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3872.0, + "prefill_tps": 1087.4, + "generated": 128.0, + "decode_ms": 1530.0, + "decode_tps": 83.7 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-c16", + "kind": "followup", + "trial": 3, + "wall_s": 2.596721009002067, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 905.0, + "prefill_tps": 274.0, + "generated": 128.0, + "decode_ms": 1688.0, + "decode_tps": 75.8 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-c16", + "kind": "fresh", + "trial": 4, + "wall_s": 9.811048677016515, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 8283.0, + "prefill_tps": 1066.1, + "generated": 128.0, + "decode_ms": 1522.0, + "decode_tps": 84.1 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c16", + "kind": "followup", + "trial": 4, + "wall_s": 1.9445385650033131, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 434.0, + "prefill_tps": 260.5, + "generated": 128.0, + "decode_ms": 1478.0, + "decode_tps": 86.6 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16i1k.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16i1k.json new file mode 100644 index 00000000000..047d6a2d714 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c16i1k.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-c16i1k", + "kind": "warmup", + "trial": 0, + "wall_s": 0.32707040203968063, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 230.0, + "prefill_tps": 65.3, + "generated": 2.0, + "decode_ms": 83.0, + "decode_tps": 24.2 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-c16i1k", + "kind": "fresh", + "trial": 1, + "wall_s": 4.798519369971473, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3161.0, + "prefill_tps": 1331.9, + "generated": 128.0, + "decode_ms": 1633.0, + "decode_tps": 78.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-c16i1k", + "kind": "followup", + "trial": 1, + "wall_s": 2.124892770021688, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 431.0, + "prefill_tps": 264.8, + "generated": 128.0, + "decode_ms": 1690.0, + "decode_tps": 75.7 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-c16i1k", + "kind": "fresh", + "trial": 2, + "wall_s": 7.820097032992635, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6319.0, + "prefill_tps": 1397.4, + "generated": 128.0, + "decode_ms": 1495.0, + "decode_tps": 85.6 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c16i1k", + "kind": "followup", + "trial": 2, + "wall_s": 1.9824172669905238, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 471.0, + "prefill_tps": 240.1, + "generated": 128.0, + "decode_ms": 1508.0, + "decode_tps": 84.9 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-c16i1k", + "kind": "fresh", + "trial": 3, + "wall_s": 5.247999338957015, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3710.0, + "prefill_tps": 1134.8, + "generated": 128.0, + "decode_ms": 1534.0, + "decode_tps": 83.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-c16i1k", + "kind": "followup", + "trial": 3, + "wall_s": 2.3458441619877703, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 651.0, + "prefill_tps": 381.1, + "generated": 128.0, + "decode_ms": 1692.0, + "decode_tps": 75.7 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-c16i1k", + "kind": "fresh", + "trial": 4, + "wall_s": 9.912156420992687, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 8401.0, + "prefill_tps": 1051.1, + "generated": 128.0, + "decode_ms": 1505.0, + "decode_tps": 85.1 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c16i1k", + "kind": "followup", + "trial": 4, + "wall_s": 1.901118580950424, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 418.0, + "prefill_tps": 270.2, + "generated": 128.0, + "decode_ms": 1479.0, + "decode_tps": 86.6 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c4.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c4.json new file mode 100644 index 00000000000..5e864fdb27a --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-c4.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-c4", + "kind": "warmup", + "trial": 0, + "wall_s": 0.3381140189594589, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 240.0, + "prefill_tps": 62.4, + "generated": 2.0, + "decode_ms": 83.0, + "decode_tps": 24.0 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-c4", + "kind": "fresh", + "trial": 1, + "wall_s": 4.840592781023588, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3206.0, + "prefill_tps": 1313.1, + "generated": 128.0, + "decode_ms": 1629.0, + "decode_tps": 78.6 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-c4", + "kind": "followup", + "trial": 1, + "wall_s": 2.126103040005546, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 433.0, + "prefill_tps": 263.2, + "generated": 128.0, + "decode_ms": 1689.0, + "decode_tps": 75.8 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-c4", + "kind": "fresh", + "trial": 2, + "wall_s": 7.814134871994611, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6316.0, + "prefill_tps": 1398.0, + "generated": 128.0, + "decode_ms": 1492.0, + "decode_tps": 85.8 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c4", + "kind": "followup", + "trial": 2, + "wall_s": 1.9270351799787022, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 418.0, + "prefill_tps": 270.1, + "generated": 128.0, + "decode_ms": 1505.0, + "decode_tps": 85.1 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-c4", + "kind": "fresh", + "trial": 3, + "wall_s": 4.761623851954937, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3226.0, + "prefill_tps": 1305.0, + "generated": 128.0, + "decode_ms": 1531.0, + "decode_tps": 83.6 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-c4", + "kind": "followup", + "trial": 3, + "wall_s": 2.3474430550122634, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 655.0, + "prefill_tps": 378.9, + "generated": 128.0, + "decode_ms": 1689.0, + "decode_tps": 75.8 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-c4", + "kind": "fresh", + "trial": 4, + "wall_s": 7.825775476987474, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6316.0, + "prefill_tps": 1398.0, + "generated": 128.0, + "decode_ms": 1503.0, + "decode_tps": 85.2 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-c4", + "kind": "followup", + "trial": 4, + "wall_s": 1.907931160996668, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 425.0, + "prefill_tps": 265.6, + "generated": 128.0, + "decode_ms": 1479.0, + "decode_tps": 86.6 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/bench-ple-i1k.json b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-i1k.json new file mode 100644 index 00000000000..bb8ca4f4430 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/bench-ple-i1k.json @@ -0,0 +1,263 @@ +[ + { + "label": "ple-i1k", + "kind": "warmup", + "trial": 0, + "wall_s": 0.32169503503246233, + "metrics": { + "prompt_tokens": 15.0, + "reused": 0.0, + "fresh": 15.0, + "prefill_ms": 225.0, + "prefill_tps": 66.6, + "generated": 2.0, + "decode_ms": 82.0, + "decode_tps": 24.4 + }, + "usage": { + "prompt_tokens": 15, + "completion_tokens": 2, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "READY" + } + }, + { + "label": "ple-i1k", + "kind": "fresh", + "trial": 1, + "wall_s": 4.825218737008981, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3187.0, + "prefill_tps": 1321.0, + "generated": 128.0, + "decode_ms": 1633.0, + "decode_tps": 78.4 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$); if this condition is true, the function returns $x + n + 1$, and if false, it returns $x - n$. Consequently, each specific rule acts as a conditional modifier that either increments the input by one more than its index or decrements it by its index, depending" + } + }, + { + "label": "ple-i1k", + "kind": "followup", + "trial": 1, + "wall_s": 2.12476084404625, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4337.0, + "fresh": 114.0, + "prefill_ms": 429.0, + "prefill_tps": 265.8, + "generated": 128.0, + "decode_ms": 1692.0, + "decode_tps": 75.7 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4337 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, if the input `x` is exactly equal to `N`, the function returns `N + 1` (a positive offset); otherwise, it returns `x - N`. This creates a discontinuity in behavior: inputs just below or above `N` result in subtraction, while `N` itself results in addition.\n\nThe **smallest useful regression test** targets this specific equality condition because it is the only case where the output" + } + }, + { + "label": "ple-i1k", + "kind": "fresh", + "trial": 2, + "wall_s": 7.869399868010078, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6369.0, + "prefill_tps": 1386.4, + "generated": 128.0, + "decode_ms": 1494.0, + "decode_tps": 85.7 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-i1k", + "kind": "followup", + "trial": 2, + "wall_s": 1.9301651229616255, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 419.0, + "prefill_tps": 269.9, + "generated": 128.0, + "decode_ms": 1508.0, + "decode_tps": 84.9 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., calling `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input:" + } + }, + { + "label": "ple-i1k", + "kind": "fresh", + "trial": 3, + "wall_s": 4.768105736002326, + "metrics": { + "prompt_tokens": 4210.0, + "reused": 0.0, + "fresh": 4210.0, + "prefill_ms": 3231.0, + "prefill_tps": 1303.1, + "generated": 128.0, + "decode_ms": 1533.0, + "decode_tps": 83.5 + }, + "usage": { + "prompt_tokens": 4210, + "completion_tokens": 128, + "total_tokens": 4338, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its name (denoted as $n$). If this condition is true, the function returns $x + n + 1$; otherwise, it returns $x - n$. Consequently, for any given function `ruleN`, the output is $2N + 1$ when the input matches $N$, and $x -" + } + }, + { + "label": "ple-i1k", + "kind": "followup", + "trial": 3, + "wall_s": 2.3504937100224197, + "metrics": { + "prompt_tokens": 4451.0, + "reused": 4203.0, + "fresh": 248.0, + "prefill_ms": 655.0, + "prefill_tps": 378.8, + "generated": 128.0, + "decode_ms": 1692.0, + "decode_tps": 75.6 + }, + "usage": { + "prompt_tokens": 4451, + "completion_tokens": 128, + "total_tokens": 4579, + "prompt_tokens_details": { + "cached_tokens": 4203 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x === n` (the function's index), because this triggers the unique \"addition\" branch (`x + n + 1`) rather than the default \"subtraction\" branch (`x - n`). This is critical because it represents the only point where the function's behavior deviates from a simple linear transformation, and it is the only case where the output is positive and larger than the input.\n\nThe **smallest useful regression test** should verify this boundary condition for at least one representative function (e.g., `rule0` or `rule13" + } + }, + { + "label": "ple-i1k", + "kind": "fresh", + "trial": 4, + "wall_s": 7.80428749701241, + "metrics": { + "prompt_tokens": 8830.0, + "reused": 0.0, + "fresh": 8830.0, + "prefill_ms": 6294.0, + "prefill_tps": 1403.0, + "generated": 128.0, + "decode_ms": 1504.0, + "decode_tps": 85.1 + }, + "usage": { + "prompt_tokens": 8830, + "completion_tokens": 128, + "total_tokens": 8958, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical pattern: it checks if the input `x` is strictly equal to the integer value embedded in its name (e.g., `rule5` checks for `5`). If this condition is true, the function returns `x + 1`; otherwise, it returns `x - 1`. Consequently, each specific rule acts as a conditional modifier that increments the input by one only when the input matches the rule's index, and decrements the input by one" + } + }, + { + "label": "ple-i1k", + "kind": "followup", + "trial": 4, + "wall_s": 1.91029199201148, + "metrics": { + "prompt_tokens": 9071.0, + "reused": 8958.0, + "fresh": 113.0, + "prefill_ms": 426.0, + "prefill_tps": 265.1, + "generated": 128.0, + "decode_ms": 1480.0, + "decode_tps": 86.5 + }, + "usage": { + "prompt_tokens": 9071, + "completion_tokens": 128, + "total_tokens": 9199, + "prompt_tokens_details": { + "cached_tokens": 8958 + } + }, + "finish_reason": "length", + "message": { + "role": "assistant", + "content": "The most relevant boundary case is the **exact match condition** where `x` equals the rule's index (e.g., `rule5(5)`), which triggers the increment branch (`x + 1`) rather than the decrement branch (`x - 1`). This distinguishes the function from a simple arithmetic operation and validates the core logic.\n\nThe **smallest useful regression test** is:\n```javascript\nassert(rule0(0) === 1); // Boundary: x=0 matches rule index, returns 0+1=1\nassert(rule0(-1) === -2); // Negative input: does" + } + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-b.json b/dirigo-results/0.2.13-port/ple-sweep/conf-b.json new file mode 100644 index 00000000000..522f818e182 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-b.json @@ -0,0 +1,63 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-b2.json b/dirigo-results/0.2.13-port/ple-sweep/conf-b2.json new file mode 100644 index 00000000000..522f818e182 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-b2.json @@ -0,0 +1,63 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-c16.json b/dirigo-results/0.2.13-port/ple-sweep/conf-c16.json new file mode 100644 index 00000000000..6576f333a11 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-c16.json @@ -0,0 +1,65 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048", + "--ple-row-cache", + "16777216" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-c16i1k.json b/dirigo-results/0.2.13-port/ple-sweep/conf-c16i1k.json new file mode 100644 index 00000000000..a1a7529ebd3 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-c16i1k.json @@ -0,0 +1,67 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048", + "--ple-row-cache", + "16777216", + "--ple-inflight", + "1024" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-c4.json b/dirigo-results/0.2.13-port/ple-sweep/conf-c4.json new file mode 100644 index 00000000000..db5e5edede2 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-c4.json @@ -0,0 +1,65 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048", + "--ple-row-cache", + "4194304" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/conf-i1k.json b/dirigo-results/0.2.13-port/ple-sweep/conf-i1k.json new file mode 100644 index 00000000000..026287a324f --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/conf-i1k.json @@ -0,0 +1,65 @@ +{ + "exe": "/home/bob/bin/strata-sycl-host.sh", + "args": [ + "--pack", + "/home/bob/XeStrata-data/packs/coder-iq1_m", + "--native", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/home/bob/strata-models/IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/expert-profile-mine.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--prompt-cache", + "6", + "--prompt-cache-every", + "4096", + "--batch", + "0", + "--spec", + "6", + "--spec-min-p", + "0.5", + "--mtp", + "/home/bob/XeStrata-data/mtp/rt", + "--mtp-draft-vocab", + "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/data/draft_vocab_en.bin", + "--mtp-window", + "8192", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vram-reserve-mib", + "700", + "--conversation-cache-mib", + "2048", + "--ple-inflight", + "1024" + ], + "cwd": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13", + "tokenizer": "/home/bob/XeStrata-data/packs/coder-iq1_m/tokenizer", + "model_name": "qwen3.8-flash-next-coder-iq1_m", + "log": "/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731/strata-0.2.13/strata-coder-iq1_m-sycl-tuned.log", + "port": 8095, + "backend": "sycl", + "open_browser": false, + "thinking_policy": "on", + "conv_route": "on", + "conv_route_batch_after": 4, + "short_answer": { + "spec": 0, + "max_tokens": 16 + }, + "spec_window": 0, + "spec_window_needle_min": 6000, + "code_tune": { + "mode": "off" + } +} \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b.json new file mode 100644 index 00000000000..db93a367a57 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-b", + "leg": "code", + "http": 200, + "wall_s": 1.38, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 584.0, + "prefill_tps": 54.8, + "generated": 90.0, + "decode_ms": 784.0, + "decode_tps": 114.8 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-b", + "leg": "long", + "http": 200, + "wall_s": 16.62, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 16583.0, + "prefill_tps": 1819.8, + "generated": 1.0, + "decode_ms": 21.0, + "decode_tps": 48.2 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-b", + "leg": "needle", + "http": 200, + "wall_s": 5.59, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5485.0, + "prefill_tps": 1432.1, + "generated": 10.0, + "decode_ms": 103.0, + "decode_tps": 97.3 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-b", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.05 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b2.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b2.json new file mode 100644 index 00000000000..bafc157f3e1 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-b2.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-b2", + "leg": "code", + "http": 200, + "wall_s": 1.36, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 557.0, + "prefill_tps": 57.4, + "generated": 90.0, + "decode_ms": 787.0, + "decode_tps": 114.4 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-b2", + "leg": "long", + "http": 200, + "wall_s": 16.68, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 16646.0, + "prefill_tps": 1812.9, + "generated": 1.0, + "decode_ms": 21.0, + "decode_tps": 48.3 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-b2", + "leg": "needle", + "http": 200, + "wall_s": 5.61, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5499.0, + "prefill_tps": 1428.5, + "generated": 10.0, + "decode_ms": 103.0, + "decode_tps": 97.5 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-b2", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.05 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16.json new file mode 100644 index 00000000000..16c9156563b --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-c16", + "leg": "code", + "http": 200, + "wall_s": 1.11, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 309.0, + "prefill_tps": 103.7, + "generated": 90.0, + "decode_ms": 784.0, + "decode_tps": 114.8 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-c16", + "leg": "long", + "http": 200, + "wall_s": 18.44, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 18405.0, + "prefill_tps": 1639.7, + "generated": 1.0, + "decode_ms": 20.0, + "decode_tps": 48.8 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-c16", + "leg": "needle", + "http": 200, + "wall_s": 5.44, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5328.0, + "prefill_tps": 1474.4, + "generated": 10.0, + "decode_ms": 102.0, + "decode_tps": 97.6 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-c16", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.05 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16i1k.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16i1k.json new file mode 100644 index 00000000000..2c2bbc1c2d9 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c16i1k.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-c16i1k", + "leg": "code", + "http": 200, + "wall_s": 1.12, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 318.0, + "prefill_tps": 100.7, + "generated": 90.0, + "decode_ms": 785.0, + "decode_tps": 114.6 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-c16i1k", + "leg": "long", + "http": 200, + "wall_s": 19.18, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 19143.0, + "prefill_tps": 1576.5, + "generated": 1.0, + "decode_ms": 20.0, + "decode_tps": 48.8 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-c16i1k", + "leg": "needle", + "http": 200, + "wall_s": 5.45, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5342.0, + "prefill_tps": 1470.4, + "generated": 10.0, + "decode_ms": 103.0, + "decode_tps": 97.5 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-c16i1k", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.05 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c4.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c4.json new file mode 100644 index 00000000000..161869fc397 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-c4.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-c4", + "leg": "code", + "http": 200, + "wall_s": 1.74, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 941.0, + "prefill_tps": 34.0, + "generated": 90.0, + "decode_ms": 784.0, + "decode_tps": 114.8 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-c4", + "leg": "long", + "http": 200, + "wall_s": 16.75, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 16679.0, + "prefill_tps": 1809.3, + "generated": 1.0, + "decode_ms": 21.0, + "decode_tps": 48.8 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-c4", + "leg": "needle", + "http": 200, + "wall_s": 5.59, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5486.0, + "prefill_tps": 1431.7, + "generated": 10.0, + "decode_ms": 103.0, + "decode_tps": 97.5 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-c4", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.06 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-i1k.json b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-i1k.json new file mode 100644 index 00000000000..cbdc08fb0e8 --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/longprobe-ple-i1k.json @@ -0,0 +1,87 @@ +[ + { + "label": "ple-i1k", + "leg": "code", + "http": 200, + "wall_s": 1.36, + "metrics": { + "prompt_tokens": 32.0, + "reused": 0.0, + "fresh": 32.0, + "prefill_ms": 563.0, + "prefill_tps": 56.8, + "generated": 90.0, + "decode_ms": 785.0, + "decode_tps": 114.7 + }, + "usage": { + "prompt_tokens": 32, + "completion_tokens": 90, + "total_tokens": 122, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "```python\nclass ListNode:\n def __init__(self, val=0, next=None):\n self.val = val\n self.next = next\n\ndef reverse_list(head: ListNode) -> ListNode:\n prev = None\n curr = head\n while curr:\n next_node = curr.next\n curr.next = prev\n prev = curr\n curr = next_node\n return prev\n```" + }, + { + "label": "ple-i1k", + "leg": "long", + "http": 200, + "wall_s": 16.63, + "metrics": { + "prompt_tokens": 30178.0, + "reused": 0.0, + "fresh": 30178.0, + "prefill_ms": 16596.0, + "prefill_tps": 1818.4, + "generated": 1.0, + "decode_ms": 21.0, + "decode_tps": 48.4 + }, + "usage": { + "prompt_tokens": 30178, + "completion_tokens": 1, + "total_tokens": 30179, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "length", + "content": "Every" + }, + { + "label": "ple-i1k", + "leg": "needle", + "http": 200, + "wall_s": 5.57, + "metrics": { + "prompt_tokens": 7855.0, + "reused": 0.0, + "fresh": 7855.0, + "prefill_ms": 5460.0, + "prefill_tps": 1438.6, + "generated": 10.0, + "decode_ms": 102.0, + "decode_tps": 97.6 + }, + "usage": { + "prompt_tokens": 7855, + "completion_tokens": 10, + "total_tokens": 7865, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "finish": "stop", + "content": "ZQ-7731-KX" + }, + { + "label": "ple-i1k", + "leg": "overflow", + "http": 400, + "body": "{\"error\": {\"type\": \"invalid_request_error\", \"message\": \"request exceeds the context window: prompt (165779 tokens) leaves no room to answer in the context (131072); requests are never truncated\"}}", + "wall_s": 0.05 + } +] diff --git a/dirigo-results/0.2.13-port/ple-sweep/strata-ple-sweep.sh b/dirigo-results/0.2.13-port/ple-sweep/strata-ple-sweep.sh new file mode 100755 index 00000000000..f541abcea7d --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/strata-ple-sweep.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# strata-ple-sweep.sh - sweep the PLE read path's two untested knobs on the Arc. +# +# The defaults are `--ple-row-cache 1048576` (rows of 90 B = 94 MB) and `--ple-inflight 256`. The engine's own +# comment records that inflight 64 left the SSD half idle (32K read 303 -> 189 ms), so this knob family is known +# to matter and has never been swept on this box. +# +# Arms, one card window each, in this order so drift lands on both ends: +# b default (row cache 1 M, inflight 256) +# c4 --ple-row-cache 4194304 (360 MB) +# c16 --ple-row-cache 16777216 (1.44 GB) +# i1k --ple-inflight 1024 +# c16i1k both +# b2 default again (drift check) +# +# Each window: stop strata-b70, run the tree's own server on :8097 with a variant config, run bench_prefill (fresh +# 4.2k/8.8k + follow-ups + warmup) and longprobe (code, needle, 30k, over-window), restore the service. +# +# UNDO: rm this file and ~/strata-ple-sweep +set -uo pipefail +S=/media/bob/3d651e2c-e9a4-4758-ba77-863725fe3731 +TREE=$S/strata-0.2.13 +BASE=$TREE/strata-coder-iq1_m-sycl-tuned.json +OUT=$HOME/strata-ple-sweep +BENCH=/home/bob/forktest/engines/strata/tools/hip/bench_prefill.py +LONG=/home/bob/bin/strata-longprobe.py +MODEL=qwen3.8-flash-next-coder-iq1_m +mkdir -p "$OUT" + +variant() { # name extra-args... + local name=$1; shift + python3 - "$BASE" "$OUT/conf-$name.json" "$@" <<'PY' +import json, sys +src, dst, extra = sys.argv[1], sys.argv[2], sys.argv[3:] +d = json.load(open(src)) +d["args"] = list(d.get("args", [])) + extra +json.dump(d, open(dst, "w"), indent=1) +print(dst, "extra:", extra) +PY +} + +arm() { # tag + local tag=$1 + local label="ple-$tag" + local conf="$OUT/conf-$tag.json" + echo "[$(date +%H:%M:%S)] ===== $label" + ~/bin/strata-card-window.sh "$TREE" 8097 --conf "$conf" -- bash -c " + EL=\$(python3 -c \"import json;print(json.load(open('$conf'))['log'])\") + curl -s -m 5 http://127.0.0.1:\$PORT/v1/status | python3 -c 'import json,sys;d=json.load(sys.stdin);print(\" engine:\",d[\"engine\"])' + /usr/bin/python3 $BENCH --url http://127.0.0.1:\$PORT --model $MODEL --engine-log \"\$EL\" \ + --output $OUT/bench-$label.json --label $label >/dev/null 2>&1 + /usr/bin/python3 $LONG --url http://127.0.0.1:\$PORT --model $MODEL --engine-log \"\$EL\" \ + --label $label --out $OUT/longprobe-$label.json >/dev/null 2>&1 + echo ' probes written:' \$(ls -1 $OUT/*-$label.json 2>/dev/null | wc -l) + curl -s -m 5 http://127.0.0.1:\$PORT/v1/status | python3 -c 'import json,sys;d=json.load(sys.stdin);print(\" status:\", {k:d.get(k) for k in (\"loaded\",\"engine\")})' + " 2>&1 | grep -E '=====|engine:|probes written|status:|restore' +} + +variant b +variant b2 # identical to b, used for the closing drift check +variant c4 --ple-row-cache 4194304 +variant c16 --ple-row-cache 16777216 +variant i1k --ple-inflight 1024 +variant c16i1k --ple-row-cache 16777216 --ple-inflight 1024 + +for tag in b c4 c16 i1k c16i1k b2; do + arm "$tag" +done +echo "[$(date +%H:%M:%S)] sweep done" diff --git a/dirigo-results/0.2.13-port/ple-sweep/summary.json b/dirigo-results/0.2.13-port/ple-sweep/summary.json new file mode 100644 index 00000000000..8b35b0cd4ac --- /dev/null +++ b/dirigo-results/0.2.13-port/ple-sweep/summary.json @@ -0,0 +1,226 @@ +[ + { + "leg": "warmup", + "prompt": 15, + "b": [ + 71.7, + 24.4 + ], + "c4": [ + 62.4, + 24.0 + ], + "c16": [ + 63.8, + 23.7 + ], + "i1k": [ + 66.6, + 24.4 + ], + "c16i1k": [ + 65.3, + 24.2 + ], + "b2": [ + 68.4, + 24.3 + ] + }, + { + "leg": "code", + "prompt": 32, + "b": [ + 54.8, + 114.8 + ], + "c4": [ + 34.0, + 114.8 + ], + "c16": [ + 103.7, + 114.8 + ], + "i1k": [ + 56.8, + 114.7 + ], + "c16i1k": [ + 100.7, + 114.6 + ], + "b2": [ + 57.4, + 114.4 + ] + }, + { + "leg": "fresh", + "prompt": 4210, + "b": [ + 1310.4499999999998, + 81.30000000000001 + ], + "c4": [ + 1309.05, + 81.1 + ], + "c16": [ + 1205.4, + 81.05000000000001 + ], + "i1k": [ + 1312.05, + 80.95 + ], + "c16i1k": [ + 1233.35, + 80.9 + ], + "b2": [ + 1312.15, + 80.85 + ] + }, + { + "leg": "followup", + "prompt": 4451, + "b": [ + 324.75, + 75.95 + ], + "c4": [ + 321.04999999999995, + 75.8 + ], + "c16": [ + 268.7, + 75.75 + ], + "i1k": [ + 322.3, + 75.65 + ], + "c16i1k": [ + 322.95000000000005, + 75.7 + ], + "b2": [ + 320.6, + 75.6 + ] + }, + { + "leg": "needle", + "prompt": 7855, + "b": [ + 1432.1, + 97.3 + ], + "c4": [ + 1431.7, + 97.5 + ], + "c16": [ + 1474.4, + 97.6 + ], + "i1k": [ + 1438.6, + 97.6 + ], + "c16i1k": [ + 1470.4, + 97.5 + ], + "b2": [ + 1428.5, + 97.5 + ] + }, + { + "leg": "fresh", + "prompt": 8830, + "b": [ + 1344.1, + 85.75 + ], + "c4": [ + 1398.0, + 85.5 + ], + "c16": [ + 1230.6999999999998, + 84.94999999999999 + ], + "i1k": [ + 1394.7, + 85.4 + ], + "c16i1k": [ + 1224.25, + 85.35 + ], + "b2": [ + 1391.6999999999998, + 85.35 + ] + }, + { + "leg": "followup", + "prompt": 9071, + "b": [ + 270.8, + 86.05000000000001 + ], + "c4": [ + 267.85, + 85.85 + ], + "c16": [ + 264.5, + 85.85 + ], + "i1k": [ + 267.5, + 85.7 + ], + "c16i1k": [ + 255.14999999999998, + 85.75 + ], + "b2": [ + 269.25, + 85.5 + ] + }, + { + "leg": "long", + "prompt": 30178, + "b": [ + 1819.8, + 48.2 + ], + "c4": [ + 1809.3, + 48.8 + ], + "c16": [ + 1639.7, + 48.8 + ], + "i1k": [ + 1818.4, + 48.4 + ], + "c16i1k": [ + 1576.5, + 48.8 + ], + "b2": [ + 1812.9, + 48.3 + ] + } +] \ No newline at end of file diff --git a/docs/PERF-NEXT.md b/docs/PERF-NEXT.md new file mode 100644 index 00000000000..14028f52985 --- /dev/null +++ b/docs/PERF-NEXT.md @@ -0,0 +1,90 @@ +# Performance: what has been tried, and what is left + +This page exists so the next session does not re-derive a month of measurements. It covers the Arc Pro B70 +(SYCL, `bmg-g31`) deployment and the fork's tree; every line points at the evidence, and the closed list is as +important as the open one. + +## 1. Tried and kept + +| lever | what it bought | recorded in | +| --- | --- | --- | +| conversation-cache routing (server) | first token **−22.1 %** at one stream against stock 0.2.13; the engine itself at parity | [`0.2.13-port/`](../dirigo-results/0.2.13-port/) | +| answer-length draft depth (`spec=4` at ≤16 tokens) | **+11.3 %** generation rate on short answers | `serve/`, release notes 0.2.10 | +| thinking suppressed for short asks | **−58 % wall, −89.5 % tokens** on direct-answer turns | `serve/prompt_kind.py` | +| prompt parking off | **−25…−30 %** on follow-ups (it recovered a cost our launcher had added) | launcher, 0.2.10 round | +| model on NVMe instead of the spinning disk | a 2,655-token code prompt went from *watchdog kill* to **3.0 s** | `PERF-IDEAS.md` §6 | +| calibration: one load, warm path, skipped confirmations | **394 s → 276 s** per calibration | `tools/calibrate.py`, `AUTOTUNE.md` | +| KV `int8` + `--kv-resident 32768` | the only KV format that pays here | release notes, `AMD_HIP_PERFORMANCE.md` | +| prefill chunks 8,192 (upstream 0.1.13), layer-major prefill (0.2.7) | prompt reads at 1.3–1.8k tok/s on long prompts | `docs/DETAILS.md` history table | +| engines 0.2.10 → 0.2.13 | decode +0.4…+1.2 % per release | [`notes/`](../dirigo-results/notes/) | + +## 2. Tried and closed — with the number that killed it + +| lever | result | +| --- | --- | +| `--batch 4` (several requests at once) | **wedges the GPU** (`UR_RESULT_ERROR_DEVICE_LOST`, Xe reset); batch 2 is stable but **−9…−36 %** throughput | +| `--kv-grow` | neutral on this box (prefill 1 172 vs 1 170 tok/s, decode 74.0 vs 69.1 within spread) | +| `--suffix-draft` 0/15/31/63, `--lookup-chain`, per-request `lookup` | inert: 71.6 / 72.7 / 72.4 / 71.7 tok/s; ±0.5 % on every lane | +| `--prefill auto:16384` / `:32768` | +8.3 % on a fresh 8.8k, but **+21 % wall** on a cached follow-up; 32k also −5.0 % on 30k | +| `--prompt-cache-tail` | every lane within ±0.5 % | +| `--conversation-cache-mib 4096` | flat (the eviction counter's advice did not survive a paired run) | +| `--spec-min-p 0.7` | **−7.0 %** decode on code; 0.5 stands | +| `--mtp-max-t` as an independent knob | it is `--spec` by another name | +| 4-bit KV (`k8v4`, `q4_0`) | prefill **−26 % / −29 %** | +| `--ple-io ram` | needs the 28.8 GB table in RAM; this box has 30 GB total, and the 92 GB box OOM'd under a 58 GB model set | +| `--expert-cache` size arms | flat: 58.19–58.21 tok/s across three arms | +| `--ple-row-cache` 4 M / 16 M rows, `--ple-inflight` 1024 | 4 M rows and inflight 1024 are **flat** (inside the 0.2–3.6 % drift band); 16 M rows (1.44 GB) costs **−8 % on fresh 4.2k, −8…−12 % on 8.8k, −17 % on a 4.4k follow-up, −10 % on 30k**. The 94 MB default and inflight 256 stand — [the sweep](../dirigo-results/0.2.13-port/ple-sweep/) | +| `spec_window` policy | recovered only half the retrieval loss (+1.3…4.8 % vs +3.6…7.5 %) — left off | +| DFlash2 / DSpark drafters | **CUDA/HIP only**; the SYCL delta in 0.2.12 is 3 lines in `prefill.cpp` | + +## 3. Where the remaining time is + +Fitted over the 18 fresh legs of the 0.2.13 battery: **prefill ≈ 336 ms fixed + 0.67 ms/token** (1,492 tok/s +marginal). A 113-token prompt costs 409 ms, of which ~330 ms is not tokens. Decode runs 48–115 tok/s depending on +the lane. The engine is within 1.1 % of stock on all eight legs, so what is left is either upstream's or in that +fixed term. + +## 4. Not tried, ranked + +| # | idea | evidence it could pay | cost | first test | +| --- | --- | --- | --- | --- | +| 1 | ship `conv-route-batch-after` 1–2 | measured **−21 % median TTFT** (0.371 → 0.293 s), summed −29 %, steady state at request 2 instead of 5; the default is still 4 | a flag + a probe | a chat-after-single-shot shape to price the flip-back | +| 2 | attribute the ~330 ms fixed per-request cost | nobody has split it on SYCL: slot init, expert warm-up for the first layers, KV allocation. The PLE read path is now excluded by measurement (§5) | an hour, no code | `bench_prefill` at 8/16/32/64/128/256 tokens | +| 3 | residency-biased router on SYCL | `route_prior.cu` is CUDA-only; here **12,288 of 24,576** experts are resident, so half of every token's experts cross PCIe. Upstream measured +15 % / +37 % on a 12 GB NVIDIA card | engine port, days | decode lanes, 5 interleaved pairs | +| 4 | prefill chunk at 4k/8k/12k with intervals | 16k/32k were tested, but 8,192 was never shown to be the knee | ~20 min | paired, three prompt sizes, per-prompt medians | +| 5 | CPU pool workers and affinity on the 7700X | 15 workers is a 16-thread default; p-core pinning was **+27 %** decode on the 13700K | 15 min | workers 8/12/15/16, `--pool-affinity` on/off | +| 6 | dispatch overhead on SYCL | the Vulkan analysis found ~88 % of per-dispatch cost is fixed; nobody has counted dispatches per token on SYCL | a day | `unitrace` kernel count per token, then fuse the five most frequent small ops | +| 7 | DFlash2 block drafter on SYCL | upstream's headline decode lever; 7 drafts per pass | weeks | only after 1–3 | + +## 5. The PLE sweep, run 2026-11-14 — closed + +Six card windows on the 0.2.13 tree, `bench_prefill` plus `longprobe` in each, `strata-b70` restored between +windows and proven with a real completion. Prefill tok/s, median per leg: + +| leg (prompt) | default | cache 4 M | cache 16 M | inflight 1024 | both | default again | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| fresh (4,210) | 1,310 | 1,309 | **1,205** | 1,312 | 1,233 | 1,312 | +| follow-up (4,451) | 325 | 321 | **269** | 322 | 323 | 321 | +| needle (7,855) | 1,432 | 1,432 | 1,474 | 1,439 | 1,470 | 1,428 | +| fresh (8,830) | 1,344 | 1,398 | **1,231** | 1,395 | 1,224 | 1,392 | +| follow-up (9,071) | 271 | 268 | 264 | 268 | 255 | 269 | +| long (30,178) | 1,820 | 1,809 | **1,640** | 1,818 | **1,576** | 1,813 | + +The two default arms bound the drift: 0.2 % on the 4.2k leg, 3.6 % on the 8.8k one, ≤1.2 % elsewhere. Against that +band, **cache 4 M rows and inflight 1024 are flat, and cache 16 M rows is 8–17 % slower** — the 1.44 GB allocation +costs more than the row hits it buys on unique prompts, and inflight 256 was not starving the drive. + +What this rules out: the fixed ~330 ms per request is **not** PLE row-read starvation. What it does not rule out: a +bigger cache helping repeat traffic that re-reads the same n-grams — no leg here does that, because follow-ups +reuse the KV cache instead of re-reading rows. + +Artifacts: [`0.2.13-port/ple-sweep/`](../dirigo-results/0.2.13-port/ple-sweep/) — the six configs, the per-leg JSON, the log with each +window's restore line, `summary.json`, and the script. + +## 6. Method rules these numbers were produced under + +- Paired or interleaved arms; median of repeats; the box's run-to-run spread stated next to the effect. +- One prompt per configuration is not a measurement: every lane here has several requests, and a single-prompt + difference is only believed when it repeats. +- A missing or failed sample makes a candidate *unknown*, not slow. +- A lever that only shows a win through a second, unexplained side effect is not adopted. diff --git a/docs/README.md b/docs/README.md index cb5aea4f39f..71897cd41d0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -59,6 +59,7 @@ Use the guide for your GPU and operating system. Model and feature support can d ## This fork +- [Performance: what has been tried, and what is left](PERF-NEXT.md) - [Verified build, measurements and evidence](../DIRIGO.md) - [Measurements, suite logs and the carry patch](../dirigo-results/) - [Glossary of every term](GLOSSARY.md)