Upgrade agentic Kimi model to K3 - #450
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #450 +/- ##
=======================================
Coverage ? 80.45%
=======================================
Files ? 150
Lines ? 20211
Branches ? 0
=======================================
Hits ? 16261
Misses ? 3950
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| target_concurrency: 8 # Submission-specific concurrency. | ||
|
|
||
| client: | ||
| num_workers: 4 |
There was a problem hiding this comment.
num workers is 10 (upto 24 if numa domain allows) by default (-1), why reduce deliberately?
| client: | ||
| num_workers: 4 | ||
| warmup_connections: 0 | ||
| max_connections: -1 |
There was a problem hiding this comment.
nit: no need to reset defaults (like max-conn -1), keep examples minimal
| warmup_connections: 0 | ||
| max_connections: -1 | ||
| max_idle_time: 0.5 | ||
| worker_initialization_timeout: 600.0 |
There was a problem hiding this comment.
q: did u need 600s for 4 client workers? that seems weird
also for future i think should auto scale worker init timeout based on num workers
(i can include in my next mr)
| | Inline accuracy | `>= 63.08%` | `>= 55.86%` | | ||
| | OSL per-turn mean | `404-494` tokens | `355-434` tokens | | ||
| | SWE-bench accuracy | `>= 76.5%` | `>= 67.5%` | | ||
| | Inline accuracy | `>= 58.32%` | `>= 55.86%` | |
There was a problem hiding this comment.
Did the inline accuracy drop from K2.6 to K3?
5 participants
Summary
Validation
kimi_agentic_benchmark.yamlthroughBenchmarkConfig.from_yaml_fileuv run pre-commit run --all-files