Benchmark OpenCode Qwen runtime performance - #4796
Merged
Merged
Conversation
atomantic
force-pushed
the
codex/model-performance-agent-benchmarks
branch
from
August 22, 2026 02:53
9e8d394 to
c49106f
Compare
atomantic
enabled auto-merge (squash)
August 22, 2026 02:53
atomantic
added a commit
that referenced
this pull request
Aug 22, 2026
…4774) #4796 gave the model sweep the reset and the per-backend capture/restore #4774 asked for, but it refuses the whole sweep when any sweepable runtime's launch line cannot be captured. `captureLlamaServerConfig` returns null whenever llama-server is stopped or was started outside PortOS — so one externally-started daemon now cancels every Ollama and LM Studio model queued alongside it, where before the sweep simply measured them. Gate the reset on having CAPTURED that runtime's launch line rather than on `isTuningSweepable` alone, and stop refusing: - A model sweep measures an uncapturable runtime under whatever is running, as it did before the reset existed. A reset renders the cleared launch line, which wipes knobs the user set on the LLMs page — only a caller holding what was running there may do that, and nothing was captured to put back. - A tuning sweep still refuses. Varying the launch line is its entire job, so an uncapturable runtime leaves it nothing to do. The asymmetry is the point. Also drops the now-dead `run.launchState` compatibility alias and the "nothing to restore" branch in `restoreUnderClaim`, which is unreachable: only captured backends reach the restore map. Adds coverage for the model-sweep dimension #4774 named — two llama targets with different stored tunings each getting the complete tuning its record names, one capture per runtime for the whole queue, a mixed llama+Ollama sweep resetting only llama, the uncapturable-runtime fallback, and a multi-runtime restore where one daemon throws and the other still comes back.
This was referenced Aug 22, 2026
atomantic
added a commit
that referenced
this pull request
Aug 22, 2026
…4774) #4796 gave the model sweep the reset and the per-backend capture/restore #4774 asked for, but it refuses the whole sweep when any sweepable runtime's launch line cannot be captured. `captureLlamaServerConfig` returns null whenever llama-server is stopped or was started outside PortOS — so one externally-started daemon now cancels every Ollama and LM Studio model queued alongside it, where before the sweep simply measured them. Gate the reset on having CAPTURED that runtime's launch line rather than on `isTuningSweepable` alone, and stop refusing: - A model sweep measures an uncapturable runtime under whatever is running, as it did before the reset existed. A reset renders the cleared launch line, which wipes knobs the user set on the LLMs page — only a caller holding what was running there may do that, and nothing was captured to put back. - A tuning sweep still refuses. Varying the launch line is its entire job, so an uncapturable runtime leaves it nothing to do. The asymmetry is the point. Also drops the now-dead `run.launchState` compatibility alias and the "nothing to restore" branch in `restoreUnderClaim`, which is unreachable: only captured backends reach the restore map. Adds coverage for the model-sweep dimension #4774 named — two llama targets with different stored tunings each getting the complete tuning its record names, one capture per runtime for the whole queue, a mixed llama+Ollama sweep resetting only llama, the uncapturable-runtime fallback, and a multi-runtime restore where one daemon throws and the other still comes back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The agent benchmark is user-triggered, runs in a disposable scratch directory, and does not persist model output or transcripts.
Test plan
cd server && npm test— 1,563 test files passed, 2 skipped; 32,731 tests passed, 33 skipped.cd client && npm test— 726 test files passed; 9,290 tests passed.npm run build— passed.git diff --check main...HEAD— passed.No merge requested; this PR is left open for review.