Repository navigation
fix(pro): use an example that is not a bench problem - #11
Merged
Merged
Conversation
The pro example conversation solved parse_duration, which is also the bench problem py-parse-duration: the model would have seen the answer and its score would have been inflated. The example now validates an IPv4 address, and a test fails if the example ever matches a bench problem again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Partial results written by the running bench; the final run replaces them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Tromset
marked this pull request as ready for review
October 8, 2026 19:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The worked example in
Modelfile.prosolvedparse_duration, which is also the bench problempy-parse-duration. Every conversation withflycoder0.3prowould carry a near-answer to that problem, so its bench score would be inflated.What changes
is_valid_ipv4), a task that is not in the bench. It keeps the same shape: assumption, code, pytest tests, run command.tests/modelfile.test.mjsfails if the example's request names the function of any bench problem.Verification
npm test: 67 tests pass.npm testfails if the leading-zero check is removed from the example.flycoder-pro-recipe9b) is rebuilt with this example, and its A/B bench is queued. The bench results will come in a follow-up commit.🤖 Generated with Claude Code
https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Generated by Claude Code