Repository navigation
Conversation
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #110. Draft, stacked on #109 (
issue-107-story-contract); this diff contains evaluation tooling and fixtures only.python -m roger.story_eval plan|run|summarize, reusing actual story prompts, schema, strict parser, source preparation, renderer and bounded revision workflow.Approved study and results
The owner approved a $100 / 196-call ceiling, staged by findings. The completed October 13 slices used 176 attempts / $2.54123660 recorded charges, plus $0.30 retained uncertain reservation. All evaluation containers exited. Production configuration and publication were outside the execution boundary.
Sol writer + Astra reviewer is the strongest candidate: 6/6 synthetic workflows accepted, compared with 0/6 Sonnet/Astra. Astra calibration agrees with all 24 current authored fixture-repeat labels, using 18 unchanged baseline results plus six targeted results; original failures remain preserved and human label review is pending.
The held-out current-evidence workflow passed parsing and claim support but failed independent agent editorial assessment for material incompleteness. This PR does not establish production acceptance. Full results, assumptions and next tasks are recorded on #110. Follow-ups: #115 completeness/public rendering, #116 production request profiles, #117 reservation accounting/capacity. #97/#107 retain their unmet live acceptance.
Validation
SDK/MockTransport sandbox behavior is tracked in R055LE/agent-platform#116; independent full tests used the real SDK boundary with dummy credentials and no provider calls.
Boundary
Keep this PR draft/unmerged. No production model, output/context limit, daily budget, schedule, destination, credentials, story activation or delivery-state change. Current source preview used a manually selected packet, not Scout acquisition. The next acceptance run must cover the production wrapper and actual public rendering after the linked fixes.