Skip to content

Add isolated story model evaluation - #113

Draft
R055LE wants to merge 4 commits into
issue-107-story-contractfrom
issue-110-model-evaluation
Draft

R055LE wants to merge 4 commits into
issue-107-story-contractfrom
issue-110-model-evaluation

Conversation

@R055LE

@R055LE R055LE commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Refs #110. Draft, stacked on #109 (issue-107-story-contract); this diff contains evaluation tooling and fixtures only.

  • Add isolated python -m roger.story_eval plan|run|summarize, reusing actual story prompts, schema, strict parser, source preparation, renderer and bounded revision workflow.
  • Freeze six reporting cases and twelve balanced reviewer fixtures, with exact model/provider/settings inventories and pending human labels.
  • Pin dated models and providers; reserve finite token/cost bounds before calls; preserve visible output, request/input snapshots, usage, route metadata and partial failures. Keep uncertain reservations, no silent retry/resume/fallback, and no credentials or hidden reasoning in artifacts.
  • Correct live-observed malformed-response handling, narrowly classify the verified passed moderation annotation, fix source-inventory identity, and repair insufficient reference citations. Refs Preserve malformed evaluation responses and classify router annotations #114.

Approved study and results

The owner approved a $100 / 196-call ceiling, staged by findings. The completed October 13 slices used 176 attempts / $2.54123660 recorded charges, plus $0.30 retained uncertain reservation. All evaluation containers exited. Production configuration and publication were outside the execution boundary.

Sol writer + Astra reviewer is the strongest candidate: 6/6 synthetic workflows accepted, compared with 0/6 Sonnet/Astra. Astra calibration agrees with all 24 current authored fixture-repeat labels, using 18 unchanged baseline results plus six targeted results; original failures remain preserved and human label review is pending.

The held-out current-evidence workflow passed parsing and claim support but failed independent agent editorial assessment for material incompleteness. This PR does not establish production acceptance. Full results, assumptions and next tasks are recorded on #110. Follow-ups: #115 completeness/public rendering, #116 production request profiles, #117 reservation accounting/capacity. #97/#107 retain their unmet live acceptance.

Validation

  • Full suite after code fixes: 627 passed, 87.14% coverage (75% required).
  • Final citation-only patch: 35 focused tests passed.
  • Ruff and diff checks passed.
  • Final-head GitHub CI: Python 3.12 and 3.14 both passed, including configured lint, tests/coverage and dependency audit.
  • Independent provider, accounting, fixture projection, request-equivalence and editorial reviews completed. Agent editorial assessment is separate from human acceptance.

SDK/MockTransport sandbox behavior is tracked in R055LE/agent-platform#116; independent full tests used the real SDK boundary with dummy credentials and no provider calls.

Boundary

Keep this PR draft/unmerged. No production model, output/context limit, daily budget, schedule, destination, credentials, story activation or delivery-state change. Current source preview used a manually selected packet, not Scout acquisition. The next acceptance run must cover the production wrapper and actual public rendering after the linked fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant