Skip to content

Constrain story output and bound validation retries - #109

Draft
R055LE wants to merge 1 commit into
mainfrom
issue-107-story-contract
Draft

R055LE wants to merge 1 commit into
mainfrom
issue-107-story-contract

Conversation

@R055LE

@R055LE R055LE commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Refs #107. Draft: live editorial acceptance is still blocked. Do not merge or activate story mode on this evidence.

  • Send provider-enforced strict schemas for story drafts/revisions and independent review, requiring parameter support while preserving the configured model order.
  • Keep local field/count/length/exact-quote validation and independent evidence/focus review. Require at least one citation for every claim; reject claims on hold/skip and duplicate optional singleton fields.
  • Count schemas within the 18,000-character request bound and policy fingerprint. Preserve full evidence and original citation ordinals.
  • Use the existing single revision allowance for initial local-validation failure as well as a negative review. Validation recovery receives original evidence and a static error, never malformed output. Paths use two, three or four calls, with no second revision.
  • Treat truncation, refusal, content filtering and blank responses as terminal. Distinguish text/quote length failures from empty text and non-exact quotes.

No production models, daily budgets, schedules, destination, database schema or story activation changes.

Validation

  • Full suite: 592 passed, 88.59% coverage, above the required 75% gate.
  • Ruff and git diff --check: passed.
  • Independent review: no code blockers, including the final diagnostic correction.
  • Actual OpenAI SDK/MockTransport tests verify the complete story schemas, provider filter, fallback array and 4,096/900 output ceilings at the HTTP JSON boundary.
  • Regressions cover malformed/overlong responses, exact quotes, mixed-topic review rejection, original source ordinals, terminal responses, one-revision limits, schema-aware request boundaries and no-send state isolation.
  • Full-source validation/revision fixtures remain below 18,000 characters with all eight source excerpts retained. Oversize requests still fail before spending.

Live acceptance remains unmet

Both no-send previews used current Scout evidence, the existing model chain and shared budget. Every completion reported finish_reason=stop under the strict schema; every response still failed local validation before independent review.

Run Outcome Tokens Cost
First structured draft Complete shape and one focused story, but reason 541 > 300, quote 764 > 500 and uncited stage 5,966 $0.0010031
Final candidate draft + one validation revision Initial change quote 514 > 500 and stage quote 25 < 40; revision reason 480 > 300, reason quote 947 > 500 and stage quote 23 < 40 11,832 $0.0019755

The final run's actual draft/revision request sizes, including schemas, were 17,827 and 17,648 characters. Claim minItems: 1 prevented empty citations, but quote-length rules still failed. The first draft's broad reason also remained a likely focus-review concern.

Zero independent-review calls and zero Discord sends occurred. Before/after comparisons matched for story, decision, delivery, seen, observation, check/lease and all Curated metadata rows. Per-call usage and costs reconciled with database accounting.

The final diagnostic-only correction separates length from substring failure. It passed automated checks but has not had a live trial. At the end of the October 3 run, 492 of 30,000 tokens remained, so further full-workflow calls stopped. This is a historical balance.

Next step, revised 2026-10-06

Keep #107 open and this PR in draft. Live acceptance remains unmet.

The owner questioned selecting Haiku and fitting the experiment into the existing token gate before establishing task fit. #110 replaces the pending Haiku-only/+15,000-token proposal with a quality-first comparison of capable current models, separate reviewer calibration, a review of adjustable limits, and budget recommendations based on complete useful workflows.

The initial shortlist is GPT-6 Astra, GPT-6.1 Sol, Claude Sonnet 5.5 and Gemini 3.8 Flash. No paid alternate-model comparison or budget override has run. Exact candidates, current provider compatibility, pricing and the evaluation contract live in #110.

A useful focused result must still pass exact support validation and independent review before production acceptance. Field-length, context and token constraints may be revised only through an explicit evaluated diff. This PR does not change production configuration or activate story mode.

Provider capability research and intermediate live evidence remain recorded in #107. Portable schemas intentionally omit unsupported maxItems/maxLength; the current parser still enforces those limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant