Skip to content

fix(harbour): strengthen repair reliability from real Anna reruns - #37

Draft
zedarvates wants to merge 34 commits into
mainfrom
agent/storycore-harbour-bootstrap
Draft

fix(harbour): strengthen repair reliability from real Anna reruns#37
zedarvates wants to merge 34 commits into
mainfrom
agent/storycore-harbour-bootstrap

Conversation

@zedarvates

@zedarvates zedarvates commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • preserve the privacy-safe evidence from the rejected cross-model experiment and the complete real Anna corpus runs;
  • keep the fail-closed A06 evidence showing that missing creative structure cannot be reconstructed deterministically;
  • strengthen the single allowed model-repair pass with a schema-complete required-field checklist and hard structural invariants;
  • retain regression coverage for exact input preservation, private-response exclusion, canonical validation, forced colours, browser flows, and Marketplace-logo paths;
  • document the direct Host LLM structured-output boundary without adding an Executa or changing the Host-API-only MVP architecture.

Current measured evidence

  • Anna default, MiniMax M3 through OpenRouter on 27 August: 6/20; median 49.13 s; p95 57.87 s; 2 repaired passes;
  • Gemma advisory preference before the schema-complete repair on 27 August: 15/20; median 21.91 s; p95 41.92 s; 6 repaired passes;
  • Gemma after the schema-complete repair on 31 August: 15/20; median 23.90 s; p95 44.00 s; 7 repaired passes;
  • post-repair failures: HBR-A01 unknown after one transport failure, plus HBR-A06/A08/A15/A20 json_invalid;
  • a bounded offline punctuation-only experiment could recover only two malformed cases, projecting at most 17/20, and was removed.

The authoritative real-model gate remains below the required 18/20. No score improvement is claimed from the repair change.

Reliability change

The repair request explicitly enumerates every required project, production-bible, character, location, scene, shot, continuity-report, and warning field. It reiterates the three-scene/one-shot budget, reference integrity, ordering, array/string types, warning severities, continuity-score bounds, original-input preservation, and 12,000-character ceiling.

The repair still:

  • discards the previous malformed/private model response;
  • rebuilds only from the original user input plus privacy-local validation errors;
  • gets exactly one attempt;
  • remains subject to the unchanged canonical validator;
  • fails closed if required creative structure is absent or JSON remains invalid.

Structured-output boundary

Anna currently documents json_object and json_schema for Executa sampling, not for direct iframe anna.llm.complete. Pinned CLI 0.1.30 and read-only inspection of CLI 0.1.49 expose no supported direct Host API response-format field. Moving Harbour into an Executa solely for structured output would violate the approved Host-API-only MVP boundary.

review/ANNA_STRUCTURED_OUTPUT_REQUEST.md contains the bounded support question and one-prompt validation gate. Nothing has been sent, no package was upgraded, and no speculative model call was made for that investigation.

Validation

  • StoryCore Harbour CI: green on head e076230;
  • SonarQube Cloud Quality Gate: green;
  • PR merge state: clean;
  • private JSONL and harness logs remain ignored and are not attached to this PR.

Release boundary

This remains a draft PR. It does not authorize an immutable Anna version, App Review submission, merge, installation claim, or publication. StoryCore Harbour remains a mutable Anna draft with zero immutable versions. The immutable release gate remains >=18/20 with median successful completion <=180 seconds on the fixed corpus.

@zedarvates zedarvates changed the title docs(harbour): record rejected cross-model repair experiment docs(harbour): record Anna reliability reruns and release boundary Aug 28, 2026
@zedarvates zedarvates changed the title docs(harbour): record Anna reliability reruns and release boundary test(harbour): refine Anna diagnostics and record reliability reruns Aug 28, 2026
@zedarvates zedarvates changed the title test(harbour): refine Anna diagnostics and record reliability reruns fix(harbour): strengthen repair reliability from real Anna reruns Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Reliability patch ready for real rerun

The repair path has now been strengthened from the measured 2026-08-27 failures without changing the immutable corpus or relaxing the canonical validator.

Changed

  • repair requests now enumerate every required project/bible/entity/scene/shot/continuity field;
  • hard rules explicitly cover array/string types, reference integrity, exact scene/shot counts, ordering, warning severities, score bounds, and preservation of the original input;
  • the previous malformed/private model response is still discarded;
  • only one repair attempt remains allowed;
  • missing creative structure still fails closed rather than being invented deterministically;
  • regression tests lock this behavior.

Automated result

GitHub Actions on head 234987a796485083bfd9dd14d881656a9b7a817b passed the full Harbour job: syntax, Node tests, sample export, mock fixture, immutable corpus, generated-bundle synchronization, strict Anna validation, browser flow, and Marketplace screenshot generation.

Next evidence to collect

Do one complete Gemma-advisory corpus rerun first, because the latest Gemma baseline is the closest to the gate (15/20, median 21.91 s) and its five failures are structural/required-field failures targeted by this patch.

Do not claim an improvement from this patch until that complete result exists. Required release gate remains >=18/20 and median <=180 s. If Gemma reaches the gate, repeat once to check stochastic stability before any readiness claim. If it remains below the gate, record only privacy-safe failing prompt IDs/categories and continue from measured evidence.

@sonarqubecloud

sonarqubecloud Bot commented Sep 1, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant