Skip to content
This repository was archived by the owner on Oct 9, 2026. It is now read-only.
This repository was archived by the owner on Oct 9, 2026. It is now read-only.

Add exec-harness cache assertion primitives #32

Closed

Description

@shiny-code-bot

Objective

Add the harness primitives needed for deterministic cache-sensitive scenarios before implementing skills-cache-continuity.

Finish Line

Exec harness supports deterministic token/cache/prompt-prefix assertions needed by cache-sensitive scenarios.

Current Status

State: Active. Next action: inspect existing evaluate_expectations and fake Responses fixtures to design the smallest assertion extensions.

Scope

In:

  • Allow fake Responses fixtures to supply usage values.
  • Add expectation support for input token bounds, cached token minimums, and cache ratios.
  • Add either prompt-prefix comparison support or an explicitly documented weaker substitute.
  • Consider a no-fail-fast mode so scenario runs report all failures.

Out:

  • No production Codex behavior changes.
  • No old Every Code harness feature expansion unless needed by the cache scenario.

Acceptance Criteria

  • Fake Responses SSE completion can carry per-response fixture usage.
  • Expectations can assert cached input token minimums and cache ratio by turn.
  • Expectations can catch catastrophic prompt bloat.
  • Prompt-prefix stability has a deterministic assertion path.
  • Unit tests cover the new harness assertion behavior.

Relationships

Child of #28. Blocks the skills-cache-continuity scenario child issue.

Validation

Run the exec-harness unit tests that cover fake Responses and expectation evaluation.

Decisions

Cache-sensitive behavior should be tested with deterministic fake usage first; speed-only signals are not enough.

Open Questions

What prefix length and cache-ratio thresholds are realistic across supported model/API behavior?

Activity

  1. added
    plan:donePlan completed or superseded
    and removed
    plan:activePlan is actionable now
    on Jun 12, 2026
  2. shiny-code-bot commented on Jun 12, 2026

    @shiny-code-bot
    CollaboratorAuthor

    Completed by PR #39, merged as e80a0b3b9b269267a43736a76f79bac49ee6068f.

    What landed:

    • Fake Responses fixtures can pass deterministic usage values through response.completed.
    • Per-turn expect.turns[].token_usage supports exact fields, *_min / *_max, cache_ratio_min, and cache_ratio_max.
    • expect.responses[].prefix_matches_request plus prefix_length provides deterministic prompt-prefix stability checks, with optional prefix_scope.
    • Unit tests cover usage passthrough, prompt bloat/cache failures, invalid assertion schemas, self-prefix references, and missing prefix scopes.
    • README documents the new scenario shape.

    Validation:

    • python3 -m unittest tools/codex-exec-harness/test_harness.py (19 tests passed)
    • python3 -m unittest discover -s tools/codex-exec-harness -p 'test_*.py' (23 tests passed)
    • just fmt (completed with pre-existing Ruff/TOML exclude-newer = "7 days" warnings)
    • PR Add exec harness cache assertion primitives #39 CI: 4/4 checks passed before merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    planDurable planning issueplan:donePlan completed or superseded

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions