Skip to content

feat(models)!: add provider-neutral decision IR foundations - #297

Open
jack-scitix-ai wants to merge 1 commit into
scitix:mainfrom
jack-scitix-ai:feat/decision-ir
Open

jack-scitix-ai wants to merge 1 commit into
scitix:mainfrom
jack-scitix-ai:feat/decision-ir

Conversation

@jack-scitix-ai

Copy link
Copy Markdown
Contributor

Type

  • feature — new benchmark, task, or capability

Summary

  • Add provider-neutral decision IR and documentation as the foundation for a separate OpenRouter Decisions dialect PR.
  • Introduce role-free ContextInput, structured JsonPart, and typed predicate, choice and score questions, answers and per-question refusals.
  • Extend request auditing and output validation to support decision responses with empty texts and one call-level finish reason; the initial contract requires sampling.n == 1 and rejects streaming.
  • Add capability aggregation and reconciliation. Configuration, task-local and request probability/confidence guarantees combine with OR; capability form sets remain serving requirements.
  • Automatically record canonical decision responses and share typed usage conversion with legacy outputs, preserving unreported fields versus reported zero. Explicit metadata retains the existing full-list override contract.
  • Integrate decision finish information with truncation detection, context-overflow diagnostics, profiling and report counts.
  • Preserve existing generation results, unused binding fingerprints and the result-directory layout. RESULT_SCHEMA_VERSION remains 1 and strict resume checks remain in force.
  • Custom dialects must extend their exhaustive declarations with a decisions capability entry and output rule. Unsupported implementations should use Unsupported(...) and OutputRule(Guarantee.NEVER); all eight built-in dialects are updated accordingly.

Related Issues

Refs #292.

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check or mypy --strict)
  • Unit tests pass (pdm run pytest)

Regression coverage includes:

  • Question/answer identity, ordering and type matching; choice membership, rubric bounds, distribution completeness, confidence requirements and valid per-question refusals.
  • Structured JSON null versus omitted guidance, optional usage versus reported zero, and shared canonical/legacy usage conversion.
  • Default capability verification, configured and task-local guarantees, rebinding, declaration-order fingerprint stability, and rejection by existing dialects before execution.
  • Automatic metadata for bare, boxed and mixed-list responses; explicit metadata precedence; finish anomaly/report agreement; token profiling; fresh disk reload and resume under both metadata and stage-recording modes.

Decision execution tests use a controlled dialect seam. Native provider wire behavior and real API validation belong to the follow-up dialect PR.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

If: Breaking Change

  • Described what breaks and migration path in Summary
  • Existing tests updated to reflect new behavior

Add typed evidence, decision questions and answers, capability reconciliation, output validation, and canonical response metadata. Integrate finish diagnostics, profiling and persistence to prepare for a separate OpenRouter Decisions dialect.

BREAKING CHANGE: custom dialects must add an explicit decisions capability entry and decisions output rule. Existing built-in dialects declare decisions unsupported.

Refs scitix#292
@ethan-scitix

Copy link
Copy Markdown
Collaborator

Thanks, this is careful work. Two parts stand out: the frozen _BINDING_FINGERPRINT_V1_CAPABILITIES baseline, and JsonPart's null-preserving encoding.

Per the discussion on #292, we won't merge this into core. Could you convert it to draft and keep the branch? decisions.py and content.py should port nearly as-is into a plugin package, and #299 is meant to give that plugin a proper subject identity and call recording.

Two notes for the plugin version:

  • The finish-diagnostics path (FINISH_REASON_TASK_TAGS, the anomaly-rule scope, n_truncated) isn't needed: OpenRouter's DecisionsResponse has no finish field. Changing rule definitions also rotates rules_hash for every task.
  • Please keep the existing AI-Generated Code attribution lines in modules you modify. The rule covers files you author, not modules you touch.

The UsageStats.to_json_value() extraction is sound. It will land with #300, which settles the Response shape before tasks migrate to Model.arun; the first task persisting a Response is its real consumer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants