Skip to content

Phase 10A: Harden live planner contract#1044

Merged
paullizer merged 1 commit into
microsoft:feature/orchestrationfrom
paullizer:feature/orchestration-phase-10a-planner-contract-hardening
Jul 16, 2026
Merged

Phase 10A: Harden live planner contract#1044
paullizer merged 1 commit into
microsoft:feature/orchestrationfrom
paullizer:feature/orchestration-phase-10a-planner-contract-hardening

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

  • align the model-facing planner schema with strict server validation using request-specific requirement aliases, candidate aliases, eligible unselected capability IDs, evidence types, and policy budgets
  • keep selected mandates server-owned, suppress redundant additions, preserve strict JSON schema when optional model parameters are unsupported, and increase the measured reasoning-model completion budget to 600 tokens
  • add a source-bound, privacy-safe, non-executing controlled-shadow runner with 19 realistic scenarios and fail-closed semantic, end-to-end, strict-schema, provider-failure, timeout, invalid-output, false-proposal, leakage, execution-surface, and latency gates
  • update planner documentation, fix documentation, release notes, combined quality gates, and application version 0.250.071

Root Cause

Phase 10A deterministic fixtures used payloads that already matched the strict validator. Controlled live calls showed that the provider-facing schema described requirement, candidate, capability, and evidence IDs too generically. Live models could therefore produce values such as req1, plan1, or candidate aliases in capability_ids; the validator correctly rejected them, preserving safety but making the invalid-output rate unsuitable for Phase 10B activation.

The prompt schema and provider response_format were also constructed separately, selected capabilities were exposed as candidate additions, and a 300-token reasoning-model budget could complete without visible JSON.

Safety And Privacy

  • planner remains non-executing and has no capability/runtime/evidence-collector imports
  • unknown, unauthorized, unavailable, blocked, selected-only, and input-not-ready additions still fail closed
  • selected mandates are restored only by server validation
  • no route, browser payload, authorization policy, execution adapter, or deterministic fallback path changed
  • evidence omits prompts, raw responses, deployment/model names, endpoint hosts, public entity text, credentials, canonical IDs, and evidence
  • report provenance is bound to manifest hash 0afbeebf616ae3b1 and evaluation-source hash 14edd8d66bf02784

Validation

Deterministic combined gate:

  • 236 passed, 5 expected environment-gated skips
  • full-file Broken Access Control guardrail passed for all 8 gate files
  • full-file XSS sink guardrail passed for all 8 gate files
  • Python compilation, editor diagnostics, CRLF-aware whitespace, privacy, provenance, and diff secret checks passed

Controlled gpt-5.6-terra gate:

  • 57/57 end-to-end samples passed across 19 scenarios and three repetitions
  • semantic accuracy: 100% in every category
  • strict JSON-schema use: 100%
  • timeout, operational-failure, invalid-output, false-proposal, and capability-leakage rates: 0%
  • prohibited execution-surface imports: 0
  • latency: p50 1.67s, p95 2.65s under the 5-second deadline
  • evidence: artifacts/phase10a_controlled_shadow_report.json

Follow-Up

After this hardening PR merges into feature/orchestration, Phase 10B can start from the new merged baseline at application version 0.250.072.

Refs #1021

Align live structured output with strict server validation, add source-bound controlled-shadow evaluation, and preserve non-executing fallback behavior.

Refs microsoft#1021
@paullizer
paullizer merged commit 3b2b2b9 into microsoft:feature/orchestration Jul 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant