Skip to content

0.1.8 Beta2: Define tool-type compatibility rules for custom Responses endpoints #65

Closed

Description

@NOirBRight

Parent

#318 (part of #248)

Outcome

Define one model-agnostic compatibility table for runtime-provided Codex tool declarations sent to custom Responses endpoints.

The table is keyed by declaration type, wire lifecycle, selected upstream protocol, and whether the request requires the tool. It is never keyed by model display name or a GLM/K2.7 allowlist.

Required table

For each structural family observed by #62, record:

  • runtime declaration kind and whether Codex or the selected Provider executes it;
  • selected-protocol native representation, if any;
  • model-visible exposure decision;
  • request encoding and inverse call/result/history mapping;
  • streaming assembly and identity requirements;
  • loss/ambiguity boundary;
  • final disposition: native, adapt, omit, or required-but-unavailable.

Rules

  • native: preserve the declaration and wire items unchanged.
  • adapt: use an Adapter only when the selected protocol cannot carry the native shape and the mapping is injective, request-scoped, and reversible.
  • omit: do not expose an optional capability that the selected Provider cannot execute or that cannot be represented safely; emit only a bounded sanitized diagnostic.
  • required-but-unavailable: fail visibly before upstream sampling when tool_choice or the request contract requires an unavailable tool.
  • Hosted tools are available only from the already-selected Provider. Unsupported hosted search, compaction, or other hosted semantics are never delegated to another Provider.
  • Execution follows the tool type; the Gateway does not add an independent executor/scheduler decision.
  • Unknown custom endpoints use the same conservative type/protocol rules. Maintained-Provider tests are evidence, not runtime eligibility.

Acceptance criteria

  • Every 0.1.8 Beta2: Inventory Codex CLI 0.146 tool types and representative Responses lifecycles #62 structural family has exactly one deterministic rule for each relevant selected-protocol condition.
  • Native-compatible tools are not unnecessarily wrapped.
  • Namespace and custom/freeform mappings define collision-free aliases/envelopes and complete inverse history behavior.
  • Client-executed tool discovery and Collaboration namespace shapes have a generic representability rule without claiming full workflow qualification.
  • Unsupported hosted tools resolve to omit or required-but-unavailable; no cross-Provider path exists.
  • Unknown or unqualified models are not denied merely because they lack a model-specific matrix row.
  • The table contains no model-name production dispatch rule and no runtime QualificationRecord/allowlist.
  • The accepted table is precise enough to implement 0.1.8 Beta2: Implement runtime-derived tool compatibility for custom Responses endpoints #58 without additional product decisions.

Blocked by

Activity

  1. NOirBRight commented on Jul 15, 2026

    @NOirBRight
    OwnerAuthor

    2026-07-16 native Responses strict_apply_patch qualification

    Source: dev a0cb8c9; Codex CLI 0.144.1. The harness used an isolated Codex CLI and isolated Gateway, native /v1/responses only, with strict_apply_patch selected in temporary provider/model configuration. No Responses-to-Chat conversion or protocol fallback was allowed. Each strict pass required shell_command -> apply_patch -> shell_command, one exact one-line target mutation, zero request errors, zero upstream retries, and zero protocol fallbacks.

    Results:

    • ollama-cloud/kimi-k2.6: strict gate 2/2 green (plus one green shakedown). Each accepted run recorded four codec adaptations, the exact tool sequence, one target-only mutation, and zero request errors/retries/fallbacks.
    • ollama-cloud/kimi-k2.7-code: not strict-green. Tool execution completed where acceptance was reached, but two independent runs recorded one UpstreamEmptyCompletedResponseError retry each. A separate readiness run completed its shell command but omitted the exact sentinel. This is not an apply_patch shape rejection, but it fails the zero-retry stability gate.
    • ollama-cloud/minimax-m3: sequential gate 2/2 green, but an earlier independent run failed after producing exact {patch: string} with an invalid patch body: zero Begin Patch markers and one End Patch marker. This proves the codec constrains the function argument envelope but does not make internal patch grammar deterministic. The earlier paired run was green.

    Additional native Responses provider rows were prepared for volc/glm-5.2, xunfei/xopglm52, and minimax-cn/MiniMax-M3. Their configured credential environment variables were not available to the isolated task process, so no live upstream call was made and no protocol verdict is claimed.

    Decision evidence: do not infer that all Ollama Cloud models, or all external providers, are safe for a provider-wide/global strict_apply_patch default from this matrix. Keep explicit model selection while #65 gathers more evidence; #58 remains the capability-driven production-default owner after #57's decision gate.

  2. NOirBRight commented on Jul 15, 2026

    @NOirBRight
    OwnerAuthor

    Correction and additional provider results

    Correction to the prior matrix comment: CodexHub had saved credentials for all four providers in the active runtime configuration. The initial harness inspected the bundled template rather than the active runtime layer. The rerun read each stored credential only in memory and passed it to its isolated Gateway child; no credential was copied into evidence or printed.

    All probes below used native Responses only; no Chat conversion or protocol fallback was used.

    • volc/glm-5.2: protocol red. Readiness shell execution reached the provider. During acceptance, strict_apply_patch was selected on every request, but Volc emitted response.output_item.added function-call items without an arguments field. The fail-closed adapter rejected six calls as function_call_fields_not_exact, producing six local 400 invalid_apply_patch_function_call results. No target mutation occurred.
    • xunfei/xopglm52: no codec verdict. Both the isolated Gateway run and a direct minimal no-tools /v1/responses probe returned upstream code 11200 AppIdNoAuthError. The request never reached tool qualification.
    • minimax-cn/MiniMax-M3: no codec verdict. Both isolated readiness runs timed out while the upstream returned HTTP 500. A direct minimal no-tools /v1/responses probe confirmed the provider account had reached its Token Plan usage limit. The request never reached tool qualification.

    Updated decision evidence: provider-wide or global strict_apply_patch default activation remains unsafe. Volc exposes a concrete Responses event-shape incompatibility; Xunfei and MiniMax.cn require account-side authorization/quota recovery before their protocol rows can be qualified.

  3. NOirBRight commented on Jul 29, 2026

    @NOirBRight
    OwnerAuthor

    This was generated by AI during triage.

    Corrected 2026-07-29 qualification result

    The configured and user-visible path is Chat Completions. Native Responses was exercised only as a control and must not be used to mark Sol's Chat path green.

    Environment: CodexHub fad95c869bf3726461d9f9a764020863a5ba0317, Codex CLI 0.145.0, and a private CLIProxyAPI-compatible endpoint. No endpoint address, credential, request ID, prompt, or generated text is included here.

    The exact model-test request defined in src-tauri/src/models.rs was replayed against Chat Completions with its non-streaming payload and eight-second timeout, alternating models to control for time-window effects:

    • Luna: 4/4 returned 200.
    • Sol: 0/4 returned 200. The sequence was an overloaded 502, two fast 503 auth_unavailable results, and—after the cooldown window—a further 502.
    • The reporter's actual model-test UI has likewise never shown Sol green.

    A separate native Responses control completed once for Sol, and one earlier synthetic Gateway Chat request also completed in a different bounded run. Neither is equivalent to this model-test path or a real Codex-over-Chat turn, so neither is a green qualification for Sol/Chat.

    Current matrix disposition: Sol + Chat Completions = red/unqualified; Luna + Chat Completions = probe-green only. Catalog exposure, protocol reachability, current auth/capacity availability, and real Codex text/tool compatibility must remain separate fields. Real Codex-over-Chat is additionally blocked by #66's deterministic request-field translation gap.

  4. 20 remaining items

  5. NOirBRight commented on Aug 4, 2026

    @NOirBRight
    OwnerAuthor

    Orchestrator dispatch status (2026-08-04)

    Campaign: campaign:0.1.8-beta2.

    Dispatch is held by #62 and #350. A single Luna Max worker will start only after the 0.146 inventory and ADR are accepted. The worker owns the generic tool-type/protocol compatibility decision artifact and tests; it must not change production routing or select/fallback a model. #249 remains the maintainer approval gate after this work.

  6. added
    orch:blockedOrchestrator V6: blocked pending a decision or external state change
    on Aug 4, 2026
  7. added a commit that references this issue on Aug 4, 2026
  8. NOirBRight commented on Aug 5, 2026

    @NOirBRight
    OwnerAuthor

    Beta2 dispatch update

    #62 and #350 are closed, so the tool-type compatibility table is now unblocked. The worker scope is the model-agnostic structural contract only: deterministic native/adapt/omit/required-but-unavailable rules for every #62 family, execution ownership, native passthrough, reversible namespace/custom mappings, hosted-tool omission/failure, and generic representability boundaries for tool_search and Collaboration V2 without claiming their full Beta3/Beta4 workflows.

    No production implementation (#58), named-model branch, cross-Provider proxy, Gateway tool execution, alternate-codec retry, model-output repair, or release action is authorized in this ticket.

  9. added
    orch:activeOrchestrator V6: claimed Worker is active or rework is in progress
    and removed
    orch:blockedOrchestrator V6: blocked pending a decision or external state change
    on Aug 5, 2026
  10. NOirBRight commented on Aug 5, 2026

    @NOirBRight
    OwnerAuthor

    Worker dispatched

  11. removed
    orch:activeOrchestrator V6: claimed Worker is active or rework is in progress
    on Aug 5, 2026
  12. NOirBRight commented on Aug 5, 2026

    @NOirBRight
    OwnerAuthor
    {
      "contract": {
        "acceptance": [
          "Every Issue #62 structural family has one deterministic rule for each relevant selected-protocol condition.",
          "Native-compatible tools are not unnecessarily wrapped.",
          "Namespace and custom/freeform mappings define collision-free aliases or envelopes and complete inverse history behavior.",
          "Client-executed tool discovery and Collaboration V2 namespace shapes have a generic representability rule without claiming full workflow qualification.",
          "Unsupported hosted tools resolve to omit or required-but-unavailable and no cross-Provider path exists.",
          "Unknown or unqualified models are not denied merely because they lack a model-specific matrix row.",
          "The table has no model-name production dispatch rule or runtime QualificationRecord/allowlist and is precise enough to implement Issue #58 without more product decisions."
        ],
        "change_claims": {
          "paths": [
            "docs/evidence/issue-65/README.md"
          ],
          "resources": [
            "contract:runtime-tool-compatibility"
          ]
        },
        "dependencies": {
          "dispatch_after": [
            62,
            350
          ],
          "merge_after": [
            62,
            350
          ]
        },
        "design": [
          "Add one durable model-agnostic compatibility table at docs/evidence/issue-65/README.md from the accepted Issue #62 inventory, ADR-0002, and Issue #64 V1/V2 boundary; do not modify production routing.",
          "For plain function, namespace, custom/freeform, client tool_search, Provider-hosted tools, Collaboration V2 representability, and unknown kinds, define execution owner, selected-protocol native shape, exposure, request encoding, inverse call/result/history mapping, streaming and identity requirements, ambiguity boundary, and exactly one disposition.",
          "Preserve native shapes unchanged; adapt only through a request-scoped injective collision-free reversible mapping; omit optional unavailable capabilities with bounded sanitized diagnostics and fail required unavailable capabilities before upstream sampling.",
          "Keep the exact selected Provider/model and never add cross-Provider hosted-tool proxying, Gateway execution or scheduling, named-model dispatch, retries, output repair, Issue #58 implementation, or Beta3/Beta4 workflow qualification."
        ],
        "difficulty": "light",
        "done_when": [
          "Inspect docs/evidence/issue-65/README.md against Issue #65, ADR-0002, and the Issue #62/#64 evidence; every acceptance item is explicit and every referenced repository path resolves.",
          "git show --check --oneline --stat HEAD exits zero.",
          "git status --porcelain is empty after the candidate commit is pushed."
        ],
        "priority": "P1",
        "risk": "low",
        "sha256": "b329854aea029e3ec8bcd9691dc208fe645a9f4f61db4017c89ec6b978f8a8c7",
        "unresolved_decisions": []
      },
      "dispatch": {
        "accepted_at": "2026-08-05T08:06:56Z",
        "attempt": 1,
        "base_sha": "dcd2f79986186cdb047388c5fb5c3f07c2bd09a9",
        "branch": "work/issue-65",
        "candidate_sha": "c92441e770cebaa06019d2cf56ef3cddc926f69b",
        "claimed_at": "2026-08-05T07:41:24.561156Z",
        "contract_sha256": "b329854aea029e3ec8bcd9691dc208fe645a9f4f61db4017c89ec6b978f8a8c7",
        "creator_agent_id": "470f6d8f-78e6-4afb-b625-257f00f09983",
        "generation": 1,
        "id": "dispatch-issue-65-a1",
        "integrating_at": "2026-08-05T08:08:48.461610Z",
        "merged_at": "2026-08-05T08:09:06Z",
        "pr_number": 362,
        "recovery_prompt_sent": true,
        "status": "merged",
        "worker_agent_id": "321f26a7-6925-4e7f-b546-c8002df3cf05",
        "workspace_id": "wks_27fcc0f94b6c1bec"
      }
    }
  13. added
    orch:readyOrchestrator V6: valid hashed contract is complete and dispatchable
    orch:activeOrchestrator V6: claimed Worker is active or rework is in progress
    and removed
    orch:readyOrchestrator V6: valid hashed contract is complete and dispatchable
    on Aug 5, 2026
  14. NOirBRight commented on Aug 5, 2026

    @NOirBRight
    OwnerAuthor

    Delivered by merged PR #362.

  15. removed
    orch:activeOrchestrator V6: claimed Worker is active or rework is in progress
    on Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1: current iteration goalcampaign:0.1.8-beta20.1.8 Beta2 generic tool compatibility campaignenhancementNew feature or requestready-for-agentFully specified, ready for an AFK agentwayfinder:researchWayfinder evidence or research ticket

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions