Skip to content

Add model discovery integration tests without managed config - #662

Merged
andy-xu-db merged 8 commits into
mainfrom
andy/model-discovery-integration-unmanaged
Sep 21, 2026
Merged

andy-xu-db merged 8 commits into
mainfrom
andy/model-discovery-integration-unmanaged

Conversation

@andy-xu-db

@andy-xu-db andy-xu-db commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Align discovery integration coverage with the fetched main at 6e5b97b (#775), merged in 043ab5e. Do not restore the obsolete discovery feature stack or change runtime behavior.

  • Current main has no UG_ENABLE_MODEL_DISCOVERY switch. Remove obsolete unmanaged disable scenarios and duplicate managed variants. Managed discovery and source-override rejection remain covered by repository Cases 1–6.
  • Current ug configure has no --model-location option. Cases 7–10 cover configured/fresh default launches: Claude automatically caches system.ai models (including recognized Anthropic gateway aliases) and opens its picker; Codex leaves model/reasoning preferences unset and exposes its native catalog without a generated scoped file.
  • Retain configured/fresh launch-time --provider and --model-location Cases 11–14 with exact scoped catalog assertions. Codex's parent catalog must exactly match an independent, parent-scoped API response and include the dedicated Codex service; it must not assume the Claude service supports the Codex API. Claude's provider models can appear as exact native Haiku 4.5, Opus 5, or Sonnet 5 picker rows.
  • Require a real unmanaged workspace for unmanaged tests. A read-only, session-scoped CodingAgentConfig List check reports published admin configuration explicitly; it does not delete, bypass, or inject configuration.
  • Update both coverage READMEs and the contract checks. No production-code changes, new skips, xfails, retries, or weakened managed-policy enforcement.

Current coverage

Cases Supported behavior Executions
1–6 Managed discovery and provider/location override rejection 12
7–10 Configured/fresh default launches without discovery flags 4
11–14 Configured/fresh provider and parent-schema discovery 8

Repository scenario numbers now run consecutively from 01 to 14, with configured/fresh variants sharing a number. Test names, artifact labels, both repository READMEs, and contract checks use the same mapping. The external Google doc and its numbering are untouched. The complete integration suite collects 97 executions: 58 live (25 Claude, 33 Codex), 27 managed-fixture, 4 managed-workspace, 1 workspace-switch, and 7 installation.

Discovery checks configure, list metadata, and open/close pickers without inference prompts. Separate existing task/agent tests do perform inference.

Latest live result and assertion fixes

At 7751cf1, all integration jobs passed in CI run 35655313004:

  • Full Claude: 25 passed; full Codex: 33 passed.
  • Managed Claude: 18 passed; managed Codex: 14 passed. These nonblocking jobs were checked individually, not inferred from the aggregate gate.
  • Installation: 7 passed. Both smoke jobs also passed.
  • All 24 numbered discovery executions passed, including configured/fresh managed discovery, default discovery, provider/parent selection, and managed override rejection.

Independent review found and fixed picker and catalog evidence gaps. Claude assertions now match numbered picker rows, not startup banners or footer text, and explicitly recognize native family/version deduplication. Managed caches/catalogs are compared with independent read-only provider-list responses. Codex parent expectations follow the API-compatible scoped response, rejecting missing, extra, duplicate, or out-of-schema app-server models.

Claude still accepts only recognized gateway aliases while preserving system.ai membership, raw-ID uniqueness, and discovered-family inclusion. Codex requires absent model/reasoning preferences for unmanaged native defaults. No production code, workspace configuration, or GitHub secrets were changed. The external Google doc remains untouched.

Verification

  • Local non-e2e suite: 2,598 passed with uv run --frozen pytest --ignore-glob='tests/test_e2e*.py'. Existing live-agent/e2e probes were explicitly excluded locally.
  • CI unit job: 2,600 passed, 10 skipped; this change adds no skips or xfails.
  • Discovery evidence, independent catalog, and contract/prerequisite checks are covered by offline regressions.
  • Ruff lint and formatting, runner --help, git diff --check, and collection of all 97 integration tests passed.
  • Saved real CI picker screens validate the row-matching and native-alias logic; banner-only and wrong-version regressions fail as intended.
  • No live agents or inference probes were run locally for this update; machine-wide agent settings were not bypassed.

@andy-xu-db andy-xu-db changed the title Add unmanaged model discovery integration tests Add model discovery integration tests without managed config Sep 16, 2026
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 5204d14 to 9751ad8 Compare September 16, 2026 15:02
@andy-xu-db
andy-xu-db changed the base branch from main to codex/model-discovery-test-stack September 16, 2026 15:02
@andy-xu-db
andy-xu-db force-pushed the codex/model-discovery-test-stack branch from d85847c to 2c0b3df Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 9751ad8 to 88b4d6f Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db changed the base branch from codex/model-discovery-test-stack to andy/model-discovery-integration-tests September 16, 2026 20:06
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch 3 times, most recently from e907761 to e4a7d5c Compare September 17, 2026 20:03
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from 8cc5eb8 to ecd7c40 Compare September 17, 2026 21:37
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from e4a7d5c to 5404b69 Compare September 17, 2026 21:37
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch 2 times, most recently from debbbfb to 6fb6e7f Compare September 17, 2026 22:54
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 6fb6e7f to 394cbde Compare September 21, 2026 18:30
@andy-xu-db
andy-xu-db changed the base branch from andy/model-discovery-integration-tests to main September 21, 2026 18:30
@andy-xu-db
andy-xu-db merged commit aa55bba into main Sep 21, 2026
21 checks passed
@andy-xu-db
andy-xu-db deleted the andy/model-discovery-integration-unmanaged branch September 21, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants