Skip to content

feat(models): api_base_env names the variable a judge's endpoint comes from - #298

Merged
ethan-scitix merged 2 commits into
mainfrom
feat/api-base-env
Nov 17, 2026
Merged

ethan-scitix merged 2 commits into
mainfrom
feat/api-base-env

Conversation

@ethan-scitix

Copy link
Copy Markdown
Collaborator

Type

  • feature — new benchmark, task, or capability

Summary

  • An inline grader / extractor may name its endpoint's variable with api_base_env (top level or args), the counterpart of api_key_env. The platform's per-job SIEVAL_JUDGE_GENAI_API can now replace the 31 literal gateway URLs in sieval-plugins.
  • Refused at plan time, naming the task, role and variable: api_base beside it (either surface), api_base_env on both surfaces, an invalid name, and an unset or empty variable.
  • Identity: normalize_inline_model_binding resolves the variable and writes the URL as api_base into the binding's identity config. A literal URL and a variable holding the same URL (same surface) therefore get the same binding id, binding-plan and deployment fingerprints, and call_key. A resume reuses the outputs. Changing the URL changes the binding.
  • Recording: Deployment.api_base_env and ModelMeta.api_base_env (beside api_base in every call's model record) say where the URL came from. Both are left out of the deployment fingerprint and call_key.
  • Scope matches the issue: only inline role bindings. A models: entry with api_base_env is refused as an unsupported binding resource (it is in BINDING_RESOURCE_KEYS), not ignored. ${VAR} expansion in api_base is still not done.
  • No RESULT_SCHEMA_VERSION bump: no layout change. The new ModelMeta key is optional and appears only when the variable is used, and existing results are unchanged.

Related Issues

Closes #296

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check, whole repo)
  • Unit tests pass: pytest tests/unit tests/integration -n 16 → 11398 passed, 1 skipped
  • scripts/check_preflight.py and sync_package_stubs.py --check pass

New tests in tests/unit/cli/test_session.py:

  • test_inline_api_base_env_calls_the_url_in_the_variable: on both surfaces, the wire request goes to the variable's URL (with OPENAI_BASE_URL set as a decoy), and meta() records both the URL and the variable.
  • test_literal_api_base_records_no_variable
  • test_inline_api_base_env_refusals_fail_at_plan_time: 6 cases. Each checks the message and that no client was built.
  • test_named_model_refuses_api_base_env
  • test_api_base_env_and_literal_api_base_share_the_binding_identity: literal vs. two different variable names, on the top level and in args.
  • test_api_base_env_value_change_changes_the_binding

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring (no new modules)
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

🤖 Generated with Claude Code

ethan-scitix and others added 2 commits November 17, 2026 21:49
…s from

An inline grader/extractor binding may now name the environment variable
holding its endpoint with `api_base_env`, the counterpart of `api_key_env`,
at the top level or in `args`.

- `api_base` beside it, the variable on both surfaces, an invalid name, or
  an unset/empty variable are refused at plan time, naming task, role and
  variable.
- The resolved URL is substituted into the binding's identity config, so a
  literal `api_base` and a variable holding the same URL are one binding:
  same binding id, plan and deployment fingerprints, and call key.
- `Deployment.api_base_env` and `ModelMeta.api_base_env` record where the
  URL came from; both stay out of the fingerprint and the call key.

Closes #296

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docs

- `session.py` reuses `resolution.ENV_VAR_NAME` for `api_key_env`
  instead of a private copy of the same regex.
- `resolve_api_base_env` and `normalize_inline_model_binding` drop the
  `environ` parameter no caller passed; both read `os.environ`.
- One helper reads `api_base_env` from either surface for both callers.
- CHANGELOG entry under Unreleased; shorter guide section, docstrings and
  comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit bce80f2 into main Nov 17, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the feat/api-base-env branch November 17, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): api_base_env names the variable a judge's endpoint comes from

1 participant