Repository navigation
feat(models): api_base_env names the variable a judge's endpoint comes from - #298
Merged
Merged
Conversation
…s from An inline grader/extractor binding may now name the environment variable holding its endpoint with `api_base_env`, the counterpart of `api_key_env`, at the top level or in `args`. - `api_base` beside it, the variable on both surfaces, an invalid name, or an unset/empty variable are refused at plan time, naming task, role and variable. - The resolved URL is substituted into the binding's identity config, so a literal `api_base` and a variable holding the same URL are one binding: same binding id, plan and deployment fingerprints, and call key. - `Deployment.api_base_env` and `ModelMeta.api_base_env` record where the URL came from; both stay out of the fingerprint and the call key. Closes #296 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docs - `session.py` reuses `resolution.ENV_VAR_NAME` for `api_key_env` instead of a private copy of the same regex. - `resolve_api_base_env` and `normalize_inline_model_binding` drop the `environ` parameter no caller passed; both read `os.environ`. - One helper reads `api_base_env` from either surface for both callers. - CHANGELOG entry under Unreleased; shorter guide section, docstrings and comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
grader/extractormay name its endpoint's variable withapi_base_env(top level orargs), the counterpart ofapi_key_env. The platform's per-jobSIEVAL_JUDGE_GENAI_APIcan now replace the 31 literal gateway URLs in sieval-plugins.api_basebeside it (either surface),api_base_envon both surfaces, an invalid name, and an unset or empty variable.normalize_inline_model_bindingresolves the variable and writes the URL asapi_baseinto the binding's identity config. A literal URL and a variable holding the same URL (same surface) therefore get the same binding id, binding-plan and deployment fingerprints, andcall_key. A resume reuses the outputs. Changing the URL changes the binding.Deployment.api_base_envandModelMeta.api_base_env(besideapi_basein every call's model record) say where the URL came from. Both are left out of the deployment fingerprint andcall_key.models:entry withapi_base_envis refused as an unsupported binding resource (it is inBINDING_RESOURCE_KEYS), not ignored.${VAR}expansion inapi_baseis still not done.RESULT_SCHEMA_VERSIONbump: no layout change. The newModelMetakey is optional and appears only when the variable is used, and existing results are unchanged.Related Issues
Closes #296
Test Plan
Automated
ruff check && ruff format --check)ty check, whole repo)pytest tests/unit tests/integration -n 16→ 11398 passed, 1 skippedscripts/check_preflight.pyandsync_package_stubs.py --checkpassNew tests in
tests/unit/cli/test_session.py:test_inline_api_base_env_calls_the_url_in_the_variable: on both surfaces, the wire request goes to the variable's URL (withOPENAI_BASE_URLset as a decoy), andmeta()records both the URL and the variable.test_literal_api_base_records_no_variabletest_inline_api_base_env_refusals_fail_at_plan_time: 6 cases. Each checks the message and that no client was built.test_named_model_refuses_api_base_envtest_api_base_env_and_literal_api_base_share_the_binding_identity: literal vs. two different variable names, on the top level and inargs.test_api_base_env_value_change_changes_the_bindingChecklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstring (no new modules)core/🤖 Generated with Claude Code