Repository navigation
Qualify the method judge with source context and native reasoning - #8
hudsonaikins wants to merge 5 commits into
Conversation
d21ed12 to
c3ddb22
Compare
hudsonaikins
changed the title
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b056994226
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "judge_context": scores.get("judge_method", {}).get("context"), | ||
| "judge_assessments": scores.get("judge_method", {}).get("assessments"), |
There was a problem hiding this comment.
When an experiment uses run_langsmith, its score dictionaries are reconstructed from LangSmith results and only metrics is copied back from evaluation_usage; the new context and assessments entries are discarded before record() reads them. Consequently, every LangSmith JSONL record writes both of these fields as null, losing the source-context and raw-assessment audit trail that local runs retain. Copy these entries back alongside the metrics, or build the record from the original scored dictionaries.
Useful? React with 👍 / 👎.
| profile = { | ||
| "name": table.name, | ||
| "row_count": table.row_count, | ||
| "columns": [{"name": c.name, "type": c.type} for c in table.columns], | ||
| "key_counts": {}, | ||
| } | ||
| if isinstance(store, SQLiteStore): | ||
| for column in table.columns: | ||
| key, name = column.name, _quoted(column.name) | ||
| if key == "id" or key.endswith("_id"): | ||
| row = store.query( | ||
| f"SELECT COUNT({name}), COUNT(DISTINCT {name}) FROM {_quoted(table.name)}" | ||
| ).rows[0] | ||
| profile["key_counts"][key] = {"non_null": row[0], "distinct": row[1]} |
There was a problem hiding this comment.
The new context sends exact row counts and distinct/non-null ID counts to the LLM judge, even when those aggregates are the scenario's reference value—for example, the transactions row count directly answers orii-l1-01, and customers.customer_unique_id's distinct count directly answers olist-l1-02. Timestamp null counts can likewise reveal pending-count answers. This lets the method judge check or infer numerical correctness instead of evaluating only reasoning, contaminating method scores on those scenarios; restrict the payload to structural/grain indicators that cannot equal requested results, or keep these checks entirely deterministic.
AGENTS.md reference: AGENTS.md:L22-L23
Useful? React with 👍 / 👎.
The method judge previously penalized valid pre-truncated snapshots and equivalent calculations, then treated answers that named their limited population as having no disclosure. This change supplies evaluator-only source context, makes caveat categories explicit, and uses native Anthropic JSON output at high effort. A partly scoped answer receives 0.5; a false broader claim still receives zero. Truncated assessments are rejected and their usage retained.
Native Haiku 5.5 passed all 10 frozen controls at the benchmark's 8192-token allowance: six original controls plus four prospectively authored sensor contrasts. The passing run cost an estimated $0.068076; earlier failures and the truncated 2048-token run remain preserved. These controls are targeted evaluator evidence, not human calibration or analyst accuracy. Numerical references, scoring tolerances and analyst code remain unchanged.
Validation: 252 tests passed, 15 optional skips; Ruff lint/format, Vulture, diff and Gitleaks checks passed. Actual SDK-payload tests verify high effort, JSON output, truncation rejection and usage accounting. LangSmith read-back matched 10 outputs and 20 feedback records. Passing experiment. Details and previous attempts:
results/judge-context-validation.md; configuration rationale:docs/judge-native-output.md.Stacked on PR #10 for native provider routing and bounded token accounting. The evaluator gate now passes; the fresh 92-response paired analyst campaign is the next separate requirement. Method scores remain uncalibrated and advisory. No release or deployment is included.