[Klaud Cold] chore(evals): disable agentic evals repo-wide behind AGENTIC_EVALS_DISABLED / 全仓库停用智能体评测（AGENTIC_EVALS_DISABLED 开关）

[functionstackx]

Turns agentic (SWE-bench) evals off repo-wide behind a single flag, until the agentic eval path is fixed.

@adibarra — TODO

Re-enable is one line: flip AGENTIC_EVALS_DISABLED to False in utils/matrix_logic/generate_sweep_configs.py (or delete it and the two if AGENTIC_EVALS_DISABLED guards). Nothing else has to change. The TODO(@adibarra) marker is on the flag itself and in utils/evals/EVALS.md.

The existing selection policy keeps its test coverage behind the new _agentic_evals_enabled() helper in test_generate_sweep_configs.py, so the behaviour you're turning back on is already asserted — you should see those tests go green against the real code path the moment you flip it.

Why one flag is enough

Both dispatch paths key off the same thing — an agentic-coding row carrying run-eval: True:

• run-sweep.yml → sweep-agentic-evals reads search-space-config.agentic_evals, which process_changelog.py fills by splitting eval rows on scenario-type == 'agentic-coding'.
• e2e-tests.yml → test-sweep-agentic-evals filters get-jobs output on the same flag.

So suppressing the mark in generate_sweep_configs.py leaves both jobs with an empty matrix, and both skip on their existing != '[]' guards. No workflow YAML changes.

Worth knowing: process_changelog.py runs a second --evals-only pass for every changelog entry (utils/process_changelog.py, the eval_groups loop), and --evals-only sets include_agentic=True. That is why agentic eval jobs were dispatching on every agentic PR regardless of labels — e.g. #2447 and #2448 each got one without asking. The flag wins over --evals-only, --all-evals, and the evals-only / all-evals PR labels alike.

Blast radius

Measured against configs/nvidia-master.yaml, before vs after:

	before	after
agentic eval rows, glm5.2-fp4-b300-sglang-agentic --evals-only	1	0
fixed-seq-len eval rows, dsv4-fp4-b200-vllm --evals-only	4	4
default throughput matrix, same key	7 rows, 0 evals	identical

Agentic throughput coverage is untouched, and gsm8k / gpqa / 8k1k swebench are untouched. Only the agentic eval job goes away.

Changes

File	Change
utils/matrix_logic/generate_sweep_configs.py	AGENTIC_EVALS_DISABLED = True + two guards: the include_agentic block in mark_eval_entries() and the unconditional agentic marking in mark_all_eval_entries().
utils/matrix_logic/test_generate_sweep_configs.py	_agentic_evals_enabled() context manager so the two existing policy tests keep testing the policy; two new tests pin that the switch beats --evals-only and --all-evals.
utils/evals/EVALS.md	Status section at the top saying what is off, what is unaffected, and how to turn it back on.

No config, no perf-changelog.yaml entry, no GPU jobs — [skip-sweep] on the head commit.

Validation

pytest utils/ — 570 passed, 104 of them in test_generate_sweep_configs.py. Two failures in utils/evals/test_run_eval_dispatch.py (test_agentic_generation_invokes_mini_swe_agent, test_agent_sandbox_cpu_knob) reproduce identically on a clean main in this environment — a missing local mini-swe-agent fixture, not this change.

[functionstackx]

@adibarra — TODO for when you're back next week: fix the agentic eval path and re-enable this.

This is a temporary repo-wide disable, not a removal. To turn agentic (SWE-bench) evals back on, flip AGENTIC_EVALS_DISABLED to False in utils/matrix_logic/generate_sweep_configs.py — that is the whole re-enable. The TODO(@adibarra) marker sits on the flag and in utils/evals/EVALS.md.

The selection policy you're re-enabling still has test coverage: the two original tests now run behind the _agentic_evals_enabled() helper in test_generate_sweep_configs.py, so they exercise the real path the moment the flag clears. Two new tests pin the disabled state, and those are the ones to delete when you flip it.

One thing you may not have known and that is worth fixing at the same time: process_changelog.py runs a second --evals-only pass for every changelog entry, and --evals-only sets include_agentic=True. That is why agentic eval jobs were dispatching on every agentic PR regardless of labels — #2447 and #2448 each got one unasked. If the intent was for agentic evals to be opt-in, that path is where the opt-in leaks.

Assigning you and requesting review so this does not get lost.

[claude]

LGTM — a well-scoped, defensively-implemented kill switch for agentic evals with test coverage to match.

What was reviewed: the two AGENTIC_EVALS_DISABLED guards in mark_eval_entries() and mark_all_eval_entries(), confirming both dispatch paths (--evals-only/--all-evals and the default include_agentic path used by process_changelog.py's second pass) are covered; ran the updated test suite locally (104 passed); and checked the EVALS.md doc contradiction flagged by the bug hunter — the new Status section explicitly says the rest of the doc describes pre-flag behavior, so it's not a real contradiction.

Extended reasoning...

Overview

This PR adds a single module-level boolean flag, AGENTIC_EVALS_DISABLED = True, in utils/matrix_logic/generate_sweep_configs.py, guards it into the two places agentic-coding rows get marked run-eval (mark_eval_entries()'s include_agentic branch, and mark_all_eval_entries()'s unconditional single-node agentic marking), and documents the change in EVALS.md. Test file changes add a _agentic_evals_enabled() context manager so the pre-existing policy tests keep exercising the real selection logic, plus two new tests that pin the disabled state overriding --evals-only/--all-evals.

Security risks

None. This is a pure CI/config-generation change with no runtime, auth, or data-handling implications — it only affects which matrix rows get marked for an eval job in generated sweep configs.

Level of scrutiny

Low-to-moderate. The change is small, additive, and reversible (the PR body and code comments both point at the single line to flip back), it ships with new unit tests that specifically pin the guarded behavior, and I independently ran the full test file (104 passed) to confirm. I also traced the third dispatch path mentioned in the PR description — process_changelog.py's per-changelog-entry --evals-only pass — and confirmed it funnels through the same mark_eval_entries(include_agentic=True) call that the new flag now gates, so the fix is complete across all three paths the author identified.

Other factors

No workflow YAML changes, no config or perf-changelog entries, and the commit is tagged [skip-sweep], so blast radius is limited to CI matrix generation logic with existing test coverage. The bug-hunting system found no real issues; the one candidate (an apparent contradiction in EVALS.md between the new Status section and older selection-policy text) was correctly ruled out since the Status section explicitly frames the older text as describing behavior that returns once the flag is cleared.