Description
Symptom
A behavioral case is ungraded because the grader call exceeds the model's context window:
tracelens-analysis-orchestrator (Linux)
[FAIL] (expected_behavior) Filled the Executive Summary of analysis.md with measured numbers [...]
-- llm_judge gave no JSON verdict: 'Prompt is too long'
This was one of the two cases in #25. That issue is about how the result is reported, and #26 fixes that part: the expectation above is now recorded as ungraded rather than published as the skill having failed it. The reason it could not be graded is untouched, so the expectation still goes unmeasured.
Cause
_grade_with_llm in skillscope/agent.py bounds two parts of the evidence it sends and not the third:
| part | bound |
|---|---|
| transcript commands and outputs | 4000 chars |
| agent final message | 1500 chars |
| tool names | small, a set of names |
workspace file list (run.files) |
none |
_list_workspace_files walks workspace.rglob("*") and keeps every path, skipping only .claude. The fixed cost of the rest of the prompt is about 5.5k characters, so the file list is effectively the whole budget. A run that installs a package, creates a virtualenv, clones a repo or writes a build tree can list tens of thousands of paths.
There is a second route to the same error that a cap would not fix. The judge is given --add-dir <workspace> and told it may read files, so it takes agent turns. Opening one multi-hundred-megabyte artifact puts that content in its conversation rather than in the prompt we built, and the API reports Prompt is too long either way. For a skill whose subject is trace files, and whose expectation invites the judge to go and check measured numbers, this looks at least as likely as the file list.
The two are not distinguishable from the error text alone. Reproducing against a real workspace would say which one is in play here, and whether both need addressing.
Note on #19
The engine in #19 already bounds this on its side, in engine/judge.py:
MAX_FILES = 20
MAX_FILE_BYTES = 20_000
MAX_TRANSCRIPT = 12_000
MAX_RESULT = 800So this is scoped to the engine currently on main. If #19 lands first, the useful residue is confirming the second route above is covered too, since those constants bound what the judge is handed, not what it then chooses to read.
Activity