Skip to content

fix(evals): bound StepEfficiency redundancy by tool calls, not steps - #143

Closed
jjcav84 wants to merge 1 commit into
future-agi:devfrom
jjcav84:fix/step-efficiency-denominator
Closed

jjcav84 wants to merge 1 commit into
future-agi:devfrom
jjcav84:fix/step-efficiency-denominator

Conversation

@jjcav84

@jjcav84 jjcav84 commented Oct 7, 2026

Copy link
Copy Markdown

Summary

StepEfficiency is documented to return 0.0–1.0 but could go negative: redundant_count is incremented once per redundant tool call while the ratio divided by steps. One step with 5 identical calls produced -0.2, and TrajectoryScore (30% weight) inherited the deficit (#60).

Per maintainer direction on the issue — option 1, no clamp: the redundancy term now divides by total_calls, matching the failed_calls / total_calls pattern directly below it. The ratio is bounded by construction.

One extra rename: details["redundant_steps"] → details["redundant_calls"]. The counter was always counting calls; the key name was part of the same confusion. Nothing in the repo reads it (grepped src/, tests/, typescript/).

Verification

  • uv run ruff check . — clean on changed files
  • Focused pytest target: uv run pytest tests/test_agent_step_efficiency.py — 6/6 pass; covers the issue repro (5 redundant calls in 1 step → 0.76), the single-call-per-step case (→ 0.8), zero-tool-call steps, and TrajectoryScore composition bounds
  • Full uv run pytest -q — not run; repo suite is large and the change is confined to one method in metrics.py + its new test file
  • TypeScript build/test — N/A, no typescript/ changes
  • agent-learn release-proof — N/A, no release-candidate behavior touched

Checklist

  • Public API names unchanged — same class/method; only the details diagnostic key renamed
  • New user-facing workflow — N/A (bug fix to documented [0,1] contract)
  • New release claim — N/A
  • No hosted-service dependency added
  • Security-sensitive output — N/A

Closes #60.

StepEfficiency counted redundant *tool calls* but divided by the number
of *steps*. A step holding several redundant calls drove the ratio below
zero and the documented 0.0-1.0 score went negative (future-agi#60), which also
pulled the composed TrajectoryScore down.

Divide by total_calls instead — the same denominator the failure term
uses — so the ratio is bounded by construction. Rename the diagnostic
details key to redundant_calls to match what is counted; nothing in the
repo consumes the old key.

Generated with Devin
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@khushalsonawat
khushalsonawat deleted the branch future-agi:dev October 9, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants