Issues

• [Feature]: Retire --sdk and the AgentRunner evaluators (appworld_eval.py, oak eval_bench.py) after cuga-agent#581

  chorecomplexity: mediumenhancementevaluation: appworldevaluation: oakpriority: medium
  cuga-project/cuga-eval#206 · Filter by author Sergey-Zeltyn opened on Sep 27, 2026

• [Feature]: AutomationBench integration pilot

  evaluation
  Feature

  cuga-project/cuga-eval#203 · Filter by author Sergey-Zeltyn opened on Sep 23, 2026
  

• [Feature]: Configurable VAKRA adapter presets for the M3 benchmark (configured CUGA, never bare main)

  enhancementevaluation: vakra
  cuga-project/cuga-eval#201 · Filter by author segevshlomovIBM opened on Sep 15, 2026

• Scheduled dependency audit: pip-audit found vulnerabilities

  type: security
  cuga-project/cuga-eval#200 · Filter by author github-actions opened on Sep 14, 2026
  5

• Leaderboard/workspace SDK runs always exit 1: eval.sh stale-results guard doesn't see workspace results, skipping finalize + official evaluate

  evaluation
  cuga-project/cuga-eval#198 · Filter by author Sergey-Zeltyn opened on Sep 3, 2026

• transformers 5.8.1 on macOS: CVE-2026-9856 (path traversal via save_pretrained), never scanned by CI

  type: security
  cuga-project/cuga-eval#193 · Filter by author haroldship opened on Sep 2, 2026

• [Evaluation experiment:] Use Indigo to improve CUGA on Appworld

  evaluation: appworld
  Task

  cuga-project/cuga-eval#184 · Filter by author Sergey-Zeltyn opened on Sep 1, 2026
  

• [Feature]: Call vendor/vakra's evaluator directly instead of maintaining a full local fork

  cuga-project/cuga-eval#177 · Filter by author haroldship opened on Aug 26, 2026

• M3 eval.sh/compare.sh no-flag defaults still hardcode the now-removed legacy data files

  cuga-project/cuga-eval#174 · Filter by author haroldship opened on Aug 26, 2026

• Extend M3_RUN_TMP_DIR-style /tmp run isolation to appworld/bpo/oak

  cuga-project/cuga-eval#173 · Filter by author haroldship opened on Aug 26, 2026

• M3 sequential domains lose all tools after the first in --no-ground-truth mode; Vakra output files not scoped to bundle

  cuga-project/cuga-eval#169 · Filter by author haroldship opened on Aug 26, 2026

• Add recurring-interruption diagnostics to M3 eval runs (SIGINT observer + stall watchdog)

  cuga-project/cuga-eval#168 · Filter by author haroldship opened on Aug 26, 2026

• [Feature]: AppWorld obstacle-escalation rider via the existing special_instructions block (stronger #706 wording, benchmark-scoped)

  enhancementevaluation: appworldneeds-triage
  cuga-project/cuga-eval#167 · Filter by author Sergey-Zeltyn opened on Aug 26, 2026
  1

  

• [Bug]: stale err after a successful retry marks a passing AppWorld SDK task as failed

  bugevaluation: appworldpriority: medium
  cuga-project/cuga-eval#153 · Filter by author Sergey-Zeltyn opened on Aug 4, 2026

• feat(m3): fail fast when no tools are loaded (extend #148 guard to m3)

  cuga-project/cuga-eval#150 · Filter by author Sergey-Zeltyn opened on Jul 30, 2026

• [Evaluation experiment]: tau2-bench baselines

  evaluation
  Task

  cuga-project/cuga-eval#147 · Filter by author Sergey-Zeltyn opened on Jul 29, 2026
  

• [Configuration decision]: Decide on optimal parameters of Appworld configuration

  evaluation: appworld
  Task

  cuga-project/cuga-eval#146 · Filter by author Sergey-Zeltyn opened on Jul 27, 2026
  

• [Skills]: Add Claude skills on analysis of individual traces and evaluation bundles

  evaluation: appworldevaluation
  Task

  cuga-project/cuga-eval#145 · Filter by author Sergey-Zeltyn opened on Jul 27, 2026
  

• [Bug]: broad except Exception in evaluate_multiturn_task_with_langfuse mislabels genuine LLM errors as Langfuse failures and silently re-runs the whole task

  bugevaluation: vakrapriority: mediumtype: performance
  cuga-project/cuga-eval#144 · Filter by author haroldship opened on Jul 22, 2026
  1

  

• [Bug]: GroundednessJudge silently scores infra/API errors (negative max_tokens crash) as hard fails

  bugevaluation: vakrapriority: high
  cuga-project/cuga-eval#143 · Filter by author haroldship opened on Jul 22, 2026
  1

• Replace hardcoded gpt-oss/Groq provider in model_profiles.sh with a selectable/config-driven mechanism

  cuga-project/cuga-eval#136 · Filter by author haroldship opened on Jul 21, 2026

• resumed M3 runs silently drop earlier console logs and may not complete all tasks (cap4-300-no-policy-check: 275/300)

  cuga-project/cuga-eval#135 · Filter by author haroldship opened on Jul 21, 2026

• [Bug]: Langfuse trace export fails with 422 for very large traces; bundle silently omits them

  bugevaluation: vakrapriority: medium
  cuga-project/cuga-eval#128 · Filter by author haroldship opened on Jul 16, 2026
  1

• [Feature]: Add capability_4 M3 dataset and clean up unlicensed legacy data files

  enhancementevaluation: vakrapriority: high
  cuga-project/cuga-eval#125 · Filter by author haroldship opened on Jul 15, 2026
  

• [Bug]: Hardcoded shared /tmp paths collide between concurrent benchmark runs (console log, summary, registry log)

  bug
  cuga-project/cuga-eval#115 · Filter by author haroldship opened on Jul 9, 2026