Skip to content

feat(evaluation): evaluate existing sessions and add RecommendationClient - #707

Closed
sundargthb wants to merge 1 commit into
aws:mainfrom
sundargthb:feat/evaluation-session-runs-and-recommendations
Closed

sundargthb wants to merge 1 commit into
aws:mainfrom
sundargthb:feat/evaluation-session-runs-and-recommendations

Conversation

@sundargthb

Copy link
Copy Markdown
Member

Adds two small pieces for working with agents that already run on AgentCore Runtime. Both are framework agnostic.

BatchEvaluationRunner.run_session_evaluation(config, start_time, end_time, session_ids=None) scores sessions already in CloudWatch without invoking the agent. If no session IDs are passed, it evaluates every session in the time window. It shares the start, poll and result path with run_dataset_evaluation, so insight_results from Builtin.FailureAnalysis come back the same way.

RecommendationClient passes through the four recommendation APIs (allowlisted, accepts snake_case kwargs) and adds start_recommendation_and_wait. It follows the same pattern as DatasetClient.

result = BatchEvaluationRunner(region="us-west-2").run_session_evaluation(
    config, start_time=start, end_time=end
)
rec = RecommendationClient(region_name="us-west-2").start_recommendation_and_wait(
    wait_config=WaitConfig(max_wait=1800),
    name="prompt-rec",
    type="SYSTEM_PROMPT_RECOMMENDATION",
    recommendation_config={"systemPromptRecommendationConfig": {
        "systemPrompt": {"text": prompt},
        "agentTraces": {"batchEvaluation": {"batchEvaluationArn": result.batch_evaluation_arn}},
        "evaluationConfig": {"evaluators": [{"evaluatorArn": goal_success_arn}]},
    }},
)

Testing: unit tests added for both. The full suite passes (3618 passed). Tested live in us-west-2: GoalSuccessRate and FailureAnalysis ran over 2 existing runtime sessions in 101 s and returned 2 insight results, then a recommendation from that batch evaluation completed in 123 s.

…ient

Add BatchEvaluationRunner.run_session_evaluation to score sessions already in
CloudWatch without invoking the agent, and RecommendationClient with
start_recommendation_and_wait for system prompt and tool description
recommendations.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.74468% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@147c5b3). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...rock_agentcore/evaluation/recommendation_client.py 94.11% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #707   +/-   ##
=======================================
  Coverage        ?   89.63%           
=======================================
  Files           ?      125           
  Lines           ?    10827           
  Branches        ?     1685           
=======================================
  Hits            ?     9705           
  Misses          ?      734           
  Partials        ?      388           
Flag Coverage Δ
unittests 89.63% <95.74%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Oct 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Oct 22, 2026
@sundargthb

sundargthb commented Oct 22, 2026 •

Copy link
Copy Markdown
Member Author

Moved to #708 so CI can run from a repo branch.

@sundargthb sundargthb closed this Oct 22, 2026
@sundargthb
sundargthb deleted the feat/evaluation-session-runs-and-recommendations branch October 22, 2026 20:10
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active
@sundargthb
sundargthb deployed to auto-approve October 22, 2026 20:19 — with GitHub Actions Active

This branch had an error being deployed

1 failed deployment
auto-approve — 2aa484b3 Deployed Oct 22, 2026 by sundargthb via Test (evaluation) #1645
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants