Repository navigation
Conversation
A read-only skill for chat and Incident RCA that investigates GuardDuty and Security Hub findings, CloudTrail security alarms and Config rule changes. It classifies the incident as Benign, At risk or compromised, or Unmonitored, and writes evidence preservation, containment, eradication and recovery commands for a person to run.
An optional parameter that adds 13 read-only actions the managed policy AIDevOpsAgentAccessPolicy (v14) does not include, for the aws-security-finding-investigation skill.
…and label any other command
liyangln-AWS
left a comment
There was a problem hiding this comment.
Line numbers refer to commit 519c38b.
What I checked and found correct
- IAM actions. All 13 actions are read-only. None of them is in
AIDevOpsAgentAccessPolicyv15, which was published on 2026-11-11, after your v14 check. v15 only addscloudwatch:*andobservabilityadmin:*reads. Every read inevidence-queries.mdis covered by the managed policy plus this policy. The resource scoping (user/*,group/*) and the use of${AWS::Partition}are fine. TheAllowedRegionsnote is correct, because IAM requests carry us-east-1 asaws:RequestedRegion. - CLI syntax. I parsed 63 commands from
response-actions.mdandevidence-queries.mdwith AWS CLI 2.36.40. I ran them offline (--no-sign-requestagainst a dead endpoint, so no request was sent), and none had a parameter error. I checked the enum values by hand. - Documented behavior. These claims match the AWS docs: the
ConsoleLoginRegions for root and IAM users,accountId = "anonymous"for unsigned S3 requests, the untracked-connection rule (TCP/UDP only), the Identity Center deny onidentitystore:userIdwith the 12-hour hold, and the existence ofAWSCompromisedKeyQuarantineV1 to V4 andPolicy:IAMUser/ShortTermRootCredentialUsage.
Before sign-off
- Session revocation: timing, order, and when to remove it (
response-actions.mdrows 2–3 and lines 83–86, SKILL.md line 224,evals.jsonline 28)- The Undo column for
DenyOlderSessionsandAWSRevokeOlderSessionssays to delete the policy, and gives no time. Deleting it early brings the stolen sessions back. Keep the user deny for at least 36 hours, becauseGetSessionTokenandGetFederationTokencredentials for an IAM user can last 129,600 seconds. Keep the role deny for at least the role'sMaxSessionDuration, which can be up to 12 hours. This is the same reasoning as the 12-hour rule you already have for Identity Center. - Order: cut off the source credential first (deactivate the key, or DenyAll), then write the
aws:TokenIssueTimedeny. If the deny is written first, the key can still mint new sessions that have a later issue time. - The value of
aws:TokenIssueTime: the console's Revoke active sessions sets it about 30 seconds in the future to cover policy propagation, and the CLI path should do the same. I would remove the "one minute after the attacker's last GetSessionToken/AssumeRole call" option. CloudTrail delivery is delayed, STS Regional-endpoint calls are logged in the Region that was called, and the key may still be active, so "the last call we can see" is not a safe cutoff. Update the eval assertion on line 28 to match.
- The Undo column for
- Revocation scope (SKILL.md line 226,
response-actions.mdline 86). "Every role the key assumed" misses chained roles (roles the role sessions assumed in turn) and roles in other accounts. Those must be revoked in the account that owns each role. Mode 1 already collects chaining, so the containment text should use it. - Coverage checks that don't confirm the needed event type is recorded. This is the skill's core rule ("no record is not proof"), and two checks miss it:
- Flow logs (
evidence-queries.mdline 58): readTrafficType. A flow log withTrafficType=REJECTnever contains ACCEPT records, so "no accepted flows" would be reported as "exposure without observed use". Also check subnet-level flow logs. Today only the ENI and VPC are checked. - S3 data events (
evidence-queries.mdline 51): readReadWriteTypeon basic selectors, and thereadOnlyandeventNamefield selectors on advanced ones. A trail that records onlyWriteOnlydata events and delivers them to CloudWatch Logs gives 0GetObject, and the skill would report that as a count.
- Flow logs (
- Root containment (
response-actions.mdline 95). The IAM docs say policies can't change the permissions of temporary credentials that root created withGetSessionTokenorGetFederationToken. Those credentials stay valid until they expire (up to 1 hour). Say so. In a member account, an SCP that denies the root user is the only control that applies to them.
Strongly recommended
- Eradication gaps. The section 4 event list has these, but section 5 has no row for them:
AddUserToGroup→remove-user-from-groupAttachRolePolicy,PutRolePolicy,AttachGroupPolicy,PutGroupPolicyon existing roles and groups (today only roles the attacker created are covered)CreateSAMLProvider(you have a row for OIDC but not for SAML)- MFA devices the attacker registered (
CreateVirtualMFADevice,EnableMFADevice) - Snapshot and AMI sharing (
ModifySnapshotAttribute,ModifyImageAttribute) has no unshare command. RDS snapshot sharing (ModifyDBSnapshotAttribute,ModifyDBClusterSnapshotAttribute) is missing from both the events and the actions.
- Defense-evasion events missing from mode 4 and section 4: GuardDuty
CreateIPSet/UpdateIPSet(a trusted IP list suppresses findings) andCreateFilter/UpdateFilter(auto-archive),DisassociateFromAdministratorAccount; Security HubDisableSecurityHubandBatchUpdateFindings; OrganizationsLeaveOrganization;PutBucketLifecycleandPutBucketPolicyon the trail's bucket; ConfigDeleteDeliveryChannel; and deletion of Route 53 Resolver query logging. - Identity Center row
- It says "merge into the inline policy" but never gives the write command. Add
aws sso-admin put-inline-policy-to-permission-set --instance-arn <instance-arn> --permission-set-arn <permission-set-arn> --inline-policy file://merged.json. I checked the syntax. - SCPs don't apply to the management account. If the user holds permission sets there, the permission set inline policy is the only option.
- With an external IdP or AD, the AWS procedure disables the user in the identity source before deleting the portal session (step 5 before step 6). Otherwise the user can start a new portal session. The current order is right only for the Identity Center directory.
- It says "merge into the inline policy" but never gives the write command. Add
- Versioning row (
response-actions.mdline 130). "Protect objects from further deletion" overstates what versioning does. A principal withs3:DeleteObjectVersioncan still delete versions permanently, and a lifecycle rule can expire noncurrent versions. Something like "keeps earlier versions of overwritten or deleted objects" is accurate. - Verify step (
response-actions.mdline 168).get-access-key-last-usedonly covers the long-term key. To confirm that session revocation worked, check CloudTrail for theASIA…keys issued before the cutoff and confirm that they now get explicit-denyAccessDenied.
The SME checklist box is already ticked while my sign-off is pending. Please untick it until I approve.
… review Session revocation: cut off the source credential first, use a time after now, keep the deny until the credentials expire, cover chained roles and other accounts, root temporary credentials, and verify through CloudTrail. Coverage checks read the flow log traffic type and the data event read/write type. More eradication rows and defense evasion events, an IAM Identity Center write command and order, and a more accurate S3 versioning row.
Thanks for the review. I fixed all nine points in faa4a11. The latest changes are under skills/aws-security-finding-investigation/ in that commit.
I also updated the AIDevOpsAgentAccessPolicy version in skill policies template to v15. I ran the new commands offline like you did, and none had a parameter error. I re-ran the key deactivation chat case 3 times and the key escalation investigation 3 times with this updated version (one of three was a re-run). All 3 chat answers keep the user deny for 36 hours and the role deny for the role's I unticked the SME box. Could you take another look when you have time? Thank you! Best, |
liyangln-AWS
left a comment
There was a problem hiding this comment.
SME sign-off for faa4a11. I re-checked the nine items from my earlier review against this commit, and all of them are addressed:
- Session revocation: the source credential is cut off first,
aws:TokenIssueTimeis set about 30 seconds after now, the deny is kept at least 36 hours for a user and at leastMaxSessionDurationfor a role, chained and cross-account roles are included, root temporary credentials are called out, and the verify step checks theASIA…keys in CloudTrail. - Coverage checks: flow logs now read
TrafficTypeat the ENI, subnet and VPC; data events now readReadWriteType,readOnlyandeventName, and a write-only trail is reported as Unknown. - The new eradication rows, the defense-evasion events, the Identity Center write command, the management-account note and identity-source order, and the versioning row are correct.
As before, I parsed the new commands and their undo forms (34 invocations) offline with AWS CLI 2.36.40, and none had a parameter error. AIDevOpsAgentAccessPolicy v15 grants every new read-only check, so the template still needs only the 13 added actions.
Two optional follow-ups, not blocking: list-virtual-mfa-devices doesn't show FIDO or hardware MFA devices (the EnableMFADevice events still do), and the shared-snapshot row could add aws rds describe-db-cluster-snapshot-attributes as the read check for cluster snapshots.
You can tick the SME box now.
|
Thanks for the review and the sign-off! I will leave the two optional follow-ups for a later change, so the commit you approved stays the PR head. |
|
Hi @yeremyjt and @ams-thakkar, this PR is ready for review, and Yang Li's SME sign-off is in the review above. I could not run the skill evaluation tool, so |
Description
This PR adds
aws-security-finding-investigation, a read-only skill that helps DevOps Agent investigate AWS security findings, both in chat and in Incident RCA. The skill starts from a GuardDuty or Security Hub finding, a security alarm built on CloudTrail events, or an AWS Config rule that turns NON_COMPLIANT. It answers four questions with evidence: who acted, what changed, what is exposed, and whether anyone exploited the exposure. It then classifies the incident as Benign, At risk or compromised, or Unmonitored, and suggests a P1 to P4 priority with its basis. Last, it generates the commands for evidence preservation, containment, eradication and recovery. The agent never runs them. A person reviews and runs each one.The PR also adds three files outside the skill:
EnableAwsSecurityFindingInvestigationparameter in the skill policies template and its README. It adds 13 read-only actions thatAIDevOpsAgentAccessPolicydoesn't include (see Testing).llms.txt.Issue: #214
What the skill fixes
Without proper guidance in skill, the agent can reach wrong root cause and give incorrect security remediation advice. These are the gaps I saw in testing without the skill:
ConsoleLoginevents can be logged in us-east-1, us-east-2 or us-west-2, so a search in one Region can miss them.The skill also covers containment for IAM Identity Center users, whose roles can't be edited in IAM.
Overlap with other skills
No existing skill investigates an AWS security finding and writes the containment, eradication and recovery steps, so this skill has no functional overlap with other skills. The closest ones are different in scope:
enrich-with-aws-security-agentandwiz-security-contextbring in findings from other security tools during an operational investigation, but they don't investigate the AWS security signal itself.network-log-analyzertroubleshoots network connectivity with flow logs, such as timeouts and REJECT records, while this skill reads flow logs only to check whether traffic reached a port after a security group was opened.guardduty-cost-optimizationandcloudtrail-cost-optimizationreduce spend on those services, while this skill investigates their findings and does not look at cost.#200 covers IAM key hygiene, while this skill only handles a key that may be compromised. #139 covers WAF logs, which are out of scope. The negative evals cover #200, #139, flow log connectivity and GuardDuty cost, and the trigger queries cover CloudTrail cost and the Wiz case.
Type of change
Required before review
/skills, MCP servers in/mcp, custom agents in/custom-agents; skills and MCP servers are never mixed in one directory.1.0.0.Testing
There are no skill evaluation tool results yet. As an external contributor, I don't have access to the tool, so I submitted #214 to request the maintainer to run it. If running the tool is not possible soon, I have a proposed
evals/exemptions.jsonready, and I would commit it only if you approve.My own with-skill and without-skill runs I wrote my own test cases (
evals/evals.jsonandeval_queries.json) and ran them against DevOps Agent myself. These are not results from the official tool. The full evidence file, with per-case results and examples, is here: https://gist.github.com/szl0144/5548d1d63b2fd8ea54ff4c344845ee83AIDevOpsAgentAccessPolicyand this skill's optional policy, and neither has an AGENTS.md or MCP servers.evals/evals.json3 times with the skill and ran every positive case 3 times without the skill. After each later fix, I re-ran the cases that the fix touched (3 runs each) and every query ineval_queries.json. The evidence file shows the skill version behind each result.AttachUserPolicyon itself, etc.evals.json, using the real account state as ground truth. I read every failed assertion myself.Known limits:
bucket-read-counts, so that case gave the same result with and without the skill.Local checks on faa4a11:
validate_skills.py --base-ref upstream/mainvalidate_skill_evals.py --skill aws-security-finding-investigationscan_aws_identifiers.pypython3 -m unittest discover -s testsmkdocs build --strictcfn-linton the policy templateiam simulate-principal-policyon the test roleUpdateAccessKeyandPutRolePolicyare deniedstructurechecks pass. n/a: I can't run the skill evaluation tool, as explained above.validate_skills.pypasses.best-practicesrun across multiple iterations, with consistent results. n/a: I can't run the skill evaluation tool, as explained above.functionalrun with and without the change, so the difference in the agent's behavior is visible. These are my own runs, not the tool's, as described above.evals/evals.jsonincludes positive cases (should trigger) and at least one negative case (should not trigger on unrelated input).Skills — complete if this PR adds or changes a skill
SKILL.mdfrontmatter is valid:namematches the directory,descriptionpresent and within length limits, body at most 500 lines. The description has 1,021 characters (the limit is 1,024), and the body has 259 lines.descriptionis written from the agent's point of view — scenarios, services, error types, symptoms. It lists the GuardDuty finding types, the CloudTrail and Config signals, and the questions users ask, such as "can we close this finding?". It also says what the skill does not cover.metadata.aws-devops-agent-skills.agent-types(INCIDENT_RCA,INCIDENT_TRIAGE,INCIDENT_MITIGATION, …), not leftGENERICwithout reason. The skill is scoped toCHATandINCIDENT_RCA.references/orassets/and loads on demand. The skill has four reference files, and each one loads at a named step.Security and safety
License confirmation