Skip to content

Add aws-security-finding-investigation skill - #236

Open
szl0144 wants to merge 4 commits into
aws:mainfrom
szl0144:feature/aws-security-finding-investigation
Open

szl0144 wants to merge 4 commits into
aws:mainfrom
szl0144:feature/aws-security-finding-investigation

Conversation

@szl0144

@szl0144 szl0144 commented Nov 11, 2026 •

Copy link
Copy Markdown

Description

This PR adds aws-security-finding-investigation, a read-only skill that helps DevOps Agent investigate AWS security findings, both in chat and in Incident RCA. The skill starts from a GuardDuty or Security Hub finding, a security alarm built on CloudTrail events, or an AWS Config rule that turns NON_COMPLIANT. It answers four questions with evidence: who acted, what changed, what is exposed, and whether anyone exploited the exposure. It then classifies the incident as Benign, At risk or compromised, or Unmonitored, and suggests a P1 to P4 priority with its basis. Last, it generates the commands for evidence preservation, containment, eradication and recovery. The agent never runs them. A person reviews and runs each one.

The PR also adds three files outside the skill:

  • An optional EnableAwsSecurityFindingInvestigation parameter in the skill policies template and its README. It adds 13 read-only actions that AIDevOpsAgentAccessPolicy doesn't include (see Testing).
  • An entry in llms.txt.

Issue: #214

What the skill fixes

Without proper guidance in skill, the agent can reach wrong root cause and give incorrect security remediation advice. These are the gaps I saw in testing without the skill:

  • When a GuardDuty finding or CloudTrail records show that an access key is being misused, the agent advised deactivating the key but did not revoke the temporary credentials and sessions already issued with it (15 of 15 runs). The attacker can keep using them after the key is deactivated. One mitigation plan said that deactivating the keys "fully cuts off its access".
  • When asked whether a finding can be closed as benign, the agent did not check whether the audit trail was complete for the whole window (3 of 3 runs). If a trail does not exist, or only recorded events for part of that window, it will reply the finding can be closed because nothing suspicious was found.
  • Investigations didn't suggest a P1 to P4 priority (9 of 9 runs).
  • For root sign-ins, the agent didn't say which Regions it checked in 4 of 6 runs. Root ConsoleLogin events can be logged in us-east-1, us-east-2 or us-west-2, so a search in one Region can miss them.

The skill also covers containment for IAM Identity Center users, whose roles can't be edited in IAM.

Overlap with other skills

No existing skill investigates an AWS security finding and writes the containment, eradication and recovery steps, so this skill has no functional overlap with other skills. The closest ones are different in scope:

enrich-with-aws-security-agent and wiz-security-context bring in findings from other security tools during an operational investigation, but they don't investigate the AWS security signal itself.
network-log-analyzer troubleshoots network connectivity with flow logs, such as timeouts and REJECT records, while this skill reads flow logs only to check whether traffic reached a port after a security group was opened. guardduty-cost-optimization and cloudtrail-cost-optimization reduce spend on those services, while this skill investigates their findings and does not look at cost.
#200 covers IAM key hygiene, while this skill only handles a key that may be compromised. #139 covers WAF logs, which are out of scope. The negative evals cover #200, #139, flow log connectivity and GuardDuty cost, and the trigger queries cover CloudTrail cost and the Wiz case.

Type of change

  • New skill
  • New custom agent
  • New MCP server
  • Update to an existing skill, agent, or MCP server
  • Documentation or infrastructure change

Required before review

  • Repository layout is correct — skills in /skills, MCP servers in /mcp, custom agents in /custom-agents; skills and MCP servers are never mixed in one directory.
  • A domain SME has reviewed and signed off. A named subject-matter expert for this domain — someone other than the author — has reviewed the tool for technical correctness and approved it in this PR. Required for any change to what the tool tells the agent; an edit that changes no technical claim can be marked n/a with the reason, as CONTRIBUTING.md describes. SME: @liyangln-AWS, AWS Security Solutions Architect, who is reviewing the response commands, the IAM actions and the other technical claims, and approved commit faa4a11 in the review above.
  • Evaluation evidence is included (see Testing).
  • No internal-only or proprietary content — no internal hostnames, account IDs, customer names, ticket numbers, internal URLs, or secrets in the tool, its examples, or its tests.
  • README is complete — prerequisites, setup/deployment, and at least one sample prompt or usage example.
  • CHANGELOG updated and the version bumped. This is a new skill at 1.0.0.
  • Named owner for ongoing maintenance. @szl0144

Testing

There are no skill evaluation tool results yet. As an external contributor, I don't have access to the tool, so I submitted #214 to request the maintainer to run it. If running the tool is not possible soon, I have a proposed evals/exemptions.json ready, and I would commit it only if you approve.

My own with-skill and without-skill runs I wrote my own test cases (evals/evals.json and eval_queries.json) and ran them against DevOps Agent myself. These are not results from the official tool. The full evidence file, with per-case results and examples, is here: https://gist.github.com/szl0144/5548d1d63b2fd8ea54ff4c344845ee83

  • To compare runs with and without the skill, I used two Agent Spaces in the same account. The only difference between them is the skill: one has no custom skill, and the other has only this skill. Both use the same IAM role, which has AIDevOpsAgentAccessPolicy and this skill's optional policy, and neither has an AGENTS.md or MCP servers.
  • I ran every case in evals/evals.json 3 times with the skill and ran every positive case 3 times without the skill. After each later fix, I re-ran the cases that the fix touched (3 runs each) and every query in eval_queries.json. The evidence file shows the skill version behind each result.
  • The account-level cases ran against real resources, including security groups open to SSH in us-east-1 and us-west-2 with flow logs on, a bucket whose S3 data events go to CloudWatch Logs, and an IAM user that created a second key and was denied AttachUserPolicy on itself, etc.
  • An LLM judge (Claude Sonnet 5 on Bedrock) graded each answer against the assertions in evals.json, using the real account state as ground truth. I read every failed assertion myself.
With skill Without skill
All positive cases, assertions passed 91.3% (189/207) 71.0% (147/207)
Investigations, real resources 92.1% (58/63) 66.7% (42/63)
Chats, records given in the question 93.5% (115/123) 74.0% (91/123)
Chat, real resources (one case; the skill didn't load) 76.2% (16/21) 66.7% (14/21)
Expected output fully met 25 of 39 runs 4 of 39 runs
Cases better / same / worse (10-point rule) 11 / 2 / 0
Skill loaded: positive cases / negative cases / trigger queries 32 of 39 / 0 of 15 / 41 of 45 should-load, 0 of 9 should-not
Commands that changed state 0 0

Known limits:

  • The scores are a guide. An LLM judge (Claude Sonnet 5 on Amazon Bedrock) graded the answers, each case ran 3 times in one test account, and the with-skill results come from four versions of the skill (The linked evidence file says which version produced the result for each case.).
  • In chat, the skill loaded in 23 of 30 positive runs. It never loaded in bucket-read-counts, so that case gave the same result with and without the skill.
  • Investigations with the skill gave a P1 to P4 priority with its basis in 6 of 9 runs.
  • Investigations with the skill took 797 s on average, against 496 s without the skill. Chats took 76 s against 55 s.

Local checks on faa4a11:

Check Result
validate_skills.py --base-ref upstream/main 0 errors and no warnings for this skill
validate_skill_evals.py --skill aws-security-finding-investigation Fails, because there are no tool results yet
scan_aws_identifiers.py 0 findings
python3 -m unittest discover -s tests 245 tests, OK
mkdocs build --strict OK
cfn-lint on the policy template No errors, only the 4 existing W2001 warnings
iam simulate-principal-policy on the test role The 13 new actions are allowed, and the managed policy alone allows none of them. Write actions such as UpdateAccessKey and PutRolePolicy are denied
  • structure checks pass. n/a: I can't run the skill evaluation tool, as explained above. validate_skills.py passes.
  • best-practices run across multiple iterations, with consistent results. n/a: I can't run the skill evaluation tool, as explained above.
  • functional run with and without the change, so the difference in the agent's behavior is visible. These are my own runs, not the tool's, as described above.
  • evals/evals.json includes positive cases (should trigger) and at least one negative case (should not trigger on unrelated input).

Skills — complete if this PR adds or changes a skill

  • SKILL.md frontmatter is valid: name matches the directory, description present and within length limits, body at most 500 lines. The description has 1,021 characters (the limit is 1,024), and the body has 259 lines.
  • The description is written from the agent's point of view — scenarios, services, error types, symptoms. It lists the GuardDuty finding types, the CloudTrail and Config signals, and the questions users ask, such as "can we close this finding?". It also says what the skill does not cover.
  • Scoped to a specific agent type in metadata.aws-devops-agent-skills.agent-types (INCIDENT_RCA, INCIDENT_TRIAGE, INCIDENT_MITIGATION, …), not left GENERIC without reason. The skill is scoped to CHAT and INCIDENT_RCA.
  • Procedures are structured as decision trees on real conditions, not flat prose. The incident state (Benign, At risk or compromised, or Unmonitored) depends on named conditions. The error handling table says what to do for each kind of failed read. The tool used to read CloudTrail depends on whether the agent runs in chat or in an investigation, and the format of the answer depends on the type of question. The skill has six steps that the agent follows in order, and the skill sends the agent back to Step 2 when it finds another credential, resource or account.
  • Reference material lives in references/ or assets/ and loads on demand. The skill has four reference files, and each one loads at a named step.
  • Non-executable content only — no scripts or code in the skill directory. Deterministic compute belongs in an MCP server. The skill directory contains only Markdown, JSON and YAML files.

Security and safety

  • Read-only by default; any action that changes customer state is explicit, documented, and approval-gated. The agent runs none of these actions, and each one comes with its impact, how to undo it, and who approves it.
  • Customer data stays in the customer's account. If an approved reputation tool is connected, the skill sends it only public IP addresses and domains.
  • No destructive operations without an explicit, documented confirmation step. The skill recommends a delete only for a resource the attacker created, after the evidence is saved, and marks it irreversible and asks for confirmation.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

A read-only skill for chat and Incident RCA that investigates GuardDuty and Security Hub findings, CloudTrail security alarms and Config rule changes. It classifies the incident as Benign, At risk or compromised, or Unmonitored, and writes evidence preservation, containment, eradication and recovery commands for a person to run.
An optional parameter that adds 13 read-only actions the managed policy AIDevOpsAgentAccessPolicy (v14) does not include, for the aws-security-finding-investigation skill.

@liyangln-AWS liyangln-AWS left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line numbers refer to commit 519c38b.

What I checked and found correct

  • IAM actions. All 13 actions are read-only. None of them is in AIDevOpsAgentAccessPolicy v15, which was published on 2026-11-11, after your v14 check. v15 only adds cloudwatch:* and observabilityadmin:* reads. Every read in evidence-queries.md is covered by the managed policy plus this policy. The resource scoping (user/*, group/*) and the use of ${AWS::Partition} are fine. The AllowedRegions note is correct, because IAM requests carry us-east-1 as aws:RequestedRegion.
  • CLI syntax. I parsed 63 commands from response-actions.md and evidence-queries.md with AWS CLI 2.36.40. I ran them offline (--no-sign-request against a dead endpoint, so no request was sent), and none had a parameter error. I checked the enum values by hand.
  • Documented behavior. These claims match the AWS docs: the ConsoleLogin Regions for root and IAM users, accountId = "anonymous" for unsigned S3 requests, the untracked-connection rule (TCP/UDP only), the Identity Center deny on identitystore:userId with the 12-hour hold, and the existence of AWSCompromisedKeyQuarantine V1 to V4 and Policy:IAMUser/ShortTermRootCredentialUsage.

Before sign-off

  1. Session revocation: timing, order, and when to remove it (response-actions.md rows 2–3 and lines 83–86, SKILL.md line 224, evals.json line 28)
    • The Undo column for DenyOlderSessions and AWSRevokeOlderSessions says to delete the policy, and gives no time. Deleting it early brings the stolen sessions back. Keep the user deny for at least 36 hours, because GetSessionToken and GetFederationToken credentials for an IAM user can last 129,600 seconds. Keep the role deny for at least the role's MaxSessionDuration, which can be up to 12 hours. This is the same reasoning as the 12-hour rule you already have for Identity Center.
    • Order: cut off the source credential first (deactivate the key, or DenyAll), then write the aws:TokenIssueTime deny. If the deny is written first, the key can still mint new sessions that have a later issue time.
    • The value of aws:TokenIssueTime: the console's Revoke active sessions sets it about 30 seconds in the future to cover policy propagation, and the CLI path should do the same. I would remove the "one minute after the attacker's last GetSessionToken/AssumeRole call" option. CloudTrail delivery is delayed, STS Regional-endpoint calls are logged in the Region that was called, and the key may still be active, so "the last call we can see" is not a safe cutoff. Update the eval assertion on line 28 to match.
  2. Revocation scope (SKILL.md line 226, response-actions.md line 86). "Every role the key assumed" misses chained roles (roles the role sessions assumed in turn) and roles in other accounts. Those must be revoked in the account that owns each role. Mode 1 already collects chaining, so the containment text should use it.
  3. Coverage checks that don't confirm the needed event type is recorded. This is the skill's core rule ("no record is not proof"), and two checks miss it:
    • Flow logs (evidence-queries.md line 58): read TrafficType. A flow log with TrafficType=REJECT never contains ACCEPT records, so "no accepted flows" would be reported as "exposure without observed use". Also check subnet-level flow logs. Today only the ENI and VPC are checked.
    • S3 data events (evidence-queries.md line 51): read ReadWriteType on basic selectors, and the readOnly and eventName field selectors on advanced ones. A trail that records only WriteOnly data events and delivers them to CloudWatch Logs gives 0 GetObject, and the skill would report that as a count.
  4. Root containment (response-actions.md line 95). The IAM docs say policies can't change the permissions of temporary credentials that root created with GetSessionToken or GetFederationToken. Those credentials stay valid until they expire (up to 1 hour). Say so. In a member account, an SCP that denies the root user is the only control that applies to them.

Strongly recommended

  1. Eradication gaps. The section 4 event list has these, but section 5 has no row for them:
    • AddUserToGroup → remove-user-from-group
    • AttachRolePolicy, PutRolePolicy, AttachGroupPolicy, PutGroupPolicy on existing roles and groups (today only roles the attacker created are covered)
    • CreateSAMLProvider (you have a row for OIDC but not for SAML)
    • MFA devices the attacker registered (CreateVirtualMFADevice, EnableMFADevice)
    • Snapshot and AMI sharing (ModifySnapshotAttribute, ModifyImageAttribute) has no unshare command. RDS snapshot sharing (ModifyDBSnapshotAttribute, ModifyDBClusterSnapshotAttribute) is missing from both the events and the actions.
  2. Defense-evasion events missing from mode 4 and section 4: GuardDuty CreateIPSet/UpdateIPSet (a trusted IP list suppresses findings) and CreateFilter/UpdateFilter (auto-archive), DisassociateFromAdministratorAccount; Security Hub DisableSecurityHub and BatchUpdateFindings; Organizations LeaveOrganization; PutBucketLifecycle and PutBucketPolicy on the trail's bucket; Config DeleteDeliveryChannel; and deletion of Route 53 Resolver query logging.
  3. Identity Center row
    • It says "merge into the inline policy" but never gives the write command. Add aws sso-admin put-inline-policy-to-permission-set --instance-arn <instance-arn> --permission-set-arn <permission-set-arn> --inline-policy file://merged.json. I checked the syntax.
    • SCPs don't apply to the management account. If the user holds permission sets there, the permission set inline policy is the only option.
    • With an external IdP or AD, the AWS procedure disables the user in the identity source before deleting the portal session (step 5 before step 6). Otherwise the user can start a new portal session. The current order is right only for the Identity Center directory.
  4. Versioning row (response-actions.md line 130). "Protect objects from further deletion" overstates what versioning does. A principal with s3:DeleteObjectVersion can still delete versions permanently, and a lifecycle rule can expire noncurrent versions. Something like "keeps earlier versions of overwritten or deleted objects" is accurate.
  5. Verify step (response-actions.md line 168). get-access-key-last-used only covers the long-term key. To confirm that session revocation worked, check CloudTrail for the ASIA… keys issued before the cutoff and confirm that they now get explicit-deny AccessDenied.

The SME checklist box is already ticked while my sign-off is pending. Please untick it until I approve.

… review

Session revocation: cut off the source credential first, use a time after now, keep the deny until the credentials expire, cover chained roles and other accounts, root temporary credentials, and verify through CloudTrail. Coverage checks read the flow log traffic type and the data event read/write type. More eradication rows and defense evasion events, an IAM Identity Center write command and order, and a more accurate S3 versioning row.
@szl0144

szl0144 commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

Line numbers refer to commit 519c38b.

What I checked and found correct

  • IAM actions. All 13 actions are read-only. None of them is in AIDevOpsAgentAccessPolicy v15, which was published on 2026-11-11, after your v14 check. v15 only adds cloudwatch:* and observabilityadmin:* reads. Every read in evidence-queries.md is covered by the managed policy plus this policy. The resource scoping (user/*, group/*) and the use of ${AWS::Partition} are fine. The AllowedRegions note is correct, because IAM requests carry us-east-1 as aws:RequestedRegion.
  • CLI syntax. I parsed 63 commands from response-actions.md and evidence-queries.md with AWS CLI 2.36.40. I ran them offline (--no-sign-request against a dead endpoint, so no request was sent), and none had a parameter error. I checked the enum values by hand.
  • Documented behavior. These claims match the AWS docs: the ConsoleLogin Regions for root and IAM users, accountId = "anonymous" for unsigned S3 requests, the untracked-connection rule (TCP/UDP only), the Identity Center deny on identitystore:userId with the 12-hour hold, and the existence of AWSCompromisedKeyQuarantine V1 to V4 and Policy:IAMUser/ShortTermRootCredentialUsage.

Before sign-off

  1. Session revocation: timing, order, and when to remove it (response-actions.md rows 2–3 and lines 83–86, SKILL.md line 224, evals.json line 28)

    • The Undo column for DenyOlderSessions and AWSRevokeOlderSessions says to delete the policy, and gives no time. Deleting it early brings the stolen sessions back. Keep the user deny for at least 36 hours, because GetSessionToken and GetFederationToken credentials for an IAM user can last 129,600 seconds. Keep the role deny for at least the role's MaxSessionDuration, which can be up to 12 hours. This is the same reasoning as the 12-hour rule you already have for Identity Center.
    • Order: cut off the source credential first (deactivate the key, or DenyAll), then write the aws:TokenIssueTime deny. If the deny is written first, the key can still mint new sessions that have a later issue time.
    • The value of aws:TokenIssueTime: the console's Revoke active sessions sets it about 30 seconds in the future to cover policy propagation, and the CLI path should do the same. I would remove the "one minute after the attacker's last GetSessionToken/AssumeRole call" option. CloudTrail delivery is delayed, STS Regional-endpoint calls are logged in the Region that was called, and the key may still be active, so "the last call we can see" is not a safe cutoff. Update the eval assertion on line 28 to match.
  2. Revocation scope (SKILL.md line 226, response-actions.md line 86). "Every role the key assumed" misses chained roles (roles the role sessions assumed in turn) and roles in other accounts. Those must be revoked in the account that owns each role. Mode 1 already collects chaining, so the containment text should use it.

  3. Coverage checks that don't confirm the needed event type is recorded. This is the skill's core rule ("no record is not proof"), and two checks miss it:

    • Flow logs (evidence-queries.md line 58): read TrafficType. A flow log with TrafficType=REJECT never contains ACCEPT records, so "no accepted flows" would be reported as "exposure without observed use". Also check subnet-level flow logs. Today only the ENI and VPC are checked.
    • S3 data events (evidence-queries.md line 51): read ReadWriteType on basic selectors, and the readOnly and eventName field selectors on advanced ones. A trail that records only WriteOnly data events and delivers them to CloudWatch Logs gives 0 GetObject, and the skill would report that as a count.
  4. Root containment (response-actions.md line 95). The IAM docs say policies can't change the permissions of temporary credentials that root created with GetSessionToken or GetFederationToken. Those credentials stay valid until they expire (up to 1 hour). Say so. In a member account, an SCP that denies the root user is the only control that applies to them.

Strongly recommended

  1. Eradication gaps. The section 4 event list has these, but section 5 has no row for them:

    • AddUserToGroup → remove-user-from-group
    • AttachRolePolicy, PutRolePolicy, AttachGroupPolicy, PutGroupPolicy on existing roles and groups (today only roles the attacker created are covered)
    • CreateSAMLProvider (you have a row for OIDC but not for SAML)
    • MFA devices the attacker registered (CreateVirtualMFADevice, EnableMFADevice)
    • Snapshot and AMI sharing (ModifySnapshotAttribute, ModifyImageAttribute) has no unshare command. RDS snapshot sharing (ModifyDBSnapshotAttribute, ModifyDBClusterSnapshotAttribute) is missing from both the events and the actions.
  2. Defense-evasion events missing from mode 4 and section 4: GuardDuty CreateIPSet/UpdateIPSet (a trusted IP list suppresses findings) and CreateFilter/UpdateFilter (auto-archive), DisassociateFromAdministratorAccount; Security Hub DisableSecurityHub and BatchUpdateFindings; Organizations LeaveOrganization; PutBucketLifecycle and PutBucketPolicy on the trail's bucket; Config DeleteDeliveryChannel; and deletion of Route 53 Resolver query logging.

  3. Identity Center row

    • It says "merge into the inline policy" but never gives the write command. Add aws sso-admin put-inline-policy-to-permission-set --instance-arn <instance-arn> --permission-set-arn <permission-set-arn> --inline-policy file://merged.json. I checked the syntax.
    • SCPs don't apply to the management account. If the user holds permission sets there, the permission set inline policy is the only option.
    • With an external IdP or AD, the AWS procedure disables the user in the identity source before deleting the portal session (step 5 before step 6). Otherwise the user can start a new portal session. The current order is right only for the Identity Center directory.
  4. Versioning row (response-actions.md line 130). "Protect objects from further deletion" overstates what versioning does. A principal with s3:DeleteObjectVersion can still delete versions permanently, and a lifecycle rule can expire noncurrent versions. Something like "keeps earlier versions of overwritten or deleted objects" is accurate.

  5. Verify step (response-actions.md line 168). get-access-key-last-used only covers the long-term key. To confirm that session revocation worked, check CloudTrail for the ASIA… keys issued before the cutoff and confirm that they now get explicit-deny AccessDenied.

The SME checklist box is already ticked while my sign-off is pending. Please untick it until I approve.
Hi Yang,

Thanks for the review. I fixed all nine points in faa4a11. The latest changes are under skills/aws-security-finding-investigation/ in that commit.

  1. Session revocation. In response-actions.md, the Undo column now keeps the user deny for at least 36 hours and each role deny for at least the role's MaxSessionDuration. The instruction says to cut off the source credential first and then write the deny. I removed the "one minute after the last call" option and now use about 30 seconds after now. The eval assertion is updated to match.

  2. Revocation scope. Added in SKILL.md and response-actions.md, and they now instruct the agent to recommend revoking the chained roles and roles in other accounts. Each role is revoked in the account that owns it.

  3. Coverage checks. In evidence-queries.md, the flow log check now instructs agent to read TrafficType at the ENI, subnet and VPC level. The S3 check now reads ReadWriteType, readOnly and eventName. If a trail records only write events, the agent now reports who read the data as Unknown, not as zero reads, because the trail does not record reads at all.

  4. Root. response-actions.md now says that temporary credentials root created with GetSessionToken or GetFederationToken are not limited by IAM policy and will last up to 1 hour. If the account is a member of an AWS Organizations organization, the only way to limit them is an SCP that denies the root user, set from the management account.

  5. Eradication gaps. I fixed all the gaps in section 5. I added the instruction in response-actions.md to instruct the agent to recommend corresponding commands for removing what attacker added, including group membership, policies on existing roles and groups, SAML providers, MFA devices, and shared EBS snapshots, AMIs and RDS snapshots. The skill policies template‘s permission set remain the same, no extra actions were added.

  6. Defense evasion. The events you listed are now in mode 4 (Defense evasion and logging changes) and in section 4 of evidence-queries.md, so the agent also investigates them when it checks whether logging or threat detection was changed.

  7. Identity Center. Identity Center row of response-actions.md, I added in the missing put-inline-policy-to-permission-set command, and a note that an SCP does not apply to the management account, so there the permission set inline policy is the only option. I also fixed the order of the revocation steps for an external identity provider or Active Directory (disable the user first, then delete the portal session).

  8. Versioning. In the data containment table of response-actions.md, I changed "Protect objects from further deletion" to "Keep earlier versions of overwritten or deleted objects". The impact column now mentions that anyone with s3:DeleteObjectVersion, and any lifecycle rule that expires noncurrent versions, can still remove those versions.

  9. Verify step. I added an extra check in CloudTrail for revoked temporary credentials: find the ASIA access keys issued before the revocation time and confirm that their later calls fail with an explicit-deny AccessDenied. I also noted that get-access-key-last-used covers the long-term key only.

I also updated the AIDevOpsAgentAccessPolicy version in skill policies template to v15. I ran the new commands offline like you did, and none had a parameter error. I re-ran the key deactivation chat case 3 times and the key escalation investigation 3 times with this updated version (one of three was a re-run). All 3 chat answers keep the user deny for 36 hours and the role deny for the role's MaxSessionDuration, set the cutoff time about 30 seconds after now, and cover chained roles. The 3 successful investigations scored 9/10, 9/10 and 10/10 on the key-escalation-attempt case in evals/evals.json.

I unticked the SME box. Could you take another look when you have time? Thank you!

Best,
Shaoyi

@liyangln-AWS liyangln-AWS left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SME sign-off for faa4a11. I re-checked the nine items from my earlier review against this commit, and all of them are addressed:

  • Session revocation: the source credential is cut off first, aws:TokenIssueTime is set about 30 seconds after now, the deny is kept at least 36 hours for a user and at least MaxSessionDuration for a role, chained and cross-account roles are included, root temporary credentials are called out, and the verify step checks the ASIA… keys in CloudTrail.
  • Coverage checks: flow logs now read TrafficType at the ENI, subnet and VPC; data events now read ReadWriteType, readOnly and eventName, and a write-only trail is reported as Unknown.
  • The new eradication rows, the defense-evasion events, the Identity Center write command, the management-account note and identity-source order, and the versioning row are correct.

As before, I parsed the new commands and their undo forms (34 invocations) offline with AWS CLI 2.36.40, and none had a parameter error. AIDevOpsAgentAccessPolicy v15 grants every new read-only check, so the template still needs only the 13 added actions.

Two optional follow-ups, not blocking: list-virtual-mfa-devices doesn't show FIDO or hardware MFA devices (the EnableMFADevice events still do), and the shared-snapshot row could add aws rds describe-db-cluster-snapshot-attributes as the read check for cluster snapshots.

You can tick the SME box now.

@szl0144

szl0144 commented Nov 14, 2026

Copy link
Copy Markdown
Author

Thanks for the review and the sign-off! I will leave the two optional follow-ups for a later change, so the commit you approved stays the PR head.

@szl0144
szl0144 marked this pull request as ready for review November 14, 2026 04:35
@szl0144

szl0144 commented Nov 14, 2026

Copy link
Copy Markdown
Author

Hi @yeremyjt and @ams-thakkar, this PR is ready for review, and Yang Li's SME sign-off is in the review above. I could not run the skill evaluation tool, so validate-skill-evals will not pass yet. When you have time, could you either run the tool on this skill, or let me know whether an evals/exemptions.json with my own test results would be acceptable? I would also appreciate any review of the skill itself. Thank you for your time.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants