Skip to content

[Security] iam-access-navigator: IAM risk review, access-key hygiene and drift detection (skill + custom agent + MCP) - #200

Open
jgq-amazon wants to merge 5 commits into
aws:mainfrom
jgq-amazon:iam-access-navigator
Open

jgq-amazon wants to merge 5 commits into
aws:mainfrom
jgq-amazon:iam-access-navigator

Conversation

@jgq-amazon

Copy link
Copy Markdown

Description

Read-only IAM review for AWS DevOps Agent. The skill triages IAM risk, rates the blast radius of a change, generates least-privilege policies from observed CloudTrail activity, reviews access-key hygiene with an identity-aware migration path, validates policies, and plans a safe rollback, teaching as it works. The custom agent packages the skill for interactive use and runs two scheduled jobs: an access-key hygiene sweep and an IAM drift check. The iam-drift-diff MCP server gives the drift job something the agent cannot do reliably on its own: hold a prior IAM state and compute an exact, complete set-difference across the whole account. In prompt-only testing the agent missed a real policy detach twice and once reported "no drift" with a false coverage claim; the server does the comparison as deterministic set math and returns machine-readable completeness flags.

The MCP reads control-plane data that use_aws can also reach. The gap it fills is computation and persistence, not data access; see "Why not just call the AWS APIs?" in mcp/iam-drift-diff/README.md. If reviewers rule that out, the skill and agent stand on their own and the drift job can be dropped.

Related work. DevOps Agent's built-in proactive prevention produces cross-service operational recommendations; the scheduled jobs here build on that recommendations surface rather than replace it, are IAM-specific, and file a recommendation only on a risk-increasing change. The drift-detection custom agent proposed in #26 is a different kind of drift: it compares resource configuration (encryption, tags, lifecycle) against a fixed baseline and does not examine IAM principals, policies or trust, while iam-drift-diff diffs IAM state against its own previous snapshot. Neither reuses the other's checks.

Type of change

  • New skill
  • New custom agent
  • New MCP server
  • Update to an existing skill, agent, or MCP server
  • Documentation or infrastructure change

Required before review

  • Repository layout is correct — skills in /skills, MCP servers in /mcp, custom agents in /custom-agents; skills and MCP servers are never mixed in one directory.
  • A domain SME has reviewed and signed off.
  • Evaluation evidence is included (see Testing).
  • No internal-only or proprietary content — no internal hostnames, account IDs, customer names, ticket numbers, internal URLs, or secrets in the tool, its examples, or its tests.
  • README is complete — prerequisites, setup/deployment, and at least one sample prompt or usage example.
  • CHANGELOG updated and the version bumped.
  • Named owner for ongoing maintenance. Owner: @jgq-amazon.

Testing

  • structure checks pass. 12/12 (skills/iam-access-navigator/evals/structure/structure-tests-results-v8.json).
  • best-practices run across multiple iterations, with consistent results. v8: 100% on all 3 iterations (14 passed, 3 skipped, 0 failed). An earlier run on a previous revision (v4) failed BP-03 once on 1 of 3 iterations at medium confidence; v8 is the committed result.
  • functional run with and without the change, so the difference in the agent's behavior is visible. v3, 10 evals x 3 iterations, 54 runs, 0 failed. Assertions: with skill 55/72 (76%), without 43/75 (57%); one with-skill run produced no final response, so 3 of its assertions were not scored. Known weak spots: the "make an IAM change" request does not load the skill (0/3; the base agent declines safely anyway); the forensics eval loads it 2 times in 3; and on the session-name eval the with-skill expected output passed 1/3 vs 3/3 without (one run had no final response, one investigated the account instead of explaining the trap).
  • evals/evals.json includes positive cases (should trigger) and at least one negative case (should not trigger on unrelated input). 8 positive, 2 negative (RDS Performance Insights, EKS node groups).

Also validated end to end in a test DevOps Agent space: interactive sessions, both schedule triggers, recommendations and the "IAM Drift Report" artifact, and the MCP over SigV4. Drift tests caught policy attach/detach, inline add/change/remove with exact grants, trust changes with high_risk, and new principals; stayed quiet on cosmetic reordering and AWS-managed churn; an ad-hoc diff between scheduled runs did not hide a change from the next report; and an instruction planted in a policy condition value was quoted as evidence and flagged, not followed. Least-privilege was also live-validated on one clean invocation per case: a role granted only S3 object actions kept them and flagged them as unverified (S3 object calls are data events, not visible to LookupEvents) instead of calling the role unused, and a zero-activity role without S3 grants was still reported as unused (a one-hour-old fixture). MCP unit tests: 54 passed. sam validate --lint: valid.

Skills — complete if this PR adds or changes a skill

  • SKILL.md frontmatter is valid: name matches the directory, description present and within length limits, body at most 500 lines.
  • The description is written from the agent's point of view — scenarios, services, error types, symptoms.
  • Scoped to a specific agent type in metadata.aws-devops-agent-skills.agent-types (INCIDENT_RCA, INCIDENT_TRIAGE, INCIDENT_MITIGATION, …), not left GENERIC without reason. CHAT, PREVENTION.
  • Procedures are structured as decision trees on real conditions, not flat prose.
  • Reference material lives in references/ or assets/ and loads on demand.
  • Non-executable content only — no scripts or code in the skill directory. Deterministic compute belongs in an MCP server.

MCP servers — complete if this PR adds or changes an MCP server

  • Transport is Streamable HTTP; auth is documented (SigV4 / OAuth / API key). SigV4 via a Lambda Function URL with AuthType: AWS_IAM.
  • Every tool is classified read-only vs mutative in the README, with mutative tools called out explicitly. All four are read-only against IAM; the only writes are the server's own snapshot objects in its own bucket.
  • Tools are fixed and named — the server does not execute arbitrary caller-supplied commands, SQL, or queries. Any query surface is allowlisted and bounded (time range, result size, timeout). Inputs are snapshot ids (validated), one ARN, and a list limit clamped to 1..50.
  • Least-privilege IAM — the execution role grants only the actions the tools need; the scoping is stated in the README and the policy/template is included. iam:GetAccountAuthorizationDetails only, object actions on its own bucket prefix, and its own KMS key.
  • No standing secrets in code or config — credentials come from Secrets Manager or equivalent. The server uses no credentials beyond its execution role.
  • Target resources are validated against an allowlist before any call. ALLOWED_ACCOUNTS (explicit, no wildcard) is checked at startup and on every call.
  • Tools return bounded summaries, not unbounded raw data. Capture returns coverage only; diff returns changed principals only; get_entity returns one principal.
  • Application security review is complete, in progress, or linked, with a threat model. An MCP server that can act in a customer account does not merge without it. In progress: the application security review is open, with the design doc and a STRIDE threat model submitted; the review is pending.
  • template.yaml (SAM) or cdk.* (CDK), src/, and tests/ are present and tests pass.

Custom agents — complete if this PR adds or changes a custom agent

  • SYSTEM_PROMPT.md is present and is what gets pasted into the agent configuration.
  • The composed tools and skills are listed in the README, and each is already in the catalog or included in this PR. The skill and MCP server are both in this PR.
  • Any mutative tool the agent can call is identified, with its approval behavior described. None; the agent is read-only.
  • Run mode (on-demand vs scheduled) and the artifact it produces are documented.

Security and safety

  • Read-only by default; any action that changes customer state is explicit, documented, and approval-gated. Nothing in this PR changes IAM.
  • Customer data stays in the customer's account.
  • No destructive operations without an explicit, documented confirmation step.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

Add a read-only IAM review bundle for AWS DevOps Agent:
- skills/iam-access-navigator: IAM risk triage, blast radius,
  least-privilege synthesis, access-key hygiene, policy validation,
  rollback planning and change forensics, with skill-eval results.
- custom-agents/iam-access-navigator: interactive agent plus scheduled
  access-key hygiene and IAM drift jobs.
- mcp/iam-drift-diff: read-only MCP server that captures IAM snapshots
  and computes an exact, deterministic diff between them (SigV4 Lambda
  Function URL, private KMS-encrypted snapshot bucket, access logging).

No component writes to IAM.
@github-actions github-actions Bot added the needs-checklist PR checklist incomplete — not reviewed until resolved label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

This PR is on hold — checklist incomplete

Required items in the PR description are unchecked. A maintainer will not review this PR until each one is checked and actually done.

The check's summary lists them. Complete the work, tick the boxes, and this clears automatically. CONTRIBUTING.md explains each item.

@szl0144 szl0144 mentioned this pull request Nov 11, 2026
20 of 26 tasks
…ty review

- Roles under /service-role/ are fully fingerprinted and diffed; only
  /aws-service-role/ (service-linked) roles are presence-only. Snapshot
  schema 2; the first diff after upgrading reports each affected role once
  as fingerprint_scope_changed (direction info).
- Baseline pointer reads fail closed: only "not found" means no baseline;
  other errors return a retryable error with nothing written, pruned or
  compared. An invalid pointer, or one naming deleted snapshots, diffs from
  the oldest retained snapshot.
- Trust high_risk tracks the condition keys guarding each principal, flags
  new unconditioned Federated principals and removed trust Deny statements.
- Policy-body diffs render NotAction/NotResource and report denies_added /
  denies_removed.
- Every S3 call passes ExpectedBucketOwner; the execution role's S3
  statement requires aws:ResourceAccount.
- Callers get AWS error codes only; refused and failed calls write an audit
  line; a malformed request-context header cannot fail a tool.
- Dedicated log group with LogRetentionDays (default 365).
- Drift prompt updated for the new fields; README and CHANGELOG updated.
- 87 unit tests (was 54), including execution-role scope and account-gate
  checks.
The invoke role has a fixed RoleName, so CloudFormation requires
CAPABILITY_NAMED_IAM. README deploy command and samconfig.toml updated.
… rule

From a live drift round on 1.0.1:
- A grant that gains a condition is reported in grants_conditioned
  (direction down) and one that loses a condition in
  grants_unconditioned (up), instead of as a removed plus an added
  grant (mixed). The 1.0.1 execution-role update itself was reported
  as mixed and drew a recommendation for a tightening.
- Drift prompt: a single authoritative recommendation rule, and the
  run must end with the full report even after updating an artifact.
- 89 unit tests.
…ey match a change

From drift round 2: one run mentioned an unrelated open recommendation and
the other did not. The prompt now says to mention an existing open
recommendation only when it matches a change in the current diff.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-checklist PR checklist incomplete — not reviewed until resolved

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant