Skip to content

chore: report security-audit skill review findings upstream #231

Open

Description

@DevSecNinja

Summary

Review of #226 found issues in the vendored security-audit skill (cloudflare/security-audit-skill, pinned at c1c8a8c1471069fb0e188eeaff69b8e8db6564a8). They can't be fixed in ai-toolkit: .agents/skills/security-audit/ is materialized by apm install, and apm-overrides.json can only replace single-line frontmatter. They should be reported upstream, and a fixed commit will then arrive as a gated Renovate bump.

Validator bugs

  • validate-coverage-ledger.cjs ~L616–624: each local artifact is added to priorArtifacts inside the per-check loop, so a second check in the same archived attempt that cites the same promoted file fails as "artifact from an earlier attempt". This fails closed. Fix: check the whole attempt, then add its artifacts. (Copilot and CodeRabbit on feat: add cloudflare security-audit skill #226.)
  • validate-coverage-ledger.cjs ~L618/627/636: priorOwners only tracks assignment owners, so a source-only check by an archived verifier can be copied into a later attempt and still pass the ownership and artifact-reuse guards. (CodeRabbit.)
  • validate-findings.test.cjs ~L166: the CLI success test lacks the skip: !HAS_SAFE_INPUT_OPEN guard its sibling ledger suite uses, so it fails on platforms without O_NOFOLLOW/O_NONBLOCK. (CodeRabbit.)

Guidance refinements

  • AI-AND-LLM.md ~L12: the reachability exclusion drops injected actions that misuse the requester's own authority, which lines 16 and 43 treat as action-binding failures.
  • CLIENT-SIDE.md ~L81: a proven lower-trust same-origin reader of another app's localStorage should count as disclosure without a lifecycle transition.
  • HUNTING.md ~L240: the critic response returns only IDs, but the parent must archive a source-backed reassignment_reason (and the same applies to resolved_prior_leads).
  • RECONNAISSANCE.md ~L156: the 5 MiB and 10,000-unit ledger limits make a single-ledger full audit of a very large target impossible to validate; consider partitioned ledgers.
  • SKILL.md ~L123: a budget-7 standard run can spend everything on reconnaissance, assign no hunter, and still not set run_status: "incomplete".

Notes

ai-toolkit sessions don't write to other repositories, so the owner needs to file these upstream (or approve doing so).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions