Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,9 @@ helper, run `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p
3. **`description` and `category` are required** in every primitive's frontmatter.
4. **Bash scripts are executable** — run as `bash script-name.sh`.
5. **No background processes** — all commands complete immediately.
6. **Never hand-edit vendored agents** in `.github/agents/` that come from
`apm.yml` dependencies; `apm install` reverts them. Put local pins in
`apm-overrides.json` (see the review gate section of `docs/apm.md`).

## Coding Style Preferences

Expand Down
63 changes: 63 additions & 0 deletions .github/labels-base.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
# Base labels shared across all DevSecNinja repositories.
# Repo-specific labels (e.g. area/* labels) should be defined in each repo's
# .github/labels.yaml file.

# Renovate Types
- name: renovate/container
color: "027fa0"
- name: renovate/github-action
color: "027fa0"
- name: renovate/github-release
color: "027fa0"
- name: renovate/github-tag
color: "027fa0"
- name: renovate/gomod
color: "027fa0"
- name: renovate/mise
color: "027fa0"
- name: renovate/ansible
color: "027fa0"

# Semantic Types
- name: type/digest
color: "ffeC19"
- name: type/patch
color: "ffeC19"
- name: type/minor
color: "ff9800"
- name: type/major
color: "f6412d"

# Automation
- name: "merge: auto"
color: "0075ca"
description: PR is eligible for automerge
aliases:
- automerge
- name: "merge: manual"
color: "5319e7"
description: PR requires manual review and merge
- name: "autorelease: pending"
color: "ededed"
description: Release Please — release pending
- name: "autorelease: tagged"
color: "0e8a16"
description: Release Please — release tagged
- name: config-sync
color: "0075ca"
- name: apm-sync
color: "0075ca"
description: Automated APM materialize PR
- name: todo-to-issue
color: "0075ca"

# Uncategorized
- name: bug
color: "d73a4a"
- name: enhancement
color: "a2eeef"
- name: hold
color: "ee0701"
- name: stale-dependency
color: "e4e669"
41 changes: 41 additions & 0 deletions .github/labels.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
# Labels specific to ai-toolkit. Label Sync merges this file with the org-wide
# .github/labels-base.yaml (synced from DevSecNinja/.github by config-sync) and
# deletes any label in neither file.

# Review gate for third-party agent text (see docs/apm.md). Set on Renovate's
# github/awesome-copilot bumps and on the APM Materialize PR; neither may
# auto-merge.
- name: external-primitives
color: "b60205"
description: Changes third-party AI primitives; owner review required

# Renovate labels vulnerability-fix PRs `security` (shared base.json5).
- name: security
color: "ee0701"
description: Security fix

# GitHub's default labels, kept so the sync doesn't delete them.
- name: dependencies
color: "ededed"
- name: documentation
color: "0075ca"
description: Improvements or additions to documentation
- name: duplicate
color: "cfd3d7"
description: This issue or pull request already exists
- name: good first issue
color: "7057ff"
description: Good for newcomers
- name: help wanted
color: "008672"
description: Extra attention is needed
- name: invalid
color: "e4e669"
description: This doesn't seem right
- name: question
color: "d876e3"
description: Further information is requested
- name: wontfix
color: "ffffff"
description: This will not be worked on
32 changes: 30 additions & 2 deletions .github/workflows/apm-materialize.yml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,12 @@ jobs:
# are not needed to materialize the agents here).
apm install --update --only apm --verbose

- name: Re-apply local overrides
# `apm install` rewrites the vendored agents, which drops local pins
# such as the security analyzer's model. apm-overrides.json keeps them.
if: steps.precheck.outputs.skip != 'true'
run: python3 scripts/apply-apm-overrides.py

- name: Fetch materialize branch
# Populate refs/remotes/origin/chore/apm-materialize so that the
# --force-with-lease push in open-pr has a valid lease to compare
Expand All @@ -117,6 +123,7 @@ jobs:
run: git fetch origin chore/apm-materialize || true

- name: Open materialize PR
id: open-pr
if: steps.precheck.outputs.skip != 'true'
# renovate: datasource=github-tags depName=DevSecNinja/.github
uses: DevSecNinja/.github/actions/open-pr@27fd66740c629ea38380dcd3bf93e7bc74c54247 # v1.9.0
Expand All @@ -125,7 +132,13 @@ jobs:
base: main
title: "chore: materialize APM primitives"
commit-message: "chore: materialize APM primitives from apm.yml"
labels: apm-sync
# Label Sync creates these from .github/labels*.yaml; `gh pr create`
# fails when one is missing. `external-primitives` marks the review
# gate and `merge: manual` that the owner merges this PR by hand.
labels: |
apm-sync
external-primitives
merge: manual
Comment thread
DevSecNinja marked this conversation as resolved.
Resolved
github-token: ${{ steps.app-token.outputs.token }}
# Stage the lockfile and deployed harness files (apm.yml itself is
# already on main; apm_modules/ and the non-committed harness output
Expand All @@ -138,6 +151,21 @@ jobs:
`main` (e.g. a Renovate SHA/tag bump).

**Review the `.github/agents/` diff** — this is the prompt-hijacking
gate: confirm the upstream agent text changed as expected.
gate: confirm the upstream agent text changed as expected. Local
pins from `apm-overrides.json` are re-applied on top.

Never auto-merge this PR: the owner reviews and merges it. See
`docs/apm.md`.

Triggered by `${{ github.workflow }}` at commit `${{ github.sha }}`.

- name: Label reused materialize PR
# The pinned open-pr only labels a PR it creates, so a PR left open by
# an earlier run would miss the review labels (DevSecNinja/.github#484
# fixes this in the action).
if: steps.open-pr.outputs.created == 'false' && steps.open-pr.outputs.pr-number != ''
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
PR_NUMBER: ${{ steps.open-pr.outputs.pr-number }}
run: |-
gh pr edit "${PR_NUMBER}" --add-label apm-sync --add-label external-primitives --add-label "merge: manual"
31 changes: 31 additions & 0 deletions .github/workflows/label-sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
name: Label Sync

# Creates and updates this repo's labels from the org-wide base set
# (.github/labels-base.yaml, delivered by config-sync from DevSecNinja/.github)
# plus the repo-specific .github/labels.yaml. Automation depends on these
# labels: `gh pr create --label` fails when a label is missing, which is what
# kept APM Materialize from opening its review PR (#210).

on:
workflow_dispatch:
push:
branches:
- main
paths:
- .github/labels.yaml
- .github/labels-base.yaml
# Weekly drift correction for labels edited in the UI; changes to the label
# files themselves are applied by the push trigger above.
schedule:
- cron: "0 0 * * 1"

permissions:
contents: read

jobs:
label-sync:
uses: DevSecNinja/.github/.github/workflows/label-sync.yml@8ef60d65001c56a496d12f6ae0a7091c05db2e06 # v3.5.0
permissions:
contents: read
issues: write
9 changes: 9 additions & 0 deletions .github/workflows/markdown-validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ on:
- '.github/prompts/**/*.md'
- 'tests/**'
- 'scripts/generate-index.sh'
- 'scripts/apply-apm-overrides.py'
- 'apm.yml'
- 'apm-overrides.json'
- '.github/agents/**'
- 'apm.lock.yaml'
- 'release-please-config.json'
- '.release-please-manifest.json'
Expand All @@ -23,7 +26,10 @@ on:
- '.github/prompts/**/*.md'
- 'tests/**'
- 'scripts/generate-index.sh'
- 'scripts/apply-apm-overrides.py'
- 'apm.yml'
- 'apm-overrides.json'
- '.github/agents/**'
- 'apm.lock.yaml'
- 'release-please-config.json'
- '.release-please-manifest.json'
Expand Down Expand Up @@ -70,6 +76,9 @@ jobs:
- name: Check apm.yml does not depend on itself
run: PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_manifest.py' -v

- name: Check vendored primitives keep their local overrides
run: PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_overrides.py' -v

consumer-install:
name: Check Consumer Install
runs-on: ubuntu-latest
Expand Down
4 changes: 4 additions & 0 deletions .lefthook.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ run = "PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_
[pre-commit.commands.apm-manifest]
run = "PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_manifest.py'"

# Vendored agents must keep their apm-overrides.json pins (see #210).
[pre-commit.commands.apm-overrides]
run = "PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_overrides.py'"

# Regenerate INDEX.md / README index from .apm/ and re-stage the result, so the
# generated index never drifts from the primitives (the common contributor miss).
[pre-commit.commands.generate-index]
Expand Down
12 changes: 12 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ What runs on **pre-commit**:
— checks release-please can still find and bump the version in `apm.yml`.
- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_manifest.py'`
— checks `apm.yml` doesn't list `ai-toolkit` as its own dependency.
- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_apm_overrides.py'`
— checks vendored agents keep their `apm-overrides.json` pins.
- `bash scripts/generate-index.sh` — regenerates `INDEX.md` / `README.md` from
`.apm/` and re-stages them, so the index never drifts from the primitives.

Expand All @@ -126,6 +128,16 @@ signing or tool denials. If hooks or filters would execute unauthorized code,
stop before committing and request scoped approval. All commits must use
1Password signing; halt if it is unavailable.

## 🛡️ Changes to external agents

The `github/awesome-copilot` agents in `apm.yml` are reviewed by the owner and
never auto-merge. A Renovate SHA bump and the follow-up
`chore: materialize APM primitives` PR both carry the `external-primitives`
label; review the `.github/agents/` diff in the materialize PR before merging.
Don't hand-edit a vendored agent: put local changes such as a model pin in
`apm-overrides.json`. See "External primitives: review gate" in
[the APM guide](docs/apm.md).

## 🚀 Submission Process

1. **Fork** this repository
Expand Down
12 changes: 12 additions & 0 deletions apm-overrides.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"$comment": "Local changes to vendored APM primitives. `apm install` rewrites files it deploys, so APM Materialize re-applies these afterwards with scripts/apply-apm-overrides.py. See docs/apm.md.",
"overrides": [
{
"file": ".github/agents/sast-sca-security-analyzer.agent.md",
"reason": "Pinned to Sonnet 5 (#135).",
"frontmatter": {
"model": "Claude Sonnet 5"
}
}
]
}
53 changes: 53 additions & 0 deletions docs/apm.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,59 @@
All commits require 1Password signing; if signing or a required tool is
unavailable or denied, halt rather than bypassing it.

## 🛡️ External primitives: review gate

The agents from `github/awesome-copilot` in `apm.yml` are third-party prompt
text, so every change to them needs the owner's approval. Neither PR in the
flow below may auto-merge.

1. **Renovate bump.** Renovate's custom manager (see
[`renovate.json5`](../renovate.json5)) bumps the pinned SHA. A package rule
re-enables these digest updates (off org-wide) in their own PR rather than
the weekly batch, turns off the shared auto-merge preset and adds the
`external-primitives` and `merge: manual` labels. The diff only shows a
hash. Renovate can't remove the preset's `merge: auto` label, but
`automerge: false` takes precedence.
2. **Materialize PR.** Merging the bump triggers
[APM Materialize](../.github/workflows/apm-materialize.yml). It runs
`apm install`, re-applies the local overrides and opens
`chore: materialize APM primitives` from `chore/apm-materialize`, labelled
`apm-sync`, `external-primitives` and `merge: manual`. **Review the
`.github/agents/` diff**: this is where upstream agent text changes become
visible. Merge it yourself once you're satisfied; close it and revert the bump
if not.

The labels come from [`.github/labels-base.yaml`](../.github/labels-base.yaml)
(org-wide, synced from `DevSecNinja/.github` by Config Sync) and
[`.github/labels.yaml`](../.github/labels.yaml), applied by
[Label Sync](../.github/workflows/label-sync.yml). `gh pr create` fails on a
missing label, so a label must exist before a workflow uses it.

### Local overrides of vendored agents

`apm install` rewrites every file it deploys, so never hand-edit
`.github/agents/` for a dependency: the next materialize run reverts it (APM
has no override setting). Add the change to
[`apm-overrides.json`](../apm-overrides.json) instead:

```json
{
"file": ".github/agents/sast-sca-security-analyzer.agent.md",
"reason": "Pinned to Sonnet 5 (#135).",
"frontmatter": { "model": "Claude Sonnet 5" }
}
```

Each entry sets top-level frontmatter keys to string values. Only a
single-line value is replaced; if upstream switches a key to a multi-line
value, the script fails and the override needs a manual update.
[`scripts/apply-apm-overrides.py`](../scripts/apply-apm-overrides.py) applies
them, and APM Materialize runs it after `apm install`. Run it yourself after a
local `apm install`. It fails when a file is missing, so drop stale entries
when an agent is removed. `tests/test_apm_overrides.py` fails if a committed
agent doesn't match its override. `apm audit` reports overridden files as
drift from the lockfile, which is expected.

## 🏷️ Versioning

Versioning is driven by [release-please](https://github.com/googleapis/release-please)
Expand Down Expand Up @@ -209,7 +262,7 @@
> line-folding change), which is fine because `apm.yml` is tool-formatted and
> excluded from yamlfmt/yamllint. `tests/test_release_please_config.py` fails if
> release-please can no longer find the version or it drifts from the manifest.

Check failure on line 265 in docs/apm.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

MD028/no-blanks-blockquote Blank line inside blockquote
> **Never add `DevSecNinja/ai-toolkit` to its own `apm.yml`** (for example by
> running the consumer `apm install DevSecNinja/ai-toolkit#vX.Y.Z` here). A
> consumer pinning a newer tag would then also need the older ref of the same
Expand Down
21 changes: 21 additions & 0 deletions renovate.json5
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,25 @@
currentValueTemplate: "main",
},
],
packageRules: [
{
// The shared autoMerge preset merges digest bumps unattended, but this
// bump PR only shows a SHA. The agent text it pulls in surfaces in the
// follow-up APM Materialize PR, so the owner reviews and merges both.
// Renovate adds labels from every matching rule and can't remove one,
// so the preset's `merge: auto` label stays; `automerge: false` wins.
// The shared presets also disable digest updates and batch non-major
// updates into one weekly PR: re-enable them here and keep this bump in
// its own PR. git-refs has no release timestamps, so the 14-day soak
// can't apply; without timestamp-optional the update stays pending.
description: "Never auto-merge github/awesome-copilot bumps: third-party agent text needs owner review",
matchDepNames: ["github/awesome-copilot"],
enabled: true,
groupName: null,
minimumReleaseAgeBehaviour: "timestamp-optional",
automerge: false,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
commitMessageSuffix: "",
addLabels: ["external-primitives", "merge: manual"],
},
],
}
Loading
Loading