Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,20 +92,22 @@ The `metadata` block with `author` and `version` fields is required, and `summar

`.github/workflows/validate-skills.yml` runs `.github/scripts/validate_skills.py` on every pull request, and `.github/workflows/validate-skills-main.yml` runs it again after each push to `main`. `.github/workflows/build-docs.yml` runs `mkdocs build --strict` on pull requests and on `main`. The rules and their reasons are in the "Skill Publishing Rules" section of [CONTRIBUTING.md](../CONTRIBUTING.md).

- The rules live in `.github/scripts/skill_checks.py` as pure functions over an in-memory skill folder (`SkillTree` of `Entry(path, mode, data)`). `validate_skills.py` reads Git objects, computes the merge base and reports. The only rules it holds look beyond one skill folder (a file or submodule directly under `skills/`, symbolic links and `.gitattributes` lines outside it, and a deprecated skill still listed in a `connectors/*/connector.yaml`); `tests/test_validate_skills.py` covers them, since a conformance case is one skill folder. Keep every per-skill rule in `skill_checks.py`, so the conformance cases and unit tests exercise exactly what CI runs.
- The rules live in `.github/scripts/skill_checks.py` as pure functions over an in-memory skill folder (`SkillTree` of `Entry(path, mode, data)`). `validate_skills.py` reads Git objects, computes the merge base and reports. The only rules it holds look beyond one skill folder (a file or submodule directly under `skills/`, symbolic links and `.gitattributes` lines outside it, a deprecated skill still listed in a `connectors/*/connector.yaml`, and the YAML of the files under `connectors/`) or need the network (`author_login_findings`, the GitHub lookup of new `metadata.author` names); `tests/test_validate_skills.py` covers them, since a conformance case is one skill folder and can't reach GitHub. Keep every per-skill rule in `skill_checks.py`, so the conformance cases and unit tests exercise exactly what CI runs. The connector version rules live the same way in `.github/scripts/connector_checks.py`, pure functions over a connector folder in the same `SkillTree`, with their cases in `skill-rules/connector-conformance-cases.json`, which the script also runs before checking anything, and unit tests in `tests/test_connector_checks.py`.
- Every skill is checked on every PR, because skills on `main` are published automatically. Errors from any skill fail; warnings print only for skills the PR touches.
- Skills are read from Git objects (`git ls-tree -r -z` plus `git cat-file --batch`), never the working tree, so modes are exact and a local run checks only committed changes.
- Rule data lives in `.github/scripts/skill-rules/` and the check fails closed (exit 2) when a file there is malformed: `agent-types.json` (the values `aws-devops-agent-skills.agent-types` may take; an unknown one is an error), `vocabulary.json` (the `aws-services` and `technical-domains` values; unknown values only warn), `published-files.json` (root names left out of the published set: `evals/`, `.skilleval.yaml` and `CHANGELOG.md`; `README.md` and `images/` are published), `internal-names.json` (hosts and terms that warn on added lines) and `conformance-cases.json`.
- Rule data lives in `.github/scripts/skill-rules/` and the check fails closed (exit 2) when a file there is malformed: `agent-types.json` (the values `aws-devops-agent-skills.agent-types` may take; an unknown one is an error), `vocabulary.json` (the `aws-services` and `technical-domains` values; an unknown value is an error, with the approved spelling when `aliases` has one, a placeholder such as `All`, `None` or `*` is an error in any case, and the file can't list one), `published-files.json` (root names left out of the published set: `evals/`, `.skilleval.yaml` and `CHANGELOG.md`; `README.md` and `images/` are published; read from the merge base when there is one, like the extension allowlist, so a change to it takes effect only once it has merged), `internal-names.json` (hosts and terms that warn on added lines) and `conformance-cases.json`.
- The extension allowlist is not duplicated: it is parsed from the `!*.<ext>` lines of `skills/.gitignore`, read from the merge base, so a new extension takes effect only once its `.gitignore` change has merged.
- Like the repository's other checks, a PR runs its own copy of the script, rules and tests, so it could edit them to pass. Code-owner review of `/.github/`, `/tests/` and `/.gitattributes` (`.github/CODEOWNERS`) is the control; don't describe any in-script measure as stopping a determined PR. The post-merge run on `main` catches pull requests that break the rules together, not a pull request that edits the check.
- Limits: at most 100 published files, each at most 1 MiB, a zip of at most 4,128,768 bytes (4 MiB less a 64 KiB margin), and frontmatter at most 32 KiB. The check warns past 90% of the file count, file size and zip limits. `validate_skills.repository_results` also rejects symbolic links anywhere in the repository and root `.gitattributes` lines that change archived bytes under `skills/`, because skills are published from an archive of the repository, which must hold exactly what Git holds.
- `check_added_content` warns about internal host or tool names only on lines a PR adds, like `scan-aws-identifiers`, so names already on `main` don't warn on every PR.
- `--check-github-logins`, which the PR workflow passes with the job's token, looks up on GitHub each `metadata.author` name a PR adds: `validate_skills.author_login_findings` with the resolver in `.github/scripts/github_logins.py`. The rules are in the `metadata.author` row of CONTRIBUTING.md; `tests/test_github_logins.py` and `tests/test_validate_skills.py` answer for GitHub, so no test reaches it.
- With `--base-ref`, `check_history` compares each skill with its merge-base copy: no removed folders, no reuse of a path deleted in `main`'s first-parent history (`git log --first-parent --no-renames --diff-filter=D`, skipped with a warning on a shallow clone), a higher version whenever `content_hash` of the published files changes, one step at a time (next patch, minor or major), and `1.0.0` for a new skill. No folder is ever deleted, not even in an emergency: deprecate it instead. The hash function is pinned by the golden fixture under `content_hash` in `conformance-cases.json`, so any other implementation can prove it computes the same hash.
- With or without a base, a deprecated skill can't stay in any `connectors/*/connector.yaml` `skills:` list (`connector_skill_references`, which composes the YAML rather than constructing it). A `connector.yaml` that can't be read, isn't a mapping, has a `skills:` value that isn't a list of skill names, or writes `skills:` more than once gets a warning; the names it does list are still checked.
- With or without a base, a deprecated skill can't stay in any `connectors/*/connector.yaml` `skills:` list (`connector_skill_references`). Every `connectors/*/connector.yaml` and `connectors/releases.yaml` is read with `skill_checks.read_yaml`, the reader the frontmatter uses, so a file that isn't UTF-8 or breaks a YAML rule is an error; the cases are in `skill-rules/yaml-conformance-cases.json`, which `tests/test_skill_checks.py` runs. A `connector.yaml` that has a `skills:` value that isn't a list of skill names gets a warning; the names it does list are still checked. One that isn't a mapping is an error from the connector version rules, since it has no version.
- `connector_checks.check_history` compares each `connectors/<name>/` folder (not `connectors/schema/`) with its merge-base copy, as the catalog publisher compares a connector with its published version, and reads `connector.yaml` as the publisher does (`skill_checks.plain_scalar_value`); without a base only `check_connector` runs. The rules are under "Connector versions" in CONTRIBUTING.md.
- Secrets in test fixtures are built at runtime (`"AKIA" + ...`) or stored as base64 in the conformance cases, so no file in the repository contains a literal key or invisible character. Internal names in tests are made up (`corp.example`, `Rivendell`), never real hosts or tools.
- `conformance-cases.json` holds language-neutral cases for these rules. Another tool that validates or publishes these skills should give the same `expect` result on the cases for the rules it implements; some rules here are stricter than any other tool's. The script runs every case before checking any skill and exits 2 on a mismatch. A rule change needs a matching case change in the same PR.
- Every rule applies to every skill; nothing is exempted. A rule that existing skills don't meet yet starts as a warning and becomes an error after a cleanup PR brings every skill into line: a missing `metadata.summary`, and dimension values outside the vocabulary. A missing `metadata.aws-devops-agent-skills.agent-types` stays a warning.
- Agent types come from `metadata.aws-devops-agent-skills.agent-types` (comma-separated values from `agent-types.json`, not from `vocabulary.json` like the other two dimensions, and an unknown value is an error rather than a warning). `GENERIC` loads the skill for every agent, so combining it with another value is an error. Without the key a skill installs as `GENERIC`, which warns.
- Every rule applies to every skill; nothing is exempted. A rule that existing skills don't meet yet starts as a warning and becomes an error after a cleanup PR brings every skill into line, as dimension values outside the vocabulary did; a missing `metadata.summary` is still a warning until its cleanup. A missing `metadata.aws-devops-agent-skills.agent-types` stays a warning.
- Agent types come from `metadata.aws-devops-agent-skills.agent-types` (comma-separated values from `agent-types.json`, not from `vocabulary.json` like the other two dimensions; an unknown value is an error in all three). `GENERIC` loads the skill for every agent, so combining it with another value is an error. Without the key a skill installs as `GENERIC`, which warns.
- Metadata values are always text, `deprecated` (`"true"`/`"false"`) and `aws-devops-agent-skills.agent-types` included, as the Agent Skills spec and the skill evaluation tool's structure test require. `metadata.tier` is `aws-managed` or `partner-managed`. Unknown top-level and metadata keys warn, naming the nearest known key; a misspelled `aws-devops-agent-skills.` prefix is an error.
- Exit codes: 0 pass (warnings allowed), 1 a rule broken, 2 could not run. Never report a 2 as the PR's fault.
- Kill switch: a maintainer's `skip-validate-skills` label sets `continue-on-error` on the PR workflow's install, test and check steps, so the check still runs and reports but can't block that PR; the `labeled`/`unlabeled` triggers re-run it. Never skip the job with `if:` instead: a skipped required check counts as passed. For a check that breaks every PR, an admin drops the required context until the fix merges.
Expand Down
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ before this check existed" in CONTRIBUTING.md.

## Skills — complete if this PR adds or changes a skill

<!-- The validate-skills check verifies the SKILL.md frontmatter, the files a skill publishes and its version history automatically; see "Skill Publishing Rules" in CONTRIBUTING.md. -->
<!-- The validate-skills check verifies the SKILL.md frontmatter, the files a skill publishes and its version history automatically, and looks up each new metadata.author name on GitHub; see "Skill Publishing Rules" in CONTRIBUTING.md. -->

- [ ] `SKILL.md` frontmatter is valid: `name` matches the directory, `description` present and within length limits, body at most 500 lines.
- [ ] The `description` is written from the agent's point of view — scenarios, services, error types, symptoms.
Expand Down Expand Up @@ -94,7 +94,7 @@ before this check existed" in CONTRIBUTING.md.

## Connectors — complete if this PR adds or changes a connector

<!-- Applies to any change inside a connectors/<name>/ directory. The partner that owns the connector opens the PR; see "Partner Connectors" in CONTRIBUTING.md. This item cannot be marked n/a. -->
<!-- Applies to any change inside a connectors/<name>/ directory. The partner that owns the connector opens the PR; see "Partner Connectors" in CONTRIBUTING.md. This item cannot be marked n/a. The validate-skills check compares the connector with main, so a change to any of its files needs a higher version. -->

- [ ] **I accept the [AWS DevOps Agent Partner Connector Terms and Conditions](https://github.com/aws/tools-for-devops-agent/blob/main/connectors/TERMS.md), version 2026-11-03, on behalf of the company that owns this connector, and I have authority to do so.**

Expand Down
13 changes: 7 additions & 6 deletions .github/scripts/check_skill_staleness.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
unlike CHANGELOG dates which are inconsistent across skills.

The script reads each ``skills/<name>/SKILL.md`` frontmatter for the ``author``
field (an internal alias, surfaced in the issue as plain text for a maintainer
to route to — it is NOT a GitHub login and is never mentioned or assigned) and
field (GitHub accounts, which validate-skills checks, surfaced in the issue as
plain text for a maintainer to route to — never mentioned or assigned, because an
author can be an organization, a team or someone who isn't a collaborator) and
prints:

* a human-readable summary to stderr, and
Expand Down Expand Up @@ -55,10 +56,10 @@ def issue_title(skill: StaleSkill) -> str:


def issue_body(skill: StaleSkill, max_age_days: int) -> str:
# The frontmatter `author` is the skill's declared owner, but in this repo it
# is an internal alias, NOT a GitHub login (e.g. "hokang" -> GitHub user
# "howard-m-k"). So it is surfaced as plain text for a maintainer to route to,
# never @-mentioned or used as an assignee.
# The frontmatter `author` is the skill's declared owner: GitHub accounts,
# which validate-skills checks. It is still surfaced as plain text for a
# maintainer to route to, never @-mentioned or used as an assignee, because an
# author can be an organization, a team or someone who isn't a collaborator.
listed_author = ", ".join(f"`{a}`" for a in skill.authors) or (
"_none listed in SKILL.md frontmatter_"
)
Expand Down
Loading
Loading