Skip to content

feat(ci): check connector versions, skill authors and tag values in PRs - #246

Draft
miscreantmoogly wants to merge 6 commits into
mainfrom
feat/skill-and-connector-checks
Draft

miscreantmoogly wants to merge 6 commits into
mainfrom
feat/skill-and-connector-checks

Conversation

@miscreantmoogly

Copy link
Copy Markdown
Contributor

Description

Five changes to the validate-skills pull request check, one commit each. Each closes a gap that let a change pass every check and then cause trouble in the skill or connector catalog after it merged. The commits can be reviewed one at a time: the first two need only #213, and the last three need the tag and author cleanup in #245.

1. Connector versions (feat(ci): check connector versions and breaking changes in PRs). validate-skills didn't look at connector versions. So #229, which changed Elastic's logos at the already-published 1.0.0, and #219, which renamed Coralogix's API key label within 1.x, both passed. Each then held up publishing for everything until #234 fixed them. The new .github/scripts/connector_checks.py compares each connectors/<name>/ folder with the merge base, the way the catalog publisher compares a connector with its published version:

  • A connector folder is never removed; deprecate it instead. Its name and type never change.
  • Every file in the folder is published, so any change needs a higher version, and the version never goes down.
  • Within a major version, no mcp.auth option is removed or changed (any field counts, a header label included). No endpoint is removed or gets another URL, and no mcp.config field is added.
  • connector.yaml needs a MAJOR.MINOR.PATCH version. A new connector that doesn't start at 1.0.0, or a version that skips a step, is a warning.

connector.yaml is compared as the publisher reads it, so comments and quoting don't count as a major change, but types do. The rules share the skill rules' helpers and in-memory folder, and their cases are in the new skill-rules/connector-conformance-cases.json, which the script runs before it checks anything.

This commit changes two existing behaviors:

  • A connector.yaml that isn't a mapping is now an error, not a warning, since it has no version.
  • Dropping a deprecated skill from a connector's skills: list now needs a connector version bump, which the publisher already requires.

2. The published-file list comes from the base branch (fix(ci): read skill-rules/published-files.json from the merge base). skill-rules/published-files.json decides which of a skill's files are published, and so which changes need a version bump. The check read it from the pull request's own copy, so a pull request that widened the list was compared on its new list. It's now read from the merge base, as the extension allowlist in skills/.gitignore already is, so a change to it takes effect once it has merged.

3. Placeholder tag values are errors (feat(ci): reject placeholder tag values such as "All"). All, Any, None, N/A, Other, General and *, in any case, are now an error in aws-services and technical-domains, and the message tells the author to leave the key out. Catalogs show every value as a filter option, so aws-services: "All" showed up as a service called "All". vocabulary.json can't approve a placeholder either: the check refuses to run if it lists one.

4. Authors must be GitHub accounts (feat(ci): check that skill authors are GitHub accounts). Catalogs link metadata.author to that name's GitHub profile, but the check only tested the form of the name, so some links 404. The new .github/scripts/github_logins.py looks a name up with GET /users/<login>. validate_skills.py --check-github-logins uses it for every name that a new skill lists, or that a changed author line adds. Names already on main aren't looked up again, so an account deleted later can't fail an unrelated PR.

  • A name GitHub doesn't know is an error.
  • When GitHub can't answer, because there's no token, a rate limit, a 5xx or a timeout, it's a warning, so an outage never blocks a PR.
  • After the first lookup GitHub can't answer, the run asks nothing more, and every later name is a warning too. An outage then costs one lookup's timeout and retry, about 20 seconds, rather than that much for every name, which could outlast the job's 10-minute limit. A rate limit isn't hit again either.
  • A team (org/team) is looked up by its organization, since only members can see a team. A team under a user account is an error.
  • The PR workflow passes the job's own token in GH_TOKEN, as the repository's other workflows do, and keeps permissions: contents: read. The token goes only to api.github.com, because redirects are refused, and it's never printed.
  • Tests use a stub resolver and a fake HTTPS handler, so no test reaches GitHub.

5. Unlisted tag values are errors (feat(ci): make unapproved tag values an error). A value missing from vocabulary.json used to be a warning, shown only on the skills a PR touched. So near-duplicates such as Database next to Databases reached main and split the filters. It's now an error. When an alias exists, the message gives the approved spelling. Otherwise it says to add the value to vocabulary.json in the same PR, where a maintainer reviews it.

Merge order

Merge this after #213, which it builds on, and only once main has no values these rules reject. That means after #245, which cleans up the tags and authors and adds the missing service names, and after #188, which adds Amazon CloudFront and AWS Elemental MediaTailor.

On main today the new rules report 12 skills: the All placeholder, 10 other unlisted values on 4 skills, and 6 author names on 9 skills that aren't GitHub accounts. Once both PRs have merged, every skill passes.

Effect on open pull requests

I checked each of the 74 open PRs merged with main, #213, #245, #188 and this branch, and compared it with the same merge without this branch:

Type of change

  • New skill
  • New custom agent
  • New MCP server
  • Update to an existing skill, agent, or MCP server
  • Documentation or infrastructure change

Required before review

  • Repository layout is correct: this PR changes only .github/, tests/, CONTRIBUTING.md, .claude/CLAUDE.md and .kiro/steering/.
  • A domain SME has reviewed and signed off. n/a: a CI change that changes no technical claim a tool makes.
  • Evaluation evidence is included. n/a: no skill, agent, or MCP server changes.
  • No internal-only or proprietary content: no internal hostnames, account IDs, customer names, ticket numbers, internal URLs, or secrets.
  • README is complete. n/a: no tool directory changes.
  • CHANGELOG updated and the version bumped. n/a: no tool directory changes, and nothing here is published.
  • Named owner for ongoing maintenance. n/a: the check is owned by the code owners of /.github/.

Testing

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

aadimch and others added 6 commits November 9, 2026 04:38
validate-skills didn't look at connector versions, so two connector
changes passed every check and then stopped the catalog publisher, which
refuses the whole run while any connector breaks its version rules: a
logo change at an already-published version, and an auth header label
renamed within the same major version. A replay of every merge on main
with this check flags exactly those two.

Add .github/scripts/connector_checks.py, which compares each
connectors/<name>/ folder with its merge-base copy the way the publisher
compares a connector with its published version:

- a connector folder is never removed (deprecate it instead), and its
  name and type never change
- every file in the folder is published, so any change to one needs a
  higher version, and the version never goes down
- within a major version, no mcp.auth option is removed or changed (any
  field, a header label included), no endpoint is removed or gets
  another URL, and no mcp.config field is added
- connector.yaml has a MAJOR.MINOR.PATCH version, and a new connector
  not at 1.0.0 or a version that skips a step is a warning

connector.yaml is compared as the publisher reads it: plain true and
false are booleans, safe integers and shortest-form decimals are
numbers, and the rest is text, so comments and quoting never count as a
major change. skill_checks.plain_scalar_value holds that typing next to
the YAML rules it depends on.

The rules work on the same in-memory SkillTree as the skill rules and
share their helpers: format_version and next_versions become public,
the lenient PyYAML reader becomes compose_lenient, and run_cases holds
the pass/fail matching for both case files. validate_skills.py reads
the connector folders with one ls-tree reader, which its other three
listings now use too, reports each finding under
connectors/<name>/connector.yaml at the version line, and runs the new
skill-rules/connector-conformance-cases.json before checking anything.
Connector findings always show, since a warning can only come from a
folder that differs from its merge-base copy.

A connector.yaml that isn't a mapping is now an error, since it has no
version to compare, and no longer also a warning from the skills: check.
The connector fixtures in tests/test_validate_skills.py gain a version,
and dropping a skill from a connector's list now bumps the connector, as
the publisher requires. CONTRIBUTING.md, the connector review steering
doc, the PR template's Connectors note, the workflow header and the
project conventions describe the rules.
published-files.json decides which files in a skill are published, and
so which changes need a version bump. The check read it from the pull
request's own copy, so a pull request that widened the list was compared
on its new list: a file it left out of the published set and changed in
the same pull request needed no bump.

Read the list from the merge base, as the check already reads the
extension allowlist in skills/.gitignore, so a change to it takes effect
only once it has merged. Both are read in the same step, through one
helper that reads a file at a revision. The conformance cases still run
on the pull request's own rules; only its skills are compared on the
base branch's list. A malformed list on the base branch stops the check
with exit 2, as malformed rule data does.

skill_checks.published_excludes holds the list's validation, which
load_rules used inline. Code-owner review of .github/ remains what
stops a pull request that edits the check itself.
A skill that lists "All" in aws-services shows up in catalogs under a
service called "All", because nothing between the skill and the filter
turns a placeholder back into "every service". The check only warned
about it, as it does for any value missing from vocabulary.json.

Make All, Any, None, N/A, Other, General and *, compared ignoring case,
an error in aws-services and technical-domains. The message tells the
author to leave the key out, since a skill without it still appears
when no filter is chosen, or, when the list also holds real values, to
remove the placeholder from it.

vocabulary.json can't approve a placeholder either. load_rules refuses
one in values or in aliases, so the check exits 2 instead of letting a
pull request approve its own placeholder.

Add the dimension-reserved-value conformance case, unit tests for the
rule and for load_rules, and describe the rule in CONTRIBUTING.md,
vocabulary.json and the project conventions.
The check only tests that metadata.author has the form of a GitHub
login, so a skill can name an account that doesn't exist. Catalogs link
the name to its GitHub profile, and that link then 404s, or goes to
whoever registers the name later.

Add .github/scripts/github_logins.py, which looks a login up with
GitHub's user and organization lookup (GET /users/<login>): a
10-second timeout, one retry for a server error or a failed request,
one lookup per login per run, and redirects refused, so the token never
goes anywhere but api.github.com. It reads the token from GH_TOKEN,
then GITHUB_TOKEN, as the GitHub CLI does.

validate_skills.py --check-github-logins looks up each name that a new
skill lists, or that a changed author line adds. Names already on the
base branch aren't looked up again, so an account deleted later can't
fail a pull request that leaves the author line alone. A 404 is an
error. When GitHub can't answer, because no token is set, the rate
limit is reached, the server fails or the request times out, the name
is a warning, so an outage never blocks a pull request. After the first
lookup GitHub can't answer, the run asks nothing more and every later
name is a warning too, so an outage costs one lookup's timeout and
retry rather than one per name, which could outlast the job's time
limit, and a rate limit isn't hit again. A team is looked up by its
organization, since GitHub shows a team only to the organization's
members, and a team of a user account is an error.

The pull request workflow passes the flag and the job's token in
GH_TOKEN, the name the repository's other workflows use, keeping
permissions at contents: read. Tests answer with a stub resolver or a
fake HTTPS handler, so none reaches GitHub. CONTRIBUTING.md now says an
author is a GitHub username, organization or org/team that exists, and
the skill staleness reminder no longer calls an author an internal
alias. The connector review steering runs the flag without a token, so
a reviewer's credentials never reach the trial merge.
A value missing from vocabulary.json, in aws-services or
technical-domains, was only a warning, and warnings show only on the
skills a pull request touches. So near-duplicates such as "Database"
next to "Databases" reached main, and catalogs, which match values
exactly, list them as separate filter options with the skills split
between them.

Make an unapproved value an error, keeping the alias hint (use
"Databases"). A value with no alias now says to use one from
vocabulary.json, or to add it there in the same pull request, where a
maintainer reviews it.

Rename the valid-unapproved-dimension-value-warns-only case to
dimension-unapproved-value, now expected to fail. Update the unit tests,
the vocabulary.json description, the CONTRIBUTING.md rules row and the
project conventions.

This must merge only once main has no unapproved values: the tag and
vocabulary cleanup, and the Amazon CloudFront and AWS Elemental
MediaTailor names from #188.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants