Repository navigation
feat(ci): check connector versions, skill authors and tag values in PRs - #246
Draft
miscreantmoogly wants to merge 6 commits into
Draft
miscreantmoogly wants to merge 6 commits into
miscreantmoogly wants to merge 6 commits into
Conversation
validate-skills didn't look at connector versions, so two connector changes passed every check and then stopped the catalog publisher, which refuses the whole run while any connector breaks its version rules: a logo change at an already-published version, and an auth header label renamed within the same major version. A replay of every merge on main with this check flags exactly those two. Add .github/scripts/connector_checks.py, which compares each connectors/<name>/ folder with its merge-base copy the way the publisher compares a connector with its published version: - a connector folder is never removed (deprecate it instead), and its name and type never change - every file in the folder is published, so any change to one needs a higher version, and the version never goes down - within a major version, no mcp.auth option is removed or changed (any field, a header label included), no endpoint is removed or gets another URL, and no mcp.config field is added - connector.yaml has a MAJOR.MINOR.PATCH version, and a new connector not at 1.0.0 or a version that skips a step is a warning connector.yaml is compared as the publisher reads it: plain true and false are booleans, safe integers and shortest-form decimals are numbers, and the rest is text, so comments and quoting never count as a major change. skill_checks.plain_scalar_value holds that typing next to the YAML rules it depends on. The rules work on the same in-memory SkillTree as the skill rules and share their helpers: format_version and next_versions become public, the lenient PyYAML reader becomes compose_lenient, and run_cases holds the pass/fail matching for both case files. validate_skills.py reads the connector folders with one ls-tree reader, which its other three listings now use too, reports each finding under connectors/<name>/connector.yaml at the version line, and runs the new skill-rules/connector-conformance-cases.json before checking anything. Connector findings always show, since a warning can only come from a folder that differs from its merge-base copy. A connector.yaml that isn't a mapping is now an error, since it has no version to compare, and no longer also a warning from the skills: check. The connector fixtures in tests/test_validate_skills.py gain a version, and dropping a skill from a connector's list now bumps the connector, as the publisher requires. CONTRIBUTING.md, the connector review steering doc, the PR template's Connectors note, the workflow header and the project conventions describe the rules.
published-files.json decides which files in a skill are published, and so which changes need a version bump. The check read it from the pull request's own copy, so a pull request that widened the list was compared on its new list: a file it left out of the published set and changed in the same pull request needed no bump. Read the list from the merge base, as the check already reads the extension allowlist in skills/.gitignore, so a change to it takes effect only once it has merged. Both are read in the same step, through one helper that reads a file at a revision. The conformance cases still run on the pull request's own rules; only its skills are compared on the base branch's list. A malformed list on the base branch stops the check with exit 2, as malformed rule data does. skill_checks.published_excludes holds the list's validation, which load_rules used inline. Code-owner review of .github/ remains what stops a pull request that edits the check itself.
A skill that lists "All" in aws-services shows up in catalogs under a service called "All", because nothing between the skill and the filter turns a placeholder back into "every service". The check only warned about it, as it does for any value missing from vocabulary.json. Make All, Any, None, N/A, Other, General and *, compared ignoring case, an error in aws-services and technical-domains. The message tells the author to leave the key out, since a skill without it still appears when no filter is chosen, or, when the list also holds real values, to remove the placeholder from it. vocabulary.json can't approve a placeholder either. load_rules refuses one in values or in aliases, so the check exits 2 instead of letting a pull request approve its own placeholder. Add the dimension-reserved-value conformance case, unit tests for the rule and for load_rules, and describe the rule in CONTRIBUTING.md, vocabulary.json and the project conventions.
The check only tests that metadata.author has the form of a GitHub login, so a skill can name an account that doesn't exist. Catalogs link the name to its GitHub profile, and that link then 404s, or goes to whoever registers the name later. Add .github/scripts/github_logins.py, which looks a login up with GitHub's user and organization lookup (GET /users/<login>): a 10-second timeout, one retry for a server error or a failed request, one lookup per login per run, and redirects refused, so the token never goes anywhere but api.github.com. It reads the token from GH_TOKEN, then GITHUB_TOKEN, as the GitHub CLI does. validate_skills.py --check-github-logins looks up each name that a new skill lists, or that a changed author line adds. Names already on the base branch aren't looked up again, so an account deleted later can't fail a pull request that leaves the author line alone. A 404 is an error. When GitHub can't answer, because no token is set, the rate limit is reached, the server fails or the request times out, the name is a warning, so an outage never blocks a pull request. After the first lookup GitHub can't answer, the run asks nothing more and every later name is a warning too, so an outage costs one lookup's timeout and retry rather than one per name, which could outlast the job's time limit, and a rate limit isn't hit again. A team is looked up by its organization, since GitHub shows a team only to the organization's members, and a team of a user account is an error. The pull request workflow passes the flag and the job's token in GH_TOKEN, the name the repository's other workflows use, keeping permissions at contents: read. Tests answer with a stub resolver or a fake HTTPS handler, so none reaches GitHub. CONTRIBUTING.md now says an author is a GitHub username, organization or org/team that exists, and the skill staleness reminder no longer calls an author an internal alias. The connector review steering runs the flag without a token, so a reviewer's credentials never reach the trial merge.
A value missing from vocabulary.json, in aws-services or technical-domains, was only a warning, and warnings show only on the skills a pull request touches. So near-duplicates such as "Database" next to "Databases" reached main, and catalogs, which match values exactly, list them as separate filter options with the skills split between them. Make an unapproved value an error, keeping the alias hint (use "Databases"). A value with no alias now says to use one from vocabulary.json, or to add it there in the same pull request, where a maintainer reviews it. Rename the valid-unapproved-dimension-value-warns-only case to dimension-unapproved-value, now expected to fail. Update the unit tests, the vocabulary.json description, the CONTRIBUTING.md rules row and the project conventions. This must merge only once main has no unapproved values: the tag and vocabulary cleanup, and the Amazon CloudFront and AWS Elemental MediaTailor names from #188.
This was referenced Nov 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Five changes to the
validate-skillspull request check, one commit each. Each closes a gap that let a change pass every check and then cause trouble in the skill or connector catalog after it merged. The commits can be reviewed one at a time: the first two need only #213, and the last three need the tag and author cleanup in #245.1. Connector versions (
feat(ci): check connector versions and breaking changes in PRs).validate-skillsdidn't look at connector versions. So #229, which changed Elastic's logos at the already-published 1.0.0, and #219, which renamed Coralogix's API key label within 1.x, both passed. Each then held up publishing for everything until #234 fixed them. The new.github/scripts/connector_checks.pycompares eachconnectors/<name>/folder with the merge base, the way the catalog publisher compares a connector with its published version:nameandtypenever change.version, and the version never goes down.mcp.authoption is removed or changed (any field counts, a header label included). No endpoint is removed or gets another URL, and nomcp.configfield is added.connector.yamlneeds aMAJOR.MINOR.PATCHversion. A new connector that doesn't start at 1.0.0, or a version that skips a step, is a warning.connector.yamlis compared as the publisher reads it, so comments and quoting don't count as a major change, but types do. The rules share the skill rules' helpers and in-memory folder, and their cases are in the newskill-rules/connector-conformance-cases.json, which the script runs before it checks anything.This commit changes two existing behaviors:
connector.yamlthat isn't a mapping is now an error, not a warning, since it has no version.skills:list now needs a connector version bump, which the publisher already requires.2. The published-file list comes from the base branch (
fix(ci): read skill-rules/published-files.json from the merge base).skill-rules/published-files.jsondecides which of a skill's files are published, and so which changes need a version bump. The check read it from the pull request's own copy, so a pull request that widened the list was compared on its new list. It's now read from the merge base, as the extension allowlist inskills/.gitignorealready is, so a change to it takes effect once it has merged.3. Placeholder tag values are errors (
feat(ci): reject placeholder tag values such as "All").All,Any,None,N/A,Other,Generaland*, in any case, are now an error inaws-servicesandtechnical-domains, and the message tells the author to leave the key out. Catalogs show every value as a filter option, soaws-services: "All"showed up as a service called "All".vocabulary.jsoncan't approve a placeholder either: the check refuses to run if it lists one.4. Authors must be GitHub accounts (
feat(ci): check that skill authors are GitHub accounts). Catalogs linkmetadata.authorto that name's GitHub profile, but the check only tested the form of the name, so some links 404. The new.github/scripts/github_logins.pylooks a name up withGET /users/<login>.validate_skills.py --check-github-loginsuses it for every name that a new skill lists, or that a changed author line adds. Names already onmainaren't looked up again, so an account deleted later can't fail an unrelated PR.org/team) is looked up by its organization, since only members can see a team. A team under a user account is an error.GH_TOKEN, as the repository's other workflows do, and keepspermissions: contents: read. The token goes only to api.github.com, because redirects are refused, and it's never printed.5. Unlisted tag values are errors (
feat(ci): make unapproved tag values an error). A value missing fromvocabulary.jsonused to be a warning, shown only on the skills a PR touched. So near-duplicates such asDatabasenext toDatabasesreachedmainand split the filters. It's now an error. When an alias exists, the message gives the approved spelling. Otherwise it says to add the value tovocabulary.jsonin the same PR, where a maintainer reviews it.Merge order
Merge this after #213, which it builds on, and only once
mainhas no values these rules reject. That means after #245, which cleans up the tags and authors and adds the missing service names, and after #188, which addsAmazon CloudFrontandAWS Elemental MediaTailor.On
maintoday the new rules report 12 skills: theAllplaceholder, 10 other unlisted values on 4 skills, and 6 author names on 9 skills that aren't GitHub accounts. Once both PRs have merged, every skill passes.Effect on open pull requests
I checked each of the 74 open PRs merged with
main, #213, #245, #188 and this branch, and compared it with the same merge without this branch:vocabulary.jsonin 16 open PRs: feat: acm-certificate-ops-review skill #20, feature: add drift-detection custom agent and baseline skill #26, feat(ec2-operation-review): Add EC2 review skill #89, feat(lambda-operational-review): Add Lambda skill #90, feat(glue-6-migration-review): Add Glue 6 skill #91, feat: add dynamodb-data-health-inspection skill #94, feat: add dynamodb-latency skill #96, Add PostgreSQL DBA diagnostics MCP #110, Add support-and-ta skill and support-and-ta-report custom agent #124, Add Prometheus (AMP) MCP server under mcp/ #131, feat(aws-waf-log-analysis): add AWS WAF log analysis skill #139, Add AWS Batch investigation and incident-correlation skills #160, [aiml-access-diagnostics] Add 8 AI/ML services (v2.0.0) #184, feat(gaming-industry-operations): add Gaming Industry Skill grounded in the Games Industry Lens #201, datalake access analyzer skill #212 and Add sustainability-operation-review skill #230. Only 4 of them, Add AWS Batch investigation and incident-correlation skills #160, feat(gaming-industry-operations): add Gaming Industry Skill grounded in the Games Industry Lens #201, datalake access analyzer skill #212 and Add sustainability-operation-review skill #230, have no other error, so those are the PRs it turns red. The other 12 fail without this branch too, for reasons such as unknown agent types. Each PR passes once it adds its names tovocabulary.jsonor uses approved ones. If that's still too many at once, drop the last commit and merge it later on its own.main, it needs 2.0.1. The other open connector PRs pass..claude/CLAUDE.mdand.kiro/steering/project-conventions.mdas the last commit here, so whichever of the two merges second needs a rebase. With both changes kept, every check passes.Type of change
Required before review
.github/,tests/,CONTRIBUTING.md,.claude/CLAUDE.mdand.kiro/steering/./.github/.Testing
python3 -I -m unittest discover -s tests -v: 323 tests pass on this branch merged withmain(53 of them new), none of them using the network. Each commit passes on its own too.python3 .github/scripts/validate_skills.py --self-check-only: every case inconformance-cases.jsonandconnector-conformance-cases.jsonpasses.I replayed all 185 first-parent commits on
mainthrough the connector rules. They flag exactly Add horizontal Elastic logo for connector #229 and HEA-107 | Document team API-key support for the Coralogix DevOps Agent connector #219, and nothing else; Unblock connector catalog publish: bump elastic to 1.0.1 and coralogix to 2.0.0 #234 passes.On
mainmerged with this branch,validate_skills.py --head-ref HEAD --check-github-loginsreports the 12 skills listed under Merge order.With fix(skills): clean up filter tags and author names #245 and Add the AWS Elemental Media Services and CloudFront to the skill vocabulary #188 merged too, all 100 skills and 12 connectors pass, both with
--base-ref origin/main --check-github-loginsand with no base. All 46 author names on that merge resolve on GitHub, including every name fix(skills): clean up filter tags and author names #245 adds. Without Add the AWS Elemental Media Services and CloudFront to the skill vocabulary #188, only its two service names fail, and without the last commit they're warnings.On that merge, with GitHub unreachable, the run with no base and
--check-github-loginsfinishes in about 20 seconds, and every author name is a warning.On that merge,
scan_aws_identifiers.py,validate_skill_evals.pyandmkdocs build --strictalso pass, and every action invalidate-skills.ymlis pinned to a commit SHA.structurechecks pass. n/a: no skill changes.best-practicesrun across multiple iterations. n/a: no skill changes.functionalrun with and without the change. n/a: no skill changes.evals/evals.jsonincludes positive and negative cases. n/a: no skill changes.License confirmation