Repository navigation
fix(license): bound GitHub licence lookups so scan jobs stop draining the repository's API quota - #342
Merged
Conversation
… the repository's API quota On the push that merged #341 the analyze preflight saw 4,913 of 5,000 requests left; by 16:08:18, three and a half minutes later, SCA's own SARIF upload failed with "GitHub API rate limit exhausted", and the forge pass found 0 left. The scan jobs' licence lookups spent it: - 46 of the 72 names a licence pass asked GitHub about cannot be repositories ($/sbom from a workflow's `uses: $/sbom`, a glob, "<your"). - Each name went through about 14 requests through gh, then 12 more with GITHUB_TOKEN as a "PAT fallback", though gh already sends that token. - Six processes per push repeat this: sca (two steps), license, sbom, and the sbom prior steps of the upload and bom-validate extras. The lookups now: - skip names GitHub cannot hold (validOwnerRepo); - run repository, /license, then root listing, ending at a repository 404, with no blind file fetches and no duplicate token pass: 1-4 requests; - spend from internal/ghquota, the forge budget moved to a shared package, seeded from the free /rate_limit, keeping 80% of the limit inside GitHub Actions (above the forge pass's half); - stop on a rate-limit answer, leave unchecked licences UNKNOWN and say how many, and log each batch's request count in Actions. gha init gives no token to a scanner run as an extra's prior step, which publishes nothing; vulnetix.yml is regenerated.
Contributor
Vulnetix config planManager
0 to create, 1 to update, 0 to delete, 0 unchanged |
0x73746F66
added a commit
to Vulnetix/platform-docs
that referenced
this pull request
Nov 13, 2026
… prior scanner steps get no token (#23) Scanners never ask GitHub about a name that cannot be a repository and stop at a repository 404. Inside GitHub Actions they stop at 4,000 of 5,000 requests left, above analyze's half, and on any rate-limit answer, leaving unchecked licences UNKNOWN and logging what each batch spent. An extra's prior scanner step gets github-token "" on every event. Matches Vulnetix/cli#342.
0x73746F66
added a commit
that referenced
this pull request
Nov 13, 2026
v4.24.33 carries the bounded GitHub licence lookups (#342): scan jobs on a push no longer spend the repository's API quota that analyze's forge pass, CodeQL's SARIF upload and the release dispatch share.
0x73746F66
added a commit
that referenced
this pull request
Nov 13, 2026
…reading as "release not found" The provenance and publish-tea jobs failed together on v4.24.8, v4.24.32 and v4.24.34 with "release not found" from `gh release upload <tag>`, minutes after the release was published, on different runners; a re-run of the failed jobs passed. The release was never missing, a draft or immutable. gh (since cli/cli#13723) looks a release up by tag over REST and as a draft over GraphQL at once, and when both fail it prefers to report "release not found". With PACKAGES_TOKEN's REST quota spent, the REST lookup is refused with 403/429 and the GraphQL lookup, which has its own quota, correctly finds no draft, so a published release reads as missing. That token is a personal token whose 5,000 requests an hour are shared with everything else using it; at 16:08:18 on the day of v4.24.32 the quota was at 0 (see #342), and the provenance upload failed at 16:16:10. - scripts/release-asset-upload.sh uploads to a release by id (the caller's, or resolved over GraphQL), replaces an existing asset of the same name, waits a spent quota out until its reset (65 minutes at most), honours Retry-After for a secondary limit, retries 404/5xx a few times, and names the HTTP status of every failure. - The release job asks GraphQL whether the release exists, uploads by id, and outputs release_id; the provenance workflow takes it as an optional input (belai's call still works by tag) and runs the script from its own commit (job_workflow_sha), so a caller in another repository runs it too. - The slsa-provenance artifact is uploaded before the release asset, so a failed release upload no longer loses the hand-off. - publish-tea needs provenance to succeed, and accepts a skipped update-packages only for a release older than the newest. Before, a provenance failure skipped canary and update-packages, and publish-tea still ran: it would have put v4.24.34 in the transparency log, which cannot be unpublished, without its bundle and with Homebrew, Scoop and the flake left on v4.24.33.
0x73746F66
added a commit
that referenced
this pull request
Nov 13, 2026
…reading as "release not found" (#344) The provenance and publish-tea jobs failed together on v4.24.8, v4.24.32 and v4.24.34 with "release not found" from `gh release upload <tag>`, minutes after the release was published, on different runners; a re-run of the failed jobs passed. The release was never missing, a draft or immutable. gh (since cli/cli#13723) looks a release up by tag over REST and as a draft over GraphQL at once, and when both fail it prefers to report "release not found". With PACKAGES_TOKEN's REST quota spent, the REST lookup is refused with 403/429 and the GraphQL lookup, which has its own quota, correctly finds no draft, so a published release reads as missing. That token is a personal token whose 5,000 requests an hour are shared with everything else using it; at 16:08:18 on the day of v4.24.32 the quota was at 0 (see #342), and the provenance upload failed at 16:16:10. - scripts/release-asset-upload.sh uploads to a release by id (the caller's, or resolved over GraphQL), replaces an existing asset of the same name, waits a spent quota out until its reset (65 minutes at most), honours Retry-After for a secondary limit, retries 404/5xx a few times, and names the HTTP status of every failure. - The release job asks GraphQL whether the release exists, uploads by id, and outputs release_id; the provenance workflow takes it as an optional input (belai's call still works by tag) and runs the script from its own commit (job_workflow_sha), so a caller in another repository runs it too. - The slsa-provenance artifact is uploaded before the release asset, so a failed release upload no longer loses the hand-off. - publish-tea needs provenance to succeed, and accepts a skipped update-packages only for a release older than the newest. Before, a provenance failure skipped canary and update-packages, and publish-tea still ran: it would have put v4.24.34 in the transparency log, which cannot be unpublished, without its bundle and with Homebrew, Scoop and the flake left on v4.24.33.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On pushes to
main, every job invulnetix.ymlspends one hourly GitHub API quota: the repository's job token. On the push that merged #341 (run 43458567106):4,913/5,000 requests remaining.code scanning sca/not-go: upload failed: GitHub API rate limit exhausted.0 of 5000 requests left.#339 and #341 only made analyze cope with a spent quota. This PR removes what spends it.
Root cause: the licence lookups in the scan jobs
Every scanner that resolves licences asks GitHub about each package that the Vulnetix store, the registry and deps.dev left
UNKNOWN. Measured locally on this repository with the network stubbed out, onevulnetix licenserun sends 72 distinct names to the GitHub stage:$/…entries fromactions-e2e.yml'suses: $/sbom,**/*.{css,scss,less}and<your./license, 11 blindcontents/LICENSE*fetches, the listing), then 12 more on the "PAT fallback" withGITHUB_TOKEN. That second pass is the same token, because gh already sendsGITHUB_TOKEN. A 404 never ended the chain early.sbomprior steps of theuploadandbom-validateextras. The prior steps take the action's default token even on pull requests.That comes to roughly 1,400 requests per process, which matches 4,913 requests gone in under 3.5 minutes. Scans run with
ghactxNoAPI: true, so apart from publishing, the licence lookups are the only GitHub API traffic the scan jobs make.Fix
internal/ghquota: the forge budget from fix(analyze): bound the forge pass to half the token quota and a short window in CI #339 and fix(analyze): a spent GitHub quota at preflight skips the forge pass instead of failing the run #341 is moved here unchanged and shared, and forge keeps its names through aliases. It gains:Keep(share)andHint(...);ObserveHTTP, which readsX-RateLimit-*and treats a primary or secondary rate-limit 403, or any 429, as a refusal (through go-github'sCheckResponseandIsRateLimit);Seed, which takes the free/rate_limitanswer;Spend, for the gh-process path.internal/license:validOwnerRepodrops impossible names before any request./license→ root listing plus the files it names. A repository 404 ends it, so a repository costs 1–4 requests.githubGet/ghAPICall, which spend from one budget. Inside GitHub Actions that budget keeps 80% of the limit (it stops at 4,000 of 5,000), which leaves room above the forge pass's 50%. Every job sees the same remaining count, so the floor holds for the whole run.UNKNOWN, with a single warning that names how many.GitHub licence lookups: N requests for M repositories (R of L left), so the run logs now carry per-job counts.FetchLicenseFromGitHubcopy of the chain is removed.gha init: a scanner run as an extra'spriorstep getsgithub-token: "", because it only writes the file the extra reads.vulnetix.ymlis regenerated (three prior steps change).Tests
TestLookupChainCostpins the request count for each repository shape.TestNamesThatCannotBeRepositoriesNeverReachGitHubTestLookupsKeepTheirShareInGitHubActionsTestLookupsSpendNothingBelowTheFloorTestRateLimitAnswerStopsLookupsghquotatests cover the HTTP headers, secondary limits,KeepandSeed.TestPriorScannerStepsGetNoTokengo test ./internal/...passes (83 packages), and so do the cmd and actiongen tests.A live run of the new binary against a token whose core quota was already at 0 made no lookup requests (the
/rate_limitseed stopped it) and printed the warning.Follow-up
vulnetix.ymlinstalls the CLI at its pinned action ref, so the CLI fix reaches this repository's CI only after the release is cut and the pin is bumped, as #340 did after #339.