Repository navigation
fix(scripts): validate delivery rules and quote the emitted crane commands - #1209
Conversation
…mands gen-delivery-image-commands.ts built the delivery script by concatenating repo, tag, dest_repositories, constant_tags and the rewritten tag into `crane copy` / `crane tag` command lines. Nothing was validated or quoted: parseImageRef only split on the last colon, the rules file was used as-is, and every tags_regex was compiled with a bare `new RegExp`, so a permissive rule such as `^(master|dedicated)(-.*)?-nextgen$` let a tag like `master-\-nextgen` through and the emitted script — run by the delivery job with push credentials for the public registries — executed it. - move the rule types and validators into packages/scripts/lib/delivery-config.ts and use them from both readers of delivery.yaml (gen-delivery-image-commands.ts and get-delivery-target-images.ts) - validate the source repo/tag, the destination repositories, the constant tags and the rewritten tag; refuse a bad value with exit 2 before writing anything - quote every argument in the emitted commands - refuse an output path that escapes through '..' - add test_gen_delivery_image_commands to .github/scripts/ci.sh: type-check, three hostile inputs (metacharacter tag, injection in dest_repositories, quadratic regex) that must be refused without writing a script, and one valid case that must produce a quoted, POSIX-clean script get-delivery-target-images.ts keeps its behaviour: same validation, same host allowlist, byte-identical output for v8.5.0.
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This pull request improves the gen-delivery-image-commands.ts script by addressing quoting and validation issues in the generated shell commands for image delivery. The changes include introducing a shared validation module for delivery rules, enhancing input validation, and ensuring all shell commands are properly quoted to prevent injection attacks. The PR also adds tests for invalid inputs, ensuring robustness against edge cases. The changes are well-structured, with appropriate modularization, but there are a few areas for improvement.
Critical Issues
-
Validation Output Path Check (Security Concern)
File:gen-delivery-image-commands.ts(Line 86 in the new version)
Issue: The check for invalid output paths (..segments) could be bypassed with relative paths likedir/../output.sh.
Suggestion: Normalize the path before checking for unsafe patterns. UseDeno.realPathor similar to resolve the absolute path and then validate.const resolvedPath = await Deno.realPath(outFile); if (resolvedPath.includes("..")) { fail(`refusing to write to output path "${outFile}"`); }
-
Regex Length and Complexity Validation (Performance Concern)
File:lib/delivery-config.ts(Line 63)
Issue: ThecompileRuleRegexfunction stops "obviously quadratic" regexes but does not account for other complex regex patterns that could cause performance issues, such as catastrophic backtracking.
Suggestion: Consider using a library likesafe-regexto validate regex safety instead of relying on custom checks.import safeRegex from "safe-regex"; if (!safeRegex(source)) { fail(`${where}: unsafe regular expression`); }
Code Improvements
-
Error Handling Consolidation
File:gen-delivery-image-commands.tsandlib/delivery-config.ts
Issue: Thefailfunction is used consistently for error reporting, but some error messages could benefit from additional context, such as the specific input file or line responsible for the error.
Suggestion: Enhance thefailfunction to accept an additionalcontextparameter and include it in the error messages.export function fail(message: string, context?: string): never { const fullMessage = context ? `❌ [${context}] ${message}` : `❌ ${message}`; console.error(fullMessage); Deno.exit(2); }
-
Testing for Valid Input Cases
File:.github/scripts/ci.sh(Lines 830-846 in the new version)
Issue: The test only checks for specific patterns in the generated output script but doesn't verify the entire script structure.
Suggestion: Use a golden file or a template-based approach to compare the entire output script with an expected result for valid input. For example:diff -q "$out" "expected_output.sh" || { echo "❌ delivery generator: output does not match expected result" exit 1 }
-
Avoiding Hardcoded Dependencies
File:gen-delivery-image-commands.tsandget-delivery-target-images.ts
Issue: TheREPOSITORY_REandTAG_REpatterns are defined in the shared module but are still hardcoded in other parts of the code. This creates potential for inconsistency if the patterns are updated in one place but not another.
Suggestion: Ensure that all regex references are imported from the shared module to prevent divergence.
Best Practices
-
Documentation for New Functions
File:lib/delivery-config.ts
Issue: While some functions are documented, others likeassertStringArraylack clear documentation about their purpose and parameters.
Suggestion: Add concise docstrings for all functions, especially exported ones. For example:/** * Validates that a value is a string array and optionally checks its length and content. * @param value - The value to validate. * @param where - Context of the validation (e.g., field name). * @param options - Validation options (minItems, pattern, label). * @returns The validated string array. */ export function assertStringArray(...) { ... }
-
Test Coverage for Edge Cases
File:.github/scripts/ci.sh
Issue: While hostile inputs are tested, edge cases such as empty rules or images with unexpected but valid formats are not covered.
Suggestion: Add test cases for:- Empty
image_copy_rulesorimages. - Valid but unusual repository formats (e.g.,
example.com:1234/repo). - Excessively long but valid tags and repository names.
- Empty
-
Avoiding Duplicated Logic
File:gen-delivery-image-commands.tsandlib/delivery-config.ts
Issue: Functions likeassertImageRefshare similar validation logic for repositories and tags as invalidateDeliveryConfig.
Suggestion: Consolidate validations into shared functions. For example,assertImageRefcould delegate to the regex checks invalidateDeliveryConfig.
Additional Notes
- Deno Dependency Management: Ensure the Deno dependencies (e.g.,
jsr:@std) are pinned to specific versions to avoid breaking changes from upstream updates. - Error Exit Codes: Consider using distinct exit codes for different error types (e.g., validation failure vs. file read error) to simplify debugging in CI pipelines.
Overall, the PR is a significant improvement in both security and maintainability, but addressing the above points will further enhance robustness and adherence to best practices.
|
/approve Delivery generator validates and quotes every crane argument; hostile tag / destination / rule regex are rejected with exit 2 and no script, valid input yields a quoted script. test-packages-cfg and zizmor pass; get-delivery-target-images.ts output is byte-identical to the previous version. (user confirmed 2026-11-12) |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
## Problem Nine pull-request-triggered jobs requested `packages: write` — the `GITHUB_TOKEN` scope that publishes to `ghcr.io/pingcap-qe/*` (bases, builders, product runtimes) — although every build inside them runs with `--push=false` and needs no push scope: - `pull-cd-builder-images.yaml` (3 jobs), `pull-cd-builder-images-centos7.yaml` (3), `pull-cd-util-images.yaml` (1), `pull-prod-runtime-images.yaml` (1). - `pull-cd-builder-msb.yaml`: the `just-build` job held `packages: write` + `pull-requests: write`, logged in to ghcr, and then ran `just "$TASK"` against the **pull request's own** `dockerfiles/Justfile`. The label allowlist gated only the task *name*; the recipe body came from the PR. An actor able to add a label could therefore run an arbitrary recipe in a runner holding publish credentials. (Its `MY_GITHUB_TOKEN` was already removed in a previous change; the write scope and the login were not.) ## Changes - Delete `.github/workflows/pull-cd-builder-msb.yaml` — the `just/<task>` preview path is gone. The multi-stage builds it exercised are covered by the release builder workflows. - Remove `packages: write` and the `docker/login-action` ghcr step from the eight remaining PR jobs; they keep `contents: read`. - Release workflows are untouched (they publish, so they keep the scope). `dockerfiles/Justfile` stays as a local development helper; it is no longer executed by CI, so its recipe body is no longer PR-controlled in a credentialed runner. ## Verification - `grep -rn ": write" .github/workflows/pull-*.yaml` → none - `actionlint` on the four edited workflows → only the pre-existing `ubuntu-26.04` runner-label warnings and pre-existing SC2086 info notes in the matrix step (identical before this change; the diff only deletes lines) - anonymous `crane manifest ghcr.io/pingcap-qe/cd/builders/tidb:<tag>` → OK without login, so the builds can pull their base images with `contents: read` - `gh pr checks` → zizmor, pre-commit, license/cla pass. The builder workflows are path-filtered to `dockerfiles/cd/builders/**`, so they do not run on a workflow-only change; the first run that exercises the new permissions is the next PR touching those paths. Until then the anonymous-pull check above is the evidence that no login is needed. ## Risk / Rollback - Risk: anonymous ghcr pulls are rate-limited per runner IP. If a build starts hitting the pull limit, the fix is a **read-only** credential (separate token, `packages: read`), not restoring `packages: write`. - Risk: anyone relying on the `just/<task>` label to preview a multi-stage build loses that trigger; `dockerfiles/Justfile` still documents the local equivalent. - Rollback: revert this commit (restores the workflow and the scopes). ## Stack - Previous: #1209 - Next: offline-package artifact provenance (same series)
Problem
packages/scripts/gen-delivery-image-commands.tswrites the shell script the delivery job runs with push credentials fordocker.io/pingcap,gcr.io/pingcap-publicanduhub.service.ucloud.cn. It built that script purely by concatenation:crane copy ${repo}:${tag} ${destRepo}:${tag},crane tag ${destRepo}:${tag} ${constTag},crane tag ${destRepo}:${tag} ${converted}— no quoting, no charset check.parseImageRefonly split on the last colon;loadImageCopyRulesused the parseddelivery.yamlas-is;getMatchingRulesForcompiled everytags_regexwith a barenew RegExp, with no length or quantifier guard.^(master|dedicated|cloud-engine)(-.*)?-nextgen$match a tag likemaster-$(curl -s host|sh)-nextgen, which was then emitted verbatim.Demonstrated on the previous version:
The same value is also an authorization problem: the delivery policy (which source may be copied to which public destination) is enforced in the sibling consumer, not in the generator that emits the credentialed commands.
Changes
packages/scripts/lib/delivery-config.ts: the rule types,REPOSITORY_RE/TAG_RE,MAX_REGEX_LENGTH,compileRuleRegex,assertStringArray,validateDeliveryConfig,getMatchingRulesFor, plusshellQuote. Both readers ofdelivery.yamlnow import it.gen-delivery-image-commands.ts: validates the source repo/tag, destination repositories, constant tags and the rewritten tag; refuses a bad value with exit 2 before writing; quotes every argument; refuses an output path containing...get-delivery-target-images.ts: local copies removed, imports the shared module. Behaviour unchanged — same validation, samecdn.jsdelivr.netallowlist, byte-identical output forv8.5.0(diffed against the previous version)..github/scripts/ci.sh: newtest_gen_delivery_image_commands— type-checks all three scripts, then feeds three hostile inputs (metacharacter tag, injection indest_repositories, quadratictags_regex) and asserts exit 2 with no script written, plus one valid case asserting a quoted, POSIX-clean script. Skipped with a message whendenois absent (the runner has no deno today; wiring deno into the workflow is deliberately left out of this PR rather than adding an unpinned install step).Verification
deno check packages/scripts/{gen-delivery-image-commands.ts,get-delivery-target-images.ts,lib/delivery-config.ts}→ PASStest_gen_delivery_image_commands→ PASS (4 ✅, including the three rejections)master-$(id)-nextgen; new script → exit 2, no file writtenget-delivery-target-images.ts --config=packages/delivery.yaml --version=v8.5.0→ identical output to the previous versiongh pr checks→ pending (this PR)Risk / Rollback
result-images.yamlcarries a repo/tag outside the accepted charset now fails fast (exit 2) instead of emitting an unquoted command. Values produced by the current builds (us-docker.pkg.dev/...,master-nextgen,v8.5.0) pass.Stack