Skip to content

fix(scripts): validate delivery rules and quote the emitted crane commands - #1209

Merged
ti-chi-bot[bot] merged 1 commit into
fix/promotion-generator-input-validation-p1from
fix/delivery-generator-input-validation-p2
Nov 12, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
fix/promotion-generator-input-validation-p1from
fix/delivery-generator-input-validation-p2

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Problem

packages/scripts/gen-delivery-image-commands.ts writes the shell script the delivery job runs with push credentials for docker.io/pingcap, gcr.io/pingcap-public and uhub.service.ucloud.cn. It built that script purely by concatenation:

  • crane copy ${repo}:${tag} ${destRepo}:${tag}, crane tag ${destRepo}:${tag} ${constTag}, crane tag ${destRepo}:${tag} ${converted} — no quoting, no charset check.
  • parseImageRef only split on the last colon; loadImageCopyRules used the parsed delivery.yaml as-is; getMatchingRulesFor compiled every tags_regex with a bare new RegExp, with no length or quantifier guard.
  • Permissive rules such as ^(master|dedicated|cloud-engine)(-.*)?-nextgen$ match a tag like master-$(curl -s host|sh)-nextgen, which was then emitted verbatim.

Demonstrated on the previous version:

$ deno run --allow-read --allow-write gen-delivery-image-commands.ts --images_file=images.yaml --yaml_file=delivery.yaml --outfile=out.sh
$ cat out.sh
crane copy us-docker.pkg.dev/hub/pingcap/tidb/image:master-$(id)-nextgen docker.io/pingcap/tidb:master-$(id)-nextgen

The same value is also an authorization problem: the delivery policy (which source may be copied to which public destination) is enforced in the sibling consumer, not in the generator that emits the credentialed commands.

Changes

  • New packages/scripts/lib/delivery-config.ts: the rule types, REPOSITORY_RE / TAG_RE, MAX_REGEX_LENGTH, compileRuleRegex, assertStringArray, validateDeliveryConfig, getMatchingRulesFor, plus shellQuote. Both readers of delivery.yaml now import it.
  • gen-delivery-image-commands.ts: validates the source repo/tag, destination repositories, constant tags and the rewritten tag; refuses a bad value with exit 2 before writing; quotes every argument; refuses an output path containing ...
  • get-delivery-target-images.ts: local copies removed, imports the shared module. Behaviour unchanged — same validation, same cdn.jsdelivr.net allowlist, byte-identical output for v8.5.0 (diffed against the previous version).
  • .github/scripts/ci.sh: new test_gen_delivery_image_commands — type-checks all three scripts, then feeds three hostile inputs (metacharacter tag, injection in dest_repositories, quadratic tags_regex) and asserts exit 2 with no script written, plus one valid case asserting a quoted, POSIX-clean script. Skipped with a message when deno is absent (the runner has no deno today; wiring deno into the workflow is deliberately left out of this PR rather than adding an unpinned install step).

Verification

  • deno check packages/scripts/{gen-delivery-image-commands.ts,get-delivery-target-images.ts,lib/delivery-config.ts} → PASS
  • test_gen_delivery_image_commands → PASS (4 ✅, including the three rejections)
  • old script, same hostile input → exit 0 and a script containing master-$(id)-nextgen; new script → exit 2, no file written
  • get-delivery-target-images.ts --config=packages/delivery.yaml --version=v8.5.0 → identical output to the previous version
  • gh pr checks → pending (this PR)

Risk / Rollback

  • Risk: an environment where result-images.yaml carries a repo/tag outside the accepted charset now fails fast (exit 2) instead of emitting an unquoted command. Values produced by the current builds (us-docker.pkg.dev/..., master-nextgen, v8.5.0) pass.
  • Rollback: revert this commit; the scripts return to the previous behaviour.

Stack

…mands

gen-delivery-image-commands.ts built the delivery script by concatenating
repo, tag, dest_repositories, constant_tags and the rewritten tag into
`crane copy` / `crane tag` command lines. Nothing was validated or quoted:
parseImageRef only split on the last colon, the rules file was used as-is, and
every tags_regex was compiled with a bare `new RegExp`, so a permissive rule
such as `^(master|dedicated)(-.*)?-nextgen$` let a tag like
`master-\-nextgen` through and the emitted script — run by
the delivery job with push credentials for the public registries — executed it.

- move the rule types and validators into packages/scripts/lib/delivery-config.ts
  and use them from both readers of delivery.yaml
  (gen-delivery-image-commands.ts and get-delivery-target-images.ts)
- validate the source repo/tag, the destination repositories, the constant tags
  and the rewritten tag; refuse a bad value with exit 2 before writing anything
- quote every argument in the emitted commands
- refuse an output path that escapes through '..'
- add test_gen_delivery_image_commands to .github/scripts/ci.sh: type-check,
  three hostile inputs (metacharacter tag, injection in dest_repositories,
  quadratic regex) that must be refused without writing a script, and one valid
  case that must produce a quoted, POSIX-clean script

get-delivery-target-images.ts keeps its behaviour: same validation, same
host allowlist, byte-identical output for v8.5.0.
@wuhuizuo
wuhuizuo added this pull request to stack #1210 November 12, 2026 11:28

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This pull request improves the gen-delivery-image-commands.ts script by addressing quoting and validation issues in the generated shell commands for image delivery. The changes include introducing a shared validation module for delivery rules, enhancing input validation, and ensuring all shell commands are properly quoted to prevent injection attacks. The PR also adds tests for invalid inputs, ensuring robustness against edge cases. The changes are well-structured, with appropriate modularization, but there are a few areas for improvement.


Critical Issues

  1. Validation Output Path Check (Security Concern)
    File: gen-delivery-image-commands.ts (Line 86 in the new version)
    Issue: The check for invalid output paths (.. segments) could be bypassed with relative paths like dir/../output.sh.
    Suggestion: Normalize the path before checking for unsafe patterns. Use Deno.realPath or similar to resolve the absolute path and then validate.

    const resolvedPath = await Deno.realPath(outFile);
    if (resolvedPath.includes("..")) {
        fail(`refusing to write to output path "${outFile}"`);
    }
  2. Regex Length and Complexity Validation (Performance Concern)
    File: lib/delivery-config.ts (Line 63)
    Issue: The compileRuleRegex function stops "obviously quadratic" regexes but does not account for other complex regex patterns that could cause performance issues, such as catastrophic backtracking.
    Suggestion: Consider using a library like safe-regex to validate regex safety instead of relying on custom checks.

    import safeRegex from "safe-regex";
    if (!safeRegex(source)) {
        fail(`${where}: unsafe regular expression`);
    }

Code Improvements

  1. Error Handling Consolidation
    File: gen-delivery-image-commands.ts and lib/delivery-config.ts
    Issue: The fail function is used consistently for error reporting, but some error messages could benefit from additional context, such as the specific input file or line responsible for the error.
    Suggestion: Enhance the fail function to accept an additional context parameter and include it in the error messages.

    export function fail(message: string, context?: string): never {
        const fullMessage = context ? `❌ [${context}] ${message}` : `❌ ${message}`;
        console.error(fullMessage);
        Deno.exit(2);
    }
  2. Testing for Valid Input Cases
    File: .github/scripts/ci.sh (Lines 830-846 in the new version)
    Issue: The test only checks for specific patterns in the generated output script but doesn't verify the entire script structure.
    Suggestion: Use a golden file or a template-based approach to compare the entire output script with an expected result for valid input. For example:

    diff -q "$out" "expected_output.sh" || {
        echo "❌ delivery generator: output does not match expected result"
        exit 1
    }
  3. Avoiding Hardcoded Dependencies
    File: gen-delivery-image-commands.ts and get-delivery-target-images.ts
    Issue: The REPOSITORY_RE and TAG_RE patterns are defined in the shared module but are still hardcoded in other parts of the code. This creates potential for inconsistency if the patterns are updated in one place but not another.
    Suggestion: Ensure that all regex references are imported from the shared module to prevent divergence.


Best Practices

  1. Documentation for New Functions
    File: lib/delivery-config.ts
    Issue: While some functions are documented, others like assertStringArray lack clear documentation about their purpose and parameters.
    Suggestion: Add concise docstrings for all functions, especially exported ones. For example:

    /**
     * Validates that a value is a string array and optionally checks its length and content.
     * @param value - The value to validate.
     * @param where - Context of the validation (e.g., field name).
     * @param options - Validation options (minItems, pattern, label).
     * @returns The validated string array.
     */
    export function assertStringArray(...) { ... }
  2. Test Coverage for Edge Cases
    File: .github/scripts/ci.sh
    Issue: While hostile inputs are tested, edge cases such as empty rules or images with unexpected but valid formats are not covered.
    Suggestion: Add test cases for:

    • Empty image_copy_rules or images.
    • Valid but unusual repository formats (e.g., example.com:1234/repo).
    • Excessively long but valid tags and repository names.
  3. Avoiding Duplicated Logic
    File: gen-delivery-image-commands.ts and lib/delivery-config.ts
    Issue: Functions like assertImageRef share similar validation logic for repositories and tags as in validateDeliveryConfig.
    Suggestion: Consolidate validations into shared functions. For example, assertImageRef could delegate to the regex checks in validateDeliveryConfig.


Additional Notes

  • Deno Dependency Management: Ensure the Deno dependencies (e.g., jsr:@std) are pinned to specific versions to avoid breaking changes from upstream updates.
  • Error Exit Codes: Consider using distinct exit codes for different error types (e.g., validation failure vs. file read error) to simplify debugging in CI pipelines.

Overall, the PR is a significant improvement in both security and maintainability, but addressing the above points will further enhance robustness and adherence to best practices.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/approve

Delivery generator validates and quotes every crane argument; hostile tag / destination / rule regex are rejected with exit 2 and no script, valid input yields a quoted script. test-packages-cfg and zizmor pass; get-delivery-target-images.ts output is byte-identical to the previous version. (user confirmed 2026-11-12)

@ti-chi-bot

ti-chi-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 12, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit 58fe713 into main Nov 12, 2026
9 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the fix/delivery-generator-input-validation-p2 branch November 12, 2026 11:59
ti-chi-bot Bot pushed a commit that referenced this pull request Nov 12, 2026
## Problem

Nine pull-request-triggered jobs requested `packages: write` — the
`GITHUB_TOKEN` scope that publishes to `ghcr.io/pingcap-qe/*` (bases,
builders, product runtimes) — although every build inside them runs with
`--push=false` and needs no push scope:

- `pull-cd-builder-images.yaml` (3 jobs),
`pull-cd-builder-images-centos7.yaml` (3), `pull-cd-util-images.yaml`
(1), `pull-prod-runtime-images.yaml` (1).
- `pull-cd-builder-msb.yaml`: the `just-build` job held `packages:
write` + `pull-requests: write`, logged in to ghcr, and then ran `just
"$TASK"` against the **pull request's own** `dockerfiles/Justfile`. The
label allowlist gated only the task *name*; the recipe body came from
the PR. An actor able to add a label could therefore run an arbitrary
recipe in a runner holding publish credentials. (Its `MY_GITHUB_TOKEN`
was already removed in a previous change; the write scope and the login
were not.)

## Changes

- Delete `.github/workflows/pull-cd-builder-msb.yaml` — the
`just/<task>` preview path is gone. The multi-stage builds it exercised
are covered by the release builder workflows.
- Remove `packages: write` and the `docker/login-action` ghcr step from
the eight remaining PR jobs; they keep `contents: read`.
- Release workflows are untouched (they publish, so they keep the
scope).

`dockerfiles/Justfile` stays as a local development helper; it is no
longer executed by CI, so its recipe body is no longer PR-controlled in
a credentialed runner.

## Verification

- `grep -rn ": write" .github/workflows/pull-*.yaml` → none
- `actionlint` on the four edited workflows → only the pre-existing
`ubuntu-26.04` runner-label warnings and pre-existing SC2086 info notes
in the matrix step (identical before this change; the diff only deletes
lines)
- anonymous `crane manifest ghcr.io/pingcap-qe/cd/builders/tidb:<tag>` →
OK without login, so the builds can pull their base images with
`contents: read`
- `gh pr checks` → zizmor, pre-commit, license/cla pass. The builder
workflows are path-filtered to `dockerfiles/cd/builders/**`, so they do
not run on a workflow-only change; the first run that exercises the new
permissions is the next PR touching those paths. Until then the
anonymous-pull check above is the evidence that no login is needed.

## Risk / Rollback

- Risk: anonymous ghcr pulls are rate-limited per runner IP. If a build
starts hitting the pull limit, the fix is a **read-only** credential
(separate token, `packages: read`), not restoring `packages: write`.
- Risk: anyone relying on the `just/<task>` label to preview a
multi-stage build loses that trigger; `dockerfiles/Justfile` still
documents the local equivalent.
- Rollback: revert this commit (restores the workflow and the scopes).

## Stack

- Previous: #1209
- Next: offline-package artifact provenance (same series)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant