Repository navigation
fix(packages): fetch offline-package artifacts over TLS and verify their digest - #1214
Conversation
…eir digest The offline packages are shipped to customers, and the composer fetched their component binaries with a bare `wget` from whatever src.url the template declared: no scheme check, no host check and no expected digest. The `sha256sum` it printed was computed after the download and compared to nothing, so it attested the bytes received rather than the bytes expected. The one concrete instance, mydumper in the 6.1-7.5 bundles, resolved to http://fileserver.pingcap.net — plaintext, and the file is no longer there. - add assert_src_url to the shared validator: https only, host allowlist, no userinfo, no '..'; applied to every 'type: http' source (url and checksum_url) by the package, package-artifact and offline-package generators - the generated scripts refuse a non-https source at run time as well, and verify the digest published next to the artifact (checksum_url -> sha256sum -c) before the bytes are extracted or copied - point the four legacy mydumper entries at the release asset rebuilt from pingcap/mydumper@0042e179 plus its sha256sums.txt, which also removes the last fileserver.pingcap.net reference from this repository - cover the new gate with negative and positive cases in verify-input-validation.sh
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR addresses a critical security gap in fetching offline-package artifacts by enforcing TLS for downloads, restricting allowed hosts, and adding SHA-256 digest verification. The approach involves introducing a new assert_src_url function in the validate.sh library, updating generator scripts to enforce secure downloads and digest verification, and modifying existing configurations to point to secure, immutable release assets. The implementation is thorough, with added validation, testing, and clearly documented changes. Overall, the PR is high-quality, but a few areas can be improved for maintainability and robustness.
Critical Issues
-
[File:
lib/validate.sh, Lines: 116-120]- Issue: The
assert_src_urlfunction does not properly handle URLs with query parameters that include... For example,https://example.com/file?path=../../etc/passwdwould pass thecasecheck but could still lead to directory traversal attacks. - Solution: Update the
casepattern to check for..anywhere in the URL string, not just the path.case "$url" in *..*) _validate_fail src_url "$url" "contains invalid '..'" ;; esac
- Issue: The
-
[File:
build-package-artifacts.sh.tmpl, Lines: 7-21]- Issue: The
wgetcommand retries three times for both the source file and checksum URL. If thewgetfor the checksum URL fails, the script exits immediately without cleaning up the partially downloaded source file, potentially leaving corrupt files. - Solution: Add cleanup logic to remove any partially downloaded files on failure.
trap 'rm -f "{{ path.Base .src.url }}"' ERR
- Issue: The
Code Improvements
-
[File:
build-package-artifacts.sh.tmpl, Lines: 14-17]- Issue: The
awkcommand used to extract the checksum does not handle cases where the file may be listed multiple times in thesha256sums.txtfile. - Solution: Add a stricter check to ensure the checksum is extracted unambiguously.
expected_sha256=$(wget --tries=3 -qO- "{{ .src.checksum_url }}" | awk -v f="{{ path.Base .src.url }}" '$2 == f { print $1 }' | head -n 1)
- Issue: The
-
[File:
gen-package-artifacts-with-config.sh, Lines: 101-107]- Issue: The while loop iterates over both
urlandchecksum_urlbut does not distinguish between them in error messages. This could make debugging harder if either fails validation. - Solution: Add context to the error messages to clarify whether the issue is with
urlorchecksum_url.while IFS= read -r src_url; do assert_src_url "$src_url" || echo "Validation failed for source: $src_url" done
- Issue: The while loop iterates over both
Best Practices
-
[File:
validate.sh, Lines: 117-118]- Issue: The
ALLOWED_SRC_HOSTSvariable is hardcoded in the script, making it less flexible for different environments. - Solution: Move
ALLOWED_SRC_HOSTSto a configuration or environment file, and load it dynamically in the script.: "${ALLOWED_SRC_HOSTS:=$(cat /path/to/allowed_hosts.txt)}"
- Issue: The
-
[File:
verify-input-validation.sh, Lines: 189-195]- Issue: The test for rejecting plaintext sources relies on modifying the template file directly, which introduces a risk of accidental template corruption during testing.
- Solution: Use a separate mock template for testing invalid cases instead of modifying the production template.
-
[File:
offline-packages.yaml.tmpl, Multiple Lines]- Issue: The comments explaining the changes reference specific GitHub tags and URLs. While helpful now, these may become outdated if the release assets are moved or updated.
- Solution: Extract these references into a centralized documentation file or README for easier future updates.
Suggested Testing Enhancements
-
Checksum Verification: Add tests for edge cases like:
- Missing
sha256sums.txtfile. - Mismatched file names in
sha256sums.txt. - Multiple entries for the same file in
sha256sums.txt.
- Missing
-
Host Allowlist: Add tests to verify that URLs with subdomains (e.g.,
cdn.github.com) or similar host names (e.g.,github-example.com) are correctly rejected.
Conclusion
This PR significantly improves the security and reliability of fetching offline-package artifacts. The changes are well-tested and well-documented, but addressing the issues above will enhance edge case handling, script robustness, and maintainability. Prioritize fixing the critical issues related to directory traversal and partial downloads before merging.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Problem
The offline packages are shipped to customers, and their composer fetched component binaries with a bare
wget {{ .src.url }}: no scheme check, no host check, no expected digest. Thesha256sumit wrote was computed after the download and compared to nothing, so it attested the bytes received rather than the bytes expected.The one concrete instance is
mydumperin the 6.1–7.5 bundles:http://fileserver.pingcap.net/download/builds/pingcap/mydumper/0042e179…/centos7/mydumper-linux-amd64.tar.gz— plaintext, and the object is no longer on the file server (NoSuchKey), so those rebuilds could not fetch it at all.Changes
lib/validate.sh: newassert_src_url— https only, host allowlist (github.com,objects.githubusercontent.com,cdn.jsdelivr.net,tiup-mirrors.pingcap.com), no userinfo, no...gen-package-images-with-config.sh,gen-package-artifacts-with-config.sh,gen-package-offline-package-with-config.sh) apply it to everytype: httpsource (urlandchecksum_url).type: ocisources are registry references, not URLs, and are untouched.checksum_url→ look the archive's basename up insha256sums.txt→sha256sum -c→ only then extract/copy. A missing checksum entry or a mismatch aborts.mydumperentries now point at the release asset rebuilt frompingcap/mydumper@0042e179plus itssha256sums.txt. This also removes the lastfileserver.pingcap.netreference from this repository.verify-input-validation.sh: negative cases for the new validator and for a plaintextsrc.urlin a real template, plus a positive case asserting the generated script carries both the https guard and the digest check.The release asset itself: https://github.com/pingcap/mydumper/releases/tag/build-0042e179 — same source commit, rebuilt on CentOS 7 (glibc 2.17), statically linked, reproducible (two clean builds produce identical
sha256). That repository's tree is unchanged; the tag only points at the existing commit.Verification
./packages/scripts/verify-input-validation.sh→ PASS (all cases, including the plaintext-source rejection and the digest-check assertions)shellcheck -S error packages/scripts/*.sh packages/scripts/lib/*.sh→ PASSv6.5.0/community parses (bash -n) and carries, formydumper:case "https://github.com/…/mydumper-linux-amd64.tar.gz" in https://*) ;; *) refuse+sha256sum -c -4eecdceb…),mydumperextracted to the archive dir,mydumper --version→mydumper 0.9.5 (0042e179…), built against MySQL 5.7.44http://…→ exits 1 withrefusing to fetch 'mydumper' from a non-https sourcegrep -rn "fileserver.pingcap.net"→ nonegh pr checks→ pending (this PR)Risk / Rollback
type: httpsource on a host outside the allowlist now fails at generation time instead of downloading; the allowlist is one variable (ALLOWED_SRC_HOSTS) at the top oflib/validate.sh.gh release delete build-0042e179 --cleanup-tag.Stack