Skip to content

fix(packages): fetch offline-package artifacts over TLS and verify their digest - #1214

Merged
ti-chi-bot[bot] merged 1 commit into
mainfrom
fix/offline-package-artifact-provenance-p6
Nov 13, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
mainfrom
fix/offline-package-artifact-provenance-p6

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Problem

The offline packages are shipped to customers, and their composer fetched component binaries with a bare wget {{ .src.url }}: no scheme check, no host check, no expected digest. The sha256sum it wrote was computed after the download and compared to nothing, so it attested the bytes received rather than the bytes expected.

The one concrete instance is mydumper in the 6.1–7.5 bundles: http://fileserver.pingcap.net/download/builds/pingcap/mydumper/0042e179…/centos7/mydumper-linux-amd64.tar.gz — plaintext, and the object is no longer on the file server (NoSuchKey), so those rebuilds could not fetch it at all.

Changes

  • lib/validate.sh: new assert_src_url — https only, host allowlist (github.com, objects.githubusercontent.com, cdn.jsdelivr.net, tiup-mirrors.pingcap.com), no userinfo, no ...
  • The three generators (gen-package-images-with-config.sh, gen-package-artifacts-with-config.sh, gen-package-offline-package-with-config.sh) apply it to every type: http source (url and checksum_url). type: oci sources are registry references, not URLs, and are untouched.
  • Generated scripts refuse a non-https source at run time as well, and verify the digest published next to the artifact: checksum_url → look the archive's basename up in sha256sums.txt → sha256sum -c → only then extract/copy. A missing checksum entry or a mismatch aborts.
  • The four legacy mydumper entries now point at the release asset rebuilt from pingcap/mydumper@0042e179 plus its sha256sums.txt. This also removes the last fileserver.pingcap.net reference from this repository.
  • verify-input-validation.sh: negative cases for the new validator and for a plaintext src.url in a real template, plus a positive case asserting the generated script carries both the https guard and the digest check.

The release asset itself: https://github.com/pingcap/mydumper/releases/tag/build-0042e179 — same source commit, rebuilt on CentOS 7 (glibc 2.17), statically linked, reproducible (two clean builds produce identical sha256). That repository's tree is unchanged; the tag only points at the existing commit.

Verification

  • ./packages/scripts/verify-input-validation.sh → PASS (all cases, including the plaintext-source rejection and the digest-check assertions)
  • shellcheck -S error packages/scripts/*.sh packages/scripts/lib/*.sh → PASS
  • generated script for v6.5.0/community parses (bash -n) and carries, for mydumper:
    case "https://github.com/…/mydumper-linux-amd64.tar.gz" in https://*) ;; *) refuse + sha256sum -c -
  • ran the generated block itself in a CentOS 7 container: asset downloaded, digest matched (4eecdceb…), mydumper extracted to the archive dir, mydumper --version → mydumper 0.9.5 (0042e179…), built against MySQL 5.7.44
  • same block with the URL rewritten to http://… → exits 1 with refusing to fetch 'mydumper' from a non-https source
  • grep -rn "fileserver.pingcap.net" → none
  • gh pr checks → pending (this PR)

Risk / Rollback

  • Risk: a future type: http source on a host outside the allowlist now fails at generation time instead of downloading; the allowlist is one variable (ALLOWED_SRC_HOSTS) at the top of lib/validate.sh.
  • Risk: the release asset is an immutable public artifact; if it ever has to change, publish a new tag rather than overwrite (documented in the vault runbook).
  • Rollback: revert this commit; the template returns to the previous source. The release can be removed with gh release delete build-0042e179 --cleanup-tag.

Stack

…eir digest

The offline packages are shipped to customers, and the composer fetched their
component binaries with a bare `wget` from whatever src.url the template
declared: no scheme check, no host check and no expected digest. The
`sha256sum` it printed was computed after the download and compared to nothing,
so it attested the bytes received rather than the bytes expected. The one
concrete instance, mydumper in the 6.1-7.5 bundles, resolved to
http://fileserver.pingcap.net — plaintext, and the file is no longer there.

- add assert_src_url to the shared validator: https only, host allowlist,
  no userinfo, no '..'; applied to every 'type: http' source (url and
  checksum_url) by the package, package-artifact and offline-package generators
- the generated scripts refuse a non-https source at run time as well, and
  verify the digest published next to the artifact (checksum_url -> sha256sum -c)
  before the bytes are extracted or copied
- point the four legacy mydumper entries at the release asset rebuilt from
  pingcap/mydumper@0042e179 plus its sha256sums.txt, which also removes the last
  fileserver.pingcap.net reference from this repository
- cover the new gate with negative and positive cases in
  verify-input-validation.sh

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR addresses a critical security gap in fetching offline-package artifacts by enforcing TLS for downloads, restricting allowed hosts, and adding SHA-256 digest verification. The approach involves introducing a new assert_src_url function in the validate.sh library, updating generator scripts to enforce secure downloads and digest verification, and modifying existing configurations to point to secure, immutable release assets. The implementation is thorough, with added validation, testing, and clearly documented changes. Overall, the PR is high-quality, but a few areas can be improved for maintainability and robustness.


Critical Issues

  • [File: lib/validate.sh, Lines: 116-120]

    • Issue: The assert_src_url function does not properly handle URLs with query parameters that include ... For example, https://example.com/file?path=../../etc/passwd would pass the case check but could still lead to directory traversal attacks.
    • Solution: Update the case pattern to check for .. anywhere in the URL string, not just the path.
      case "$url" in
        *..*) _validate_fail src_url "$url" "contains invalid '..'" ;;
      esac
  • [File: build-package-artifacts.sh.tmpl, Lines: 7-21]

    • Issue: The wget command retries three times for both the source file and checksum URL. If the wget for the checksum URL fails, the script exits immediately without cleaning up the partially downloaded source file, potentially leaving corrupt files.
    • Solution: Add cleanup logic to remove any partially downloaded files on failure.
      trap 'rm -f "{{ path.Base .src.url }}"' ERR

Code Improvements

  • [File: build-package-artifacts.sh.tmpl, Lines: 14-17]

    • Issue: The awk command used to extract the checksum does not handle cases where the file may be listed multiple times in the sha256sums.txt file.
    • Solution: Add a stricter check to ensure the checksum is extracted unambiguously.
      expected_sha256=$(wget --tries=3 -qO- "{{ .src.checksum_url }}" | awk -v f="{{ path.Base .src.url }}" '$2 == f { print $1 }' | head -n 1)
  • [File: gen-package-artifacts-with-config.sh, Lines: 101-107]

    • Issue: The while loop iterates over both url and checksum_url but does not distinguish between them in error messages. This could make debugging harder if either fails validation.
    • Solution: Add context to the error messages to clarify whether the issue is with url or checksum_url.
      while IFS= read -r src_url; do
          assert_src_url "$src_url" || echo "Validation failed for source: $src_url"
      done

Best Practices

  • [File: validate.sh, Lines: 117-118]

    • Issue: The ALLOWED_SRC_HOSTS variable is hardcoded in the script, making it less flexible for different environments.
    • Solution: Move ALLOWED_SRC_HOSTS to a configuration or environment file, and load it dynamically in the script.
      : "${ALLOWED_SRC_HOSTS:=$(cat /path/to/allowed_hosts.txt)}"
  • [File: verify-input-validation.sh, Lines: 189-195]

    • Issue: The test for rejecting plaintext sources relies on modifying the template file directly, which introduces a risk of accidental template corruption during testing.
    • Solution: Use a separate mock template for testing invalid cases instead of modifying the production template.
  • [File: offline-packages.yaml.tmpl, Multiple Lines]

    • Issue: The comments explaining the changes reference specific GitHub tags and URLs. While helpful now, these may become outdated if the release assets are moved or updated.
    • Solution: Extract these references into a centralized documentation file or README for easier future updates.

Suggested Testing Enhancements

  • Checksum Verification: Add tests for edge cases like:

    • Missing sha256sums.txt file.
    • Mismatched file names in sha256sums.txt.
    • Multiple entries for the same file in sha256sums.txt.
  • Host Allowlist: Add tests to verify that URLs with subdomains (e.g., cdn.github.com) or similar host names (e.g., github-example.com) are correctly rejected.


Conclusion

This PR significantly improves the security and reliability of fetching offline-package artifacts. The changes are well-tested and well-documented, but addressing the issues above will enhance edge case handling, script robustness, and maintainability. Prioritize fixing the critical issues related to directory traversal and partial downloads before merging.

@ti-chi-bot ti-chi-bot Bot added the size/L label Nov 13, 2026

@wuhuizuo wuhuizuo left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@ti-chi-bot

ti-chi-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 13, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit 2b99683 into main Nov 13, 2026
4 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the fix/offline-package-artifact-provenance-p6 branch November 13, 2026 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant