Skip to content

chore(ci): pin the scripts CI fetches at run time to commit SHAs - #5561

Merged
ti-chi-bot[bot] merged 2 commits into
mainfrom
fix/pin-runtime-scripts
Nov 12, 2026
Merged

ti-chi-bot[bot] merged 2 commits into
mainfrom
fix/pin-runtime-scripts

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Problem

Many jobs resolve a helper script at run time from a branch ref on jsDelivr
and execute it in a pod that holds credentials:

job fetched and executed credentials in scope
periodic-crawl-ci-run-data ee-apps@main/insight/crawlers/ci/prow-jobs.ts CI database DSN (env and argv)
periodic-reporter-flaky-tests-* ci@main/tools/reporters/ci/flaky-tests/main.ts org PAT, CI DB DSN, SMTP
release/hotfix Tekton tasks deno run --allow-all over ci@main + artifacts@main scripts release / publication credentials
pingcap/*, tikv/* postsubmits ci@main/scripts/plugins/*.ts GitHub token
pull_unit_test (7 branches) ci@main/scripts/tikv/tikv/gen_test_binary_json.py repo token
pull_tiflash_test (release-6.5) artifacts@main/dockerfiles/products/tidb/tidb.Dockerfile image build
tiflash-scripts / tici presubmits ci@main/scripts/artifacts/download_pingcap_oci_artifact.sh registry access

@main is resolved when the job runs, so a commit that lands on PingCAP-QE/ci,
PingCAP-QE/ee-apps or PingCAP-QE/artifacts changes what executes next to
those credentials immediately — the job definition gets reviewed, the code it
actually runs does not. Nothing is checked: there is no checksum, no --lock,
no deno.lock on these invocations.

Change

Pin all 37 URLs to a commit — today's head of each source repository:

  • PingCAP-QE/ci@10bcc6134c65… (26 URLs)
  • PingCAP-QE/ee-apps@ba65735ce3e9… (1)
  • PingCAP-QE/artifacts@01c93d68ad2c… (10)

30 files, no other change: every pinned URL serves exactly what the branch
served before, so job behaviour is identical today, and the code a job runs is
now the code that was in the reviewed commit.

When a script changes, the refs that name it move in the same commit that
changes it — the diff is the review.

Deliberately left on the branch ref

  • pingcap/docs@master/scripts/… and tikv/tikv@master/rust-toolchain.toml:
    upstream content consumed by upstream-owned jobs, one of them data rather than
    code.
  • artifacts@main/packages/delivery.yaml: data that drives which images a
    release delivers. Freezing that here would silently change delivery behaviour
    when a new artifact is registered upstream, and the task already exposes
    delivery-config as an override.

Verification

  • All 20 distinct pinned URLs return 200 from jsDelivr.
  • .ci/verify-prow-jobs-run.sh passes on the touched Prow job files, which
    includes parsing and syntax-checking every modified args block.

Rows of jobs pull a helper script off jsDelivr at run time and execute it inside
a pod that holds credentials:

- `periodic-crawl-ci-run-data` runs `ee-apps@main/insight/crawlers/ci/prow-jobs.ts`
  with the CI database DSN both in the environment and on the argv;
- `periodic-reporter-flaky-tests-*` runs `ci@main/tools/reporters/ci/flaky-tests/main.ts`
  with `--allow-env` while the org PAT and the SMTP credentials are set;
- the release/hotfix Tekton tasks run `deno run --allow-all` over `ci@main` and
  `artifacts@main` scripts with the publishing credentials mounted;
- the postsubmits in `pingcap/*` and `tikv/*` run `ci@main/scripts/plugins/*.ts`
  with a GitHub token, and the tiflash-scripts / tici presubmits run
  `ci@main/scripts/artifacts/download_pingcap_oci_artifact.sh`;
- the `pull_unit_test` jobs (seven release branches) wget
  `ci@main/scripts/tikv/tikv/gen_test_binary_json.py`, and
  `pull_tiflash_test` on release-6.5 builds an image from
  `artifacts@main/dockerfiles/products/tidb/tidb.Dockerfile`.

A branch ref is resolved at run time, in every job, against whatever the branch
holds at that moment: anyone who can land a commit on those repositories (or
influence the CDN response) decides what runs next to those credentials, without
the review that the job definition itself gets.

Pin all 37 URLs to a commit — the current head of each source repository
(`PingCAP-QE/ci@10bcc61`, `PingCAP-QE/ee-apps@ba65735ce3e9`,
`PingCAP-QE/artifacts@01c93d68ad2c`) — so the code a job runs is the code its
review approved.

Two kinds of URL are deliberately left on the branch ref:

- `pingcap/docs@master/scripts/…` and `tikv/tikv@master/rust-toolchain.toml` are
  upstream content consumed by upstream-owned jobs, one of them data rather than
  code;
- the `artifacts@main/packages/delivery.yaml` manifests stay live. They are data
  that drives which images a release delivers, and freezing them here would
  silently change delivery behaviour when a new artifact is added upstream —
  the task already exposes `delivery-config` as an override.

Verified: all 20 distinct pinned URLs return `200` from jsDelivr, and
`.ci/verify-prow-jobs-run.sh` passes on the touched Prow job files (including the
shell syntax of every modified `args` block).

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This pull request improves the security and consistency of CI/CD pipeline scripts by pinning runtime-fetched external scripts to specific commit SHAs. This ensures that the scripts executed during CI jobs are locked to reviewed versions, preventing potential security risks from unreviewed upstream changes. The implementation modifies 30 files to replace branch references (@main) in script URLs with specific commit SHAs, effectively freezing their versions. The change is well-documented and includes a verification process, but some minor improvements can be made.


Critical Issues

No critical issues were identified. The changes address a significant security concern by pinning specific commit SHAs for external scripts.


Code Improvements

  • Future Script Updates:

    • Files: All modified files
    • Issue: While pinning SHAs improves security, it also imposes a maintenance burden, as updates to scripts will require manually updating the SHAs in multiple locations for every change.
    • Suggestion: Consider implementing a script or process to automate SHA updates and verify integrity. For example, a CI job could:
      • Detect when a script dependency is updated in the source repository.
      • Automatically replace the SHA in dependent files.
      • Run a test suite to ensure compatibility.
  • Centralized Management of URLs:

    • Files: All modified files
    • Issue: URLs with pinned SHAs are scattered across multiple files, which could lead to inconsistencies if manual updates are required.
    • Suggestion: Consolidate URLs into a single configuration file (e.g., a JSON or YAML file) that the jobs reference. This would centralize management of script URLs and make future updates easier.

Best Practices

  • Testing for Script Changes:

    • Files: All modified files
    • Issue: While the PR locks script versions, there is no clear process for testing or verifying new script versions before updating SHAs.
    • Suggestion: Define a testing strategy for script updates. This could include:
      • Running impacted jobs in a staging environment with the new script version before updating the SHA.
      • Adding a checksum validation step to ensure the fetched script matches the reviewed commit.
  • Documentation on Updating SHAs:

    • Files: PR description and/or repository documentation (e.g., README or CONTRIBUTING file)
    • Issue: The process for updating SHAs when script changes are made is not explicitly documented.
    • Suggestion: Add documentation that explains:
      • How to update SHAs when upstream scripts are modified.
      • How to verify the integrity of new script versions (e.g., by comparing the script's hash to the commit's content).
  • Consider Using Package Managers:

    • Files: All modified files
    • Issue: Using curl and wget to fetch scripts from URLs can be error-prone and lacks dependency management features (e.g., versioning, dependency resolution).
    • Suggestion: For deno scripts, consider using deno.land with versioned imports (e.g., https://deno.land/x/[module]@[version]/[file].ts). For other scripts, consider packaging them in a dependency management solution like npm or pip.

Additional Suggestions

  • Checksum Validation:
    • Files: All modified files (scripts fetched using curl or wget)
    • Issue: The current implementation does not validate the integrity of fetched scripts. If the content of a pinned SHA is tampered with at the source, there is no mechanism to detect it.
    • Suggestion: Add checksum validation to ensure the script fetched matches the expected content. For example:
      curl -fsSL https://cdn.jsdelivr.net/gh/PingCAP-QE/ci@10bcc6134c6540cb7996f7313c96d0a6eadc2683/scripts/artifacts/download_pingcap_oci_artifact.sh -o "${oci_script}"
      echo "expected_checksum  ${oci_script}" | sha256sum -c -

This PR is a significant improvement in terms of security, but attention should be given to minimizing the maintenance burden and ensuring long-term manageability. Addressing the above concerns will further strengthen the approach.

@ti-chi-bot ti-chi-bot Bot added the size/M label Nov 12, 2026
… a branch

`serverless-builds.test.ts` asserted the exact `deno run` command of the
version-CLI step, including the `@main` ref — so pinning that step to a commit
turned the assertion red, and the assertion would have kept the branch ref in
place.

Assert the shape instead: the URL must carry a 40-character commit, which keeps
the test meaningful (an unpinned ref fails) without pinning the test to the SHA
it happens to see today.

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR addresses a critical security and reliability concern by pinning URLs for scripts fetched during CI/CD jobs to specific commit SHAs, ensuring the exact version of code being executed is reviewed and stable. The approach involves replacing branch references (e.g., @main) with specific commit SHAs across 30 files, affecting 37 URLs. This change adds significant security benefits by eliminating the risk of unreviewed or unintended changes being executed during runtime. The implementation is clear and straightforward, maintaining backward compatibility by not altering job behavior. Overall, the PR is well-executed but has a few minor areas for improvement.


Critical Issues

None identified. This PR effectively mitigates the described security vulnerability without introducing new risks or regressions.


Code Improvements

  • Avoiding Hardcoding SHAs for Maintainability
    • Files: All affected files (e.g., jenkins/jobs/*, tekton/v1/tasks/*, prow-jobs/*).
    • Issue: Hardcoding SHAs directly into multiple files can make future updates cumbersome and error-prone, as every SHA update will require manual changes across all files.
    • Suggestion: Use a single central configuration file to store the commit SHAs (e.g., .ci/commit-shas.yaml) and reference it in all scripts. For example:
      ci_scripts:
        download_oci_artifact: "10bcc6134c6540cb7996f7313c96d0a6eadc2683"
        gen_test_binary_json: "10bcc6134c6540cb7996f7313c96d0a6eadc2683"
      Update scripts to dynamically fetch SHAs from this file, reducing maintenance overhead.

Best Practices

  • Testing Coverage for Locked URLs

    • File: tekton/tests/serverless-builds.test.ts (and potentially other test files).
    • Issue: While the use of assertMatch ensures the pinned SHA format is correct, there are no additional tests to verify that the URLs are accessible or fetch the expected content.
    • Suggestion: Add an integration test to periodically validate that all pinned URLs are live and return the correct content. Example:
      Deno.test("Verify pinned URLs are accessible", async () => {
        const urls = [
          "https://cdn.jsdelivr.net/gh/PingCAP-QE/ci@10bcc6134c6540cb7996f7313c96d0a6eadc2683/scripts/artifacts/download_pingcap_oci_artifact.sh",
          // Add other URLs here
        ];
        for (const url of urls) {
          const response = await fetch(url);
          assertEquals(response.status, 200, `Failed to fetch ${url}`);
        }
      });
  • Error Handling for Script Fetch Failures

    • Files: Multiple files (e.g., jenkins/jobs/*, tekton/v1/tasks/*, prow-jobs/*).
    • Issue: Many wget and curl commands lack robust error handling, which could cause downstream issues if a script fails to download or is corrupted.
    • Suggestion: Add checksum verification for downloaded scripts to ensure integrity. For example:
      EXPECTED_HASH="abc123..."
      curl -fsSL "https://cdn.jsdelivr.net/gh/PingCAP-QE/ci@10bcc6134c6540cb7996f7313c96d0a6eadc2683/scripts/artifacts/download_pingcap_oci_artifact.sh" -o "${oci_script}"
      echo "${EXPECTED_HASH} ${oci_script}" | sha256sum -c -
  • Document the Change Process for SHAs

    • File: Repository README or .ci/README.md.
    • Issue: The process for updating the pinned SHAs is not described, which could lead to inconsistencies or mistakes during future updates.
    • Suggestion: Add a section to the documentation detailing:
      • How to update the SHAs.
      • The expected steps for verification (e.g., running tests, checksum validation).
      • How to regenerate the central configuration file (if implemented).
  • Use of Explicit Retry Logic

    • Files: Scripts where curl or wget is used (e.g., scripts/pingcap/tidb-binlog/pull_integration_test.sh).
    • Issue: While curl and wget retry logic is sometimes included (--retry 3), it is inconsistently applied.
    • Suggestion: Standardize the retry logic for all download commands to ensure resilience against transient network issues. Example:
      curl --retry 5 --retry-delay 5 -fsSL ...

Conclusion

This PR resolves a critical vulnerability effectively, ensuring CI/CD jobs execute only reviewed and approved code. While the implementation is sound, introducing centralized SHA management, improved error handling, and additional tests would enhance maintainability and robustness.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/approve

@ti-chi-bot

ti-chi-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 12, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit 1d837e2 into main Nov 12, 2026
14 of 15 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the fix/pin-runtime-scripts branch November 12, 2026 11:32
ti-chi-bot Bot pushed a commit that referenced this pull request Nov 12, 2026
…5565)

The version-CLI step runs `scripts/flow/build/versioning-strategy.ts`
from a
commit-pinned URL (added in #5561), so the fix in #5564 — a released
version now
requires a tag trigger — only reaches Tekton once the pin moves to it.

Move the pin from `10bcc6134c65…` to `32e6febe3b96…`, the merge commit
of #5564.

Verified: the pinned URL returns `200` and serves the fixed script (it
contains
the `triggeredByTag` check), and `deno test -A tekton/tests/` passes (37
tests,
0 failures).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant