Skip to content

fix(dockerfiles): stop granting the build user passwordless root - #5531

Merged
ti-chi-bot[bot] merged 6 commits into
mainfrom
fix/drop-passwordless-sudo
Nov 11, 2026
Merged

ti-chi-bot[bot] merged 6 commits into
mainfrom
fix/drop-passwordless-sudo

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Problem

dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile
append jenkins ALL=(ALL) NOPASSWD: ALL to /etc/sudoers. The images are what
the Jenkins agent fleet runs in, and 370 Jenkinsfiles execute the pull request's
own Makefile/build/test scripts inside them — so any external contributor
reaches uid 0 in the container with one sudo. That makes the pod policy's
runAsUser != 0 rule (.ci/verify-pod-privilege-policy.sh) cosmetic, and in the
pods allowlisted to run privileged while keeping the ServiceAccount token it
turns "container escape" into "node/API-server escalation".

Changes

  • Drop the sudoers line and the sudo package from both images.
  • Pre-create /var/lib/mysql owned by jenkins in both images.
  • The eight jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile
    pipelines used sudo mkdir/chmod/chown/cp to stage MySQL TLS fixtures there;
    they now copy them as the unprivileged user. That was the only consumer: no
    sudo call remains under jenkins/, libraries/ or scripts/.

Verification

  • .github/workflows/pull-ci-runtime-images.yaml rebuilds the base and agent
    images for both architectures on this PR — that proves the Dockerfile changes
    and the install -d line.
  • pull-replay-jenkins-pipelines replays the dm integration jobs, which exercise
    the TLS fixture staging and the test harness inside the image.

Risk

Any script inside a component repository that calls sudo will now fail
instead of silently escalating. That is the intended effect, but it can surface
as a dm integration test failure; the replay above is how we find out.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR removes the passwordless sudo access for the jenkins user in Docker images used by Jenkins agents, addressing a potential security vulnerability that allowed external contributors to escalate privileges to root. The change involves pre-configuring /var/lib/mysql with the correct permissions and modifying Jenkins pipelines to avoid using sudo. The overall implementation is sound, with clear explanations of the changes and associated risks. However, there are minor areas for improvement in documentation and validation to ensure robustness.


Critical Issues

No critical issues identified. The PR effectively mitigates the identified security vulnerability without introducing apparent bugs or regressions.


Code Improvements

  • File: dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile
    Issue: The removal of the sudo package is a significant change. While the PR description mentions that no sudo calls remain in the codebase, a runtime check or script validation during the build process to confirm the absence of sudo usage in dependent scripts would provide additional confidence.
    Suggestion: Add a validation step in the Dockerfile or CI workflow to ensure no scripts in the Jenkins pipelines call sudo.
    Example:

    RUN ! grep -r 'sudo' /path/to/your/scripts || (echo "Error: sudo usage detected" && exit 1)
  • File: jenkins/jobs/pingcap/tiflow/*/Jenkinsfile
    Issue: The removal of sudo commands simplifies the scripts, but the reliance on /var/lib/mysql being pre-configured in the image introduces a hidden dependency. If this setup changes in the future, it could lead to failures.
    Suggestion: Add an explicit check in the Jenkins pipelines to validate the existence and ownership of /var/lib/mysql before proceeding with the tests.
    Example:

    if [ ! -d /var/lib/mysql ] || [ "$(stat -c '%U' /var/lib/mysql)" != "jenkins" ]; then
        echo "Error: /var/lib/mysql is not correctly configured."
        exit 1
    fi

Best Practices

  • File: dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile
    Issue: The install -d command is used to create /var/lib/mysql with specific permissions, but this could benefit from a comment explaining why this directory is pre-created (e.g., for MySQL TLS fixtures).
    Suggestion: Add a brief comment to clarify the purpose of this setup.
    Example:

    # Pre-create /var/lib/mysql with appropriate permissions to support MySQL TLS fixtures
    install -d -o jenkins -g jenkins -m 0755 /var/lib/mysql
  • File: All modified Jenkinsfile scripts
    Issue: The current comments explaining the removal of sudo are helpful but could be more explicit about the security implications and why this change was necessary.
    Suggestion: Update the comments to highlight the security rationale behind removing sudo.
    Example:

    // The /var/lib/mysql directory is pre-created and owned by the jenkins user in the CI image.
    // This change removes the need for sudo and avoids potential privilege escalation vulnerabilities.
  • Testing Coverage
    Issue: While the PR mentions replaying the affected pipelines, there is no mention of testing for other dependent scripts that may invoke sudo.
    Suggestion: Expand testing coverage to include any other scripts or workflows that might rely on sudo or root privileges, even if they are not part of the immediate changes in this PR.


Conclusion

The PR effectively addresses a significant security concern with minimal disruption to existing functionality. The changes are well-documented and tested within the scope of the updated pipelines. To further enhance the robustness of this change, consider adding validations to detect sudo calls during the build process and in the Jenkins pipelines. Additionally, improving comments and documentation will make the rationale for these changes clearer to future developers.

@ti-chi-bot ti-chi-bot Bot added the size/L label Nov 11, 2026
Both Jenkins agent images appended `jenkins ALL=(ALL) NOPASSWD: ALL` to
`/etc/sudoers`. That made the pod policy's `runAsUser != 0` rule cosmetic: 370
Jenkinsfiles execute the pull request's own build and test scripts inside these
images, and any of them reached uid 0 with one `sudo` (and in the pods that are
allowlisted to run privileged while keeping the ServiceAccount token, that is a
node/API-server escalation rather than a contained build).

- Drop the sudoers line and the `sudo` package from both images.
- Pre-create `/var/lib/mysql` owned by `jenkins` so the one in-repo consumer can
  drop its root steps: the eight `pull_dm_integration_test*` pipelines used
  `sudo mkdir/chmod/chown/cp` to stage MySQL TLS fixtures there. They now copy
  the fixtures as the unprivileged user; the directory is created by the image.

No `sudo` call remains under `jenkins/`, `libraries/` or `scripts/`, so nothing in
this repository needs root inside the container.
@wuhuizuo
wuhuizuo force-pushed the fix/drop-passwordless-sudo branch from c40b1e2 to 43a4989 Compare November 11, 2026 07:16
@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR removes passwordless root access for the jenkins user in Docker images used by Jenkins agents, thereby addressing a significant security vulnerability that allowed containerized Jenkins jobs to escalate privileges to uid 0. The approach includes dropping the sudo package, pre-creating necessary directories with proper ownership, and updating Jenkins pipelines to operate without sudo. The changes are well-structured and thoroughly documented, with verification steps outlined. Overall, the PR significantly improves security without introducing unnecessary complexity, though minor improvements can be made to enhance maintainability.


Critical Issues

  • None Identified: The PR effectively addresses the stated security issue. No breaking changes or regressions were observed in the provided diff.

Code Improvements

  1. Directory Creation in Dockerfiles:

    • File: dockerfiles/ci/jenkins/Dockerfile (lines 8-9), dockerfiles/ci/jenkins/tikv/Dockerfile (lines 8-9)
    • Issue: Hardcoding /var/lib/mysql creation in the Dockerfile increases coupling between the image and specific scenarios like MySQL TLS fixture staging.
    • Suggestion: Consider parameterizing the directory creation or documenting its purpose more clearly. Alternatively, create this directory dynamically in the relevant test scripts to decouple the image from specific test needs.
    # Add a comment explaining why /var/lib/mysql is pre-created in the image.
    install -d -o jenkins -g jenkins -m 0755 /var/lib/mysql
  2. Pipeline Error Handling:

    • File: Multiple Jenkinsfile (e.g., jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile line 159)
    • Issue: The updated cp command assumes that TLS fixture files always exist in ${WORKSPACE}/mysql-ssl/*.pem. If the directory is empty or missing, the command will fail.
    • Suggestion: Add a check to ensure the source files are present before attempting the cp.
    if [ -n "$(ls -A ${WORKSPACE}/mysql-ssl/*.pem 2>/dev/null)" ]; then
        cp -r ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/
    else
        echo "No TLS fixtures found to copy."
    fi

Best Practices

  1. Testing for Unexpected sudo Use:

    • File: Not directly in the diff, but relevant to the broader PR impact.
    • Issue: The PR changes the baseline assumption for Jenkins jobs by removing sudo. While the replay verifies existing pipelines, it doesn't protect against regressions where new sudo calls might be introduced inadvertently in future PRs.
    • Suggestion: Implement static analysis or CI checks to detect any new sudo calls in Jenkins pipeline scripts or other CI-related files.
  2. Documentation on Behavior Change:

    • File: Not included in the diff but relevant to maintainability.
    • Issue: The removal of sudo and the requirement for non-root operations represent a significant behavior change. This may catch future developers off guard.
    • Suggestion: Update any relevant README or documentation for the CI images and Jenkins pipelines to highlight the new restrictions and explain how to work within them.
  3. Cleanup of Unused Variables or Comments:

    • File: dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile
    • Issue: While the explanatory comments are useful, some of them could be streamlined or rephrased for conciseness.
    • Suggestion: Consider revising comments for clarity and conciseness.
    # The `jenkins` user is created without sudo privileges to prevent privilege escalation
    # in the container by external contributors. The user is intended for Jenkins agent operations only.

Summary of Recommendations

  • Consider decoupling /var/lib/mysql creation from the Dockerfiles or enhance documentation to clarify its purpose.
  • Add error handling in pipeline scripts to avoid failures if mysql-ssl fixtures are missing.
  • Implement static analysis or CI checks to prevent new sudo calls from being introduced in the future.
  • Update documentation to reflect the new behavior of the Jenkins images and pipelines.
  • Streamline comments where necessary to improve readability.

These suggestions aim to future-proof the changes, enhance maintainability, and ensure that the new security model is respected across future contributions.

Removing passwordless sudo exposed that the dm integration pipelines prepared
`/var/lib/mysql` as root for a reason: the directory is the mysql sidecar's
volume, owned by the mysql user, and the certificates copied into the workspace
are 0600 and owned by that same user. The unprivileged agent could neither write
the directory nor read the copies, so TLS_GROUP failed and its fail-fast took the
other groups down with it (replay build 268).

Keep the flow root-free by using the owner of each side:

- the prepare step (`container("mysql1")`) makes the workspace copies readable
  (`chmod -R a+rX`) instead of relying on a later `chown` from the agent;
- the TLS group stages the certificates from inside `container("mysql1")`, which
  owns `/var/lib/mysql`, instead of `sudo cp` from the agent container.

No `sudo` remains, and the image pre-creates no directory it does not own.
The dm TLS fixtures are staged from the mysql sidecar, which owns that volume, so
pre-creating the directory in the agent image would only mislead: an emptyDir
mounts over it anyway.
@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR removes the use of passwordless sudo for the jenkins user in Docker images, effectively preventing privilege escalation by pull request build scripts. It also refactors the staging of MySQL TLS fixtures to work without sudo, ensuring compatibility with unprivileged containers. The changes are well-documented and follow a security-first approach. Overall, the PR significantly improves the security posture of the CI environment, but there are a few areas for improvement in terms of code consistency, readability, and potential edge cases.


1. Critical Issues

  • File: jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile (and similar files for other versions)
    • Issue: The chmod -R a+rX on ${WORKSPACE}/mysql-ssl ensures readability but might inadvertently expose sensitive files to unintended users if additional files are added to this directory in the future.
    • Why it’s an issue: This could lead to a security vulnerability if other scripts or jobs add files to this directory with sensitive information.
    • Suggested Solution:
      Instead of a blanket chmod -R a+rX, explicitly set permissions on the required .pem files:
      chmod 0644 ${WORKSPACE}/mysql-ssl/*.pem

2. Code Improvements

  • File: dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile

    • Issue: The install -d -o jenkins -g jenkins -m 0755 /var/lib/mysql line is directly coupled to the Jenkins user's configuration, which could lead to maintenance challenges if the user ID or configuration changes.
    • Why it’s an issue: Hardcoding user/group IDs (jenkins and 1000) makes the setup less flexible and harder to adapt to future changes.
    • Suggested Solution:
      Use environment variables or arguments to make the configuration more dynamic:
      ARG JENKINS_USER=jenkins
      ARG JENKINS_UID=1000
      ARG JENKINS_GID=1000
      
      RUN groupadd -g $JENKINS_GID $JENKINS_USER && \
          useradd -u $JENKINS_UID -g $JENKINS_GID -m -s /bin/bash $JENKINS_USER && \
          install -d -o $JENKINS_USER -g $JENKINS_USER -m 0755 /var/lib/mysql
  • File: jenkins/jobs/.../Jenkinsfile (all modified versions)

    • Issue: The repeated logic for staging MySQL TLS certs (cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/) is duplicated across multiple files.
    • Why it’s an issue: Code duplication increases maintenance overhead and the risk of inconsistencies.
    • Suggested Solution:
      Extract the shared logic into a reusable function or script:
      def stageMysqlCerts() {
          sh label: "Stage MySQL TLS certs", script: """
              cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/
              ls -alh /var/lib/mysql/
          """
      }

3. Best Practices

  • File: jenkins/jobs/.../Jenkinsfile (all modified versions)

    • Issue: Some parts of the pipeline scripts are tightly coupled to specific container names (e.g., mysql1).
    • Why it’s an issue: This reduces flexibility when reusing the pipeline for different setups or container configurations.
    • Suggested Solution:
      Define container names as variables at the top of the script so that they can be easily updated:
      def mysqlContainer = "mysql1"
      container(mysqlContainer) {
          sh // ...
      }
  • File: dockerfiles/ci/jenkins/Dockerfile and dockerfiles/ci/jenkins/tikv/Dockerfile

    • Issue: Comments explaining the removal of sudo are verbose and could be streamlined for clarity.
    • Why it’s an issue: Excessively detailed comments may reduce readability for future maintainers.
    • Suggested Solution:
      Simplify comments, focusing only on the most critical points:
      # Create unprivileged Jenkins user without sudo to prevent privilege escalation.
  • File: .github/workflows/pull-ci-runtime-images.yaml (referenced in the PR description)

    • Issue: The PR mentions that this workflow rebuilds images but does not explicitly confirm whether all affected image variations are tested.
    • Why it’s an issue: Without explicit validation, there’s a risk that some images might break unnoticed.
    • Suggested Solution:
      Confirm in the PR description or add to the CI script that all combinations of base images and architectures are tested.

4. Testing Coverage

  • Issue: The PR description mentions replaying dm_integration_test pipelines to verify changes, but there’s no mention of broader regression testing for other scripts or tasks in the Jenkins environment.
  • Why it’s an issue: Changes to the Dockerfiles could impact other scripts or jobs that are not tested by the specific replayed pipelines.
  • Suggested Solution:
    • Add a note in the PR description verifying whether other jobs in the Jenkins ecosystem were tested for compatibility.
    • Alternatively, include a fallback plan for monitoring and rolling back if issues arise post-deployment.

Conclusion

This PR addresses a critical security issue with clear and effective changes, but it can be improved by reducing code duplication, parameterizing hardcoded values, and refining permission handling for sensitive files. The verification plan is reasonable, but additional testing or documentation on broader impact would further strengthen the PR.

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR removes passwordless root access for the jenkins user in Docker images used by Jenkins agents, thereby addressing a significant security vulnerability. It also updates build scripts and pipeline configurations to operate without sudo, ensuring compatibility with the new unprivileged user setup. The approach is well-executed, with changes consistently applied across multiple Dockerfiles and Jenkinsfiles. The overall quality is solid, with clear improvements to security. However, there are opportunities to refine the implementation for readability and maintainability.


Critical Issues

None identified. The PR adequately addresses the security risk of passwordless sudo access and ensures compatibility across affected pipelines.


Code Improvements

  • File: Jenkinsfiles (multiple locations)

    • Issue: The repeated block for copying and staging MySQL TLS certificates introduces duplication across Jenkinsfiles for different branches/releases.
    • Why it's an issue: Duplication increases maintenance overhead and the risk of inconsistent updates in future changes.
    • Suggested solution: Extract the repeated logic into a reusable library function or a shared script. For example:
      // Shared logic for TLS cert handling
      def stageMysqlTlsCerts(workspace) {
          sh """
              mkdir -p ${workspace}/mysql-ssl
              cp -r /var/lib/mysql/*.pem ${workspace}/mysql-ssl/
              chmod -R a+rX ${workspace}/mysql-ssl
              ls -alh ${workspace}/mysql-ssl/
          """
      }
      
      // Usage in Jenkinsfile
      container("mysql1") {
          stageMysqlTlsCerts(WORKSPACE)
      }
      This would simplify the Jenkinsfiles and ensure consistent behavior across pipelines.
  • File: Dockerfiles (dockerfiles/ci/jenkins/Dockerfile, dockerfiles/ci/jenkins/tikv/Dockerfile)

    • Issue: The /var/lib/mysql directory is pre-created in the Dockerfiles but lacks clear documentation on the rationale.
    • Why it's an issue: It may cause confusion for future maintainers regarding why this directory is initialized upfront.
    • Suggested solution: Add a comment explicitly stating the purpose of pre-creating this directory, e.g.:
      # Pre-create /var/lib/mysql for staging MySQL TLS fixtures during Jenkins pipelines

Best Practices

  • File: Jenkinsfiles (multiple locations)

    • Issue: The inline comments explaining the removal of sudo use are helpful but could be centralized in the shared function/script suggested above.
    • Why it's an issue: Inline comments in multiple places can lead to inconsistencies if the explanation needs updating later.
    • Suggested solution: Add a single, detailed comment in the shared function or script to document why sudo was removed and how the new approach ensures security.
  • Testing Coverage

    • Issue: The PR description mentions replaying Jenkins pipelines for verification, but no automated tests are included for validating the removal of sudo and ensuring compatibility with the unprivileged user setup.
    • Why it's an issue: Reliance solely on manual testing can lead to missed edge cases in future updates.
    • Suggested solution: Introduce automated tests to validate the Docker images and pipeline behavior. For instance:
      • A test to ensure Jenkinsfiles do not contain sudo commands.
      • A container-level test to verify that jenkins cannot escalate privileges.
  • Naming Conventions

    • Issue: The variable WORKSPACE is used inconsistently in scripts.
    • Why it's an issue: Naming conventions should be consistent for clarity and maintainability.
    • Suggested solution: Ensure consistent capitalization or adopt a standard naming convention across scripts, e.g., workspaceDir.

Overall Assessment

The PR effectively addresses a critical security vulnerability and ensures compatibility with the updated setup. The changes are methodical and well-documented. Refining the code to reduce duplication, adding automated tests, and improving documentation will further enhance the robustness and maintainability of the changes.

A declarative pipeline only accepts steps in 'steps'; the conditional has to be
wrapped in 'script', which the Jenkinsfile validator reported as
'Expected a step @ line 193'.
@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-verify-jenkins-pipelines

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

The PR addresses a critical security issue by removing passwordless root access for the jenkins user in Docker images, ensuring that external contributors cannot escalate privileges within Jenkins agents. The approach involves removing the sudo package, dropping the corresponding /etc/sudoers entry, and refactoring scripts to avoid sudo commands. Overall, the changes are well-justified and effectively improve container security without introducing unnecessary complexity. However, some minor improvements can enhance maintainability and clarity.


Critical Issues

  • None identified. The changes effectively mitigate the security vulnerability without compromising functionality.

Code Improvements

  1. Error Handling for File Operations

    • Files:
      jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile (and other affected Jenkinsfiles)
      • Problem: Operations like cp and chmod assume success but lack error handling. If these fail (e.g., due to missing files), the pipeline may silently proceed to incorrect behavior or fail later with unrelated errors.
      • Suggested Fix: Add error handling to critical file operations to surface issues earlier. For example:
        sh label: "stage mysql TLS certs", script: """
            set -e
            cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/ || { echo 'Failed to copy certs'; exit 1; }
            chmod -R a+rX ${WORKSPACE}/mysql-ssl || { echo 'Failed to adjust permissions'; exit 1; }
            ls -alh /var/lib/mysql/
        """
  2. Pre-Creation of /var/lib/mysql in Dockerfiles

    • Files:
      dockerfiles/ci/jenkins/Dockerfile, dockerfiles/ci/jenkins/tikv/Dockerfile
    • Problem: While /var/lib/mysql is pre-created in the images, the PR does not explicitly show where or how this happens. This may cause confusion for future maintainers.
    • Suggested Fix: Add a direct RUN mkdir -p /var/lib/mysql && chown jenkins:jenkins /var/lib/mysql line in the Dockerfiles for clarity:
      RUN mkdir -p /var/lib/mysql && chown jenkins:jenkins /var/lib/mysql

Best Practices

  1. Consistency in Comments

    • Files:
      Various Jenkinsfiles (e.g., jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile)
    • Problem: Comments explaining why chmod -R a+rX or similar steps are necessary are inconsistent across files. Consistent documentation helps future contributors understand why specific steps are included.
    • Suggested Fix: Use a uniform comment format, e.g.:
      # Ensure the agent user can read these files without root privileges.
      chmod -R a+rX ${WORKSPACE}/mysql-ssl
  2. Use of mkdir -p in Workspace Paths

    • Files:
      Various Jenkinsfiles (e.g., jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile)
    • Problem: While mkdir -p is correctly used in most cases, some earlier versions of the Jenkinsfile diff show inconsistencies.
    • Suggested Fix: Ensure all directory creations use mkdir -p for idempotence.
  3. Testing Coverage

    • Files: Not directly in the PR, but related to downstream impact.
    • Problem: The PR mentions potential risks from component repository scripts calling sudo. While the replay verification partially mitigates this, there is no explicit mention of automated tests to catch such failures in other pipelines.
    • Suggested Fix: Add a pre-check to identify sudo usage in downstream repositories' scripts, e.g., a static analysis or a grep-based validation step.
  4. Documentation of Risk Impact

    • Files: PR description.
    • Problem: While the description notes the risk of failures in component repositories due to removed sudo, it lacks concrete suggestions for identifying or resolving such issues.
    • Suggested Fix: Enhance documentation by specifying how contributors can detect and resolve such failures, e.g.,:
      • Provide a grep command to identify sudo usage in scripts.
      • Suggest fallback approaches for developers encountering failures.

Final Remarks

The PR is thorough and addresses a significant security vulnerability without introducing regressions. The actionable feedback mainly focuses on improving robustness, consistency, and maintainability, ensuring the changes are future-proof and well-documented. Addressing the above points will further enhance the quality and clarity of this important fix.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

The staging step compared `env.TEST_GROUP`, which the matrix axis does not set:
the step never ran, while the shell block it replaced compared the Groovy binding
(`\"${TEST_GROUP}\"`) and did run. Use the binding so TLS_GROUP keeps the
preconditions it had before the sudo removal.
@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This PR removes the sudo package and passwordless sudo permissions from Docker images used by Jenkins, thus addressing a critical security vulnerability where external contributors could escalate to root privileges within containers. The PR ensures that all impacted scripts and Jenkins pipelines are updated to function without requiring sudo. The implementation is thorough, with redundant sudo commands replaced by unprivileged alternatives, and verification steps outlined. Overall, the changes are well-executed, but a few areas could benefit from clarification or minor improvements.


Critical Issues

  • Risk of Unintended Failures in Unverified Scripts:
    • Files: Any external repository scripts executed within the Jenkins containers.
    • Issue: While the PR focuses on updating scripts under jenkins/jobs to remove sudo usage, any external repository scripts relying on sudo will now fail silently. This is acknowledged in the "Risk" section, but no automated detection mechanism is proposed.
    • Recommendation: Introduce logging or a fallback mechanism in the images to detect and alert on any sudo usage in external scripts. For example:
      alias sudo='echo "Error: sudo is disabled in this container." >&2; exit 1'
      Add this to the Dockerfile to explicitly fail on sudo usage.

Code Improvements

  1. Privilege Management in /var/lib/mysql:

    • Files: jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile (multiple instances)
    • Issue: The chmod -R a+rX ${WORKSPACE}/mysql-ssl command ensures read permissions for all users, but this may inadvertently expose sensitive data if the workspace is shared or logs are leaked.
    • Recommendation: Limit permissions to the user running the Jenkins agent instead:
      chmod -R u+rwX ${WORKSPACE}/mysql-ssl
  2. Redundant mkdir Commands:

    • Files: jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile (multiple instances)
    • Issue: mkdir -p is used to create directories that are immediately overwritten by cp -r. This redundancy can be avoided.
    • Recommendation: Replace with a single cp command that creates the target directory, e.g.:
      cp -r --parents /var/lib/mysql/*.pem ${WORKSPACE}/mysql-ssl
  3. Hardcoded Permissions in chmod:

    • Files: jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile (multiple instances)
    • Issue: Using chmod -R on a directory can lead to inconsistent behavior if the directory contains files with specific permissions.
    • Recommendation: Use find with -type to explicitly apply permissions only to files or directories as needed:
      find ${WORKSPACE}/mysql-ssl -type f -exec chmod u+rw {} \;
      find ${WORKSPACE}/mysql-ssl -type d -exec chmod u+rwx {} \;

Best Practices

  1. Documentation for Verification Steps:

    • Files: .github/workflows/pull-ci-runtime-images.yaml
    • Issue: The verification of the Dockerfile changes relies on CI workflows, but the exact validation coverage (e.g., image-based tests, script execution) is unclear.
    • Recommendation: Add documentation in the Dockerfiles or CI workflows describing the expected validation checks (e.g., ensuring no sudo calls in any script).
  2. Testing Coverage for Edge Cases:

    • Files: jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile
    • Issue: The PR assumes no sudo usage outside the updated scripts, but this assumption is not explicitly tested.
    • Recommendation: Add a smoke test in the CI pipeline to scan for sudo usage in any script executed within the containers. For example, use:
      grep -r sudo ${WORKSPACE} || true
  3. Consistency in Comments:

    • Files: Multiple Jenkinsfiles (e.g., jenkins/jobs/pingcap/tiflow/...)
    • Issue: Comments explaining why chmod and cp adjustments are made are present in some places but missing in others, leading to inconsistency.
    • Recommendation: Standardize comments to explain the rationale for privilege adjustments, such as:
      // Ensure unprivileged user can read staged TLS certs
  4. Avoid Code Duplication:

    • Files: Multiple Jenkinsfiles (jenkins/jobs/pingcap/tiflow/...)
    • Issue: Repeated blocks of code for handling TLS certificates and staging MySQL files introduce duplication.
    • Recommendation: Extract the common logic into a shared utility script or Groovy function that can be reused across pipelines.

Summary of Suggested Changes

  • Add a safety mechanism (e.g., alias) to fail explicitly on sudo usage in external scripts.
  • Adjust permissions in /var/lib/mysql and mysql-ssl to be more restrictive and avoid over-permissioning.
  • Avoid redundant mkdir calls when using cp -r.
  • Improve testing and verification documentation to clarify coverage of edge cases.
  • Standardize and consolidate duplicated logic in Jenkinsfiles for maintainability.

…ficates

The previous step copied `/var/lib/mysql/*.pem` into the workspace so the TLS
group could copy them back into the mysql sidecar's data directory. Both sides
are the same files: mysqld generates that key material in its own data directory,
and the replay of `pull_dm_integration_test` (#271) passed TLS_GROUP with the step
never executing.

Keep only what the unprivileged agent actually needs — `chmod -R a+rX` on the
workspace copies, so the tests can read them without sudo — and drop the round
trip and its conditional.
@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary

This pull request removes unnecessary and potentially dangerous passwordless sudo privileges from Jenkins CI Docker images and adjusts associated scripts to run without root. The approach involves removing the sudo package, changing file staging commands to operate under an unprivileged user, and ensuring proper file permissions for non-root users. The changes are well-justified and address a significant security vulnerability while maintaining functionality. Overall, the PR is well-structured, but a few areas require attention to ensure robustness and maintainability.


1. Critical Issues

  • No Critical Issues Identified.

2. Code Improvements

  • File Permission Hardening (Multiple Jenkinsfiles):

    • Files: jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile
    • Issue: The use of chmod -R a+rX grants read and execute permissions to all users indiscriminately. While this addresses the issue of unprivileged user access, it could expose sensitive files if the container is compromised.
    • Suggestion: Limit permissions to only the Jenkins user and group. Use chmod -R u+rwX,g+rX,o-rwx to ensure tighter control.
      chmod -R u+rwX,g+rX,o-rwx ${WORKSPACE}/mysql-ssl
  • Error Handling in Shell Commands (Multiple Jenkinsfiles):

    • Files: jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile
    • Issue: Commands like mkdir -p and cp do not handle errors explicitly. If these commands fail, subsequent steps may execute with incomplete setup, leading to hard-to-debug failures.
    • Suggestion: Add error checks after critical commands using set -e or inline error handling with || exit.
      mkdir -p ${WORKSPACE}/mysql-ssl || exit 1
      cp -r /var/lib/mysql/*.pem ${WORKSPACE}/mysql-ssl/ || exit 1
  • Single Source of Truth for Permissions (Jenkinsfiles):

    • Files: jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile
    • Issue: The permissions logic is repeated across multiple Jenkinsfiles. This redundancy increases maintenance overhead and the risk of inconsistencies.
    • Suggestion: Extract the permissions logic into a shared script (e.g., set_permissions.sh) that can be reused across Jenkinsfiles.
      # Example shared script: scripts/set_permissions.sh
      #!/bin/bash
      chmod -R u+rwX,g+rX,o-rwx "$1"

3. Best Practices

  • Testing for Unintended Breakages:

    • Issue: While the PR description mentions replaying DM integration tests, there’s no explicit mention of testing for other potential impacts (e.g., unrelated scripts or pipelines inadvertently relying on sudo).
    • Suggestion: Perform a broader audit or add a pre-merge verification step to ensure no other repositories or scripts are affected by the removal of sudo.
  • Documentation Updates (Dockerfiles):

    • Files: dockerfiles/ci/jenkins/Dockerfile, dockerfiles/ci/jenkins/tikv/Dockerfile
    • Issue: The removal of sudo and its implications (e.g., unprivileged user constraints, file permission requirements) are not documented clearly in the Dockerfiles.
    • Suggestion: Add comments in the Dockerfiles to explain the rationale for removing sudo and any operational constraints this introduces.
      # Note: The `sudo` package has been removed to prevent privilege escalation.
      # Ensure that all operations inside the container are compatible with 
      # running as an unprivileged user.
  • Style Consistency in Shell Commands:

    • Files: jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile
    • Issue: Shell commands sometimes use inconsistent quoting (e.g., ${WORKSPACE} is sometimes quoted, sometimes not). This inconsistency could lead to issues if variables contain spaces or special characters.
    • Suggestion: Always quote variables in shell scripts.
      mkdir -p "${WORKSPACE}/mysql-ssl"
      cp -r /var/lib/mysql/*.pem "${WORKSPACE}/mysql-ssl/"
  • Dependency Audit (Dockerfiles):

    • Files: dockerfiles/ci/jenkins/Dockerfile, dockerfiles/ci/jenkins/tikv/Dockerfile
    • Issue: The PR removes sudo, but it’s unclear if any other tools or scripts in the CI pipeline implicitly depend on it.
    • Suggestion: Perform a dependency audit to ensure no residual dependencies on sudo exist.

Summary of Suggested Changes

  1. Harden file permissions by limiting access to the Jenkins user and group.
  2. Add error handling to critical shell commands in Jenkinsfiles.
  3. Consolidate repeated permission-setting logic into a shared script.
  4. Document the removal of sudo and its implications in the Dockerfiles.
  5. Ensure consistent variable quoting in shell commands.
  6. Conduct a dependency audit to verify no unintended reliance on sudo.

These changes will further improve the robustness, security, and maintainability of the updated code.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/test pull-replay-jenkins-pipelines

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/approve

@ti-chi-bot

ti-chi-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 11, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

@wuhuizuo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-replay-jenkins-pipelines 3b0dfae link false /test pull-replay-jenkins-pipelines

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ti-chi-bot
ti-chi-bot Bot merged commit 426de7b into main Nov 11, 2026
19 of 20 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the fix/drop-passwordless-sudo branch November 11, 2026 10:15
dillon-zheng added a commit that referenced this pull request Nov 18, 2026
## Problem

DM integration tests fail in `TLS_GROUP` when creating the TLS source,
with `open /var/lib/mysql/ca.pem: no such file or directory`. This
blocks pingcap/tiflow#12990; the same error occurs in release-8.5 builds
99, 103, and 106.

PR #5531 removed the block that copied MySQL certificates from the
shared workspace into the golang container. Certificates are still
staged by mysql1 under `${WORKSPACE}/mysql-ssl`, but the test harness
reads them from `/var/lib/mysql` in the golang container. The current
agent Dockerfile does not pre-create that path for the build user.

## Changes

- Mount a dedicated `emptyDir` at `/var/lib/mysql` in the golang
container. The existing pod-level `fsGroup: 1000` makes the volume
writable by the Jenkins user.
- Copy the staged certificates into that volume before running
`TLS_GROUP`, using ordinary `cp` without sudo, root access, or an agent
image rebuild.
- Keep the fixtures separate from the live MySQL data directory so the
HA tests can rename and restore `ca.pem`.
- Apply the fix to all eight affected DM integration pipelines,
including latest/next-gen and release branches.
- Export only `ca.pem`, `client-cert.pem`, and `client-key.pem`; keep
CA/server private keys inside mysql1. Set workspace fixtures to group
1000, directory mode 0750, and file mode 0640. Use umask 077 for the
copies in golang.
- Add narrowly scoped resource-policy exemptions for the 19 pre-existing
container/template violations in the six release DM integration test
Pods. These expire after 2026-12-31; requests, limits, and the policy
gate itself remain unchanged. Right-sizing is deferred to a job-owner
review with usage evidence.

## Validation

- `git diff --check` — passed.
- `.ci/check-jenkins-job-references.sh --quiet` — passed.
- Jenkins `/pipeline-model-converter/validate` — all eight changed
Jenkinsfiles successfully validated.
- Parsed all eight Pod YAML files with PyYAML: verified `fsGroup: 1000`,
the emptyDir definition, and that only golang mounts the fixture volume
at `/var/lib/mysql`.
- `bash -n` on all 16 certificate-staging and test shell blocks —
passed.
- Revalidated all eight Jenkinsfiles after the permission hardening —
passed.

Full DM integration execution and Jenkins replay have not been run: this
session has no Jenkins replay credentials and the local Docker daemon is
unavailable. The TLS_GROUP jobs still need end-to-end verification in
CI.

## Security review

This preserves the goal of #5531: no sudo/sudoers changes, root user
overrides, privileged containers, hostPath mounts, capabilities,
Secrets, or ServiceAccount permissions are added. The new emptyDir is
mounted only in golang, is local to the test Pod, and does not expose
mysql1 or mysql2 data directories. The existing fsGroup remains 1000.
Certificates come from the disposable upstream MySQL instance; no
production credentials are introduced. The source staging step runs in
the existing mysql1 container; the final copy runs as the existing
Jenkins user.

The first revision passed `pull-verify-pod-privilege-policy`,
`pull-verify-secret-scan`, `pull-verify-storage-class-policy`, Pod YAML
validation, and Jenkins validation. The latest commit restricts key
selection and permissions further; its CI must complete again. Full
TLS_GROUP execution remains necessary to establish that the regression
is fixed.

### Resource-policy follow-up

The original `pull-verify-pod-resource-policy` failure came from 19
pre-existing violations in the six release DM test templates, exposed by
the TLS volume change. The follow-up adds explicit container/path
exemptions in `.ci/pod-resource-policy-allowlist.txt` through 2026-12-31
instead of changing runtime resource budgets or weakening the gate. No
container wildcards are used; central approval is still required for the
exemption ledger.

Validation of the follow-up:

- Reviewed-base `pod-resource` gate on all eight changed Pod templates:
passed (CPU mismatch warnings remain non-blocking).
- `.ci/test-verify-pod-resource-policy.sh`: all nine cases passed.
- Scope check against repository paths: exactly the intended 19
container/template pairs matched.
- Review-date boundary: passed on 2026-12-31; all 19 violations block
again on 2027-01-01.
- `git diff --check`: passed.

The updated commit `a08d9fc35` has been pushed, and
[pull-verify-pod-resource-policy](https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/PingCAP-QE_ci/5596/pull-verify-pod-resource-policy/2122963764872482816)
now reports **SUCCESS**. Jenkins replay and the remaining pending checks
still need to complete; this PR has not been merged.

---------

Co-authored-by: weizheng <wei.zheng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant