Repository navigation
fix(dockerfiles): stop granting the build user passwordless root - #5531
Conversation
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR removes the passwordless sudo access for the jenkins user in Docker images used by Jenkins agents, addressing a potential security vulnerability that allowed external contributors to escalate privileges to root. The change involves pre-configuring /var/lib/mysql with the correct permissions and modifying Jenkins pipelines to avoid using sudo. The overall implementation is sound, with clear explanations of the changes and associated risks. However, there are minor areas for improvement in documentation and validation to ensure robustness.
Critical Issues
No critical issues identified. The PR effectively mitigates the identified security vulnerability without introducing apparent bugs or regressions.
Code Improvements
-
File:
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfile
Issue: The removal of thesudopackage is a significant change. While the PR description mentions that nosudocalls remain in the codebase, a runtime check or script validation during the build process to confirm the absence ofsudousage in dependent scripts would provide additional confidence.
Suggestion: Add a validation step in the Dockerfile or CI workflow to ensure no scripts in the Jenkins pipelines callsudo.
Example:RUN ! grep -r 'sudo' /path/to/your/scripts || (echo "Error: sudo usage detected" && exit 1)
-
File:
jenkins/jobs/pingcap/tiflow/*/Jenkinsfile
Issue: The removal ofsudocommands simplifies the scripts, but the reliance on/var/lib/mysqlbeing pre-configured in the image introduces a hidden dependency. If this setup changes in the future, it could lead to failures.
Suggestion: Add an explicit check in the Jenkins pipelines to validate the existence and ownership of/var/lib/mysqlbefore proceeding with the tests.
Example:if [ ! -d /var/lib/mysql ] || [ "$(stat -c '%U' /var/lib/mysql)" != "jenkins" ]; then echo "Error: /var/lib/mysql is not correctly configured." exit 1 fi
Best Practices
-
File:
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfile
Issue: Theinstall -dcommand is used to create/var/lib/mysqlwith specific permissions, but this could benefit from a comment explaining why this directory is pre-created (e.g., for MySQL TLS fixtures).
Suggestion: Add a brief comment to clarify the purpose of this setup.
Example:# Pre-create /var/lib/mysql with appropriate permissions to support MySQL TLS fixtures install -d -o jenkins -g jenkins -m 0755 /var/lib/mysql -
File: All modified
Jenkinsfilescripts
Issue: The current comments explaining the removal ofsudoare helpful but could be more explicit about the security implications and why this change was necessary.
Suggestion: Update the comments to highlight the security rationale behind removingsudo.
Example:// The /var/lib/mysql directory is pre-created and owned by the jenkins user in the CI image. // This change removes the need for sudo and avoids potential privilege escalation vulnerabilities.
-
Testing Coverage
Issue: While the PR mentions replaying the affected pipelines, there is no mention of testing for other dependent scripts that may invokesudo.
Suggestion: Expand testing coverage to include any other scripts or workflows that might rely onsudoor root privileges, even if they are not part of the immediate changes in this PR.
Conclusion
The PR effectively addresses a significant security concern with minimal disruption to existing functionality. The changes are well-documented and tested within the scope of the updated pipelines. To further enhance the robustness of this change, consider adding validations to detect sudo calls during the build process and in the Jenkins pipelines. Additionally, improving comments and documentation will make the rationale for these changes clearer to future developers.
Both Jenkins agent images appended `jenkins ALL=(ALL) NOPASSWD: ALL` to `/etc/sudoers`. That made the pod policy's `runAsUser != 0` rule cosmetic: 370 Jenkinsfiles execute the pull request's own build and test scripts inside these images, and any of them reached uid 0 with one `sudo` (and in the pods that are allowlisted to run privileged while keeping the ServiceAccount token, that is a node/API-server escalation rather than a contained build). - Drop the sudoers line and the `sudo` package from both images. - Pre-create `/var/lib/mysql` owned by `jenkins` so the one in-repo consumer can drop its root steps: the eight `pull_dm_integration_test*` pipelines used `sudo mkdir/chmod/chown/cp` to stage MySQL TLS fixtures there. They now copy the fixtures as the unprivileged user; the directory is created by the image. No `sudo` call remains under `jenkins/`, `libraries/` or `scripts/`, so nothing in this repository needs root inside the container.
c40b1e2 to
43a4989
Compare
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR removes passwordless root access for the jenkins user in Docker images used by Jenkins agents, thereby addressing a significant security vulnerability that allowed containerized Jenkins jobs to escalate privileges to uid 0. The approach includes dropping the sudo package, pre-creating necessary directories with proper ownership, and updating Jenkins pipelines to operate without sudo. The changes are well-structured and thoroughly documented, with verification steps outlined. Overall, the PR significantly improves security without introducing unnecessary complexity, though minor improvements can be made to enhance maintainability.
Critical Issues
- None Identified: The PR effectively addresses the stated security issue. No breaking changes or regressions were observed in the provided diff.
Code Improvements
-
Directory Creation in Dockerfiles:
- File:
dockerfiles/ci/jenkins/Dockerfile(lines 8-9),dockerfiles/ci/jenkins/tikv/Dockerfile(lines 8-9) - Issue: Hardcoding
/var/lib/mysqlcreation in the Dockerfile increases coupling between the image and specific scenarios like MySQL TLS fixture staging. - Suggestion: Consider parameterizing the directory creation or documenting its purpose more clearly. Alternatively, create this directory dynamically in the relevant test scripts to decouple the image from specific test needs.
# Add a comment explaining why /var/lib/mysql is pre-created in the image. install -d -o jenkins -g jenkins -m 0755 /var/lib/mysql - File:
-
Pipeline Error Handling:
- File: Multiple
Jenkinsfile(e.g.,jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfileline 159) - Issue: The updated
cpcommand assumes that TLS fixture files always exist in${WORKSPACE}/mysql-ssl/*.pem. If the directory is empty or missing, the command will fail. - Suggestion: Add a check to ensure the source files are present before attempting the
cp.
if [ -n "$(ls -A ${WORKSPACE}/mysql-ssl/*.pem 2>/dev/null)" ]; then cp -r ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/ else echo "No TLS fixtures found to copy." fi
- File: Multiple
Best Practices
-
Testing for Unexpected
sudoUse:- File: Not directly in the diff, but relevant to the broader PR impact.
- Issue: The PR changes the baseline assumption for Jenkins jobs by removing
sudo. While the replay verifies existing pipelines, it doesn't protect against regressions where newsudocalls might be introduced inadvertently in future PRs. - Suggestion: Implement static analysis or CI checks to detect any new
sudocalls in Jenkins pipeline scripts or other CI-related files.
-
Documentation on Behavior Change:
- File: Not included in the diff but relevant to maintainability.
- Issue: The removal of
sudoand the requirement for non-root operations represent a significant behavior change. This may catch future developers off guard. - Suggestion: Update any relevant README or documentation for the CI images and Jenkins pipelines to highlight the new restrictions and explain how to work within them.
-
Cleanup of Unused Variables or Comments:
- File:
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfile - Issue: While the explanatory comments are useful, some of them could be streamlined or rephrased for conciseness.
- Suggestion: Consider revising comments for clarity and conciseness.
# The `jenkins` user is created without sudo privileges to prevent privilege escalation # in the container by external contributors. The user is intended for Jenkins agent operations only.
- File:
Summary of Recommendations
- Consider decoupling
/var/lib/mysqlcreation from the Dockerfiles or enhance documentation to clarify its purpose. - Add error handling in pipeline scripts to avoid failures if
mysql-sslfixtures are missing. - Implement static analysis or CI checks to prevent new
sudocalls from being introduced in the future. - Update documentation to reflect the new behavior of the Jenkins images and pipelines.
- Streamline comments where necessary to improve readability.
These suggestions aim to future-proof the changes, enhance maintainability, and ensure that the new security model is respected across future contributions.
Removing passwordless sudo exposed that the dm integration pipelines prepared
`/var/lib/mysql` as root for a reason: the directory is the mysql sidecar's
volume, owned by the mysql user, and the certificates copied into the workspace
are 0600 and owned by that same user. The unprivileged agent could neither write
the directory nor read the copies, so TLS_GROUP failed and its fail-fast took the
other groups down with it (replay build 268).
Keep the flow root-free by using the owner of each side:
- the prepare step (`container("mysql1")`) makes the workspace copies readable
(`chmod -R a+rX`) instead of relying on a later `chown` from the agent;
- the TLS group stages the certificates from inside `container("mysql1")`, which
owns `/var/lib/mysql`, instead of `sudo cp` from the agent container.
No `sudo` remains, and the image pre-creates no directory it does not own.
The dm TLS fixtures are staged from the mysql sidecar, which owns that volume, so pre-creating the directory in the agent image would only mislead: an emptyDir mounts over it anyway.
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR removes the use of passwordless sudo for the jenkins user in Docker images, effectively preventing privilege escalation by pull request build scripts. It also refactors the staging of MySQL TLS fixtures to work without sudo, ensuring compatibility with unprivileged containers. The changes are well-documented and follow a security-first approach. Overall, the PR significantly improves the security posture of the CI environment, but there are a few areas for improvement in terms of code consistency, readability, and potential edge cases.
1. Critical Issues
- File:
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile(and similar files for other versions)- Issue: The
chmod -R a+rXon${WORKSPACE}/mysql-sslensures readability but might inadvertently expose sensitive files to unintended users if additional files are added to this directory in the future. - Why it’s an issue: This could lead to a security vulnerability if other scripts or jobs add files to this directory with sensitive information.
- Suggested Solution:
Instead of a blanketchmod -R a+rX, explicitly set permissions on the required.pemfiles:chmod 0644 ${WORKSPACE}/mysql-ssl/*.pem
- Issue: The
2. Code Improvements
-
File:
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfile- Issue: The
install -d -o jenkins -g jenkins -m 0755 /var/lib/mysqlline is directly coupled to the Jenkins user's configuration, which could lead to maintenance challenges if the user ID or configuration changes. - Why it’s an issue: Hardcoding user/group IDs (
jenkinsand1000) makes the setup less flexible and harder to adapt to future changes. - Suggested Solution:
Use environment variables or arguments to make the configuration more dynamic:ARG JENKINS_USER=jenkins ARG JENKINS_UID=1000 ARG JENKINS_GID=1000 RUN groupadd -g $JENKINS_GID $JENKINS_USER && \ useradd -u $JENKINS_UID -g $JENKINS_GID -m -s /bin/bash $JENKINS_USER && \ install -d -o $JENKINS_USER -g $JENKINS_USER -m 0755 /var/lib/mysql
- Issue: The
-
File:
jenkins/jobs/.../Jenkinsfile(all modified versions)- Issue: The repeated logic for staging MySQL TLS certs (
cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/) is duplicated across multiple files. - Why it’s an issue: Code duplication increases maintenance overhead and the risk of inconsistencies.
- Suggested Solution:
Extract the shared logic into a reusable function or script:def stageMysqlCerts() { sh label: "Stage MySQL TLS certs", script: """ cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/ ls -alh /var/lib/mysql/ """ }
- Issue: The repeated logic for staging MySQL TLS certs (
3. Best Practices
-
File:
jenkins/jobs/.../Jenkinsfile(all modified versions)- Issue: Some parts of the pipeline scripts are tightly coupled to specific container names (e.g.,
mysql1). - Why it’s an issue: This reduces flexibility when reusing the pipeline for different setups or container configurations.
- Suggested Solution:
Define container names as variables at the top of the script so that they can be easily updated:def mysqlContainer = "mysql1" container(mysqlContainer) { sh // ... }
- Issue: Some parts of the pipeline scripts are tightly coupled to specific container names (e.g.,
-
File:
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfile- Issue: Comments explaining the removal of
sudoare verbose and could be streamlined for clarity. - Why it’s an issue: Excessively detailed comments may reduce readability for future maintainers.
- Suggested Solution:
Simplify comments, focusing only on the most critical points:# Create unprivileged Jenkins user without sudo to prevent privilege escalation.
- Issue: Comments explaining the removal of
-
File:
.github/workflows/pull-ci-runtime-images.yaml(referenced in the PR description)- Issue: The PR mentions that this workflow rebuilds images but does not explicitly confirm whether all affected image variations are tested.
- Why it’s an issue: Without explicit validation, there’s a risk that some images might break unnoticed.
- Suggested Solution:
Confirm in the PR description or add to the CI script that all combinations of base images and architectures are tested.
4. Testing Coverage
- Issue: The PR description mentions replaying
dm_integration_testpipelines to verify changes, but there’s no mention of broader regression testing for other scripts or tasks in the Jenkins environment. - Why it’s an issue: Changes to the Dockerfiles could impact other scripts or jobs that are not tested by the specific replayed pipelines.
- Suggested Solution:
- Add a note in the PR description verifying whether other jobs in the Jenkins ecosystem were tested for compatibility.
- Alternatively, include a fallback plan for monitoring and rolling back if issues arise post-deployment.
Conclusion
This PR addresses a critical security issue with clear and effective changes, but it can be improved by reducing code duplication, parameterizing hardcoded values, and refining permission handling for sensitive files. The verification plan is reasonable, but additional testing or documentation on broader impact would further strengthen the PR.
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR removes passwordless root access for the jenkins user in Docker images used by Jenkins agents, thereby addressing a significant security vulnerability. It also updates build scripts and pipeline configurations to operate without sudo, ensuring compatibility with the new unprivileged user setup. The approach is well-executed, with changes consistently applied across multiple Dockerfiles and Jenkinsfiles. The overall quality is solid, with clear improvements to security. However, there are opportunities to refine the implementation for readability and maintainability.
Critical Issues
None identified. The PR adequately addresses the security risk of passwordless sudo access and ensures compatibility across affected pipelines.
Code Improvements
-
File: Jenkinsfiles (multiple locations)
- Issue: The repeated block for copying and staging MySQL TLS certificates introduces duplication across Jenkinsfiles for different branches/releases.
- Why it's an issue: Duplication increases maintenance overhead and the risk of inconsistent updates in future changes.
- Suggested solution: Extract the repeated logic into a reusable library function or a shared script. For example:
This would simplify the Jenkinsfiles and ensure consistent behavior across pipelines.
// Shared logic for TLS cert handling def stageMysqlTlsCerts(workspace) { sh """ mkdir -p ${workspace}/mysql-ssl cp -r /var/lib/mysql/*.pem ${workspace}/mysql-ssl/ chmod -R a+rX ${workspace}/mysql-ssl ls -alh ${workspace}/mysql-ssl/ """ } // Usage in Jenkinsfile container("mysql1") { stageMysqlTlsCerts(WORKSPACE) }
-
File: Dockerfiles (
dockerfiles/ci/jenkins/Dockerfile,dockerfiles/ci/jenkins/tikv/Dockerfile)- Issue: The
/var/lib/mysqldirectory is pre-created in the Dockerfiles but lacks clear documentation on the rationale. - Why it's an issue: It may cause confusion for future maintainers regarding why this directory is initialized upfront.
- Suggested solution: Add a comment explicitly stating the purpose of pre-creating this directory, e.g.:
# Pre-create /var/lib/mysql for staging MySQL TLS fixtures during Jenkins pipelines
- Issue: The
Best Practices
-
File: Jenkinsfiles (multiple locations)
- Issue: The inline comments explaining the removal of
sudouse are helpful but could be centralized in the shared function/script suggested above. - Why it's an issue: Inline comments in multiple places can lead to inconsistencies if the explanation needs updating later.
- Suggested solution: Add a single, detailed comment in the shared function or script to document why
sudowas removed and how the new approach ensures security.
- Issue: The inline comments explaining the removal of
-
Testing Coverage
- Issue: The PR description mentions replaying Jenkins pipelines for verification, but no automated tests are included for validating the removal of
sudoand ensuring compatibility with the unprivileged user setup. - Why it's an issue: Reliance solely on manual testing can lead to missed edge cases in future updates.
- Suggested solution: Introduce automated tests to validate the Docker images and pipeline behavior. For instance:
- A test to ensure Jenkinsfiles do not contain
sudocommands. - A container-level test to verify that
jenkinscannot escalate privileges.
- A test to ensure Jenkinsfiles do not contain
- Issue: The PR description mentions replaying Jenkins pipelines for verification, but no automated tests are included for validating the removal of
-
Naming Conventions
- Issue: The variable
WORKSPACEis used inconsistently in scripts. - Why it's an issue: Naming conventions should be consistent for clarity and maintainability.
- Suggested solution: Ensure consistent capitalization or adopt a standard naming convention across scripts, e.g.,
workspaceDir.
- Issue: The variable
Overall Assessment
The PR effectively addresses a critical security vulnerability and ensures compatibility with the updated setup. The changes are methodical and well-documented. Refining the code to reduce duplication, adding automated tests, and improving documentation will further enhance the robustness and maintainability of the changes.
A declarative pipeline only accepts steps in 'steps'; the conditional has to be wrapped in 'script', which the Jenkinsfile validator reported as 'Expected a step @ line 193'.
|
/test pull-verify-jenkins-pipelines |
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
The PR addresses a critical security issue by removing passwordless root access for the jenkins user in Docker images, ensuring that external contributors cannot escalate privileges within Jenkins agents. The approach involves removing the sudo package, dropping the corresponding /etc/sudoers entry, and refactoring scripts to avoid sudo commands. Overall, the changes are well-justified and effectively improve container security without introducing unnecessary complexity. However, some minor improvements can enhance maintainability and clarity.
Critical Issues
- None identified. The changes effectively mitigate the security vulnerability without compromising functionality.
Code Improvements
-
Error Handling for File Operations
- Files:
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile(and other affected Jenkinsfiles)- Problem: Operations like
cpandchmodassume success but lack error handling. If these fail (e.g., due to missing files), the pipeline may silently proceed to incorrect behavior or fail later with unrelated errors. - Suggested Fix: Add error handling to critical file operations to surface issues earlier. For example:
sh label: "stage mysql TLS certs", script: """ set -e cp -f ${WORKSPACE}/mysql-ssl/*.pem /var/lib/mysql/ || { echo 'Failed to copy certs'; exit 1; } chmod -R a+rX ${WORKSPACE}/mysql-ssl || { echo 'Failed to adjust permissions'; exit 1; } ls -alh /var/lib/mysql/ """
- Problem: Operations like
- Files:
-
Pre-Creation of
/var/lib/mysqlin Dockerfiles- Files:
dockerfiles/ci/jenkins/Dockerfile,dockerfiles/ci/jenkins/tikv/Dockerfile - Problem: While
/var/lib/mysqlis pre-created in the images, the PR does not explicitly show where or how this happens. This may cause confusion for future maintainers. - Suggested Fix: Add a direct
RUN mkdir -p /var/lib/mysql && chown jenkins:jenkins /var/lib/mysqlline in the Dockerfiles for clarity:RUN mkdir -p /var/lib/mysql && chown jenkins:jenkins /var/lib/mysql
- Files:
Best Practices
-
Consistency in Comments
- Files:
Various Jenkinsfiles (e.g.,jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile) - Problem: Comments explaining why
chmod -R a+rXor similar steps are necessary are inconsistent across files. Consistent documentation helps future contributors understand why specific steps are included. - Suggested Fix: Use a uniform comment format, e.g.:
# Ensure the agent user can read these files without root privileges. chmod -R a+rX ${WORKSPACE}/mysql-ssl
- Files:
-
Use of
mkdir -pin Workspace Paths- Files:
Various Jenkinsfiles (e.g.,jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile) - Problem: While
mkdir -pis correctly used in most cases, some earlier versions of theJenkinsfilediff show inconsistencies. - Suggested Fix: Ensure all directory creations use
mkdir -pfor idempotence.
- Files:
-
Testing Coverage
- Files: Not directly in the PR, but related to downstream impact.
- Problem: The PR mentions potential risks from component repository scripts calling
sudo. While the replay verification partially mitigates this, there is no explicit mention of automated tests to catch such failures in other pipelines. - Suggested Fix: Add a pre-check to identify
sudousage in downstream repositories' scripts, e.g., a static analysis or a grep-based validation step.
-
Documentation of Risk Impact
- Files: PR description.
- Problem: While the description notes the risk of failures in component repositories due to removed
sudo, it lacks concrete suggestions for identifying or resolving such issues. - Suggested Fix: Enhance documentation by specifying how contributors can detect and resolve such failures, e.g.,:
- Provide a grep command to identify
sudousage in scripts. - Suggest fallback approaches for developers encountering failures.
- Provide a grep command to identify
Final Remarks
The PR is thorough and addresses a significant security vulnerability without introducing regressions. The actionable feedback mainly focuses on improving robustness, consistency, and maintainability, ensuring the changes are future-proof and well-documented. Addressing the above points will further enhance the quality and clarity of this important fix.
|
/test pull-replay-jenkins-pipelines |
The staging step compared `env.TEST_GROUP`, which the matrix axis does not set:
the step never ran, while the shell block it replaced compared the Groovy binding
(`\"${TEST_GROUP}\"`) and did run. Use the binding so TLS_GROUP keeps the
preconditions it had before the sudo removal.
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR removes the sudo package and passwordless sudo permissions from Docker images used by Jenkins, thus addressing a critical security vulnerability where external contributors could escalate to root privileges within containers. The PR ensures that all impacted scripts and Jenkins pipelines are updated to function without requiring sudo. The implementation is thorough, with redundant sudo commands replaced by unprivileged alternatives, and verification steps outlined. Overall, the changes are well-executed, but a few areas could benefit from clarification or minor improvements.
Critical Issues
- Risk of Unintended Failures in Unverified Scripts:
- Files: Any external repository scripts executed within the Jenkins containers.
- Issue: While the PR focuses on updating scripts under
jenkins/jobsto removesudousage, any external repository scripts relying onsudowill now fail silently. This is acknowledged in the "Risk" section, but no automated detection mechanism is proposed. - Recommendation: Introduce logging or a fallback mechanism in the images to detect and alert on any
sudousage in external scripts. For example:Add this to the Dockerfile to explicitly fail onalias sudo='echo "Error: sudo is disabled in this container." >&2; exit 1'
sudousage.
Code Improvements
-
Privilege Management in
/var/lib/mysql:- Files:
jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile(multiple instances) - Issue: The
chmod -R a+rX ${WORKSPACE}/mysql-sslcommand ensures read permissions for all users, but this may inadvertently expose sensitive data if the workspace is shared or logs are leaked. - Recommendation: Limit permissions to the user running the Jenkins agent instead:
chmod -R u+rwX ${WORKSPACE}/mysql-ssl
- Files:
-
Redundant
mkdirCommands:- Files:
jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile(multiple instances) - Issue:
mkdir -pis used to create directories that are immediately overwritten bycp -r. This redundancy can be avoided. - Recommendation: Replace with a single
cpcommand that creates the target directory, e.g.:cp -r --parents /var/lib/mysql/*.pem ${WORKSPACE}/mysql-ssl
- Files:
-
Hardcoded Permissions in
chmod:- Files:
jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile(multiple instances) - Issue: Using
chmod -Ron a directory can lead to inconsistent behavior if the directory contains files with specific permissions. - Recommendation: Use
findwith-typeto explicitly apply permissions only to files or directories as needed:find ${WORKSPACE}/mysql-ssl -type f -exec chmod u+rw {} \; find ${WORKSPACE}/mysql-ssl -type d -exec chmod u+rwx {} \;
- Files:
Best Practices
-
Documentation for Verification Steps:
- Files:
.github/workflows/pull-ci-runtime-images.yaml - Issue: The verification of the Dockerfile changes relies on CI workflows, but the exact validation coverage (e.g., image-based tests, script execution) is unclear.
- Recommendation: Add documentation in the Dockerfiles or CI workflows describing the expected validation checks (e.g., ensuring no
sudocalls in any script).
- Files:
-
Testing Coverage for Edge Cases:
- Files:
jenkins/jobs/.../pull_dm_integration_test*/Jenkinsfile - Issue: The PR assumes no
sudousage outside the updated scripts, but this assumption is not explicitly tested. - Recommendation: Add a smoke test in the CI pipeline to scan for
sudousage in any script executed within the containers. For example, use:grep -r sudo ${WORKSPACE} || true
- Files:
-
Consistency in Comments:
- Files: Multiple Jenkinsfiles (e.g.,
jenkins/jobs/pingcap/tiflow/...) - Issue: Comments explaining why
chmodandcpadjustments are made are present in some places but missing in others, leading to inconsistency. - Recommendation: Standardize comments to explain the rationale for privilege adjustments, such as:
// Ensure unprivileged user can read staged TLS certs
- Files: Multiple Jenkinsfiles (e.g.,
-
Avoid Code Duplication:
- Files: Multiple Jenkinsfiles (
jenkins/jobs/pingcap/tiflow/...) - Issue: Repeated blocks of code for handling TLS certificates and staging MySQL files introduce duplication.
- Recommendation: Extract the common logic into a shared utility script or Groovy function that can be reused across pipelines.
- Files: Multiple Jenkinsfiles (
Summary of Suggested Changes
- Add a safety mechanism (e.g., alias) to fail explicitly on
sudousage in external scripts. - Adjust permissions in
/var/lib/mysqlandmysql-sslto be more restrictive and avoid over-permissioning. - Avoid redundant
mkdircalls when usingcp -r. - Improve testing and verification documentation to clarify coverage of edge cases.
- Standardize and consolidate duplicated logic in Jenkinsfiles for maintainability.
…ficates The previous step copied `/var/lib/mysql/*.pem` into the workspace so the TLS group could copy them back into the mysql sidecar's data directory. Both sides are the same files: mysqld generates that key material in its own data directory, and the replay of `pull_dm_integration_test` (#271) passed TLS_GROUP with the step never executing. Keep only what the unprivileged agent actually needs — `chmod -R a+rX` on the workspace copies, so the tests can read them without sudo — and drop the round trip and its conditional.
|
/test pull-replay-jenkins-pipelines |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This pull request removes unnecessary and potentially dangerous passwordless sudo privileges from Jenkins CI Docker images and adjusts associated scripts to run without root. The approach involves removing the sudo package, changing file staging commands to operate under an unprivileged user, and ensuring proper file permissions for non-root users. The changes are well-justified and address a significant security vulnerability while maintaining functionality. Overall, the PR is well-structured, but a few areas require attention to ensure robustness and maintainability.
1. Critical Issues
- No Critical Issues Identified.
2. Code Improvements
-
File Permission Hardening (Multiple Jenkinsfiles):
- Files:
jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile - Issue: The use of
chmod -R a+rXgrants read and execute permissions to all users indiscriminately. While this addresses the issue of unprivileged user access, it could expose sensitive files if the container is compromised. - Suggestion: Limit permissions to only the Jenkins user and group. Use
chmod -R u+rwX,g+rX,o-rwxto ensure tighter control.chmod -R u+rwX,g+rX,o-rwx ${WORKSPACE}/mysql-ssl
- Files:
-
Error Handling in Shell Commands (Multiple Jenkinsfiles):
- Files:
jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile - Issue: Commands like
mkdir -pandcpdo not handle errors explicitly. If these commands fail, subsequent steps may execute with incomplete setup, leading to hard-to-debug failures. - Suggestion: Add error checks after critical commands using
set -eor inline error handling with|| exit.mkdir -p ${WORKSPACE}/mysql-ssl || exit 1 cp -r /var/lib/mysql/*.pem ${WORKSPACE}/mysql-ssl/ || exit 1
- Files:
-
Single Source of Truth for Permissions (Jenkinsfiles):
- Files:
jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile - Issue: The permissions logic is repeated across multiple Jenkinsfiles. This redundancy increases maintenance overhead and the risk of inconsistencies.
- Suggestion: Extract the permissions logic into a shared script (e.g.,
set_permissions.sh) that can be reused across Jenkinsfiles.# Example shared script: scripts/set_permissions.sh #!/bin/bash chmod -R u+rwX,g+rX,o-rwx "$1"
- Files:
3. Best Practices
-
Testing for Unintended Breakages:
- Issue: While the PR description mentions replaying DM integration tests, there’s no explicit mention of testing for other potential impacts (e.g., unrelated scripts or pipelines inadvertently relying on
sudo). - Suggestion: Perform a broader audit or add a pre-merge verification step to ensure no other repositories or scripts are affected by the removal of
sudo.
- Issue: While the PR description mentions replaying DM integration tests, there’s no explicit mention of testing for other potential impacts (e.g., unrelated scripts or pipelines inadvertently relying on
-
Documentation Updates (Dockerfiles):
- Files:
dockerfiles/ci/jenkins/Dockerfile,dockerfiles/ci/jenkins/tikv/Dockerfile - Issue: The removal of
sudoand its implications (e.g., unprivileged user constraints, file permission requirements) are not documented clearly in the Dockerfiles. - Suggestion: Add comments in the Dockerfiles to explain the rationale for removing
sudoand any operational constraints this introduces.# Note: The `sudo` package has been removed to prevent privilege escalation. # Ensure that all operations inside the container are compatible with # running as an unprivileged user.
- Files:
-
Style Consistency in Shell Commands:
- Files:
jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfile - Issue: Shell commands sometimes use inconsistent quoting (e.g.,
${WORKSPACE}is sometimes quoted, sometimes not). This inconsistency could lead to issues if variables contain spaces or special characters. - Suggestion: Always quote variables in shell scripts.
mkdir -p "${WORKSPACE}/mysql-ssl" cp -r /var/lib/mysql/*.pem "${WORKSPACE}/mysql-ssl/"
- Files:
-
Dependency Audit (Dockerfiles):
- Files:
dockerfiles/ci/jenkins/Dockerfile,dockerfiles/ci/jenkins/tikv/Dockerfile - Issue: The PR removes
sudo, but it’s unclear if any other tools or scripts in the CI pipeline implicitly depend on it. - Suggestion: Perform a dependency audit to ensure no residual dependencies on
sudoexist.
- Files:
Summary of Suggested Changes
- Harden file permissions by limiting access to the Jenkins user and group.
- Add error handling to critical shell commands in Jenkinsfiles.
- Consolidate repeated permission-setting logic into a shared script.
- Document the removal of
sudoand its implications in the Dockerfiles. - Ensure consistent variable quoting in shell commands.
- Conduct a dependency audit to verify no unintended reliance on
sudo.
These changes will further improve the robustness, security, and maintainability of the updated code.
|
/test pull-replay-jenkins-pipelines |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@wuhuizuo: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
## Problem DM integration tests fail in `TLS_GROUP` when creating the TLS source, with `open /var/lib/mysql/ca.pem: no such file or directory`. This blocks pingcap/tiflow#12990; the same error occurs in release-8.5 builds 99, 103, and 106. PR #5531 removed the block that copied MySQL certificates from the shared workspace into the golang container. Certificates are still staged by mysql1 under `${WORKSPACE}/mysql-ssl`, but the test harness reads them from `/var/lib/mysql` in the golang container. The current agent Dockerfile does not pre-create that path for the build user. ## Changes - Mount a dedicated `emptyDir` at `/var/lib/mysql` in the golang container. The existing pod-level `fsGroup: 1000` makes the volume writable by the Jenkins user. - Copy the staged certificates into that volume before running `TLS_GROUP`, using ordinary `cp` without sudo, root access, or an agent image rebuild. - Keep the fixtures separate from the live MySQL data directory so the HA tests can rename and restore `ca.pem`. - Apply the fix to all eight affected DM integration pipelines, including latest/next-gen and release branches. - Export only `ca.pem`, `client-cert.pem`, and `client-key.pem`; keep CA/server private keys inside mysql1. Set workspace fixtures to group 1000, directory mode 0750, and file mode 0640. Use umask 077 for the copies in golang. - Add narrowly scoped resource-policy exemptions for the 19 pre-existing container/template violations in the six release DM integration test Pods. These expire after 2026-12-31; requests, limits, and the policy gate itself remain unchanged. Right-sizing is deferred to a job-owner review with usage evidence. ## Validation - `git diff --check` — passed. - `.ci/check-jenkins-job-references.sh --quiet` — passed. - Jenkins `/pipeline-model-converter/validate` — all eight changed Jenkinsfiles successfully validated. - Parsed all eight Pod YAML files with PyYAML: verified `fsGroup: 1000`, the emptyDir definition, and that only golang mounts the fixture volume at `/var/lib/mysql`. - `bash -n` on all 16 certificate-staging and test shell blocks — passed. - Revalidated all eight Jenkinsfiles after the permission hardening — passed. Full DM integration execution and Jenkins replay have not been run: this session has no Jenkins replay credentials and the local Docker daemon is unavailable. The TLS_GROUP jobs still need end-to-end verification in CI. ## Security review This preserves the goal of #5531: no sudo/sudoers changes, root user overrides, privileged containers, hostPath mounts, capabilities, Secrets, or ServiceAccount permissions are added. The new emptyDir is mounted only in golang, is local to the test Pod, and does not expose mysql1 or mysql2 data directories. The existing fsGroup remains 1000. Certificates come from the disposable upstream MySQL instance; no production credentials are introduced. The source staging step runs in the existing mysql1 container; the final copy runs as the existing Jenkins user. The first revision passed `pull-verify-pod-privilege-policy`, `pull-verify-secret-scan`, `pull-verify-storage-class-policy`, Pod YAML validation, and Jenkins validation. The latest commit restricts key selection and permissions further; its CI must complete again. Full TLS_GROUP execution remains necessary to establish that the regression is fixed. ### Resource-policy follow-up The original `pull-verify-pod-resource-policy` failure came from 19 pre-existing violations in the six release DM test templates, exposed by the TLS volume change. The follow-up adds explicit container/path exemptions in `.ci/pod-resource-policy-allowlist.txt` through 2026-12-31 instead of changing runtime resource budgets or weakening the gate. No container wildcards are used; central approval is still required for the exemption ledger. Validation of the follow-up: - Reviewed-base `pod-resource` gate on all eight changed Pod templates: passed (CPU mismatch warnings remain non-blocking). - `.ci/test-verify-pod-resource-policy.sh`: all nine cases passed. - Scope check against repository paths: exactly the intended 19 container/template pairs matched. - Review-date boundary: passed on 2026-12-31; all 19 violations block again on 2027-01-01. - `git diff --check`: passed. The updated commit `a08d9fc35` has been pushed, and [pull-verify-pod-resource-policy](https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/PingCAP-QE_ci/5596/pull-verify-pod-resource-policy/2122963764872482816) now reports **SUCCESS**. Jenkins replay and the remaining pending checks still need to complete; this PR has not been merged. --------- Co-authored-by: weizheng <wei.zheng@pingcap.com>
Problem
dockerfiles/ci/jenkins/Dockerfileanddockerfiles/ci/jenkins/tikv/Dockerfileappend
jenkins ALL=(ALL) NOPASSWD: ALLto/etc/sudoers. The images are whatthe Jenkins agent fleet runs in, and 370 Jenkinsfiles execute the pull request's
own
Makefile/build/test scripts inside them — so any external contributorreaches uid 0 in the container with one
sudo. That makes the pod policy'srunAsUser != 0rule (.ci/verify-pod-privilege-policy.sh) cosmetic, and in thepods allowlisted to run privileged while keeping the ServiceAccount token it
turns "container escape" into "node/API-server escalation".
Changes
sudopackage from both images./var/lib/mysqlowned byjenkinsin both images.jenkins/jobs/pingcap/tiflow/*/pull_dm_integration_test*/Jenkinsfilepipelines used
sudo mkdir/chmod/chown/cpto stage MySQL TLS fixtures there;they now copy them as the unprivileged user. That was the only consumer: no
sudocall remains underjenkins/,libraries/orscripts/.Verification
.github/workflows/pull-ci-runtime-images.yamlrebuilds the base and agentimages for both architectures on this PR — that proves the Dockerfile changes
and the
install -dline.pull-replay-jenkins-pipelinesreplays the dm integration jobs, which exercisethe TLS fixture staging and the test harness inside the image.
Risk
Any script inside a component repository that calls
sudowill now failinstead of silently escalating. That is the intended effect, but it can surface
as a dm integration test failure; the replay above is how we find out.