Repository navigation
fix(tiflow): restore DM TLS fixtures without sudo - #5596
Conversation
Restore certificate staging removed by PingCAP-QE#5531. Mount a group-writable emptyDir at /var/lib/mysql in the golang container and copy the staged MySQL certificates before TLS_GROUP runs. Keep the fixture copy separate from the MySQL data directory so HA tests can rename certificates safely.
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR addresses integration test failures in the DM TLS_GROUP by restoring TLS fixtures without requiring elevated privileges or container rebuilds. It introduces a new emptyDir volume mounted at /var/lib/mysql in the Jenkins pod configuration, allowing writable access to the build user for copying certificates. The changes span multiple pipelines and branches, ensuring consistency across different versions. The approach is generally robust and avoids introducing security risks, but there are minor areas for improvement regarding error handling, maintainability, and documentation.
Critical Issues
1. Missing Error Handling for cp Command
- File:
Jenkinsfile(e.g.,latest/pull_dm_integration_test/Jenkinsfile, line 159) - Issue: The
cpcommand assumes that the certificate files exist in${WORKSPACE}/mysql-ssl, but if they are missing or unreadable, the script will fail silently and potentially lead to test errors later. - Solution: Add explicit error checking for the existence of the files before copying them.
if [ "TLS_GROUP" = "${TEST_GROUP}" ]; then if [ -z "$(ls "${WORKSPACE}/mysql-ssl/"*.pem 2>/dev/null)" ]; then echo "Error: TLS certificates not found in ${WORKSPACE}/mysql-ssl/" exit 1 fi cp "${WORKSPACE}/mysql-ssl/"*.pem /var/lib/mysql/ fi
Code Improvements
1. Refactor Repeated Code Across Jenkinsfiles
- Files: All modified
Jenkinsfilescripts - Issue: The logic to copy certificates into
/var/lib/mysqlis duplicated across all pipelines, making maintenance difficult and error-prone. - Solution: Extract the shared logic into a shell script or Groovy function and call it consistently across pipelines.
def copyTlsCertificates(testGroup, workspace) { if (testGroup == "TLS_GROUP") { sh """ if [ -z "$(ls "${workspace}/mysql-ssl/"*.pem 2>/dev/null)" ]; then echo "Error: TLS certificates not found in ${workspace}/mysql-ssl/" exit 1 fi cp "${workspace}/mysql-ssl/"*.pem /var/lib/mysql/ """ } }
2. Potential Redundancy in emptyDir Volumes
- Files:
pod.yamlfiles (e.g.,latest/pull_dm_integration_test/pod.yaml, line 110) - Issue: The
emptyDirvolume is introduced for TLS fixtures, but it appears to be exclusively used for/var/lib/mysql. If this directory is only used for staging certificates, consider consolidating its purpose or re-evaluating the directory naming for clarity. - Solution: Document the purpose of
/var/lib/mysqlin the pod configuration or use a more descriptive mount path like/tls-fixtures.
Best Practices
1. Testing Coverage for Changes
- Files: All Jenkins pipelines
- Issue: The PR description mentions that end-to-end validation in CI has not been performed. This is critical for ensuring the changes work across all branches.
- Solution: Run full integration tests on at least one representative branch (e.g.,
latest) and confirm that the TLS_GROUP tests complete successfully before merging.
2. Missing Comments in Pod YAML
- Files: All
pod.yamlfiles - Issue: The new
mysql-tls-fixturesvolume is added without documentation explaining its purpose. - Solution: Add comments to describe the reason for the new volume and its role in the integration tests.
volumes: - name: mysql-tls-fixtures emptyDir: {} # Volume used to stage TLS certificates for DM integration tests
3. File Permissions for TLS Certificates
- File:
release-6.5/pull_dm_integration_test/Jenkinsfile, line 174 - Issue: The
chmod -R a+rXcommand makes all files in${WORKSPACE}/mysql-sslreadable by all users, which might be overly permissive for sensitive files like.pem. - Solution: Limit permissions to the Jenkins user or group where possible.
chmod -R u+rX ${WORKSPACE}/mysql-ssl
Summary of Recommendations
- Critical Fixes:
- Implement error handling for the
cpcommand to avoid silent failures.
- Implement error handling for the
- Code Improvements:
- Refactor duplicate certificate handling logic into a shared function or script.
- Evaluate the naming and necessity of the
/var/lib/mysqlmount path inpod.yaml.
- Best Practices:
- Ensure end-to-end CI validation for at least one pipeline before merging.
- Document the new
mysql-tls-fixturesvolume inpod.yaml. - Restrict file permissions for sensitive certificate files.
These changes will improve the reliability, maintainability, and security of the integration pipelines.
Stage only the CA certificate and client certificate/key. Limit workspace fixture access to group 1000 and keep private copies in the test container, without exporting CA or server private keys.
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This pull request addresses the issue of DM integration tests failing due to missing TLS certificates in the /var/lib/mysql directory within the Golang container. The approach involves mounting a writable emptyDir volume at /var/lib/mysql and copying the certificates into it using non-root operations, ensuring compatibility with the Jenkins user permissions. The changes are applied across multiple pipelines and include modifications to shell scripts and Kubernetes pod configurations. The overall approach is sound, but there are areas that could benefit from refinement and further testing.
Critical Issues
-
Incomplete Validation of End-to-End Execution
- Files: Entire PR
- Issue: The PR description mentions that full integration execution and CI replay were not conducted due to lack of credentials and access to a local Docker daemon. This is a blocker since the changes involve critical pipeline functionality across multiple branches. Without full validation, potential regressions or compatibility issues could go unnoticed.
- Suggestion: Ensure end-to-end testing of all affected pipelines in a staging or CI environment before merging. If access to Jenkins replay is an issue, coordinate with the appropriate team to perform these tests.
-
Potential Permission Issues with
chgrp- Files: Multiple Jenkinsfiles (e.g.,
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile, line 143) - Issue: The use of
chgrp 1000assumes that the build user belongs to group1000. This could fail if group1000does not exist or is not assigned to the user in some environments. - Suggestion: Validate the group ID dynamically in the script or use a more generic approach, such as ensuring the files are readable by all users (
chmod a+r), while avoiding unnecessary security risks.
- Files: Multiple Jenkinsfiles (e.g.,
Code Improvements
-
Error Handling in Shell Commands
- Files: Multiple Jenkinsfiles (e.g.,
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile, line 140) - Issue: Shell commands like
cpandchmodmay fail silently, potentially leading to undetected errors. For example, a missing source file forcpcould halt the pipeline later. - Suggestion: Add error handling for critical commands using
set -eor|| exit <error_code>to fail fast if any step encounters an issue. For example:cp /var/lib/mysql/ca.pem /var/lib/mysql/client-cert.pem /var/lib/mysql/client-key.pem "${WORKSPACE}/mysql-ssl/" || exit 1
- Files: Multiple Jenkinsfiles (e.g.,
-
Duplication Across Jenkinsfiles
- Files: All modified Jenkinsfiles
- Issue: The same block for copying certificates and setting permissions is repeated across multiple files, leading to redundancy and maintainability concerns.
- Suggestion: Refactor the common shell logic into a shared script (e.g.,
copy_tls_certs.sh) and invoke it in each Jenkinsfile. Example:sh label: "copy mysql certs", script: """ source ./scripts/copy_tls_certs.sh """
-
Improper Use of
umask- Files: Multiple Jenkinsfiles (e.g.,
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile, line 157) - Issue: The use of
umask 077applies stricter permissions (owner-only access) than what is ultimately set bychmod. This could lead to confusion or inconsistencies. - Suggestion: Remove
umaskand directly set permissions usingchmodto avoid ambiguity.
- Files: Multiple Jenkinsfiles (e.g.,
Best Practices
-
Testing Coverage
- Files: Entire PR
- Issue: The PR does not include automated tests to verify the correctness of the TLS certificate mounting and copying process, especially the permissions.
- Suggestion: Add automated tests that validate the presence and permissions of the files in
/var/lib/mysqlafter theTLS_GROUPtest runs. This can be done by running a simple test command within the container:test -f /var/lib/mysql/ca.pem && test -r /var/lib/mysql/ca.pem || exit 1
-
Documentation and Comments
- Files: Multiple Jenkinsfiles and pod YAML files (e.g.,
jenkins/jobs/pingcap/tiflow/latest/pull_dm_integration_test/Jenkinsfile, lines 140-157) - Issue: The comments in the shell script blocks are helpful but could be more concise. Additionally, the
pod.yamlfiles lack comments explaining the purpose of themysql-tls-fixturesvolume. - Suggestion: Refactor comments to focus on the "why" rather than the "how." Add a comment in
pod.yamlto describe the purpose of the new volume. Example:volumes: # Temporary volume for storing TLS certificates required by DM integration tests - name: mysql-tls-fixtures emptyDir: {}
- Files: Multiple Jenkinsfiles and pod YAML files (e.g.,
Conclusion
The changes are well-targeted to address the issue of missing TLS certificates without requiring elevated permissions, making the solution broadly applicable across pipelines. However, critical validation steps are missing, and the implementation could benefit from better error handling, reduced redundancy, and clearer documentation. Addressing these points will improve the overall robustness and maintainability of the code.
|
@wk989898: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR addresses a regression in DM integration tests related to TLS setup by introducing a new emptyDir mount at /var/lib/mysql in the golang container. It ensures that MySQL certificates are properly staged and copied without requiring elevated privileges or root access. The solution is applied across multiple release pipelines and hardens permissions to improve security. While the approach is thorough and well-documented, there are areas for optimization and potential risks that need to be addressed.
Critical Issues
-
Potential Race Condition in Certificate Copying
- Files: All
Jenkinsfilescripts (latest,next-gen,release-*) - Lines: e.g.,
cp "${WORKSPACE}"/mysql-ssl/*.pem /var/lib/mysql/ - Issue: The use of
cpwithout validation can lead to race conditions if the required certificates are missing or malformed due to prior steps. This can result in partial or failed integration tests. - Suggested Solution: Validate the presence and integrity of required certificate files before copying. For example:
required_files=("ca.pem" "client-cert.pem" "client-key.pem") for file in "${required_files[@]}"; do if [ ! -f "${WORKSPACE}/mysql-ssl/$file" ]; then echo "Error: $file is missing!" exit 1 fi done cp "${WORKSPACE}"/mysql-ssl/*.pem /var/lib/mysql/
- Files: All
-
Limited Validation of End-to-End Execution
- Files: N/A (PR description)
- Issue: The PR description notes that full TLS_GROUP execution in CI has not yet been completed. Without this validation, there is no guarantee that the issue is fully resolved.
- Suggested Solution: Prioritize running full end-to-end tests in CI before merging the PR to ensure the fix behaves as expected in all pipelines.
Code Improvements
-
Repetitive Code Across Multiple Jenkinsfiles
- Files: All
Jenkinsfilescripts (latest,next-gen,release-*) - Issue: The new logic for copying certificates and setting permissions is repeated across multiple files, making maintenance cumbersome and error-prone.
- Suggested Solution: Extract the certificate copying logic into a reusable shell script or shared Jenkins library function. For example:
Then call this script from each Jenkinsfile:
# shell script: copy_mysql_certs.sh mkdir -p "$1/mysql-ssl" cp /var/lib/mysql/ca.pem /var/lib/mysql/client-cert.pem /var/lib/mysql/client-key.pem "$1/mysql-ssl/" chgrp 1000 "$1/mysql-ssl" "$1/mysql-ssl/*.pem" chmod 0750 "$1/mysql-ssl" chmod 0640 "$1/mysql-ssl/*.pem"
sh label: "copy mysql certs", script: "./scripts/copy_mysql_certs.sh ${WORKSPACE}"
- Files: All
-
Error Handling During Certificate Operations
- Files: All
Jenkinsfilescripts (latest,next-gen,release-*) - Lines: e.g.,
mkdir -p "${WORKSPACE}/mysql-ssl" - Issue: Commands like
mkdirandcpsilently fail if issues occur, such as permission errors or disk space issues. - Suggested Solution: Add error handling to ensure failed commands are logged and the pipeline exits early. For example:
mkdir -p "${WORKSPACE}/mysql-ssl" || { echo "Failed to create directory"; exit 1; }
- Files: All
-
Hardcoded Permissions
- Files: All
Jenkinsfilescripts (latest,next-gen,release-*) - Lines: e.g.,
chmod 0750 "${WORKSPACE}/mysql-ssl" - Issue: Hardcoded permissions like
0750and0640may not account for edge cases where a stricter or looser permission model might be required (e.g., for group execution). - Suggested Solution: Introduce a configuration variable or parameter for permissions, allowing flexibility across environments.
- Files: All
Best Practices
-
Testing Coverage for Permission Hardening
- Files: N/A (PR description)
- Issue: While permissions for certificates have been hardened, there is no explicit mention of test coverage for scenarios involving incorrect or missing permissions.
- Suggested Solution: Include tests in the CI pipeline that validate the permissions on the staged certificates. For example:
ls -l "${WORKSPACE}/mysql-ssl"/*.pem | awk '{print $1}' | grep -v -E '^.{4}r--r--$' && echo "Permission error detected!" && exit 1
-
Documentation for New TLS Fixture Handling
- Files: N/A
- Issue: The PR introduces significant changes to how TLS fixtures are staged and mounted, but there is no mention of updates to documentation for developers or CI maintainers.
- Suggested Solution: Update the relevant documentation to include:
- Details on the new
emptyDirmount. - Explanation of certificate permissions and staging process.
- Steps for debugging TLS_GROUP failures related to certificates.
- Details on the new
-
Consistency in YAML Formatting
- Files: All
pod.yamlfiles (latest,next-gen,release-*) - Issue: Indentation and formatting inconsistencies, such as mismatched alignment between
volumesand other fields, can reduce readability. - Suggested Solution: Standardize YAML formatting using tools like
yamllintor IDE plugins.
- Files: All
Summary of Suggested Changes
- Add validation steps for certificate presence and integrity before copying.
- Extract repetitive certificate logic into a reusable script or Jenkins shared library.
- Enhance error handling for shell commands.
- Improve documentation of TLS fixture handling.
- Standardize YAML formatting for better readability.
These changes will improve maintainability, reduce risks of regressions, and ensure the robustness of the fix across pipelines.
Preserve the pre-existing memory budgets of the six release DM test templates while restoring TLS fixtures. Exempt only the 19 affected container/template pairs through 2026-12-31, pending sizing review with the job owners and usage evidence.
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR addresses a regression in DM integration tests that fail in the TLS_GROUP due to missing TLS certificates. The solution involves creating a dedicated emptyDir volume in the golang container, copying the required certificates without sudo, and configuring permissions to allow the Jenkins user access while maintaining security. The approach appears well-documented and security-conscious, but there are areas for improvement in terms of maintainability and validation coverage.
Critical Issues
- Validation Gap for End-to-End Testing:
- File(s): All modified Jenkinsfiles and related pod configurations
- Issue: The PR description notes that full end-to-end verification of the TLS_GROUP execution in CI has not been done, leaving uncertainty about the actual fix's effectiveness.
- Suggestion: Ensure that CI pipelines are replayed and tested end-to-end to confirm the changes work as intended. Without this, the PR risks introducing new regressions.
Code Improvements
-
Hardcoding Group ID (1000):
- File(s):
jenkins/jobs/pingcap/tiflow/*/Jenkinsfile(multiple instances) - Issue: Hardcoding the group ID
1000assumes uniform user/group configurations across environments, which could break portability or future compatibility. - Suggestion: Consider abstracting
1000into a constant or environment variable that can be configured dynamically, e.g.:TLS_GROUP_ID=1000 chgrp $TLS_GROUP_ID "${WORKSPACE}/mysql-ssl" "${WORKSPACE}/mysql-ssl/*.pem"
- File(s):
-
Redundant Volume Definition:
- File(s):
pod.yamlfiles across releases - Issue: The definition of the
emptyDirvolume is repeated across multiple files. This duplication can lead to inconsistencies if changes are made in the future. - Suggestion: Abstract the volume configuration into a common template or shared configuration file for reuse across all pipelines.
- File(s):
Best Practices
-
Missing Tests for Edge Cases:
- File(s): Integration tests (implicit)
- Issue: The PR does not specify additional test cases for edge scenarios, such as missing or corrupted certificate files.
- Suggestion: Add tests to simulate scenarios where the required certificate files (
ca.pem,client-cert.pem,client-key.pem) are missing, inaccessible, or corrupted. Ensure proper error handling and logging.
-
Documentation Improvement:
- File(s): PR description and Jenkinsfiles
- Issue: While the PR description is comprehensive, documentation within the code (e.g., inline comments in Jenkinsfiles) about the rationale for permissions and
emptyDirusage could be clearer. - Suggestion: Add brief comments explaining why specific permissions (e.g.,
0750for directories and0640for files) are set and their security implications.
-
Avoid Hardcoded File Paths:
- File(s):
jenkins/jobs/pingcap/tiflow/*/Jenkinsfile(multiple instances) - Issue: The file paths, such as
/var/lib/mysqland${WORKSPACE}/mysql-ssl, are hardcoded, making the code less flexible. - Suggestion: Use variables or configurations to define paths dynamically, e.g.:
MYSQL_TLS_CERT_DIR="/var/lib/mysql" SSL_WORKSPACE_DIR="${WORKSPACE}/mysql-ssl" cp "$MYSQL_TLS_CERT_DIR/ca.pem" "$SSL_WORKSPACE_DIR/"
- File(s):
Summary of Key Suggestions
- Perform end-to-end testing to validate the fix fully in CI.
- Abstract hardcoded values like
1000and file paths into configurable variables. - Consolidate redundant volume definitions into shared templates for maintainability.
- Add edge case tests for missing/corrupted TLS certificates.
- Improve inline documentation for clarity and maintainability.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dillon-zheng, wk989898 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
Problem
DM integration tests fail in
TLS_GROUPwhen creating the TLS source, withopen /var/lib/mysql/ca.pem: no such file or directory. This blocks pingcap/tiflow#12990; the same error occurs in release-8.5 builds 99, 103, and 106.PR #5531 removed the block that copied MySQL certificates from the shared workspace into the golang container. Certificates are still staged by mysql1 under
${WORKSPACE}/mysql-ssl, but the test harness reads them from/var/lib/mysqlin the golang container. The current agent Dockerfile does not pre-create that path for the build user.Changes
emptyDirat/var/lib/mysqlin the golang container. The existing pod-levelfsGroup: 1000makes the volume writable by the Jenkins user.TLS_GROUP, using ordinarycpwithout sudo, root access, or an agent image rebuild.ca.pem.ca.pem,client-cert.pem, andclient-key.pem; keep CA/server private keys inside mysql1. Set workspace fixtures to group 1000, directory mode 0750, and file mode 0640. Use umask 077 for the copies in golang.Validation
git diff --check— passed..ci/check-jenkins-job-references.sh --quiet— passed./pipeline-model-converter/validate— all eight changed Jenkinsfiles successfully validated.fsGroup: 1000, the emptyDir definition, and that only golang mounts the fixture volume at/var/lib/mysql.bash -non all 16 certificate-staging and test shell blocks — passed.Full DM integration execution and Jenkins replay have not been run: this session has no Jenkins replay credentials and the local Docker daemon is unavailable. The TLS_GROUP jobs still need end-to-end verification in CI.
Security review
This preserves the goal of #5531: no sudo/sudoers changes, root user overrides, privileged containers, hostPath mounts, capabilities, Secrets, or ServiceAccount permissions are added. The new emptyDir is mounted only in golang, is local to the test Pod, and does not expose mysql1 or mysql2 data directories. The existing fsGroup remains 1000. Certificates come from the disposable upstream MySQL instance; no production credentials are introduced. The source staging step runs in the existing mysql1 container; the final copy runs as the existing Jenkins user.
The first revision passed
pull-verify-pod-privilege-policy,pull-verify-secret-scan,pull-verify-storage-class-policy, Pod YAML validation, and Jenkins validation. The latest commit restricts key selection and permissions further; its CI must complete again. Full TLS_GROUP execution remains necessary to establish that the regression is fixed.Resource-policy follow-up
The original
pull-verify-pod-resource-policyfailure came from 19 pre-existing violations in the six release DM test templates, exposed by the TLS volume change. The follow-up adds explicit container/path exemptions in.ci/pod-resource-policy-allowlist.txtthrough 2026-12-31 instead of changing runtime resource budgets or weakening the gate. No container wildcards are used; central approval is still required for the exemption ledger.Validation of the follow-up:
pod-resourcegate on all eight changed Pod templates: passed (CPU mismatch warnings remain non-blocking)..ci/test-verify-pod-resource-policy.sh: all nine cases passed.git diff --check: passed.The updated commit
a08d9fc35has been pushed, and pull-verify-pod-resource-policy now reports SUCCESS. Jenkins replay and the remaining pending checks still need to complete; this PR has not been merged.