Repository navigation
fix(prow-jobs): run the Tekton validator under sh again and capture kubectl output - #5550
Conversation
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR fixes a failing presubmit job caused by the use of bash in an Alpine 3.24 image, which lacks bash. The change reverts the shell from bash -ceo pipefail to sh -ce and modifies the pipeline to capture kubectl output and its exit code properly without relying on pipefail. The approach is sound and addresses the root cause effectively. The updated logic now correctly detects failures and formats the output for easier debugging. The YAML change is minimal and just triggers the validation job. Overall, the fix is well-justified and clean.
Critical Issues
- None found. The fix correctly addresses the broken container startup and failure detection.
Code Improvements
-
prow-jobs/pingcap-qe/ci/presubmits.yaml (lines ~315-332):
-
The current snippet uses a combination of capturing stdout/stderr and checking the return code in a separate variable
rc:rc=0 out=$(kubectl apply --dry-run=server -f "$f" 2>&1) || rc=$? printf '%s\n' "$out" | sed 's/^/ /' [ "$rc" -eq 0 ] || failed=1
-
Suggestion: To improve readability and robustness, consider explicitly redirecting stderr only if needed, or adding comments clarifying why
2>&1is necessary (to capture both stdout and stderr inout). Also, use||to assign rc only if the command fails, which is done correctly here, but might be more idiomatic as:out=$(kubectl apply --dry-run=server -f "$f" 2>&1) rc=$?
-
This is more straightforward and equivalent.
-
Minor: Add a blank line before the loop for clarity.
-
Optional: In case of multiple files failing, it might be helpful to print a summary of all failures, but this is not critical.
-
Best Practices
-
prow-jobs/pingcap-qe/ci/presubmits.yaml (lines ~315-332):
-
The inline shell script is moderately complex; consider extracting this into a dedicated script file or reusable snippet if it grows, to improve maintainability.
-
Comment clarity is good, but the comment inside the loop could be formatted better:
# Capture kubectl output before formatting it. # The image lacks bash, so pipefail is unavailable, and sed's status # would otherwise mask kubectl's failure.
-
This makes the intent clearer.
-
-
Testing coverage:
-
The PR description mentions manual testing in Alpine 3.24 and reproducing the failure, which is good.
-
However, consider adding or updating any CI test to verify the new shell command snippet works as intended, or add a unit test if applicable.
-
-
YAML file (tekton/v1/tasks/pingcap-get-set-release-version.yaml):
-
The added comment
# touched to exercise the Tekton validation presubmit in CIis helpful. -
No issues here.
-
Summary of actionable suggestions
# prow-jobs/pingcap-qe/ci/presubmits.yaml, around line 320:
# Add blank line before loop for readability
...
failed=0
for f in $changed_files; do
echo "Validating: $f"
# Capture kubectl output before formatting it.
# The image lacks bash, so pipefail is unavailable, and sed's status
# would otherwise mask kubectl's failure.
out=$(kubectl apply --dry-run=server -f "$f" 2>&1)
rc=$?
printf '%s\n' "$out" | sed 's/^/ /'
[ "$rc" -eq 0 ] || failed=1
done- Consider extracting this shell snippet into a separate script file if it grows.
- Add or ensure automated CI tests cover this validation logic.
- Improve comments for clarity as shown above.
Overall, this is a well-targeted and effective fix addressing a real blocker with minimal disruption. The suggestions above are minor improvements to clarity and maintainability.
|
@wuhuizuo: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
…ubectl output The validation presubmit ran under `bash -ceo pipefail`, but the image is `alpine:3.24`, which has no bash — the container failed to start (`exec: "bash": executable file not found`), so the job never ran at all. Go back to `sh -ce` and get the failure detection without pipefail: capture `kubectl`'s output first, then format it. `kubectl ... | sed` reported sed's status, which is how the original check could never fail; `out=$(...)` keeps the command's own status. Verified in `alpine:3.24` with busybox ash: a failing command sets `failed=1` and the step exits 1.
d1ab089 to
7520cca
Compare
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR fixes a presubmit job that was broken by switching from sh to bash in an Alpine 3.24 image that does not have bash installed. The fix reverts the shell to sh -ce and modifies the pipeline command to capture and print kubectl output while still preserving the correct exit status without relying on pipefail or bash. The approach is sound, solves the problem effectively, and the code is clear and well-commented.
Critical Issues
- None detected.
Code Improvements
-
Shell idiom for capturing exit code (prow-jobs/pingcap-qe/ci/presubmits.yaml, lines ~324-331)
The pattern used to capture the exit code is fine, but slightly more idiomatic and concise is:if ! out=$(kubectl apply --dry-run=server -f "$f" 2>&1); then rc=$? else rc=0 fi printf '%s\n' "$out" | sed 's/^/ /' [ "$rc" -eq 0 ] || failed=1
This makes it clearer that the command is attempted inside
if !and the exit code is captured only on failure. -
Avoiding use of
sedfor indentation
Since Alpine'ssedis available, this is fine. However, for better portability or readability, consider using shell parameter expansion orawkif the codebase grows. -
Use of
apk addevery run (line ~319)
The job installskubectl yq-go giton every run. If the job runs frequently and image size or startup time matters, consider building a custom image with these pre-installed. This is not a blocking concern but can improve job performance.
Best Practices
-
Add a comment about why
pipefailis not used (line ~322)
The added comment is good; consider making it even clearer by explicitly stating thatshin Alpine's busybox shell does not supportpipefailandbashis not present, so this workaround is necessary. -
Test coverage for failure cases
The PR description states local manual testing. Consider adding a unit or integration test snippet or documenting how to reproduce the failure in CI to ensure this fix is verified automatically. -
Style / YAML Formatting
The YAML indentation and quoting look correct. No issues here.
Summary of actionable suggestions
- rc=0
- out=$(kubectl apply --dry-run=server -f "$f" 2>&1) || rc=$?
- printf '%s\n' "$out" | sed 's/^/ /'
- [ "$rc" -eq 0 ] || failed=1
+ if ! out=$(kubectl apply --dry-run=server -f "$f" 2>&1); then
+ rc=$?
+ else
+ rc=0
+ fi
+ printf '%s\n' "$out" | sed 's/^/ /'
+ [ "$rc" -eq 0 ] || failed=1- Optionally document the lack of
pipefailsupport explicitly for maintainers unfamiliar with Alpine shell limitations. - Consider caching dependencies or using a custom image for faster job startup in presubmit jobs.
- Add or document automated testing for this fix in CI.
Overall, this is a well-scoped, high-impact fix addressing a real breakage with minimal disruption.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…a cluster (#5553) ## Problem #5550 fixed the shell (the job ran bash on an alpine image and could not start), and the canary then showed the second half: `kubectl apply --dry-run=server` cannot work in this pod — no kubeconfig is mounted, so kubectl dials `localhost:8080` and fails with `connect: connection refused`. That also explains the original defect: the `kubectl … | sed` pipeline reported sed's status, so a validation failure was masked. Had it not been masked, the job would have failed on every Tekton change for a reason unrelated to the manifest. ## Change Validate what can be validated without a cluster: - the file parses as YAML; - `kind` is a Tekton kind (`Task`, `TaskRun`, `Pipeline`, `PipelineRun`, `Trigger`, `TriggerTemplate`, `TriggerBinding`, `ClusterTriggerBinding`, `EventListener`, `Interceptor`); - `apiVersion` and `metadata.name` are set. This is a real gate — a typo, a missing field or a non-Tekton document fails it — and it needs no credentials. The step now also uses the same capture-then-format pattern as #5550, so a failure actually reaches the exit status. ## Verification - Locally: a valid task passes; a file with `kind: NotATektonKind` fails with `unknown or missing kind` and exit 1. - This PR does not touch `tekton/` on purpose: the presubmit reads its definition from the base branch, so a Tekton change here would run the *old* body. A canary PR follows once this is merged.
Problem
#5546 changed this presubmit to
command: [bash, -ceo, pipefail]to give thekubectl … | sedpipeline a meaningful status. The job's image isalpine:3.24, which does not ship bash, so the container never started:Every pull request touching
tekton/v1/**/*.yamlfailed at container creation.Change
Back to
sh -ce, and detect failure without pipefail by capturing the command'soutput before formatting it:
kubectl … | sedreported sed's status — the reason the original check couldnever fail. Capturing first keeps kubectl's own status, which is what the gate
needs, without requiring a shell the image does not have.
Verification
docker run --rm alpine:3.24 bash -c …→
exec: "bash": executable file not found.alpine:3.24(busybox ash) with a failing command:output is captured and indented,
failed=1, exit status 1.# touched …comment) sopull-verify-tekton-yamlactually runs here — the job is the check.