Skip to content

fix(prow-jobs): run the Tekton validator under sh again and capture kubectl output - #5550

Merged
ti-chi-bot[bot] merged 1 commit into
mainfrom
fix/tekton-validator-sh
Nov 12, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
mainfrom
fix/tekton-validator-sh

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Problem

#5546 changed this presubmit to command: [bash, -ceo, pipefail] to give the
kubectl … | sed pipeline a meaningful status. The job's image is
alpine:3.24, which does not ship bash, so the container never started:

exec: "bash": executable file not found in $PATH

Every pull request touching tekton/v1/**/*.yaml failed at container creation.

Change

Back to sh -ce, and detect failure without pipefail by capturing the command's
output before formatting it:

rc=0
out=$(kubectl apply --dry-run=server -f "$f" 2>&1) || rc=$?
printf '%s\n' "$out" | sed 's/^/  /'
[ "$rc" -eq 0 ] || failed=1

kubectl … | sed reported sed's status — the reason the original check could
never fail. Capturing first keeps kubectl's own status, which is what the gate
needs, without requiring a shell the image does not have.

Verification

  • Reproduced the original break locally: docker run --rm alpine:3.24 bash -c …
    → exec: "bash": executable file not found.
  • Ran the new snippet in alpine:3.24 (busybox ash) with a failing command:
    output is captured and indented, failed=1, exit status 1.
  • This PR touches one Tekton file (# touched … comment) so
    pull-verify-tekton-yaml actually runs here — the job is the check.

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary
This PR fixes a failing presubmit job caused by the use of bash in an Alpine 3.24 image, which lacks bash. The change reverts the shell from bash -ceo pipefail to sh -ce and modifies the pipeline to capture kubectl output and its exit code properly without relying on pipefail. The approach is sound and addresses the root cause effectively. The updated logic now correctly detects failures and formats the output for easier debugging. The YAML change is minimal and just triggers the validation job. Overall, the fix is well-justified and clean.


Critical Issues

  • None found. The fix correctly addresses the broken container startup and failure detection.

Code Improvements

  • prow-jobs/pingcap-qe/ci/presubmits.yaml (lines ~315-332):

    • The current snippet uses a combination of capturing stdout/stderr and checking the return code in a separate variable rc:

      rc=0
      out=$(kubectl apply --dry-run=server -f "$f" 2>&1) || rc=$?
      printf '%s\n' "$out" | sed 's/^/  /'
      [ "$rc" -eq 0 ] || failed=1
    • Suggestion: To improve readability and robustness, consider explicitly redirecting stderr only if needed, or adding comments clarifying why 2>&1 is necessary (to capture both stdout and stderr in out). Also, use || to assign rc only if the command fails, which is done correctly here, but might be more idiomatic as:

      out=$(kubectl apply --dry-run=server -f "$f" 2>&1)
      rc=$?
    • This is more straightforward and equivalent.

    • Minor: Add a blank line before the loop for clarity.

    • Optional: In case of multiple files failing, it might be helpful to print a summary of all failures, but this is not critical.


Best Practices

  • prow-jobs/pingcap-qe/ci/presubmits.yaml (lines ~315-332):

    • The inline shell script is moderately complex; consider extracting this into a dedicated script file or reusable snippet if it grows, to improve maintainability.

    • Comment clarity is good, but the comment inside the loop could be formatted better:

      # Capture kubectl output before formatting it.
      # The image lacks bash, so pipefail is unavailable, and sed's status
      # would otherwise mask kubectl's failure.
    • This makes the intent clearer.

  • Testing coverage:

    • The PR description mentions manual testing in Alpine 3.24 and reproducing the failure, which is good.

    • However, consider adding or updating any CI test to verify the new shell command snippet works as intended, or add a unit test if applicable.

  • YAML file (tekton/v1/tasks/pingcap-get-set-release-version.yaml):

    • The added comment # touched to exercise the Tekton validation presubmit in CI is helpful.

    • No issues here.


Summary of actionable suggestions

# prow-jobs/pingcap-qe/ci/presubmits.yaml, around line 320:

# Add blank line before loop for readability
...

failed=0

for f in $changed_files; do
  echo "Validating: $f"
  # Capture kubectl output before formatting it.
  # The image lacks bash, so pipefail is unavailable, and sed's status
  # would otherwise mask kubectl's failure.
  out=$(kubectl apply --dry-run=server -f "$f" 2>&1)
  rc=$?
  printf '%s\n' "$out" | sed 's/^/  /'
  [ "$rc" -eq 0 ] || failed=1
done
  • Consider extracting this shell snippet into a separate script file if it grows.
  • Add or ensure automated CI tests cover this validation logic.
  • Improve comments for clarity as shown above.

Overall, this is a well-targeted and effective fix addressing a real blocker with minimal disruption. The suggestions above are minor improvements to clarity and maintainability.

@ti-chi-bot ti-chi-bot Bot added the size/S label Nov 12, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

@wuhuizuo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-verify-tekton-yaml d1ab089 link true /test pull-verify-tekton-yaml

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

…ubectl output

The validation presubmit ran under `bash -ceo pipefail`, but the image is
`alpine:3.24`, which has no bash — the container failed to start
(`exec: "bash": executable file not found`), so the job never ran at all.

Go back to `sh -ce` and get the failure detection without pipefail: capture
`kubectl`'s output first, then format it. `kubectl ... | sed` reported sed's
status, which is how the original check could never fail; `out=$(...)` keeps the
command's own status.

Verified in `alpine:3.24` with busybox ash: a failing command sets `failed=1` and
the step exits 1.
@wuhuizuo
wuhuizuo force-pushed the fix/tekton-validator-sh branch from d1ab089 to 7520cca Compare November 12, 2026 06:43

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary
This PR fixes a presubmit job that was broken by switching from sh to bash in an Alpine 3.24 image that does not have bash installed. The fix reverts the shell to sh -ce and modifies the pipeline command to capture and print kubectl output while still preserving the correct exit status without relying on pipefail or bash. The approach is sound, solves the problem effectively, and the code is clear and well-commented.


Critical Issues

  • None detected.

Code Improvements

  • Shell idiom for capturing exit code (prow-jobs/pingcap-qe/ci/presubmits.yaml, lines ~324-331)
    The pattern used to capture the exit code is fine, but slightly more idiomatic and concise is:

    if ! out=$(kubectl apply --dry-run=server -f "$f" 2>&1); then
      rc=$?
    else
      rc=0
    fi
    printf '%s\n' "$out" | sed 's/^/  /'
    [ "$rc" -eq 0 ] || failed=1

    This makes it clearer that the command is attempted inside if ! and the exit code is captured only on failure.

  • Avoiding use of sed for indentation
    Since Alpine's sed is available, this is fine. However, for better portability or readability, consider using shell parameter expansion or awk if the codebase grows.

  • Use of apk add every run (line ~319)
    The job installs kubectl yq-go git on every run. If the job runs frequently and image size or startup time matters, consider building a custom image with these pre-installed. This is not a blocking concern but can improve job performance.


Best Practices

  • Add a comment about why pipefail is not used (line ~322)
    The added comment is good; consider making it even clearer by explicitly stating that sh in Alpine's busybox shell does not support pipefail and bash is not present, so this workaround is necessary.

  • Test coverage for failure cases
    The PR description states local manual testing. Consider adding a unit or integration test snippet or documenting how to reproduce the failure in CI to ensure this fix is verified automatically.

  • Style / YAML Formatting
    The YAML indentation and quoting look correct. No issues here.


Summary of actionable suggestions

- rc=0
- out=$(kubectl apply --dry-run=server -f "$f" 2>&1) || rc=$?
- printf '%s\n' "$out" | sed 's/^/  /'
- [ "$rc" -eq 0 ] || failed=1
+ if ! out=$(kubectl apply --dry-run=server -f "$f" 2>&1); then
+   rc=$?
+ else
+   rc=0
+ fi
+ printf '%s\n' "$out" | sed 's/^/  /'
+ [ "$rc" -eq 0 ] || failed=1
  • Optionally document the lack of pipefail support explicitly for maintainers unfamiliar with Alpine shell limitations.
  • Consider caching dependencies or using a custom image for faster job startup in presubmit jobs.
  • Add or document automated testing for this fix in CI.

Overall, this is a well-scoped, high-impact fix addressing a real breakage with minimal disruption.

@wuhuizuo

Copy link
Copy Markdown
Contributor Author

/approve

@ti-chi-bot

ti-chi-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 12, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit a39f5d1 into main Nov 12, 2026
10 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the fix/tekton-validator-sh branch November 12, 2026 06:50
ti-chi-bot Bot pushed a commit that referenced this pull request Nov 12, 2026
…a cluster (#5553)

## Problem

#5550 fixed the shell (the job ran bash on an alpine image and could not
start),
and the canary then showed the second half: `kubectl apply
--dry-run=server`
cannot work in this pod — no kubeconfig is mounted, so kubectl dials
`localhost:8080` and fails with `connect: connection refused`.

That also explains the original defect: the `kubectl … | sed` pipeline
reported
sed's status, so a validation failure was masked. Had it not been
masked, the job
would have failed on every Tekton change for a reason unrelated to the
manifest.

## Change

Validate what can be validated without a cluster:

- the file parses as YAML;
- `kind` is a Tekton kind (`Task`, `TaskRun`, `Pipeline`, `PipelineRun`,
`Trigger`,
`TriggerTemplate`, `TriggerBinding`, `ClusterTriggerBinding`,
`EventListener`,
  `Interceptor`);
- `apiVersion` and `metadata.name` are set.

This is a real gate — a typo, a missing field or a non-Tekton document
fails it —
and it needs no credentials. The step now also uses the same
capture-then-format
pattern as #5550, so a failure actually reaches the exit status.

## Verification

- Locally: a valid task passes; a file with `kind: NotATektonKind` fails
with
  `unknown or missing kind` and exit 1.
- This PR does not touch `tekton/` on purpose: the presubmit reads its
definition
from the base branch, so a Tekton change here would run the *old* body.
A canary
  PR follows once this is merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant