Repository navigation
chore(tekton): drop task retries and raise build timeouts - #5588
Conversation
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR modifies Tekton build pipelines and triggers for v1 by removing task-level retries on package build tasks to encourage failing fast and shifting retries to the pipeline level, and by increasing build timeouts for slower repositories in env-gcp triggers to prevent premature timeouts. The approach is straightforward YAML editing with clear targeted changes. The PR appears clean and focused, with no obvious errors in the diffs. The changes align with good CI/CD practices by reducing redundant retries and accommodating slow builds.
Critical Issues
- No critical bugs or broken functionality found. The removal of
retries: 2is consistent across relevant tasks, and timeout increases are clearly targeted.
Code Improvements
-
Consider adding comments in YAML to explain rationale for removal of retries and increased timeouts
Files:tekton/v1/pipelines/pingcap-build-package-linux.yaml(lines ~180-260)tekton/v1/pipelines/pingcap-build-package-darwin-mac.yaml(lines ~80-90)tekton/v1/triggers/triggers/env-gcp/_/*.yaml(where timeouts changed)
Why: Adding short comments near these changes would help future maintainers understand why retries were removed (to fail fast and avoid duplication) and why timeouts were increased (slow repos). This is especially helpful for YAML pipelines that may lack context.
Suggested:
# Removed retries to fail fast and let pipeline-level retries handle failures - name: build-binaries # retries: 2 # Removed retries # Increased timeout from 2h to 3h due to observed slow builds on pingcap/tiflash 'pingcap/tiflash': { 'timeout': '3h', ... }
-
Validate timeout format consistency
The timeouts use strings like'3h'and'30m'. Ensure this format matches Tekton's expected duration format exactly (e.g.,3h0m0sor3h). If'3h'is valid, no action needed; otherwise, consider making explicit full durations to avoid ambiguity. -
Edge case: no fallback or warning if task retries are removed
Removing task retries means transient failures will cause immediate task failure. Consider adding pipeline-level retry logic or documenting how pipeline retries should be configured to avoid build flakiness.
Best Practices
-
Testing coverage and verification
The PR description notes manual verification steps but no automated tests. Since these are CI config changes, recommend adding automated test coverage if you have pipeline YAML validation or integration testing to catch syntax or config errors early. -
Consistency in YAML indentation and style
The YAML indentation and style appear consistent. Continue enforcing YAML linter checks in CI to avoid indentation errors that are hard to spot. -
Naming conventions
Task and pipeline names are consistent with existing conventions. -
Documentation updates
If there is a central README or docs describing build retries and timeouts, update them to reflect this new practice of no task retries and longer timeouts for slow repos.
Overall, this is a focused and well-scoped chore PR that improves build pipeline behavior. Adding a few clarifying comments and confirming timeout format validity would strengthen maintainability and prevent confusion in the future. Consider pipeline-level retry strategy documentation as a follow-up.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…ry (#5591) ## Why Since #5474 removed the `ce-paramGithubRepo` allowlist bypass, the devbuild filters have also required `ce-paramProduct` to equal `body.repository.name`. That equality is neither a security boundary nor a routing input: - the repository allowlist is body-only, keyed on `body.repository.full_name`; no product header widens it; - every non-Serverless route derives `component` from the allowlisted repository (`repository.name`, plus the `tidbcloud/*` and `pingcap/ticse` overrides), so the header cannot change what is built either. What it did do was reject legitimate builds. tibuild labels a component that lives in a repository with another name, and `githubRepo` may be passed explicitly: | product | repository | | --- | --- | | `ticdc-newarch` | `pingcap/ticdc` | | `br`, `dumpling`, `tidb-lightning` | `pingcap/tidb` | | `drainer`, `pump` | `pingcap/tidb-binlog` | | `dm` | `pingcap/tiflow` | A rejected event creates no `PipelineRun` and the listener logs nothing, so the devbuild stays `PENDING` forever. Five of them are stuck right now (every devbuild since 2026-11-06 whose product does not equal its repository name; all 120 that do match have real statuses): | devbuild | created (UTC) | product | repository | ref | kind | | --- | --- | --- | --- | --- | --- | | 3030009 | 2026-11-13 09:35 | `ticdc-newarch` | `pingcap/ticdc` | `tag/v8.5.4-20261113-5691a52` | hotfix | | 2970069 | 2026-11-11 08:11 | `br` | `pingcap/tidb` | `tag/v8.5.4-20261111-4d32ecb` | hotfix | | 3030012 | 2026-11-13 10:53 | `ticdc` | `pingcap/tiflow` | `branch/master` | devbuild | | 2970059 | 2026-11-10 09:55 | `ticdc` | `pingcap/tiflow` | `pull/12954` | devbuild | | 2970060 | 2026-11-10 09:56 | `ticdc` | `pingcap/tiflow` | `pull/12954` | devbuild | The last `ticdc-newarch` hotfix that actually ran is 2880034 (2026-11-03), i.e. before the merge. ## Change Drop the product/repository equality from all nine `fake-github-*` routes (`branch-push`, `pr`, `tag-create` × plain, `-darwin`, `-single-platform`). The Serverless routes keep it: there the product *is* the component (`extensions.build.serverless ? extensions.build.product : ...`), so it still has to name the Serverless repository. `tekton/tests/trigger-security.test.ts` now asserts the replacement invariant — whatever the header says, the route and the component come from the repository: ``` ACCEPT pingcap/ticdc product=ticdc-newarch component=ticdc ACCEPT pingcap/tidb product=br component=tidb ACCEPT pingcap/tiflow product=ticdc component=tiflow REJECT attacker/evil product=tidb component=- ``` Note the third line: with the product decoupled, `ticdc` on `pingcap/tiflow` builds the **tiflow** component. Nothing about the request's product label influences the build, which is the point. ## Security Unchanged and unaffected: - the allowlist is still `body.repository.full_name in [...]`, and `trigger-security.test.ts` still asserts that no route reads `ce-paramGithubRepo`, keys the allowlist on `repository.name`, or binds `repository.clone_url`; - a forged event for a non-allowlisted repository is still rejected with arbitrary product headers; - a Serverless product still cannot route a non-Serverless repository (that branch takes the component from the header, so it keeps the equality). The `fake-github-*` triggers are only selected by `el-tibuild`, which verifies tibuild's CloudEvent HMAC (PingCAP-QE/ee-ops#2619). ## Verification ``` kustomize build tekton/v1/triggers/triggers/env-gcp -> OK deno test --frozen --lock=tekton/tests/deno.lock --allow-read --allow-write \ --allow-env --allow-run=git,sh,deno tekton/tests/*.test.ts -> ok | 43 passed (302 steps) | 0 failed ``` ## Second commit (please review separately) `deno test tekton/tests/*.test.ts` is red on `main`, independent of this change — I reproduced it on a clean `origin/main` worktree (3f2c3d0). The second commit repairs the TiCSE formal-route assertions: - #5533 added `pingcap/ticse` to `cseRepositories`, so the tag and branch-creation assertions ran with a `branch-push` payload; those triggers require `body.ref_type` (#5569) and threw `field not found: ref_type`. They now get payloads that carry it. - #5588 raised `tidbcloud/cloud-storage-engine` to `1h` on the branch-push route only, while the test asserted a single `30m` for both routes. It now asserts each route's own value. If you would rather land that as its own PR, say so and I will split it out. ## Follow-ups (not in this PR) - **tibuild (PingCAP-QE/ee-apps)**: a devbuild whose CloudEvents produced no `PipelineRun` stays `PENDING` forever — 80 such records are in the database, the oldest from 2026-08-10. A watchdog (`PENDING` + `triggersEventIDs` non-empty + `pipelines` empty after N minutes → error) would have turned this incident into a visible failure instead of a stalled queue. - **`git-create-tag-build-ng.yaml`**: the tag route still carries the old `30m` budget for `tidbcloud/cloud-storage-engine` while the branch-push route has `1h` after #5588. Looks unintended; worth a one-line follow-up.
Summary
Two Tekton adjustments on the v1 build pipelines/triggers:
retries: 2from the package build pipelines(
pingcap-build-package-linux,pingcap-build-package-darwin-mac).Retrying whole tasks duplicates work and hides failures; the affected
tasks (
build-binaries,build-images,deliver-binaries) should failfast and be re-run from the pipeline level when needed.
triggers:
pingcap/tiflash:2h->3h(create-tag, push-branch, community,hotfix and ng triggers)
tikv/pd:20m->30mtidbcloud/cloud-storage-engine:30m->1hFiles changed
tekton/v1/pipelines/pingcap-build-package-linux.yamltekton/v1/pipelines/pingcap-build-package-darwin-mac.yamltekton/v1/triggers/triggers/env-gcp/_/git-create-tag-build-ng.yamltekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-ng.yamltekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-community-linux.yamltekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-hotfix-linux.yamlTest Plan
PipelineRuntimeouts take effectbuild-binariesno longer retries inside thePipelineRun