Skip to content

chore(tekton): drop task retries and raise build timeouts - #5588

Merged
ti-chi-bot[bot] merged 2 commits into
mainfrom
chore/tekton-drop-retries-and-raise-build-timeouts
Nov 13, 2026
Merged

ti-chi-bot[bot] merged 2 commits into
mainfrom
chore/tekton-drop-retries-and-raise-build-timeouts

Conversation

@wuhuizuo

Copy link
Copy Markdown
Contributor

Summary

Two Tekton adjustments on the v1 build pipelines/triggers:

  1. Drop task-level retries: 2 from the package build pipelines
    (pingcap-build-package-linux, pingcap-build-package-darwin-mac).
    Retrying whole tasks duplicates work and hides failures; the affected
    tasks (build-binaries, build-images, deliver-binaries) should fail
    fast and be re-run from the pipeline level when needed.
  2. Raise build timeouts for the slow repositories in the env-gcp
    triggers:
    • pingcap/tiflash: 2h -> 3h (create-tag, push-branch, community,
      hotfix and ng triggers)
    • tikv/pd: 20m -> 30m
    • tidbcloud/cloud-storage-engine: 30m -> 1h

Files changed

  • tekton/v1/pipelines/pingcap-build-package-linux.yaml
  • tekton/v1/pipelines/pingcap-build-package-darwin-mac.yaml
  • tekton/v1/triggers/triggers/env-gcp/_/git-create-tag-build-ng.yaml
  • tekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-ng.yaml
  • tekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-community-linux.yaml
  • tekton/v1/triggers/triggers/env-gcp/_/git-push-branch-build-hotfix-linux.yaml

Test Plan

  • YAML only; no resource add/remove, so the Tekton kustomizations stay unchanged
  • Verify on the next tiflash / pd / cloud-storage-engine build that the new PipelineRun timeouts take effect
  • Verify that a failing build-binaries no longer retries inside the PipelineRun

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary
This PR modifies Tekton build pipelines and triggers for v1 by removing task-level retries on package build tasks to encourage failing fast and shifting retries to the pipeline level, and by increasing build timeouts for slower repositories in env-gcp triggers to prevent premature timeouts. The approach is straightforward YAML editing with clear targeted changes. The PR appears clean and focused, with no obvious errors in the diffs. The changes align with good CI/CD practices by reducing redundant retries and accommodating slow builds.


Critical Issues

  • No critical bugs or broken functionality found. The removal of retries: 2 is consistent across relevant tasks, and timeout increases are clearly targeted.

Code Improvements

  • Consider adding comments in YAML to explain rationale for removal of retries and increased timeouts
    Files:

    • tekton/v1/pipelines/pingcap-build-package-linux.yaml (lines ~180-260)
    • tekton/v1/pipelines/pingcap-build-package-darwin-mac.yaml (lines ~80-90)
    • tekton/v1/triggers/triggers/env-gcp/_/*.yaml (where timeouts changed)

    Why: Adding short comments near these changes would help future maintainers understand why retries were removed (to fail fast and avoid duplication) and why timeouts were increased (slow repos). This is especially helpful for YAML pipelines that may lack context.

    Suggested:

    # Removed retries to fail fast and let pipeline-level retries handle failures
    - name: build-binaries
      # retries: 2  # Removed retries
    
    # Increased timeout from 2h to 3h due to observed slow builds on pingcap/tiflash
    'pingcap/tiflash': {
      'timeout': '3h',
      ...
    }
  • Validate timeout format consistency
    The timeouts use strings like '3h' and '30m'. Ensure this format matches Tekton's expected duration format exactly (e.g., 3h0m0s or 3h). If '3h' is valid, no action needed; otherwise, consider making explicit full durations to avoid ambiguity.

  • Edge case: no fallback or warning if task retries are removed
    Removing task retries means transient failures will cause immediate task failure. Consider adding pipeline-level retry logic or documenting how pipeline retries should be configured to avoid build flakiness.


Best Practices

  • Testing coverage and verification
    The PR description notes manual verification steps but no automated tests. Since these are CI config changes, recommend adding automated test coverage if you have pipeline YAML validation or integration testing to catch syntax or config errors early.

  • Consistency in YAML indentation and style
    The YAML indentation and style appear consistent. Continue enforcing YAML linter checks in CI to avoid indentation errors that are hard to spot.

  • Naming conventions
    Task and pipeline names are consistent with existing conventions.

  • Documentation updates
    If there is a central README or docs describing build retries and timeouts, update them to reflect this new practice of no task retries and longer timeouts for slow repos.


Overall, this is a focused and well-scoped chore PR that improves build pipeline behavior. Adding a few clarifying comments and confirming timeout format validity would strengthen maintainability and prevent confusion in the future. Consider pipeline-level retry strategy documentation as a follow-up.

@ti-chi-bot ti-chi-bot Bot added the size/S label Nov 13, 2026

@wuhuizuo wuhuizuo left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@ti-chi-bot

ti-chi-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 13, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit 9682e51 into main Nov 13, 2026
8 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the chore/tekton-drop-retries-and-raise-build-timeouts branch November 13, 2026 09:03
ti-chi-bot Bot pushed a commit that referenced this pull request Nov 13, 2026
…ry (#5591)

## Why

Since #5474 removed the `ce-paramGithubRepo` allowlist bypass, the
devbuild filters have also required `ce-paramProduct` to equal
`body.repository.name`. That equality is neither a security boundary nor
a routing input:

- the repository allowlist is body-only, keyed on
`body.repository.full_name`; no product header widens it;
- every non-Serverless route derives `component` from the allowlisted
repository (`repository.name`, plus the `tidbcloud/*` and
`pingcap/ticse` overrides), so the header cannot change what is built
either.

What it did do was reject legitimate builds. tibuild labels a component
that lives in a repository with another name, and `githubRepo` may be
passed explicitly:

| product | repository |
| --- | --- |
| `ticdc-newarch` | `pingcap/ticdc` |
| `br`, `dumpling`, `tidb-lightning` | `pingcap/tidb` |
| `drainer`, `pump` | `pingcap/tidb-binlog` |
| `dm` | `pingcap/tiflow` |

A rejected event creates no `PipelineRun` and the listener logs nothing,
so the devbuild stays `PENDING` forever. Five of them are stuck right
now (every devbuild since 2026-11-06 whose product does not equal its
repository name; all 120 that do match have real statuses):

| devbuild | created (UTC) | product | repository | ref | kind |
| --- | --- | --- | --- | --- | --- |
| 3030009 | 2026-11-13 09:35 | `ticdc-newarch` | `pingcap/ticdc` |
`tag/v8.5.4-20261113-5691a52` | hotfix |
| 2970069 | 2026-11-11 08:11 | `br` | `pingcap/tidb` |
`tag/v8.5.4-20261111-4d32ecb` | hotfix |
| 3030012 | 2026-11-13 10:53 | `ticdc` | `pingcap/tiflow` |
`branch/master` | devbuild |
| 2970059 | 2026-11-10 09:55 | `ticdc` | `pingcap/tiflow` | `pull/12954`
| devbuild |
| 2970060 | 2026-11-10 09:56 | `ticdc` | `pingcap/tiflow` | `pull/12954`
| devbuild |

The last `ticdc-newarch` hotfix that actually ran is 2880034
(2026-11-03), i.e. before the merge.

## Change

Drop the product/repository equality from all nine `fake-github-*`
routes (`branch-push`, `pr`, `tag-create` × plain, `-darwin`,
`-single-platform`).

The Serverless routes keep it: there the product *is* the component
(`extensions.build.serverless ? extensions.build.product : ...`), so it
still has to name the Serverless repository.

`tekton/tests/trigger-security.test.ts` now asserts the replacement
invariant — whatever the header says, the route and the component come
from the repository:

```
ACCEPT  pingcap/ticdc    product=ticdc-newarch  component=ticdc
ACCEPT  pingcap/tidb     product=br             component=tidb
ACCEPT  pingcap/tiflow   product=ticdc          component=tiflow
REJECT  attacker/evil    product=tidb           component=-
```

Note the third line: with the product decoupled, `ticdc` on
`pingcap/tiflow` builds the **tiflow** component. Nothing about the
request's product label influences the build, which is the point.

## Security

Unchanged and unaffected:

- the allowlist is still `body.repository.full_name in [...]`, and
`trigger-security.test.ts` still asserts that no route reads
`ce-paramGithubRepo`, keys the allowlist on `repository.name`, or binds
`repository.clone_url`;
- a forged event for a non-allowlisted repository is still rejected with
arbitrary product headers;
- a Serverless product still cannot route a non-Serverless repository
(that branch takes the component from the header, so it keeps the
equality).

The `fake-github-*` triggers are only selected by `el-tibuild`, which
verifies tibuild's CloudEvent HMAC (PingCAP-QE/ee-ops#2619).

## Verification

```
kustomize build tekton/v1/triggers/triggers/env-gcp   -> OK
deno test --frozen --lock=tekton/tests/deno.lock --allow-read --allow-write \
  --allow-env --allow-run=git,sh,deno tekton/tests/*.test.ts
  -> ok | 43 passed (302 steps) | 0 failed
```

## Second commit (please review separately)

`deno test tekton/tests/*.test.ts` is red on `main`, independent of this
change — I reproduced it on a clean `origin/main` worktree (3f2c3d0).
The second commit repairs the TiCSE formal-route assertions:

- #5533 added `pingcap/ticse` to `cseRepositories`, so the tag and
branch-creation assertions ran with a `branch-push` payload; those
triggers require `body.ref_type` (#5569) and threw `field not found:
ref_type`. They now get payloads that carry it.
- #5588 raised `tidbcloud/cloud-storage-engine` to `1h` on the
branch-push route only, while the test asserted a single `30m` for both
routes. It now asserts each route's own value.

If you would rather land that as its own PR, say so and I will split it
out.

## Follow-ups (not in this PR)

- **tibuild (PingCAP-QE/ee-apps)**: a devbuild whose CloudEvents
produced no `PipelineRun` stays `PENDING` forever — 80 such records are
in the database, the oldest from 2026-08-10. A watchdog (`PENDING` +
`triggersEventIDs` non-empty + `pipelines` empty after N minutes →
error) would have turned this incident into a visible failure instead of
a stalled queue.
- **`git-create-tag-build-ng.yaml`**: the tag route still carries the
old `30m` budget for `tidbcloud/cloud-storage-engine` while the
branch-push route has `1h` after #5588. Looks unintended; worth a
one-line follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant