Skip to content

feat(jobs): pre-enable pingcap/ticse with legacy CSE compatibility - #5533

Merged
ti-chi-bot[bot] merged 1 commit into
mainfrom
prep/ticse-dual-repo-ci
Nov 13, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
mainfrom
prep/ticse-dual-repo-ci

Conversation

@dillon-zheng

Copy link
Copy Markdown
Contributor

Summary

Prepare CI for the upcoming transfer and rename of tidbcloud/cloud-storage-engine to pingcap/ticse (TiDB Cloud Storage Engine, TiCSE).

This is additive pre-cutover compatibility: both repository identities remain accepted. The product/component stays tikv; no binaries, image repositories, delivery destinations, Jenkins job names, or check contexts are renamed.

Changes

  • Add the pingcap/ticse Prow presubmit key, sharing the existing job definition and backend with the old key.
  • Resolve Jenkins pod templates from the existing CI job directory, independently of source REFS.org/repo. Checkout, workspace, and cache identity continue to use the actual source repository.
  • Extend the existing 12 shared Tekton Trigger resources rather than duplicating them: 11 packaging routes and one version-placeholder route.
  • Normalize the new repository's component/resource lookup to existing CSE settings, preserving its 48Gi memory profile and all existing branch, tag, profile, revision, and registry behavior.
  • Add regression coverage for both identities, Product aliases and mismatches, Linux/Darwin/all-platform selection, payload-derived source identity/revision, resources, formal branches/tags, and placeholder behavior. Arbitrary forks and the personal rehearsal repository remain rejected.

Validation

  • Local routing/versioning suite: 40 tests, 302 steps passed using Deno 2.9.6, the existing frozen lockfile, and the workflow's permission allowlist. The workflow-pinned 2.9.8 was not available in the local npm registry; no runtime pins or dependencies were changed.
  • Regression sensitivity: against the original Trigger YAMLs, the new tests pass all 108 old-identity cases and fail all 108 new-identity cases; both identities pass with this patch.
  • deno fmt --check tekton/tests/serverless-builds.test.ts.
  • .ci/check-jenkins-job-references.sh --quiet and .ci/verify-jenkins-credential-policy.sh.
  • Both Prow and Tekton kustomization regeneration checks pass without changing generated inventories.
  • Local kubectl kustomize tekton/v1/triggers/triggers/env-gcp succeeds; no Kubernetes API calls are made.
  • YAML parsing and git diff --check pass.

Rollout and limits

  • Deploy this compatibility change together with the companion Prow policy/selector configuration before the actual Transfer: feat(prow): pre-enable pingcap/ticse with existing CSE policy ti-community-infra/configs#1313.
  • This PR does not transfer/create the target repository, change repository/org permissions, modify Actions settings, or deploy a workload. No CSE build, publish, or Jenkins replay was triggered during preparation.
  • New-identity private checkout, GitHub App/broker credentials, PR status reporting, and real build/delivery remain runtime acceptance checks; local CEL tests are not an end-to-end build claim.
  • Keep the old identity after cutover for in-flight jobs and historical reruns. Removing it is a separate decision, not part of this PR.
  • Existing limitation, unchanged for both identities: the version-placeholder Trigger accepts calendar-style nextgen branches, but its shared Task only parses numeric nextgen branches. This is a non-packaging TaskRun and a pre-existing mismatch, not an additional migration fix in this PR.
  • Draft for review; no merge or deployment is authorized by this PR preparation.

Signed-off-by: weizheng <wei.zheng@pingcap.com>
@ti-chi-bot

ti-chi-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@dillon-zheng dillon-zheng left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated pi-DeepSeek review-loop result: CLEAN

Reviewed head: 8797b993b7e05a95060a56e2fa54bae92246256d.

The canonical review-loop script ran with the actual pi-deepseek read-only reviewer and pi-codex-company validation role (maximum 3 rounds). Round 1 returned Critical: none, Warnings: none, Suggestions: none. The normal reviewer-clean condition stopped the loop; there were no findings to fix, so no fixer/adjudicator phase or new commit was needed. This is a COMMENTED review, not an approval or merge authorization.

Reviewed the 15-file patch and surrounding bindings/templates/tooling for dual source identity, unchanged Jenkins/check naming, source-true REFS and clone/revision handling, exact repository admission, CSE tikv/48Gi resource selection, platform uniqueness, preserved legacy behavior and negative cases.

Validation rerun:

  • Frozen/cached routing and versioning suite: 40 tests, 302 steps passed.
  • Modified test formatting, Jenkins reference resolution, credential policy and whitespace checks passed.
  • Test coverage percentage: not measured; test counts are not a coverage percentage.

The already documented calendar-style placeholder Task parser mismatch remains pre-existing and outside this migration patch. Actual private checkout, credentials, PR status reporting and artifact delivery still require authorized runtime acceptance. No build/replay/publish, deployment, Transfer, permission change, merge or source modification was performed by this review loop.

@dillon-zheng
dillon-zheng marked this pull request as ready for review November 13, 2026 09:19
@ti-chi-bot

ti-chi-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

@dillon-zheng: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-replay-jenkins-pipelines 8797b99 link false /test pull-replay-jenkins-pipelines

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot Bot pushed a commit to PingCAP-QE/ee-ops that referenced this pull request Nov 13, 2026
## Summary

Prepare TiDB Cloud Storage Engine (TiCSE) for the repository transfer
from `tidbcloud/cloud-storage-engine` to `pingcap/ticse`.

Add **only** the exact new fullname to `devbuild.github_repo_allowlist`
in the GCP and Tencent Cloud tibuild-v2 configurations. Keep the old
fullname for existing callers and historical records/reruns. Do not add
an organization wildcard.

This is the **pre-transfer compatibility** part of the CD changes. It
does not switch the source used by existing requests and does not
require the new repository to exist before the configuration can be
deployed. Actual new-source builds still require the transfer and the
shared [CI/Tekton compatibility
PR](PingCAP-QE/ci#5533). The related [Prow
policy PR](ti-community-infra/configs#1313)
preserves the repository's existing review/governance behavior. These
are separate draft PRs, not changes duplicated here.

## Scope and invariants

- Two configuration files, one additional allowlist entry each.
- Keep `product_repo_map.tikv: tikv/tikv`; TiCSE requests continue to
specify `product=tikv` and an explicit `githubRepo`.
- Preserve every existing allowlist entry, product default,
builder-image policy, endpoint, credential reference, registry and
delivery target.
- No new prod2 build admission: its API remains disabled, and it did not
admit the old CSE source.
- Do not duplicate the 12 shared Tekton Trigger resources already
covered by the CI preparation.
- No tibuild service code, image, workflow, package or component
renaming.

Checked against the [declared runtime's repository-policy
implementation](https://github.com/PingCAP-QE/ee-apps/blob/4491f5bbae3eeb657536597122eace288ab5558f/experiments/tibuild-v2/pkg/config/repo_allowlist.go#L32-L117):
redirects do not make the old fullname admit the new spelling.
Historical reruns also recheck their stored repo against the current
policy, so retain the old entry.

## Validation

- `git diff --check`: passed.
- `yq eval 'true'` on both changed manifests: passed.
- `kubectl kustomize apps/gcp/tibuild/v2`: passed (local rendering
only).
- `kubectl kustomize apps/tencentcloud/tibuild/v2`: passed (local
rendering only).
- Parsed the ExternalSecrets and embedded `config.yaml` using inert
placeholders, without resolving or reading Secret values. Semantic
checks verify both fullnames are allowed, the sole change is the exact
new entry, all product defaults remain covered, and the public TiKV
default is unchanged. The new-identity assertion fails against the
original configuration and passes after this patch.

No build, publish, Kubernetes apply, Flux reconciliation or deployment
was executed. This PR does not demonstrate effective production
credentials or completed artifact delivery. Full repository manifest
validation remains for the normal PR checks.

## Deployment ordering

After review, this additive configuration can be deployed **before** the
transfer. Verify the API actually loads the updated allowlist; use the
existing controlled rollout procedure if the mounted configuration does
not refresh. Do not request a real build from the new path until the
transfer and required Tekton compatibility are complete.

The canonical `github_repo` values in the TiDB Cloud release-metadata
configuration intentionally stay on the old fullname here. They are
handled by the separate **[post-transfer metadata
PR](#2680. Neither PR
authorizes the transfer, a deployment or a merge.

Signed-off-by: weizheng <wei.zheng@pingcap.com>
@ti-chi-bot ti-chi-bot Bot added the lgtm label Nov 13, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Nov 13, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-11-13 09:39:00.898177807 +0000 UTC m=+1437065.523328088: ☑️ agreed by wuhuizuo.

@ti-chi-bot
ti-chi-bot Bot merged commit 682912b into main Nov 13, 2026
14 of 15 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the prep/ticse-dual-repo-ci branch November 13, 2026 09:50
ti-chi-bot Bot pushed a commit that referenced this pull request Nov 13, 2026
…ry (#5591)

## Why

Since #5474 removed the `ce-paramGithubRepo` allowlist bypass, the
devbuild filters have also required `ce-paramProduct` to equal
`body.repository.name`. That equality is neither a security boundary nor
a routing input:

- the repository allowlist is body-only, keyed on
`body.repository.full_name`; no product header widens it;
- every non-Serverless route derives `component` from the allowlisted
repository (`repository.name`, plus the `tidbcloud/*` and
`pingcap/ticse` overrides), so the header cannot change what is built
either.

What it did do was reject legitimate builds. tibuild labels a component
that lives in a repository with another name, and `githubRepo` may be
passed explicitly:

| product | repository |
| --- | --- |
| `ticdc-newarch` | `pingcap/ticdc` |
| `br`, `dumpling`, `tidb-lightning` | `pingcap/tidb` |
| `drainer`, `pump` | `pingcap/tidb-binlog` |
| `dm` | `pingcap/tiflow` |

A rejected event creates no `PipelineRun` and the listener logs nothing,
so the devbuild stays `PENDING` forever. Five of them are stuck right
now (every devbuild since 2026-11-06 whose product does not equal its
repository name; all 120 that do match have real statuses):

| devbuild | created (UTC) | product | repository | ref | kind |
| --- | --- | --- | --- | --- | --- |
| 3030009 | 2026-11-13 09:35 | `ticdc-newarch` | `pingcap/ticdc` |
`tag/v8.5.4-20261113-5691a52` | hotfix |
| 2970069 | 2026-11-11 08:11 | `br` | `pingcap/tidb` |
`tag/v8.5.4-20261111-4d32ecb` | hotfix |
| 3030012 | 2026-11-13 10:53 | `ticdc` | `pingcap/tiflow` |
`branch/master` | devbuild |
| 2970059 | 2026-11-10 09:55 | `ticdc` | `pingcap/tiflow` | `pull/12954`
| devbuild |
| 2970060 | 2026-11-10 09:56 | `ticdc` | `pingcap/tiflow` | `pull/12954`
| devbuild |

The last `ticdc-newarch` hotfix that actually ran is 2880034
(2026-11-03), i.e. before the merge.

## Change

Drop the product/repository equality from all nine `fake-github-*`
routes (`branch-push`, `pr`, `tag-create` × plain, `-darwin`,
`-single-platform`).

The Serverless routes keep it: there the product *is* the component
(`extensions.build.serverless ? extensions.build.product : ...`), so it
still has to name the Serverless repository.

`tekton/tests/trigger-security.test.ts` now asserts the replacement
invariant — whatever the header says, the route and the component come
from the repository:

```
ACCEPT  pingcap/ticdc    product=ticdc-newarch  component=ticdc
ACCEPT  pingcap/tidb     product=br             component=tidb
ACCEPT  pingcap/tiflow   product=ticdc          component=tiflow
REJECT  attacker/evil    product=tidb           component=-
```

Note the third line: with the product decoupled, `ticdc` on
`pingcap/tiflow` builds the **tiflow** component. Nothing about the
request's product label influences the build, which is the point.

## Security

Unchanged and unaffected:

- the allowlist is still `body.repository.full_name in [...]`, and
`trigger-security.test.ts` still asserts that no route reads
`ce-paramGithubRepo`, keys the allowlist on `repository.name`, or binds
`repository.clone_url`;
- a forged event for a non-allowlisted repository is still rejected with
arbitrary product headers;
- a Serverless product still cannot route a non-Serverless repository
(that branch takes the component from the header, so it keeps the
equality).

The `fake-github-*` triggers are only selected by `el-tibuild`, which
verifies tibuild's CloudEvent HMAC (PingCAP-QE/ee-ops#2619).

## Verification

```
kustomize build tekton/v1/triggers/triggers/env-gcp   -> OK
deno test --frozen --lock=tekton/tests/deno.lock --allow-read --allow-write \
  --allow-env --allow-run=git,sh,deno tekton/tests/*.test.ts
  -> ok | 43 passed (302 steps) | 0 failed
```

## Second commit (please review separately)

`deno test tekton/tests/*.test.ts` is red on `main`, independent of this
change — I reproduced it on a clean `origin/main` worktree (3f2c3d0).
The second commit repairs the TiCSE formal-route assertions:

- #5533 added `pingcap/ticse` to `cseRepositories`, so the tag and
branch-creation assertions ran with a `branch-push` payload; those
triggers require `body.ref_type` (#5569) and threw `field not found:
ref_type`. They now get payloads that carry it.
- #5588 raised `tidbcloud/cloud-storage-engine` to `1h` on the
branch-push route only, while the test asserted a single `30m` for both
routes. It now asserts each route's own value.

If you would rather land that as its own PR, say so and I will split it
out.

## Follow-ups (not in this PR)

- **tibuild (PingCAP-QE/ee-apps)**: a devbuild whose CloudEvents
produced no `PipelineRun` stays `PENDING` forever — 80 such records are
in the database, the oldest from 2026-08-10. A watchdog (`PENDING` +
`triggersEventIDs` non-empty + `pipelines` empty after N minutes →
error) would have turned this incident into a visible failure instead of
a stalled queue.
- **`git-create-tag-build-ng.yaml`**: the tag route still carries the
old `30m` budget for `tidbcloud/cloud-storage-engine` while the
branch-push route has `1h` after #5588. Looks unintended; worth a
one-line follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

2 participants