Repository navigation
feat(jobs): pre-enable pingcap/ticse with legacy CSE compatibility - #5533
Conversation
Signed-off-by: weizheng <wei.zheng@pingcap.com>
|
Skipping CI for Draft Pull Request. |
dillon-zheng
left a comment
There was a problem hiding this comment.
Automated pi-DeepSeek review-loop result: CLEAN
Reviewed head: 8797b993b7e05a95060a56e2fa54bae92246256d.
The canonical review-loop script ran with the actual pi-deepseek read-only reviewer and pi-codex-company validation role (maximum 3 rounds). Round 1 returned Critical: none, Warnings: none, Suggestions: none. The normal reviewer-clean condition stopped the loop; there were no findings to fix, so no fixer/adjudicator phase or new commit was needed. This is a COMMENTED review, not an approval or merge authorization.
Reviewed the 15-file patch and surrounding bindings/templates/tooling for dual source identity, unchanged Jenkins/check naming, source-true REFS and clone/revision handling, exact repository admission, CSE tikv/48Gi resource selection, platform uniqueness, preserved legacy behavior and negative cases.
Validation rerun:
- Frozen/cached routing and versioning suite: 40 tests, 302 steps passed.
- Modified test formatting, Jenkins reference resolution, credential policy and whitespace checks passed.
- Test coverage percentage: not measured; test counts are not a coverage percentage.
The already documented calendar-style placeholder Task parser mismatch remains pre-existing and outside this migration patch. Actual private checkout, credentials, PR status reporting and artifact delivery still require authorized runtime acceptance. No build/replay/publish, deployment, Transfer, permission change, merge or source modification was performed by this review loop.
|
@dillon-zheng: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
## Summary Prepare TiDB Cloud Storage Engine (TiCSE) for the repository transfer from `tidbcloud/cloud-storage-engine` to `pingcap/ticse`. Add **only** the exact new fullname to `devbuild.github_repo_allowlist` in the GCP and Tencent Cloud tibuild-v2 configurations. Keep the old fullname for existing callers and historical records/reruns. Do not add an organization wildcard. This is the **pre-transfer compatibility** part of the CD changes. It does not switch the source used by existing requests and does not require the new repository to exist before the configuration can be deployed. Actual new-source builds still require the transfer and the shared [CI/Tekton compatibility PR](PingCAP-QE/ci#5533). The related [Prow policy PR](ti-community-infra/configs#1313) preserves the repository's existing review/governance behavior. These are separate draft PRs, not changes duplicated here. ## Scope and invariants - Two configuration files, one additional allowlist entry each. - Keep `product_repo_map.tikv: tikv/tikv`; TiCSE requests continue to specify `product=tikv` and an explicit `githubRepo`. - Preserve every existing allowlist entry, product default, builder-image policy, endpoint, credential reference, registry and delivery target. - No new prod2 build admission: its API remains disabled, and it did not admit the old CSE source. - Do not duplicate the 12 shared Tekton Trigger resources already covered by the CI preparation. - No tibuild service code, image, workflow, package or component renaming. Checked against the [declared runtime's repository-policy implementation](https://github.com/PingCAP-QE/ee-apps/blob/4491f5bbae3eeb657536597122eace288ab5558f/experiments/tibuild-v2/pkg/config/repo_allowlist.go#L32-L117): redirects do not make the old fullname admit the new spelling. Historical reruns also recheck their stored repo against the current policy, so retain the old entry. ## Validation - `git diff --check`: passed. - `yq eval 'true'` on both changed manifests: passed. - `kubectl kustomize apps/gcp/tibuild/v2`: passed (local rendering only). - `kubectl kustomize apps/tencentcloud/tibuild/v2`: passed (local rendering only). - Parsed the ExternalSecrets and embedded `config.yaml` using inert placeholders, without resolving or reading Secret values. Semantic checks verify both fullnames are allowed, the sole change is the exact new entry, all product defaults remain covered, and the public TiKV default is unchanged. The new-identity assertion fails against the original configuration and passes after this patch. No build, publish, Kubernetes apply, Flux reconciliation or deployment was executed. This PR does not demonstrate effective production credentials or completed artifact delivery. Full repository manifest validation remains for the normal PR checks. ## Deployment ordering After review, this additive configuration can be deployed **before** the transfer. Verify the API actually loads the updated allowlist; use the existing controlled rollout procedure if the mounted configuration does not refresh. Do not request a real build from the new path until the transfer and required Tekton compatibility are complete. The canonical `github_repo` values in the TiDB Cloud release-metadata configuration intentionally stay on the old fullname here. They are handled by the separate **[post-transfer metadata PR](#2680. Neither PR authorizes the transfer, a deployment or a merge. Signed-off-by: weizheng <wei.zheng@pingcap.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wuhuizuo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
…ry (#5591) ## Why Since #5474 removed the `ce-paramGithubRepo` allowlist bypass, the devbuild filters have also required `ce-paramProduct` to equal `body.repository.name`. That equality is neither a security boundary nor a routing input: - the repository allowlist is body-only, keyed on `body.repository.full_name`; no product header widens it; - every non-Serverless route derives `component` from the allowlisted repository (`repository.name`, plus the `tidbcloud/*` and `pingcap/ticse` overrides), so the header cannot change what is built either. What it did do was reject legitimate builds. tibuild labels a component that lives in a repository with another name, and `githubRepo` may be passed explicitly: | product | repository | | --- | --- | | `ticdc-newarch` | `pingcap/ticdc` | | `br`, `dumpling`, `tidb-lightning` | `pingcap/tidb` | | `drainer`, `pump` | `pingcap/tidb-binlog` | | `dm` | `pingcap/tiflow` | A rejected event creates no `PipelineRun` and the listener logs nothing, so the devbuild stays `PENDING` forever. Five of them are stuck right now (every devbuild since 2026-11-06 whose product does not equal its repository name; all 120 that do match have real statuses): | devbuild | created (UTC) | product | repository | ref | kind | | --- | --- | --- | --- | --- | --- | | 3030009 | 2026-11-13 09:35 | `ticdc-newarch` | `pingcap/ticdc` | `tag/v8.5.4-20261113-5691a52` | hotfix | | 2970069 | 2026-11-11 08:11 | `br` | `pingcap/tidb` | `tag/v8.5.4-20261111-4d32ecb` | hotfix | | 3030012 | 2026-11-13 10:53 | `ticdc` | `pingcap/tiflow` | `branch/master` | devbuild | | 2970059 | 2026-11-10 09:55 | `ticdc` | `pingcap/tiflow` | `pull/12954` | devbuild | | 2970060 | 2026-11-10 09:56 | `ticdc` | `pingcap/tiflow` | `pull/12954` | devbuild | The last `ticdc-newarch` hotfix that actually ran is 2880034 (2026-11-03), i.e. before the merge. ## Change Drop the product/repository equality from all nine `fake-github-*` routes (`branch-push`, `pr`, `tag-create` × plain, `-darwin`, `-single-platform`). The Serverless routes keep it: there the product *is* the component (`extensions.build.serverless ? extensions.build.product : ...`), so it still has to name the Serverless repository. `tekton/tests/trigger-security.test.ts` now asserts the replacement invariant — whatever the header says, the route and the component come from the repository: ``` ACCEPT pingcap/ticdc product=ticdc-newarch component=ticdc ACCEPT pingcap/tidb product=br component=tidb ACCEPT pingcap/tiflow product=ticdc component=tiflow REJECT attacker/evil product=tidb component=- ``` Note the third line: with the product decoupled, `ticdc` on `pingcap/tiflow` builds the **tiflow** component. Nothing about the request's product label influences the build, which is the point. ## Security Unchanged and unaffected: - the allowlist is still `body.repository.full_name in [...]`, and `trigger-security.test.ts` still asserts that no route reads `ce-paramGithubRepo`, keys the allowlist on `repository.name`, or binds `repository.clone_url`; - a forged event for a non-allowlisted repository is still rejected with arbitrary product headers; - a Serverless product still cannot route a non-Serverless repository (that branch takes the component from the header, so it keeps the equality). The `fake-github-*` triggers are only selected by `el-tibuild`, which verifies tibuild's CloudEvent HMAC (PingCAP-QE/ee-ops#2619). ## Verification ``` kustomize build tekton/v1/triggers/triggers/env-gcp -> OK deno test --frozen --lock=tekton/tests/deno.lock --allow-read --allow-write \ --allow-env --allow-run=git,sh,deno tekton/tests/*.test.ts -> ok | 43 passed (302 steps) | 0 failed ``` ## Second commit (please review separately) `deno test tekton/tests/*.test.ts` is red on `main`, independent of this change — I reproduced it on a clean `origin/main` worktree (3f2c3d0). The second commit repairs the TiCSE formal-route assertions: - #5533 added `pingcap/ticse` to `cseRepositories`, so the tag and branch-creation assertions ran with a `branch-push` payload; those triggers require `body.ref_type` (#5569) and threw `field not found: ref_type`. They now get payloads that carry it. - #5588 raised `tidbcloud/cloud-storage-engine` to `1h` on the branch-push route only, while the test asserted a single `30m` for both routes. It now asserts each route's own value. If you would rather land that as its own PR, say so and I will split it out. ## Follow-ups (not in this PR) - **tibuild (PingCAP-QE/ee-apps)**: a devbuild whose CloudEvents produced no `PipelineRun` stays `PENDING` forever — 80 such records are in the database, the oldest from 2026-08-10. A watchdog (`PENDING` + `triggersEventIDs` non-empty + `pipelines` empty after N minutes → error) would have turned this incident into a visible failure instead of a stalled queue. - **`git-create-tag-build-ng.yaml`**: the tag route still carries the old `30m` budget for `tidbcloud/cloud-storage-engine` while the branch-push route has `1h` after #5588. Looks unintended; worth a one-line follow-up.
Summary
Prepare CI for the upcoming transfer and rename of
tidbcloud/cloud-storage-enginetopingcap/ticse(TiDB Cloud Storage Engine, TiCSE).This is additive pre-cutover compatibility: both repository identities remain accepted. The product/component stays
tikv; no binaries, image repositories, delivery destinations, Jenkins job names, or check contexts are renamed.Changes
pingcap/ticseProw presubmit key, sharing the existing job definition and backend with the old key.REFS.org/repo. Checkout, workspace, and cache identity continue to use the actual source repository.Validation
deno fmt --check tekton/tests/serverless-builds.test.ts..ci/check-jenkins-job-references.sh --quietand.ci/verify-jenkins-credential-policy.sh.kubectl kustomize tekton/v1/triggers/triggers/env-gcpsucceeds; no Kubernetes API calls are made.git diff --checkpass.Rollout and limits