From 51ffdd99f13383e1d0b1d0454cd5870c8073dbcb Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Fri, 13 Nov 2026 19:08:23 +0000 Subject: [PATCH 1/7] docs: add 0.17.x to 1.0.0 upgrade notes (TRUNK-23152) UPGRADING.md lists what a 0.17.x user has to change for 1.0.0: a v2 API key or JWT, a required test collection, the removed flags and variables, the fork lane with no v1 fallback, the summary on stdout, GUID links, the telemetry host, and which analytics-uploader versions pick up 1.0.0. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- UPGRADING.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 1 deletion(-) create mode 100644 UPGRADING.md diff --git a/README.md b/README.md index fe0800bd..a5d83e84 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ This is downloaded automatically using the [Trunk launcher][launcher] and when u - **A v2 API key** (`trunk_v2_...`). Create one at https://app.trunk.io/settings/developer/api-tokens, and give it the **Upload test results** (`upload_test_results`) permission. The CLI sends it as `Authorization: Bearer`. - **A Trunk JWT**. The CLI sends it on `x-trunk-token`. With a key stored by `trunk auth api-key set`, `trunk flakytests upload` exchanges it for a JWT just before the upload and passes that in `TRUNK_API_TOKEN`, while `trunk flakytests test` passes the key itself. A JWT lasts at most 15 minutes, and the one `trunk flakytests upload` passes can have about a minute left, so use a JWT only for the upload itself: a long `trunk-analytics-cli test` run, or a test-runner plugin that looks up quarantined tests early and uploads at the end, can outlive it, so give those a v2 API key. When Trunk refuses an expired JWT, the CLI says so; it does not refresh it. -analytics-cli 1.0.0 refuses any other token, the organization token included, before it calls Trunk. Pin analytics-cli 0.17.x to keep using the organization token until December 1, 2027. +analytics-cli 1.0.0 refuses any other token, the organization token included, before it calls Trunk. Pin analytics-cli 0.17.x to keep using the organization token until December 1, 2027. To upgrade from 0.17.x, see [UPGRADING.md](UPGRADING.md). If Trunk answers `403` because the key lacks the Upload test results permission, edit the key on the API Keys screen to add it. diff --git a/UPGRADING.md b/UPGRADING.md new file mode 100644 index 00000000..2391560a --- /dev/null +++ b/UPGRADING.md @@ -0,0 +1,55 @@ +# Upgrading from 0.17.x to 1.0.0 + +analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collections and takes only a v2 API key or a Trunk JWT. Most runs need three changes: a v2 API key, a test collection id, and no removed flags. + +0.17.x stays available. Pin it to keep the organization token while you move: it gets important fixes on the `0.17.x` branch, and no new features. + +## If you use analytics-uploader + +- **v3** installs the trunk CLI and runs `trunk flakytests`, which resolves analytics-cli 1.x. Move to it to get 1.0.0. +- **v2.1.8 and later**, and the floating `@v2` tag, stay on 0.x: their `latest` reads `channel-0.json`. +- **v2.1.7 and earlier, pinned by SHA or exact tag,** resolve `latest` to the newest release and so get 1.0.0. Their uploads then fail with an organization token, without `test-collection-id`, or with the `quarantine` or `hide-banner` input set, which they pass as flags 1.0.0 removed. Move to v2.1.8 or v3 before 1.0.0 is the latest release, or set `cli-version: 0.17.0`. + +## What you have to change + +### Credentials + +`--token` / `TRUNK_API_TOKEN` must hold one of: + +- **A v2 API key** (`trunk_v2_...`) with the **Upload test results** permission. Create it in the Trunk web app under Settings → Developer → API Keys. +- **A Trunk JWT**, such as the one `trunk flakytests upload` passes. A JWT lasts at most 15 minutes and the CLI does not refresh it, so use a v2 API key for a long `test` run or a test-runner plugin. + + 1.0.0 refuses the organization token before it calls Trunk; `test` refuses before it runs your tests. A key without Upload test results gets a `403` that names the permission. See [Credentials](README.md#credentials). + +### Test collection + +`--test-collection-id` / `TRUNK_TEST_COLLECTION_ID` is required for `upload` and `test`. A blank value counts as missing. 0.17.x inferred the collection from the repository; 1.0.0 does not. The rspec gem stays off, with a warning, without one. + +### Removed flags and variables + +| Removed | Instead | +| ------------------------------------------------------------------- | --------------------------------------------------------- | +| `--use-quarantining` | Nothing for `true`; `--disable-quarantining` for `false`. | +| `--team`, `--print-files`, `--hide-banner` | Drop them; they did nothing. | +| `--public-repo-id` / `TRUNK_PUBLIC_REPO_ID` | `--allow-forked-pr-uploads` with `--test-collection-id`. | +| `--hide-test-collection-links` / `TRUNK_HIDE_TEST_COLLECTION_LINKS` | Drop it. Every link is now a test collection link. | + +`upload` and `validate` exit with code 2 on a removed flag. `test` reads an unknown flag before the test command as part of the command, so the run fails with exit code 70. + +### Fork pull requests + +`--allow-forked-pr-uploads` with no token works only in a GitHub Actions run of a pull request from a fork. The CLI swaps the job's `GITHUB_TOKEN` for a Trunk login, so the job needs it in the environment. There is no fallback any more: a refused swap, a missing `GITHUB_TOKEN`, or the flag outside a fork pull request uploads nothing and says why. + +### Output + +- The summary goes to **stdout**; logs, notices and progress go to stderr. A script that reads only stderr loses the summary. +- `--output-format json` (or `TRUNK_OUTPUT_FORMAT=json`) on `upload` and `validate` prints the summary as the JSON that `--summary-output-file` writes. +- Test links use the test case's GUID (`/flaky-tests/collections//tests/`) when Trunk returns the collection, and the `/t/?repo=` short link otherwise. The repository-scoped `/flaky-tests/test/` link is gone. + +### Network + +1.0.0 sends its telemetry to `api.trunk.io` (`/v2/cloud/telemetry/v1/logs`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. + +## Test-runner plugins + +`test_report` and the rspec gem share the CLI's API client, so the same rules apply: a v2 API key (or a JWT that outlasts the run) and `TRUNK_TEST_COLLECTION_ID`. Without either, the gem prints a warning and stays off. From 5feff4ef32ec210aba39068b1c0cde06e066e2eb Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Fri, 13 Nov 2026 20:21:28 +0000 Subject: [PATCH 2/7] docs: name the telemetry route's new path in the upgrade notes (TRUNK-23152) Co-Authored-By: Claude Opus 5.5 --- UPGRADING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/UPGRADING.md b/UPGRADING.md index 2391560a..052163fe 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -48,7 +48,7 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti ### Network -1.0.0 sends its telemetry to `api.trunk.io` (`/v2/cloud/telemetry/v1/logs`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. +1.0.0 sends its telemetry to `api.trunk.io` (`/v2/telemetry/v1/logs`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. ## Test-runner plugins From c94fb5449482299b7d2970207c887542f475f71d Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Mon, 16 Nov 2026 18:16:16 +0000 Subject: [PATCH 3/7] docs: name the metrics telemetry route in the upgrade notes (TRUNK-23152) Co-Authored-By: Claude Opus 5.5 --- UPGRADING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/UPGRADING.md b/UPGRADING.md index 052163fe..e3b1db28 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -48,7 +48,7 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti ### Network -1.0.0 sends its telemetry to `api.trunk.io` (`/v2/telemetry/v1/logs`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. +1.0.0 sends its telemetry, as OTLP metrics, to `api.trunk.io` (`/v2/telemetry/v1/metrics`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. ## Test-runner plugins From 4b3f6e1352f981e58d3538a0b32df7d6a3e775e7 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Tue, 17 Nov 2026 13:25:52 +0000 Subject: [PATCH 4/7] docs: link the API keys page and date the 0.17.x pin in the upgrade notes (TRUNK-23152) Co-Authored-By: Claude Opus 5.5 --- UPGRADING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/UPGRADING.md b/UPGRADING.md index e3b1db28..dc6c44fd 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -2,7 +2,7 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collections and takes only a v2 API key or a Trunk JWT. Most runs need three changes: a v2 API key, a test collection id, and no removed flags. -0.17.x stays available. Pin it to keep the organization token while you move: it gets important fixes on the `0.17.x` branch, and no new features. +0.17.x stays available and keeps taking the organization token until December 1, 2027. Pin it while you move: it gets important fixes on the `0.17.x` branch, and no new features. ## If you use analytics-uploader @@ -16,7 +16,7 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti `--token` / `TRUNK_API_TOKEN` must hold one of: -- **A v2 API key** (`trunk_v2_...`) with the **Upload test results** permission. Create it in the Trunk web app under Settings → Developer → API Keys. +- **A v2 API key** (`trunk_v2_...`) with the **Upload test results** permission. Create it at https://app.trunk.io/settings/developer/api-tokens. - **A Trunk JWT**, such as the one `trunk flakytests upload` passes. A JWT lasts at most 15 minutes and the CLI does not refresh it, so use a v2 API key for a long `test` run or a test-runner plugin. 1.0.0 refuses the organization token before it calls Trunk; `test` refuses before it runs your tests. A key without Upload test results gets a `403` that names the permission. See [Credentials](README.md#credentials). From 8481e97c6a47ccaf8f62a1f9e6fb73c58b7041f3 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Tue, 17 Nov 2026 16:36:36 +0000 Subject: [PATCH 5/7] docs: the rspec gem fails the run, and 1.0.0 reaches old uploaders only when promoted to latest (TRUNK-23152) - The rspec gem 1.0.0 raises RSpec::Trunk::ConfigurationError without a collection id or with an organization token (#1337), with DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true to opt out and a ~> 0.17.0 pin to stay. - analytics-uploader v2.1.7 and older follow channel.json or GitHub's latest, which move only when 1.0.0 is promoted to latest (#1339). - trunk CLI 0.0.14 runs analytics-cli 1.x unless pinned (trunk-io/trunk2#8640). Co-Authored-By: Claude Opus 5.5 --- UPGRADING.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/UPGRADING.md b/UPGRADING.md index dc6c44fd..cd2409a6 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -8,7 +8,11 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti - **v3** installs the trunk CLI and runs `trunk flakytests`, which resolves analytics-cli 1.x. Move to it to get 1.0.0. - **v2.1.8 and later**, and the floating `@v2` tag, stay on 0.x: their `latest` reads `channel-0.json`. -- **v2.1.7 and earlier, pinned by SHA or exact tag,** resolve `latest` to the newest release and so get 1.0.0. Their uploads then fail with an organization token, without `test-collection-id`, or with the `quarantine` or `hide-banner` input set, which they pass as flags 1.0.0 removed. Move to v2.1.8 or v3 before 1.0.0 is the latest release, or set `cli-version: 0.17.0`. +- **v2.1.7 and earlier, pinned by SHA or exact tag,** follow `channel.json` or GitHub's latest release. Both stay on 0.17.x until 1.0.0 is promoted to latest overall, which is a separate step after the 1.0.0 release. From then on these versions get 1.0.0, and their uploads fail with an organization token, without `test-collection-id`, or with the `quarantine` or `hide-banner` input set, which they pass as flags 1.0.0 removed. Move to v2.1.8 or v3 before then, or set `cli-version: 0.17.0`. + +## If you use the trunk CLI + +From trunk CLI 0.0.14, `trunk flakytests` runs the newest analytics-cli 1.x unless `TRUNK_ANALYTICS_CLI_VERSION` pins a version. Earlier trunk CLI versions follow `channel.json`, so they stay on 0.17.x until 1.0.0 is promoted to latest; run `trunk update` to get 0.0.14. ## What you have to change @@ -23,7 +27,7 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti ### Test collection -`--test-collection-id` / `TRUNK_TEST_COLLECTION_ID` is required for `upload` and `test`. A blank value counts as missing. 0.17.x inferred the collection from the repository; 1.0.0 does not. The rspec gem stays off, with a warning, without one. +`--test-collection-id` / `TRUNK_TEST_COLLECTION_ID` is required for `upload` and `test`. A blank value counts as missing. 0.17.x inferred the collection from the repository; 1.0.0 does not. The rspec gem fails the run without one (see Test-runner plugins). ### Removed flags and variables @@ -52,4 +56,9 @@ analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collecti ## Test-runner plugins -`test_report` and the rspec gem share the CLI's API client, so the same rules apply: a v2 API key (or a JWT that outlasts the run) and `TRUNK_TEST_COLLECTION_ID`. Without either, the gem prints a warning and stays off. +`test_report` and the rspec gem share the CLI's API client, so the same rules apply: a v2 API key (or a JWT that outlasts the run) and `TRUNK_TEST_COLLECTION_ID`. + +The rspec gem 1.0.0 **fails the run** when it is set up to upload but can't: with `TRUNK_ORG_URL_SLUG` and a token set, a missing `TRUNK_TEST_COLLECTION_ID` or an organization token raises `RSpec::Trunk::ConfigurationError` while `spec_helper` loads, so no examples run and RSpec exits non-zero. 0.17.x only warned and turned itself off. + +- To run without Trunk, set `DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true`. An empty `TRUNK_API_TOKEN`, such as a fork pull request's missing secret, still turns the gem off quietly. +- The gem is published to RubyGems with each release, with no separate promotion step, so an unpinned `gem 'rspec_trunk_flaky_tests'` or a Dependabot or Renovate bump picks up 1.0.0 as soon as it is released. Pin `gem 'rspec_trunk_flaky_tests', '~> 0.17.0'` to stay on 0.17.x while you move. From 0b30f05c218a74573c4afca2d3912eea1c87d915 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Tue, 17 Nov 2026 18:37:59 +0000 Subject: [PATCH 6/7] docs: the trunk CLI reads channel-1.json, not channel.json (TRUNK-23152) trunk-io/trunk2#8640 moves the trunk CLI's flakytests to channel-1.json, so promote-latest no longer holds it back, and channel-1.json becomes a pointer that must not be removed. Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 537b2394..24d00b4f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -147,13 +147,13 @@ The pattern is: Releases are published in two places: GitHub releases, and `https://trunk.io/releases/analytics-cli/prod/` (the `trunk-releases` S3 bucket). Trunk's own consumers (the `analytics-uploader` action and the `trunk` CLI's `flakytests` command) read the S3 copy. GitHub releases stay, because the frozen legacy `trunk` launcher and customers' own scripts download from them. 1. **Cut**: run the `Release` workflow with the version. It builds, smoke-tests, creates a GitHub **prerelease**, and uploads the same assets to `releases/analytics-cli/prod//`. A version is immutable once published. -2. **Promote**: run the `Promote Release` workflow with the version. It points `channel-.json` at it. With **`promote-latest`** ticked (off by default), and when the version is the newest release overall, it also points `channel.json` at it and marks it latest on GitHub. Leave it off to hold `channel.json` and GitHub's latest back: analytics-uploader v2.1.7 and older follow one of them, and many orgs pin those versions. With it off, the `trunk` CLI's `flakytests` also keeps resolving `latest` to the previous release, unless the caller sets `TRUNK_ANALYTICS_CLI_VERSION`. Promoting a 0.x backport after 1.x moves only `channel-0.json` either way. A pointer never moves back unless you tick `allow-downgrade`, for a rollback. Do **not** mark a release latest in the GitHub UI, since that leaves the pointers behind. +2. **Promote**: run the `Promote Release` workflow with the version. It points `channel-.json` at it. With **`promote-latest`** ticked (off by default), and when the version is the newest release overall, it also points `channel.json` at it and marks it latest on GitHub. Leave it off to hold `channel.json` and GitHub's latest back: analytics-uploader v2.1.7 and older follow one of them, and many orgs pin those versions. The `trunk` CLI's `flakytests` (0.0.14 and later) reads `channel-1.json`, so it gets a 1.x release whether or not the box is ticked. Promoting a 0.x backport after 1.x moves only `channel-0.json` either way. A pointer never moves back unless you tick `allow-downgrade`, for a rollback. Do **not** mark a release latest in the GitHub UI, since that leaves the pointers behind. 3. **Backfill**: run `Mirror Releases to S3` to copy releases that predate the S3 lane (or that a failed run left incomplete). Pass space-separated tags or `all`. Re-running it is safe because complete versions are skipped. Each download is first checked against the sha256 digest GitHub recorded for it; releases from before June 2025 have no digest and are copied unchecked. Layout under `releases/analytics-cli/prod/`: -- `channel.json`: `{"latest": ""}`, the newest release overall. Read by the `trunk` CLI and the docs. -- `channel-.json`: the same shape, for the newest release of one major version (`channel-0.json`, `channel-1.json`). Each analytics-uploader major reads its own, so `@v2` stays on 0.x. +- `channel.json`: `{"latest": ""}`, the newest release overall. Read by the docs and by `trunk` CLI releases before 0.0.14. +- `channel-.json`: the same shape, for the newest release of one major version (`channel-0.json`, `channel-1.json`). Each analytics-uploader major reads its own, so `@v2` stays on 0.x. The `trunk` CLI reads `channel-1.json`, so that pointer must never be removed or pointed at another major. - Both are written with `Cache-Control: max-age=60` and are the only mutable objects. - `/`: each GitHub release asset under the same filename, e.g. `trunk-analytics-cli-x86_64-unknown-linux.tar.gz`. - `/manifest.json`: `{"version": "", "artifacts": {"": {"sha256": ""}}}`, uploaded last, so its presence means the version is complete. From ab35bbd8f456c05c3fbf32789c001781367a13a7 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Tue, 17 Nov 2026 19:16:17 +0000 Subject: [PATCH 7/7] docs: keep implementation details out of the upgrade notes (TRUNK-23152) Users only ever pass a v2 API key: the Trunk JWT is how the trunk CLI hands a credential to analytics-cli, so the user-facing docs drop it. Also drop the channel pointer names, the fork lane's token swap, the telemetry route and format, and the test link paths, keeping what a user has to do. Co-Authored-By: Claude Opus 5.5 --- README.md | 7 +++---- UPGRADING.md | 23 ++++++++++------------- rspec-trunk-flaky-tests/README.md | 2 +- 3 files changed, 14 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index a5d83e84..53af6568 100644 --- a/README.md +++ b/README.md @@ -20,16 +20,15 @@ This is downloaded automatically using the [Trunk launcher][launcher] and when u ## Credentials -`--token` / `TRUNK_API_TOKEN` takes one of two credentials. The CLI reads which one before it calls Trunk: +`--token` / `TRUNK_API_TOKEN` takes a **v2 API key** (`trunk_v2_...`). Create one at https://app.trunk.io/settings/developer/api-tokens, and give it the **Upload test results** (`upload_test_results`) permission. -- **A v2 API key** (`trunk_v2_...`). Create one at https://app.trunk.io/settings/developer/api-tokens, and give it the **Upload test results** (`upload_test_results`) permission. The CLI sends it as `Authorization: Bearer`. -- **A Trunk JWT**. The CLI sends it on `x-trunk-token`. With a key stored by `trunk auth api-key set`, `trunk flakytests upload` exchanges it for a JWT just before the upload and passes that in `TRUNK_API_TOKEN`, while `trunk flakytests test` passes the key itself. A JWT lasts at most 15 minutes, and the one `trunk flakytests upload` passes can have about a minute left, so use a JWT only for the upload itself: a long `trunk-analytics-cli test` run, or a test-runner plugin that looks up quarantined tests early and uploads at the end, can outlive it, so give those a v2 API key. When Trunk refuses an expired JWT, the CLI says so; it does not refresh it. +With the trunk CLI, store the key once with `trunk auth api-key set`, and `trunk flakytests` passes a credential for you. It may pass a short-lived Trunk token instead of the key; you don't need to manage it. analytics-cli 1.0.0 refuses any other token, the organization token included, before it calls Trunk. Pin analytics-cli 0.17.x to keep using the organization token until December 1, 2027. To upgrade from 0.17.x, see [UPGRADING.md](UPGRADING.md). If Trunk answers `403` because the key lacks the Upload test results permission, edit the key on the API Keys screen to add it. -`--allow-forked-pr-uploads` uploads with no token, only from a GitHub Actions run of a pull request from a fork: the CLI swaps the job's `GITHUB_TOKEN` for a Trunk login. +`--allow-forked-pr-uploads` uploads with no token, only from a GitHub Actions run of a pull request from a fork. The job needs `GITHUB_TOKEN` in its environment. ## Development diff --git a/UPGRADING.md b/UPGRADING.md index cd2409a6..83ba6336 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -1,29 +1,26 @@ # Upgrading from 0.17.x to 1.0.0 -analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collections and takes only a v2 API key or a Trunk JWT. Most runs need three changes: a v2 API key, a test collection id, and no removed flags. +analytics-cli 1.0.0 uses only the Trunk API v2. It uploads only to test collections and takes only a v2 API key. Most runs need three changes: a v2 API key, a test collection id, and no removed flags. 0.17.x stays available and keeps taking the organization token until December 1, 2027. Pin it while you move: it gets important fixes on the `0.17.x` branch, and no new features. ## If you use analytics-uploader - **v3** installs the trunk CLI and runs `trunk flakytests`, which resolves analytics-cli 1.x. Move to it to get 1.0.0. -- **v2.1.8 and later**, and the floating `@v2` tag, stay on 0.x: their `latest` reads `channel-0.json`. -- **v2.1.7 and earlier, pinned by SHA or exact tag,** follow `channel.json` or GitHub's latest release. Both stay on 0.17.x until 1.0.0 is promoted to latest overall, which is a separate step after the 1.0.0 release. From then on these versions get 1.0.0, and their uploads fail with an organization token, without `test-collection-id`, or with the `quarantine` or `hide-banner` input set, which they pass as flags 1.0.0 removed. Move to v2.1.8 or v3 before then, or set `cli-version: 0.17.0`. +- **v2.1.8 and later**, and the floating `@v2` tag, stay on 0.17.x. +- **v2.1.7 and earlier, pinned by SHA or exact tag,** follow the latest analytics-cli release, which stays 0.17.x for now. 1.0.0 becomes the latest release in a later, separate step. From then on these versions get 1.0.0, and their uploads fail with an organization token, without `test-collection-id`, or with the `quarantine` or `hide-banner` input set, which they pass as flags 1.0.0 removed. Move to v2.1.8 or v3 before then, or set `cli-version: 0.17.0`. ## If you use the trunk CLI -From trunk CLI 0.0.14, `trunk flakytests` runs the newest analytics-cli 1.x unless `TRUNK_ANALYTICS_CLI_VERSION` pins a version. Earlier trunk CLI versions follow `channel.json`, so they stay on 0.17.x until 1.0.0 is promoted to latest; run `trunk update` to get 0.0.14. +From trunk CLI 0.0.14, `trunk flakytests` runs the newest analytics-cli 1.x unless `TRUNK_ANALYTICS_CLI_VERSION` pins a version. Earlier trunk CLI versions stay on 0.17.x until 1.0.0 becomes the latest release; run `trunk update` to get 0.0.14. ## What you have to change ### Credentials -`--token` / `TRUNK_API_TOKEN` must hold one of: +`--token` / `TRUNK_API_TOKEN` must hold a **v2 API key** (`trunk_v2_...`) with the **Upload test results** permission. Create it at https://app.trunk.io/settings/developer/api-tokens. With `trunk flakytests`, store the key once with `trunk auth api-key set` and the trunk CLI passes it for you. -- **A v2 API key** (`trunk_v2_...`) with the **Upload test results** permission. Create it at https://app.trunk.io/settings/developer/api-tokens. -- **A Trunk JWT**, such as the one `trunk flakytests upload` passes. A JWT lasts at most 15 minutes and the CLI does not refresh it, so use a v2 API key for a long `test` run or a test-runner plugin. - - 1.0.0 refuses the organization token before it calls Trunk; `test` refuses before it runs your tests. A key without Upload test results gets a `403` that names the permission. See [Credentials](README.md#credentials). +1.0.0 refuses the organization token before it calls Trunk; `test` refuses before it runs your tests. A key without Upload test results gets a `403` that names the permission. See [Credentials](README.md#credentials). ### Test collection @@ -42,21 +39,21 @@ From trunk CLI 0.0.14, `trunk flakytests` runs the newest analytics-cli 1.x unle ### Fork pull requests -`--allow-forked-pr-uploads` with no token works only in a GitHub Actions run of a pull request from a fork. The CLI swaps the job's `GITHUB_TOKEN` for a Trunk login, so the job needs it in the environment. There is no fallback any more: a refused swap, a missing `GITHUB_TOKEN`, or the flag outside a fork pull request uploads nothing and says why. +`--allow-forked-pr-uploads` with no token works only in a GitHub Actions run of a pull request from a fork, and the job needs `GITHUB_TOKEN` in its environment. There is no fallback any more: Trunk refusing the job's token, a missing `GITHUB_TOKEN`, or the flag outside a fork pull request uploads nothing and says why. ### Output - The summary goes to **stdout**; logs, notices and progress go to stderr. A script that reads only stderr loses the summary. - `--output-format json` (or `TRUNK_OUTPUT_FORMAT=json`) on `upload` and `validate` prints the summary as the JSON that `--summary-output-file` writes. -- Test links use the test case's GUID (`/flaky-tests/collections//tests/`) when Trunk returns the collection, and the `/t/?repo=` short link otherwise. The repository-scoped `/flaky-tests/test/` link is gone. +- Test links point at the test in its test collection. The old repository-scoped test links are gone. ### Network -1.0.0 sends its telemetry, as OTLP metrics, to `api.trunk.io` (`/v2/telemetry/v1/metrics`), the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. +1.0.0 sends its telemetry to `api.trunk.io`, the host it already uploads to, instead of `telemetry.trunk.io`. A firewall allowlist needs only `api.trunk.io` and the storage host uploads go to. ## Test-runner plugins -`test_report` and the rspec gem share the CLI's API client, so the same rules apply: a v2 API key (or a JWT that outlasts the run) and `TRUNK_TEST_COLLECTION_ID`. +The same rules apply to `test_report` and the rspec gem: a v2 API key and `TRUNK_TEST_COLLECTION_ID`. The rspec gem 1.0.0 **fails the run** when it is set up to upload but can't: with `TRUNK_ORG_URL_SLUG` and a token set, a missing `TRUNK_TEST_COLLECTION_ID` or an organization token raises `RSpec::Trunk::ConfigurationError` while `spec_helper` loads, so no examples run and RSpec exits non-zero. 0.17.x only warned and turned itself off. diff --git a/rspec-trunk-flaky-tests/README.md b/rspec-trunk-flaky-tests/README.md index 54d8621a..a3236af3 100644 --- a/rspec-trunk-flaky-tests/README.md +++ b/rspec-trunk-flaky-tests/README.md @@ -106,7 +106,7 @@ require 'trunk_spec_helper' ### Environment Variables -For a complete list of environment variables that the gem accepts, see [`lib/trunk_spec_helper.rb`](lib/trunk_spec_helper.rb). The gem uses the same environment variables as the Trunk Analytics CLI for configuration overrides. `TRUNK_ORG_URL_SLUG` and a non-empty `TRUNK_API_TOKEN` activate the plugin; without either it stays off. Once active it needs `TRUNK_TEST_COLLECTION_ID` too, and `TRUNK_API_TOKEN` must be a v2 API key with the Upload test results permission, or a Trunk JWT that outlasts the run (one lasts at most 15 minutes). With no test collection, or with an organization token, loading the plugin raises `RSpec::Trunk::ConfigurationError`, so RSpec runs no examples and exits non-zero; set `DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true` to run without Trunk. See [Credentials](../README.md#credentials). +For a complete list of environment variables that the gem accepts, see [`lib/trunk_spec_helper.rb`](lib/trunk_spec_helper.rb). The gem uses the same environment variables as the Trunk Analytics CLI for configuration overrides. `TRUNK_ORG_URL_SLUG` and a non-empty `TRUNK_API_TOKEN` activate the plugin; without either it stays off. Once active it needs `TRUNK_TEST_COLLECTION_ID` too, and `TRUNK_API_TOKEN` must be a v2 API key with the Upload test results permission. With no test collection, or with an organization token, loading the plugin raises `RSpec::Trunk::ConfigurationError`, so RSpec runs no examples and exits non-zero; set `DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true` to run without Trunk. See [Credentials](../README.md#credentials). #### `TRUNK_QUARANTINE_QUERY_FAILURE_EXIT`