Skip to content

feat(rspec)!: fail the run when Trunk is configured but can't upload - #1337

Merged
trunk-io[bot] merged 2 commits into
mainfrom
tylerbeebe/gem-fail-loudly
Nov 17, 2026
Merged

trunk-io[bot] merged 2 commits into
mainfrom
tylerbeebe/gem-fail-loudly

Conversation

@acatxnamedvirtue

Copy link
Copy Markdown
Contributor

Summary

The rspec gem now fails the run, instead of warning and staying off, when it is configured to upload but can't: a run with TRUNK_ORG_URL_SLUG and a token, but no TRUNK_TEST_COLLECTION_ID or an organization token. This reverses the "stays off, with a warning" choice from #1298 and #1314.

Situation Before After
Slug and token set, no TRUNK_TEST_COLLECTION_ID warning, gem off, build green RSpec::Trunk::ConfigurationError while spec_helper loads: no examples run, non-zero exit
Slug set, organization token warning, gem off, build green same error, with the existing "not a v2 API key… pin analytics-cli 0.17.x" text
DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true, the Knapsack detector, or no slug quietly off unchanged
Empty TRUNK_API_TOKEN warning, gem off quietly off (counts as no token)

Both errors end with "To run without Trunk, set DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true."

Why

The gem is published to RubyGems as part of every release, with no prerelease or channel step, so 1.0.0 becomes the newest stable gem the day it ships. Our docs install it unpinned (gem "rspec_trunk_flaky_tests") and suggest bundle update rspec_trunk_flaky_tests. A Dependabot or Renovate bump to 1.0.0, for a repo without a collection id or still on the organization token, would pass CI and merge, and then uploads and quarantining would stop with only a warning in the log, so quarantined flaky tests start failing builds with nothing pointing at the bump. Failing the load makes that PR's CI red instead. It matches the CLI, whose test command exits 2 before running the tests when --test-collection-id is missing.

An empty token stays quiet on purpose: it is what a fork pull request's missing secret turns into, in customers' CI and this repo's.

Follow-up

The 1.0.0 upgrade notes in #1329 say the gem "stays off, with a warning" (Test collection, and Test-runner plugins). They need to say it fails the run once this merges.

Test plan

  • test/test_collection_spec.rb and test/token_spec.rb now expect the load to fail with the message, and cover the empty-token and turned-off cases. One case runs a real rspec against a probe spec and checks it exits non-zero without running the example.
  • bundle exec rake test (with TRUNK_LOCAL_UPLOAD_DIR, as CI sets): 64 examples, 0 failures.
  • trunk check and trunk fmt clean on the changed files.
  • CI's gem upload steps set a literal collection id and a v2 token, or an empty token on a fork, so they keep working.

🤖 Generated with Claude Code

With TRUNK_ORG_URL_SLUG and a token set, the plugin now raises
RSpec::Trunk::ConfigurationError while spec_helper loads, instead of
warning and staying off, when the run names no test collection or the
token is an organization token. RSpec runs no examples and exits
non-zero, as the CLI's `test` command does for a missing
--test-collection-id.

A warning left the build green while uploads and quarantining stopped,
so a dependency bot's upgrade to 1.0.0 would merge unnoticed.

An empty TRUNK_API_TOKEN now counts as none and keeps the plugin off
quietly, since that is what a fork pull request's missing secret turns
into. DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true still turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

pyo3's build links the runner's python3, and the macos-latest image now has a
Python 3.15 with no libpython to link against (ld: library 'python3.15' not
found), so codeowners, context-js and context-py fail to link before any test
runs. Pin the version setup_build_pyo3 already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
acatxnamedvirtue added a commit that referenced this pull request Nov 17, 2026
…ly when promoted to latest (TRUNK-23152)

- The rspec gem 1.0.0 raises RSpec::Trunk::ConfigurationError without a
  collection id or with an organization token (#1337), with
  DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true to opt out and a ~> 0.17.0 pin to stay.
- analytics-uploader v2.1.7 and older follow channel.json or GitHub's latest,
  which move only when 1.0.0 is promoted to latest (#1339).
- trunk CLI 0.0.14 runs analytics-cli 1.x unless pinned (trunk-io/trunk2#8640).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.05%. Comparing base (e5df5d4) to head (2142bc0).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1337      +/-   ##
==========================================
+ Coverage   85.85%   86.05%   +0.19%     
==========================================
  Files          85       85              
  Lines       19538    19538              
==========================================
+ Hits        16775    16814      +39     
+ Misses       2763     2724      -39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trunk-io
trunk-io Bot merged commit dbfb00a into main Nov 17, 2026
33 checks passed
acatxnamedvirtue added a commit that referenced this pull request Nov 17, 2026
…ly when promoted to latest (TRUNK-23152)

- The rspec gem 1.0.0 raises RSpec::Trunk::ConfigurationError without a
  collection id or with an organization token (#1337), with
  DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true to opt out and a ~> 0.17.0 pin to stay.
- analytics-uploader v2.1.7 and older follow channel.json or GitHub's latest,
  which move only when 1.0.0 is promoted to latest (#1339).
- trunk CLI 0.0.14 runs analytics-cli 1.x unless pinned (trunk-io/trunk2#8640).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trunk-io Bot pushed a commit that referenced this pull request Nov 17, 2026
* docs: add 0.17.x to 1.0.0 upgrade notes (TRUNK-23152)

UPGRADING.md lists what a 0.17.x user has to change for 1.0.0: a v2 API key
or JWT, a required test collection, the removed flags and variables, the
fork lane with no v1 fallback, the summary on stdout, GUID links, the
telemetry host, and which analytics-uploader versions pick up 1.0.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: name the telemetry route's new path in the upgrade notes (TRUNK-23152)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: name the metrics telemetry route in the upgrade notes (TRUNK-23152)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: link the API keys page and date the 0.17.x pin in the upgrade notes (TRUNK-23152)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: the rspec gem fails the run, and 1.0.0 reaches old uploaders only when promoted to latest (TRUNK-23152)

- The rspec gem 1.0.0 raises RSpec::Trunk::ConfigurationError without a
  collection id or with an organization token (#1337), with
  DISABLE_RSPEC_TRUNK_FLAKY_TESTS=true to opt out and a ~> 0.17.0 pin to stay.
- analytics-uploader v2.1.7 and older follow channel.json or GitHub's latest,
  which move only when 1.0.0 is promoted to latest (#1339).
- trunk CLI 0.0.14 runs analytics-cli 1.x unless pinned (trunk-io/trunk2#8640).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: the trunk CLI reads channel-1.json, not channel.json (TRUNK-23152)

trunk-io/trunk2#8640 moves the trunk CLI's flakytests to channel-1.json,
so promote-latest no longer holds it back, and channel-1.json becomes a
pointer that must not be removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: keep implementation details out of the upgrade notes (TRUNK-23152)

Users only ever pass a v2 API key: the Trunk JWT is how the trunk CLI
hands a credential to analytics-cli, so the user-facing docs drop it.
Also drop the channel pointer names, the fork lane's token swap, the
telemetry route and format, and the test link paths, keeping what a
user has to do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants