Skip to content

feat(rspec): report shared and library-defined examples at their spec location - #1365

Open
TylerJang27 wants to merge 3 commits into
mainfrom
tyler/rspec-spec-location
Open

TylerJang27 wants to merge 3 commits into
mainfrom
tyler/rspec-spec-location

Conversation

@TylerJang27

@TylerJang27 TylerJang27 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

A customer asked for this. The gem reports each example at metadata[:file_path] and metadata[:line_number], which is where the example block is written:

  • For a shared example, that's the shared examples file.
  • For rswag's run_test!, it's the rswag gem's own example_group_helpers.rb. That path is outside the repo, so no CODEOWNERS rule matches it.

Code owners are resolved in the gem from that file. So these tests get the owners of the shared file, or no owners at all.

Change

A new opt-in, TRUNK_REPORT_SPEC_LOCATION=true, changes how an example defined outside its spec file is reported. That's any example where file_path differs from rerun_file_path:

  • File: rerun_file_path, the spec file RSpec reruns the example from. The dotted classname is derived from it as before.
  • Line:
    • For it_behaves_like or include_examples, the line of that call, read from the outermost shared_group_inclusion_backtrace frame in that file.
    • Otherwise, the line of the nearest enclosing group defined in the spec. For rswag, that's the response block around run_test!, because RSpec doesn't record where run_test! was called.
  • Code owners: these follow the file, with no Rust change.

The quarantine lookup uses the same file and classname as the upload, so the two stay consistent.

An example defined in its own spec file is reported exactly as before, whether or not the variable is set.

What this means for other customers

The change is opt-in, so nothing changes for anyone who doesn't set the variable. For anyone who does, or if it ever becomes the default:

  • New test ids. The file and classname are part of a test's id. Affected examples become new test cases with no history. Quarantine overrides, flaky status, linked tickets, and test-case GUIDs in webhooks or API scripts all stay on the old ids, and failures stop being quarantined until the new tests are quarantined. Examples with generated descriptions (it { is_expected.to ... }) keep their ids, because their id is built from trunk:<id>-<location> and doesn't include the file.
  • Tests can split. The same shared group included under the same description in two spec files had one id, because both reported the shared file. It now becomes one test per spec file.
  • Tests can merge (rare). Two different shared groups, each defining an example with the same full description and both included with include_examples in the same context, used to have separate ids because their files differed. They now share one id.
  • Code owners change for every affected example. Owner-based routing, notifications, and dashboards follow the spec file instead of the shared file.
  • rswag line numbers point at the enclosing response group, not the run_test! line.
  • Default-on is a breaking change. If this ever became the default, every customer with shared examples would get new ids on upgrade. That should only happen in a major version, with migration notes in UPGRADING.md.

Moving quarantines to the new ids. After the first upload with the variable set, we match old and new test cases on parent and name within the same collection, repo, and variant, and copy quarantines with POST /v2/tests/quarantine:

  • User overrides (always and never quarantine) are copied permanently.
  • Auto-quarantined flaky tests get a temporary always-quarantine override.
  • About a week later, we clear the temporary overrides whose latest override is still ours, so detection takes over.

To avoid failures before the copy, make the first upload from a branch that doesn't gate merges.

Testing

  • New test/location_spec.rb, run in a sandbox. Shared examples are defined in a support file. The run_test! example comes from the real rswag-specs gem (MIT), using path, get and response. It covers:
    • an example in its own spec file
    • it_behaves_like
    • include_examples in a nested context
    • rswag's run_test!
    • the classname
    • unchanged behaviour when the variable is unset
  • Each of the three parts of the location lookup (inclusion line, enclosing group line, file) was broken in turn, and each break failed the spec.
  • bundle exec rspec test/: 67 examples, 0 failures (Ruby 3.3, rspec-core 3.13.2, native extension built locally).
  • rubocop 1.87.0 reports nothing new beyond the metrics cops this file already suppresses. Prettier passes on the README.
  • trunk check couldn't run here because trunk didn't find the system Rust runtime.

rswag-specs is a test-only dependency, and it's heavy.

  • It's in the gem's Gemfile, not the gemspec, so nothing changes for gem users.
  • It pulls about 50 gems, including Rails components and nokogiri.
  • The lockfile was resolved under Ruby 3.1 (Rails 7.2.4, nokogiri 1.18.10), because the rake test job runs Ruby 3.1. Under Ruby 3.3 it resolves to Rails 8.1, which needs Ruby 3.2 or newer.
  • The upload harness Gemfile also adds it, unpinned and with no lockfile. The main-branch smoke run executes this suite there on Ruby 3.1 to 4.0, so each Ruby resolves its own version. Every run of the harness now installs those gems.

🤖 Generated with Claude Code

… location

With TRUNK_REPORT_SPEC_LOCATION=true, an example defined outside its spec
file (a shared example, or rswag's run_test!) is reported at the spec file
RSpec reruns it from: the it_behaves_like or include_examples line, or else
the nearest enclosing group in the spec. Code owners follow that file. It is
opt-in because the file is part of a test's id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Nov 18, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

The location spec now defines its rswag example with rswag-specs 2.17.0
instead of a stand-in. The lockfile is resolved under Ruby 3.1, which the
unit-test job runs, so it takes Rails 7.2 components. The upload harness
Gemfile gets rswag-specs too, since the main-branch smoke run executes the
same suite there.

A shared-group inclusion frame with no location no longer raises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.84%. Comparing base (05937c7) to head (a9e50a4).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1365      +/-   ##
==========================================
+ Coverage   85.62%   85.84%   +0.21%     
==========================================
  Files          85       85              
  Lines       19192    19204      +12     
==========================================
+ Hits        16434    16485      +51     
+ Misses       2758     2719      -39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rswag-specs brings in nokogiri, which ships native builds for many
platforms, and Bundler puts every one of them in the harness lockfile. The
built gem exists for one platform only, so `bundle install --local` could
not resolve it. The harness now locks to the gem's own platform before
adding it to the Gemfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants