Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,8 @@
"GET /v2/test-cases",
"GET /v2/test-cases/{testCaseId}",
"GET /v2/test-cases/{testCaseId}/status-history",
"GET /v2/test-cases/{testCaseId}/test-runs"
"GET /v2/test-cases/{testCaseId}/test-runs",
"POST /v2/test-cases/{testCaseId}/ticket"
]
},
{
Expand Down
10 changes: 6 additions & 4 deletions flaky-tests/detection/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,22 +39,24 @@ Trunk groups monitors into two categories based on what they do when they activa
- **Health classification monitors** determine a test's overall health status (healthy, flaky, or broken). When one activates, the test's status changes across the dashboard, CI annotations, and notifications.
- **Lifecycle and performance monitors** apply labels to tests based on lifecycle events or performance characteristics. They do not affect health status. These monitors appear in a separate section of the monitors page.

**Default state** below is what a new [test collection](../test-collections#what-a-new-collection-starts-with) starts with.

### Health Classification Monitors

| Monitor | What it detects | Available actions | Default state |
| --- | --- | --- | --- |
| [**Pass-on-Retry**](./pass-on-retry-monitor) | A test fails then passes on the same commit (retry after failure) | Classify (flaky) or [apply labels](../management/test-labels#automatic-labeling-from-monitors) | Enabled |
| [**Failure Rate**](./failure-rate-monitor) | Failure rate exceeds a configured percentage over a time window | Classify (flaky or broken) or [apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |
| [**Failure Count**](./failure-count-monitor) | A test accumulates a configured number of failures in a rolling window | Classify (flaky or broken) or [apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |
| [**Failure Rate**](./failure-rate-monitor) | Failure rate exceeds a configured percentage over a time window | Classify (flaky or broken) or [apply labels](../management/test-labels#automatic-labeling-from-monitors) | Enabled |
| [**Failure Count**](./failure-count-monitor) | A test accumulates a configured number of failures in a rolling window | Classify (flaky or broken) or [apply labels](../management/test-labels#automatic-labeling-from-monitors) | Enabled |

### Lifecycle and Performance Monitors

These monitors apply labels based on lifecycle events or performance characteristics. They do not classify tests as flaky or broken, and they do not contribute to the test's overall health status.

| Monitor | What it detects | Available actions | Default state |
| --- | --- | --- | --- |
| [**New Test**](./new-test-monitor) | A test case seen for the first time, tracked for a configurable grace period | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |
| [**Skipped Test**](./skipped-test-monitor) | A test is consistently skipped across runs within a time window | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |
| [**New Test**](./new-test-monitor) | A test case seen for the first time, tracked for a configurable grace period | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Enabled |
| [**Skipped Test**](./skipped-test-monitor) | A test is consistently skipped across runs within a time window | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Enabled |
| [**Slow Test**](./slow-test-monitor) | A test's average duration exceeds a configured threshold | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |
| [**Timeout Inflation**](./timeout-inflation-monitor) | A test's typical failure duration is much larger than its passing duration, exposing an inflated timeout | [Apply labels](../management/test-labels#automatic-labeling-from-monitors) | Disabled |

Expand Down
4 changes: 2 additions & 2 deletions flaky-tests/get-started/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ If not, configure your test frameworks to output JUnit XML:

### Step 2: Configure CI uploads

Add test result uploads to all CI jobs that run tests.
[Create a test collection](../test-collections#create-a-collection) and copy its **Collection ID**, then add test result uploads to all CI jobs that run tests. Every upload step passes that ID, as `--test-collection-id` or the `TRUNK_TEST_COLLECTION_ID` environment variable.

1. See [**CI Providers**](./ci-providers/) for integration instructions
2. Configure uploads in jobs that run on:
Expand Down Expand Up @@ -51,7 +51,7 @@ Trunk automatically recognizes `main`, `master`, and `develop` as stable branche

After uploads are flowing, open your [test collection](../test-collections)'s **Monitors** tab to set up detection.

**Pass-on-retry** is enabled by default and is the recommended baseline for everyone. It catches the most common flakiness pattern — a test that fails and then passes on retry within the same commit — without any configuration needed.
A collection arrives with [five monitors already seeded](../test-collections#what-a-new-collection-starts-with), including pass-on-retry, failure count, and failure rate. Your job here is to check them against how you actually run tests — the seeded branch patterns in particular are defaults, not a reading of your repository.

**Failure rate monitors** let you detect flakiness based on failure rate over a rolling time window. How you configure them depends on your CI setup:

Expand Down
41 changes: 35 additions & 6 deletions flaky-tests/get-started/migrate-to-test-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,40 @@ Flaky Tests opens on your test collections. To reach the repository view, click
<img className="hidden dark:block" src="/assets/flaky-tests/migration-legacy-view-button-dark.png" alt="The Collections list header in Flaky Tests, with a Legacy view button on the right, a collections search box, and a Create Collection button." />
</Frame>

## Nothing carries over from your repositories
## What carries over, and what doesn't

Monitors, quarantine overrides, ticketing configuration, and infrastructure-failure thresholds are all per-collection. A repository's configuration does not transfer to a collection, so anything you have tuned on a repository has to be set up again on the collection you want it on.
Configuration is per-collection. Nothing transfers on its own, but two things can be brought across deliberately while you are migrating.

That is often what you want. You may wish to create different monitors for each collection, such as more sensitive flakiness detection for unit tests than for end-to-end tests.
| | Brought across |
| --- | --- |
| Monitors | Yes — from the collection's **Monitors** tab, see below |
| Ticketing connections | Yes — from **Settings → Organization → Ticketing**, see below |
| Quarantine overrides | No |
| Ticket automation settings | No |
| Infrastructure-failure thresholds | No |

Starting fresh is often what you want. You may wish to create different monitors for each collection, such as more sensitive flakiness detection for unit tests than for end-to-end tests.

### Bring your repository monitors across

While your organization has both views, the collection's **Monitors** tab offers **Migrate from a repository**. Pick a repository and every one of its monitors is sorted into three groups:

- **Not in this collection** — selected by default, with the configuration the migrated monitor will have.
- **Settings differ** — the same monitor name on both sides with different settings. Not selectable; edit the collection's monitor instead, so you don't end up with two monitors sharing a name.
- **Already in this collection** — same name, same settings.

A fourth group lists monitors only the collection has, so the comparison reads both ways. Migrating runs one way only: your repositories' monitors are never changed.

<Frame caption="Every monitor on the repository, sorted by whether the collection already has it.">
<img className="block dark:hidden" src="/assets/flaky-tests/get-started/migrate-monitors-light.png" alt="The Migrate monitors from a repository page, with a repository chosen in the Migrate from repository dropdown and a Migrate 1 monitor button. Three sections list the comparison: Not in this collection, with a checkbox beside each monitor's name, type, and settings; Already in this collection, showing one monitor marked Already configured; and Only in this collection, listing the monitors the repository does not have." />
<img className="hidden dark:block" src="/assets/flaky-tests/get-started/migrate-monitors-dark.png" alt="The Migrate monitors from a repository page, with a repository chosen in the Migrate from repository dropdown and a Migrate 1 monitor button. Three sections list the comparison: Not in this collection, with a checkbox beside each monitor's name, type, and settings; Already in this collection, showing one monitor marked Already configured; and Only in this collection, listing the monitors the repository does not have." />
</Frame>

### Bring a ticketing connection across

An organization ticketing integration can be copied from a repository that already has one, rather than re-entering an API token and re-picking a project. In the add-connection picker on **Settings → Organization → Ticketing**, choose **Copy from a repo**.

The copy is named `<Provider> (<repo>)`, because a collection picks a connection by name. Ticket automation does not copy — repository automation stays on and the collection's stays off, so you can time the cutover yourself.

## Migrate a CI job

Expand All @@ -56,11 +85,11 @@ Open the collection's **Uploads** tab. An upload appears as soon as Trunk accept
</Step>

<Step title="Review the collection's monitors">
Open the collection's **Monitors** tab and check the seeded defaults against what you run on the repository today. Adjust thresholds and branch patterns here rather than assuming the repository's carried over.
Open the collection's **Monitors** tab and check the [seeded defaults](../test-collections#what-a-new-collection-starts-with) against what you run on the repository today. Either adjust them here, or use **Migrate from a repository** to bring the repository's monitors across.
</Step>

<Step title="Turn on quarantining, then re-apply your overrides">
Quarantining is configured per collection, and enabling it replaces the repository's quarantining for every upload routed to that collection. Your repository's **Always Quarantine** and **Never Quarantine** overrides do not follow.
While you are migrating, a new collection starts with quarantining off, because the repository's settings are still governing your uploads. Enabling it replaces the repository's quarantining for every upload routed to that collection, and your repository's **Always Quarantine** and **Never Quarantine** overrides do not follow.

Overrides can only be set once quarantining is enabled on the collection, so re-apply the ones that matter as soon as you turn it on.
</Step>
Expand Down Expand Up @@ -111,7 +140,7 @@ Collections don't backfill history — metrics start at the collection's first u
</Accordion>

<Accordion title="Why aren't tests being quarantined?">
Quarantining has to be enabled on the collection, and a test has to have been detected by one of the collection's monitors. Check both on the collection's **Settings** tab and **Monitors** tab.
Check **Settings → Quarantining** on the collection. While you are migrating, a collection starts with quarantining off, and auto-quarantine is off even once you turn quarantining on — so until you turn it on too, only tests you have set an **Always Quarantine** override on are quarantined. Auto-quarantine covers tests a monitor has flagged **flaky**; a **broken** test is never a candidate.
</Accordion>

<Accordion title="Can a collection contain tests from several repositories?">
Expand Down
2 changes: 1 addition & 1 deletion flaky-tests/management/ticketing/asana-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Asana tasks are created in a single project that you choose during setup. Throug

To connect an Asana project:

1. In the Trunk app, navigate to **Settings** → **Repositories** → **[your repository]** → **Ticketing**.
1. In the Trunk app, navigate to **Settings** → **Organization** → **Ticketing**. (On a repository, use **Settings** → **Repositories** → **[your repository]** → **Ticketing** instead.)
2. Click **Connect** on the **Asana** row.
3. Add an [Asana personal access token](#personal-access-token) and click **Connect**.
4. In the **Configuration** section that appears, select a **Workspace**, then select a **Project**, and click **Update**.
Expand Down
4 changes: 2 additions & 2 deletions flaky-tests/management/ticketing/jira-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Webhook payloads will also contain ticket information when a Jira ticket is crea
<img className="hidden dark:block" src="/assets/flaky-tests/management/ticketing/jira-connect-form-dark.png" alt="" />
</Frame>

To connect a Jira Cloud project, navigate to **Settings** → **Repositories** → **[your repository]** → **Ticketing** and select **Jira** as your Ticketing System.
To connect a Jira Cloud project, navigate to **Settings** → **Organization** → **Ticketing** and select **Jira** as your Ticketing System. (On a repository, use **Settings** → **Repositories** → **[your repository]** → **Ticketing** instead.)

Then complete the form and click Connect to Jira Cloud with the following information.

Expand Down Expand Up @@ -116,7 +116,7 @@ Some Jira projects require additional fields beyond the standard fields (summary

### Configuring custom fields

In the Jira integration settings (**Settings** → **Repositories** → **[your repository]** → **Ticketing**), select an issue type. Trunk fetches all available fields for that issue type from the Jira API and displays inputs for each supported field.
In the Jira integration settings, select an issue type. Trunk fetches all available fields for that issue type from the Jira API and displays inputs for each supported field.

For each field, you can:

Expand Down
2 changes: 1 addition & 1 deletion flaky-tests/management/ticketing/linear-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Webhook payloads will also contain ticket information when a Linear ticket is cr

To connect a Linear project:

1. Navigate to **Settings** → **Repositories** → **[your repository]** → **Ticketing**.
1. Navigate to **Settings** → **Organization** → **Ticketing**. (On a repository, use **Settings** → **Repositories** → **[your repository]** → **Ticketing** instead.)
2. Select **Linear** as your Ticketing System.
3. Add a [Linear API key](./linear-integration#api-key-permissions)
4. Select a Team and **Connect to Linear**.
Expand Down
6 changes: 4 additions & 2 deletions flaky-tests/quarantining/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ With quarantining enabled, the Analytics Uploader will compare failed test cases

### Quarantining settings

To enable quarantining, navigate to **Settings** → **Repositories** → **[repository]** → **Flaky Tests** → toggle **Enable Test Quarantining** on.
A new [test collection](../test-collections) starts with quarantining enabled, unless your organization is still [migrating](../get-started/migrate-to-test-collections), in which case it starts off. To change it, navigate to **Flaky Tests** → **Collections** → **[collection]** → **Settings** → **Quarantining**.

On a repository, navigate to **Settings** → **Repositories** → **[repository]** → **Flaky Tests** → toggle **Enable Test Quarantining** on. This route is unavailable once your organization has [fully migrated to collections](../get-started/migrate-to-test-collections).

<Frame>
![](/assets/flaky-tests/quarantining/enable_test_quarantining.png)
Expand Down Expand Up @@ -296,7 +298,7 @@ Once quarantining is active, the **Quarantining** tab provides a central hub for

## Audit logs

Trunk provides audit logs for all setting changes and overwrites for individual tests. You can access the audit log by navigating to **Settings** → **Repositories** → **[repository]** → **Flaky Tests** → **Audit logs** under the Enable Test Quarantining heading.
Trunk provides audit logs for all setting changes and overwrites for individual tests. On a collection, the audit log is on its **Settings** → **Quarantining** page. On a repository, navigate to **Settings** → **Repositories** → **[repository]** → **Flaky Tests** → **Audit logs** under the Enable Test Quarantining heading.

<Frame>
![](/assets/flaky-tests/quarantining/qurantine-audit-logs.png)
Expand Down
34 changes: 25 additions & 9 deletions flaky-tests/test-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,18 +34,27 @@ The ID is what CI needs. Nothing else about a collection is used to route upload

Any member of your organization can create a collection.

1. In Flaky Tests, open **Collections**.
2. Click **Create Collection**.
3. Enter a **Collection name**, and a description if you want one.
4. Click **Create collection**.
Enter a **Collection name**, and a description if you want one. Your first collection is created from the form Flaky Tests opens on; later ones from **Create collection** in the actions menu on the collections list.

You land on the new collection. Its **Tests** and **Uploads** tabs stay disabled until test results arrive.
You land on the new collection. Its **Uploads** tab opens when an upload arrives; **Tests** waits until those results are ingested.

## What a new collection starts with

A new collection starts with a basic set of flake-detection monitors, so detection starts working as soon as test results arrive.
A new collection starts with five monitors, so detection works as soon as test results arrive.

You can change them, and everything else about the collection, from its **Monitors** and **Settings** tabs.
| Monitor | Configuration | Action |
| --- | --- | --- |
| Pass-on-retry | 1 retry, 7 day resolve, every branch | Classify as flaky |
| Failure count | 5 failures, 7 day resolve | Classify as flaky |
| Failure rate | 90% over 24 hours, 5 sample minimum | Classify as broken |
| New test | 7 days | Apply the **New** label |
| Skipped test | 24 hour window, 7 day resolve | Apply the **Skipped** label |

Every monitor but pass-on-retry watches `main`, `master`, `develop`, `dev`, and `release/*`. A collection has no repository until its first upload, so check those branch names against your own.

[Quarantining](./quarantining/) also starts enabled, with auto-quarantine off, so nothing is quarantined until you turn auto-quarantine on or set an override on a test. While your organization is [migrating](./get-started/migrate-to-test-collections), it starts off instead, because your repositories' settings are still governing uploads.

You can change any of it from the collection's **Monitors** and **Settings** tabs.

## Upload test results to a collection

Expand Down Expand Up @@ -80,21 +89,28 @@ A CI job uploads to exactly one collection. To send results to more than one col

## Finish setting up a collection

Each collection has a setup checklist, on its **Overview** tab before results arrive and on its **Settings** tab permanently. It tracks four steps:
Each collection has a setup checklist, on its **Overview** tab before results arrive and on its **Settings** tab permanently. It tracks five steps:

1. **Send your first upload**
2. **Ingest your test results**
3. **Review flake detection**
4. **Review quarantining**
5. **Review ticketing**

The first two complete on their own once uploads are flowing. You complete the last three yourself, from the **Monitors** tab, **Settings → Quarantining**, and **Settings → Ticketing**. If you are not using ticketing, say so and that step is complete.

The first two complete on their own once uploads are flowing. The rest can be configured in the app.
Ticketing may already be done. Connections belong to your organization, so another collection may have set one up.

## Permissions

| Action | Admin | Member |
| --- | --- | --- |
| View collections | Yes | Yes |
| Create a collection | Yes | Yes |
| Create, edit, delete, or mute a monitor | Yes | Yes |
| Connect, copy, or remove an organization ticketing integration | Yes | Yes |
| Choose a collection's default ticketing connection | Yes | Yes |
| Create, link, or unlink a ticket | Yes | Yes |
| Edit a collection's name or description | Yes | No |
| Change quarantining settings | Yes | No |
| Delete a collection | Yes | No |
Expand Down
Loading
Loading