Skip to content

ci(release): resolve dd-trace-rs test matrix from remote tags - #2334

Draft
iunanua wants to merge 1 commit into
mainfrom
igor/versioning/release-proposal-testing-v2
Draft

ci(release): resolve dd-trace-rs test matrix from remote tags#2334
iunanua wants to merge 1 commit into
mainfrom
igor/versioning/release-proposal-testing-v2

Conversation

@iunanua

@iunanua iunanua commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

The release-proposal-test job pinned a hardcoded list of dd-trace-rs tags.

Resolve the matrix at run time instead: list the remote tags, group them into release lines (MAJOR from 1.0.0 on, MAJOR.MINOR while MAJOR is 0, since a 0.x minor bump is breaking), and take the newest patch of each of the most recent lines.

The release-proposal test job pinned a hardcoded list of dd-trace-rs tags,
which had already gone stale (it pinned v0.4.0 as newest while v0.5.0
exists, and tested two patches of the same 0.3 line).

Resolve the matrix at run time instead: list the remote tags, group them
into release lines (MAJOR from 1.0.0 on, MAJOR.MINOR while MAJOR is 0,
since a 0.x minor bump is breaking), and take the newest patch of each of
the most recent lines. Repo, tag prefix and line count are workflow-level
env vars.

The resolve step runs unconditionally, unlike the neighbouring steps:
a skipped step yields an empty output, and fromJSON('') in a matrix is a
hard workflow error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 7, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 1 Pipeline job failed

Required checks pass | allchecks   View in Datadog   GitHub Actions

See error Some checks have failed or timed out. Please check the workflow run summary for details.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 75.90% (-0.01%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: edf3cc7 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.20 MB 8.20 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 89.19 MB 89.19 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 100.37 MB 100.37 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.06 MB 11.06 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.63 MB 26.63 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 94.56 KB 94.56 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 191.43 MB 191.43 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 1001.70 MB 1001.70 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.71 MB 8.71 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 94.56 KB 94.56 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.62 MB 25.62 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 51.01 MB 51.01 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 23.22 MB 23.22 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 96.04 KB 96.04 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 196.35 MB 196.33 MB -0% (-16.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 990.83 MB 990.83 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.73 MB 6.73 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 96.04 KB 96.04 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 27.55 MB 27.55 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 48.56 MB 48.56 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 79.47 MB 79.47 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 9.13 MB 9.13 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 95.12 MB 95.12 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.14 MB 11.14 MB 0% (0 B) 👌

@pr-commenter

pr-commenter Bot commented Aug 7, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-08-07 16:16:49

Comparing candidate commit edf3cc7 in PR branch igor/versioning/release-proposal-testing-v2 with baseline commit ea75b04 in branch main.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 136 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:profiles_dictionary/profile_string_inserts/threads/1

  • 🟥 execution_time [+17.034µs; +18.456µs] or [+5.647%; +6.119%]
  • 🟥 throughput [-196375.433op/s; -180891.750op/s] or [-5.785%; -5.328%]

scenario:vec_map/get_hit/8

  • 🟥 execution_time [+4.233ns; +4.280ns] or [+7.844%; +7.931%]
  • 🟥 throughput [-10901343.317op/s; -10775758.694op/s] or [-7.354%; -7.269%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant