Cover the ETP/v1 ↔ Client-Side Stats intersection in both directions

[darccio]

Motivation

CSS (Client-Side Stats) and the ETP/v1 trace protocol were only ever exercised separately. No scenario configured both, so the combination was reached solely by accident — via whichever protocol a tracer happened to default to.

The two features genuinely interact. CSS drops P0s client-side, changing which traces are emitted; ETP/v1 changes how they are encoded. Both act on the same trace-emission path — which is also why several parametric tests have to set DD_TRACE_STATS_COMPUTATION_ENABLED=false to observe P0s at all (see the comment at tests/parametric/test_span_sampling.py:862).

And the interaction was not hypothetical. Released dd-trace-go gates v1 on CSS capability and silently downgrades to /v0.4/traces the moment stats computation is disabled; DataDog/dd-trace-go#5122 removes that gate. system-tests never reported it, because the one place it would have surfaced — Test_V1PayloadByDefault, which runs in the DEFAULT scenario where DD_TRACE_STATS_COMPUTATION_ENABLED=false — is declared missing_feature, i.e. a non-strict xfail that cannot fail. Worse, its stated reason (not implemented by default yet) is inaccurate: Go had implemented v1-by-default; the test failed because the scenario disables CSS. The declaration hid the bug and misattributed its cause.

This PR makes the intersection explicit in both directions.

Changes

1. TRACE_STATS_COMPUTATION_V1 — CSS on, v1 protocol

Identical to TRACE_STATS_COMPUTATION apart from two lines: weblog DD_TRACE_AGENT_PROTOCOL_VERSION=1.0 and agent DD_APM_ENABLE_V1_TRACE_ENDPOINT=true. Keeping everything else byte-for-byte identical is deliberate: a failure here points at a CSS/ETP interaction, not at a sampling or obfuscation difference.

The format-sensitive stats classes now run in it too:

Class	Why
Test_Client_Stats	hit / top-level-hit counting and IsTraceRoot depend on top-level span detection, which v1 relocates into attributes
Test_Peer_Tags	peer tag extraction depends on span structure
Test_Transport_Headers	stats request headers

Deliberately excluded: Test_Time_Bucketing (timing-sensitive — would double an existing flake surface) and Test_Agent_Info_Endpoint (asserts /info capabilities, format-independent).

2. APM_TRACING_EFFICIENT_PAYLOAD_STATS_DISABLED — CSS off, v1 protocol

Scenario 1 alone is not a regression test for the coupling: CSS-on + v1 worked even while v1 was gated on CSS, so it cannot discriminate. The discriminating case is the mirror — v1 pinned while CSS is off — and no existing scenario did that. Only two scenarios pin the protocol, and neither disabled CSS.

APM_TRACING_EFFICIENT_PAYLOAD plus DD_TRACE_STATS_COMPUTATION_ENABLED=false, with Test_V1PayloadWithStatsDisabled asserting every trace still goes to /v1.0/traces in v1 format. It first asserts that no /v0.6/stats payloads exist, so it fails loudly rather than passing vacuously if CSS were somehow active.

This also settles something the suite currently cannot answer: whether the old code honoured an explicit protocol pin or overrode it — the difference between a default-selection bug and a config-precedence bug.

3. Fixed endpoint probing in Test_Client_Drop_P0s

The sequential if len(...) == 0 chain stopped at the first non-empty endpoint, so a tracer emitting on both v0.4 and v1.0 only ever had v0.4 header-checked. get_data accepts a list of path filters and matches any of them, so all trace payloads are now collected and every one is verified.

Manifests

Test_V1PayloadWithStatsDisabled is declared missing_feature for:

• golang — until Test for the exclusion of documentation modifications in library sele… #5122 ships. This is the bug being fixed, so the test is expected to fail on every current release.
• nodejs and ruby — matching every other class in this file; neither has working v1 today.

Java is deliberately left inheriting the file-level >=1.62.0. It supports v1 when the protocol is pinned and has no CSS coupling, so it is the language that proves the new test is meaningful rather than vacuous.

Plumbing

Both scenarios registered in .github/workflows/run-end-to-end.yml; affected entries updated in tests/test_the_test/scenarios.json.

Longer term: this becomes the permanent guard

Once DataDog/dd-trace-go#5122 ships, Test_V1PayloadByDefault in the DEFAULT scenario becomes a real regression guard for the decoupling, at no extra cost — DEFAULT already disables CSS and already asserts v1, which is precisely the property #5122 establishes.

That gives a concrete acceptance criterion, in this order:

1. Test for the exclusion of documentation modifications in library sele… #5122 merges and is released. It is not in v2.10.0-rc.1 — verified with git merge-base --is-ancestor — so it lands in v2.11.0 unless backported.
2. Flip Test_V1PayloadWithStatsDisabled for golang from missing_feature to that release version.
3. Flip Test_V1PayloadByDefault for golang from missing_feature (not implemented by default yet) to the same version, replacing the inaccurate reason.

⚠️ Do not set either to v2.10.0-rc.1. That tag predates the decoupling, so CSS-off still downgrades to v0.4 and both tests must fail on net-http / net-http-orchestrion. Until #5122 is released, missing_feature is the correct declaration — just not for the reason currently written down.

Worth noting for whoever does step 3: because missing_feature is a non-strict xfail, these tests will XPASS silently the moment Go starts passing them. Nothing will tell you it is time to flip them. That is the same gap that let the original coupling go unreported, and it is being tracked separately.

Notes for the reviewer

This needs R&P review on two counts — scenarios are added, and files outside tests//manifests/ are touched (utils/_context/, .github/workflows/).

One open decision. Python, .NET and PHP have CSS coverage but no ETP/v1 support, so DD_TRACE_AGENT_PROTOCOL_VERSION=1.0 will likely be ignored and they will emit legacy payloads — making TRACE_STATS_COMPUTATION_V1 a duplicate run of TRACE_STATS_COMPUTATION for them. It passes, it just costs CI time. Left ungated on purpose: when those tracers gain ETP/v1 the coverage activates by itself with no manifest edit, and it surfaces the day a tracer starts honouring the flag. Happy to add missing_feature gating for those three if you would rather pay less CI.

Verification done — the full ./format.sh --check equivalent is green locally:

• mypy 2.1.0 (as pinned): Success: no issues found in 528 source files
• ruff format --check: 612 files already formatted
• ruff check: all checks passed
• no trailing whitespace
• all three edited manifests parse through the repo's own parser (utils/manifest/_internal/parser.load), including the quoted # in the golang reason string
• Test_V1PayloadWithStatsDisabled resolves to exactly one scenario marker, APM_TRACING_EFFICIENT_PAYLOAD_STATS_DISABLED
• test_disable correctly still maps to DEFAULT — its method-level @scenarios.default overrides the class markers, per conftest.py:344

Not verified: neither scenario has been executed against a real tracer, so they are unproven end-to-end. CI is the first real proof, and the manifest declarations above are my best reading of each tracer's current state — expect to adjust them.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

🤖 Generated with Claude Code

[github-actions]

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
tests/k8s_lib_injection/test_k8s_lib_injection_profiling.py             @DataDog/injection-platform
tests/schemas/test_schemas.py                                           @DataDog/system-tests-core
tests/stats/test_stats.py                                               @DataDog/system-tests-core
tests/test_the_test/scenarios.json                                      @DataDog/system-tests-core
tests/test_v1_payloads.py                                               @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

[datadog-official]

 

⚠️ Warnings

🚦 22 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #2 / laravel11x 2     

See error

Job failed due to missing jUnit report files.

🧪 1 Test failed

tests.stats.test_stats.Test_Peer_Tags.test_peer_tags[laravel11x] from system_tests_suite  

AssertionError: Client spans should have peer tags, found: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.stats.test_stats.Test_Peer_Tags object at 0x7f18acd94350>

    def test_peer_tags(self):
        """Test that client spans include peer tags while server spans don't"""
        client_stats_found = False
        server_stats_found = False
...

DataDog/system-tests | Amazon_Linux_2_arm64.CHA: [test-app-php]     

DataDog/system-tests | Amazon_Linux_2_arm64.CO1B: [test-app-php-alpine]     

View all 22 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 10 jobs - 10 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e458535 | Docs | Datadog PR Page | Give us feedback!}