[SLES-2971] fix(traces): don't propagate X-Ray Sampled=0 as a drop decision

[lym953]

https://datadoghq.atlassian.net/browse/SLES-2971

Overview

A customer's Go Lambda dropped from ~14 spans per invocation to 2 after moving from extension v87 + DD_EXTENSION_VERSION=compatibility to v98. The spans weren't lost in transit — the tracer built the full tree, but the whole trace was marked sampled-out.

With no readable _datadog attribute on the SQS record, extract_span_context falls back to the X-Ray AWSTraceHeader and mapped its Sampled flag straight onto a Datadog priority:

let sampling_priority = i8::from(sampled == "1");

The function is X-Ray PassThrough, so Sampled != 1 yields priority 0. That gets injected into the start-invocation response (listener.rs:216-224), dd-trace-go adopts it for the whole local trace, and the backend drops the tracer's chunk. The extension's own inferred + invocation spans survive because a priority is only read for them when trace_id == 0 (processor.rs:1401-1419) — hence exactly 2 spans left.

From the customer's flare: Extracted trace context from inferred span 0×, Returning generated span context 24×, and every app span logged with _sampling_priority_v1:0 in an otherwise correct tree.

v87 avoided this because the Go extension guards the same fallback with rootRegex = "Root=1-[0-9a-fA-F]{8}-00000000[0-9a-fA-F]{16}" (datadog-agent, carriers.go:37), rejecting AWS-generated ids and returning no headers, so the tracer sampled for itself.

Fix

Only honor Sampled=1 as a keep; leave the priority unset otherwise so the tracer decides. The trace id still comes from the header, so correlation is unchanged.

I did not port Go's regex: #452 appears to accept AWS-native ids deliberately (the eventbridge_sqs_java_header_event.json fixture would fail that regex), so porting it would likely regress that correlation.

Open questions

• Drop the priority, or port the regex instead? This keeps X-Ray correlation; the regex would restore v87 exactly but likely undo feat(bottlecap): add aws trace header for java and sqs #452. Input welcome from whoever knows feat(bottlecap): add aws trace header for java and sqs #452's intent.
• Behavior change: traces X-Ray sampled out are now subject to the tracer's sampling, so ingestion volume rises for anyone relying on X-Ray's sampler to suppress it.
• Inferred, not observed: that this customer's AWSTraceHeader is AWS-generated rather than Datadog-planted. The flare never logs the raw value; a raw SQS record would confirm.

Testing

• cargo test --lib 543 pass, clippy -D warnings clean, fmt clean
• Updated extract_java_sqs_header_context (fixture carries Sampled=0) and added explicit Sampled=0 / Sampled=1 cases

Verified on a live Lambda (Go, arm64, provided.al2023, SQS trigger, TracingConfig Mode=PassThrough, us-east-2, datadog-lambda-go v1.32.0 + dd-trace-go/v2 v2.9.2 to match the customer). Messages sent with an AWS-native AWSTraceHeader and no _datadog attribute:

Case	Layer	Sampled	Span priorities
before	public Datadog-Extension-ARM:98	0	all 14 spans _sampling_priority_v1:0
after	this branch, built + published	0	all 14 spans _sampling_priority_v1:1
after	this branch	1	all 14 spans _sampling_priority_v1:1

The before-case reproduces the customer's signature exactly — every span including the aws.lambda root stamped priority 0. Both cases log Returning generated span context, so the fallback still runs; and the tracer's trace id still equals the header's low 64 bits before and after, so X-Ray correlation is unchanged:

before: Root=1-bd6d9dbd-c3f0773b7a5b471c2cc4d662 -> low64 8816718881843631714 == dd.trace_id
after:  Root=1-1ba0fb24-3e6494513706243a464ac3af -> low64 3964896354652308399 == dd.trace_id

Not re-verified: the resulting backend span count (the customer's "14 -> 2"). Traces flushed fine, but the sandbox API key's org isn't one I can query, so I could not count spans in the UI. That step follows from p0 traces being dropped rather than from a measurement here.

🤖 Generated with Claude Code

[datadog-prod-us1-5]

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64)     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 00c42b5 | Docs | Datadog PR Page | Give us feedback!}