test(otel): OTel HTTP semantics coverage against the cross-tracer RFC

[link04]

Motivation

End-to-end coverage for the OTel HTTP semantic conventions the tracers are adding behind DD_TRACE_OTEL_SEMANTICS_ENABLED, checked against the cross-tracer RFC rather than against whatever any one tracer currently emits.

This supersedes #7139, which was closed unmerged. This branch is cut from its head, so its commits are included here.

Changes

31 tests to 44, across six classes: 20 server, 11 client, 5 OTLP-only server, 4 OTLP-only client, 2 error-status config, 2 stats.

One new scenario: OTEL_SEMANTICS_STATS, separate rather than extra env on OTEL_SEMANTICS, because DD_TRACE_COMPUTE_STATS also turns on client-side P0 dropping, which would break every span assertion in the existing scenario.

Substance is in tests/test_otel_http_semantics.py: derived span names, a helper that reads an int attribute from either meta or metrics depending on export path, shared legacy-key tuples, and the server/client split of the OTLP-only class.

What is in the suite, scenario by scenario

44 tests in tests/test_otel_http_semantics.py, six classes, four scenarios. Two classes are shared between OTEL_SEMANTICS and OTEL_SEMANTICS_OTLP, so the same 44 tests produce 75 executions per weblog.

Scenario	Classes	Tests run	Extra env on top of the flag
OTEL_SEMANTICS	Server, Client	31	DD_TRACE_CLIENT_IP_ENABLED, DD_TRACE_RESOURCE_RENAMING_ENABLED, obfuscation regexp
OTEL_SEMANTICS_OTLP	Server, Client, OTLP_Server, OTLP_Client	40	same, plus DD_TRACE_OTEL_ENABLED and the OTLP exporter pointed at the proxy
OTEL_SEMANTICS_OTLP_CUSTOM_ERROR_STATUSES	ErrorStatusConfiguration	2	OTLP, plus DD_TRACE_HTTP_SERVER_ERROR_STATUSES=200
OTEL_SEMANTICS_STATS	Stats_Consistency	2	DD_TRACE_COMPUTE_STATS, DD_TRACE_FEATURES=discovery, client_drop_p0s=False

Every class asserts against the same span twice when it runs in both scenarios: once as Datadog MsgPack meta, once as OTLP attributes. That is deliberate, it is how the two-representation rule gets checked without duplicating the test bodies.

Test_OtelSemantics_Spans_Http_Server (20): OTEL_SEMANTICS, OTEL_SEMANTICS_OTLP

Test	Request	What it asserts
test_otel_attributes_present	GET /	http.request.method, url.path, url.scheme, status code all present with the right values
test_legacy_attributes_absent	GET / + X-Forwarded-For	none of http.method, http.url, http.status_code, http.useragent, http.client_ip, http.hostname, network.client.ip, http.query.string survive, in meta and metrics
test_span_name_with_route	GET /sample_rate_route/1	span name is {method} {http.route}, derived from the attribute rather than hardcoded
test_span_name_without_route	GET /no_such_route_xyz	name is {method} {route} conditioned on whatever route the span itself reports, so a framework catch-all still passes, and the URI path never appears in the name
test_span_name_unknown_method	PROPFIND /	method normalizes to _OTHER, http.request.method_original keeps PROPFIND, and the raw verb never appears in the span name
test_span_name_route_invariance	/sample_rate_route/1 vs /99999	two different path params produce the same span name, the low-cardinality guarantee
test_status_500_no_exception_is_status_code_error	GET /status?code=500	5xx alone makes a server span an error, with error.type set to the status as a string
test_status_400_is_not_error	GET /status?code=400	4xx does not make a server span an error, this is where server and client rules differ
test_status_3xx_is_not_error	GET /status?code=302	3xx is not an error
test_url_query_present_with_query_string	GET /?otel=visible	url.query present, without the leading ?
test_url_query_absent_without_query_string	GET /	url.query absent rather than empty-string
test_url_query_obfuscation	GET /?token=<sensitive>	the configured obfuscation regexp applies to url.query, value replaced by redacted
test_user_agent	GET /	user_agent.original carries the request UA
test_client_address	GET / + X-Forwarded-For	client.address resolves from the forwarded header, requires DD_TRACE_CLIENT_IP_ENABLED
test_network_peer_address	GET /	network.peer.address is the socket peer, distinct from client.address
test_server_address	GET /	server.address is the host the request was sent to
test_server_port	GET /	server.port is 7777, int on OTLP, string in meta, required once server.address is set
test_http_endpoint_retained	GET /no_such_route_xyz	the Datadog-only http.endpoint fallback is not stripped by the flag
test_span_kind_is_server	GET /	span.kind=server / OTLP SERVER
test_http_route_retained	GET /sample_rate_route/1	http.route is present and is a template, not the raw path with the parameter baked in

Test_OtelSemantics_Spans_Http_Client (11): OTEL_SEMANTICS, OTEL_SEMANTICS_OTLP

All of these drive the weblog's outbound /make_distant_call endpoint back at itself.

Test	Request	What it asserts
test_otel_attributes_present	outbound GET	http.request.method, url.full, server.address, server.port, status code
test_legacy_attributes_absent	outbound GET	http.method, http.url, http.status_code, out.host, peer.hostname, network.destination.name all gone, meta and metrics
test_url_path_absent_on_client	outbound to a routed path	url.full present and url.path / url.scheme / url.query absent, the client table is url.full only
test_span_name_is_method	outbound GET	name is {method} (no tracer emits url.template), and specifically not HTTP GET, which is what upstream otelhttp produces
test_span_name_unknown_method	outbound PROPFIND	_OTHER + method_original, raw verb absent from the name
test_span_kind_is_client	outbound GET	span.kind=client / OTLP CLIENT
test_status_400_is_error	outbound /status?code=400	4xx is an error on a client span, the deliberate divergence from the Datadog default
test_status_500_is_error	outbound /status?code=500	5xx is an error
test_status_3xx_not_error	outbound /status?code=302	3xx is not
test_url_full_credential_redaction	outbound with user:pass@	userinfo never reaches url.full; if the HTTP API kept it, both halves read REDACTED
test_url_full_query_obfuscation	outbound ?token=<sensitive>	the obfuscation regexp applies inside url.full, not just to url.query

Test_OtelSemantics_OTLP_Server (5): OTEL_SEMANTICS_OTLP only

Split from the client class on purpose: a tracer converts one span kind at a time, and a combined test reports half-done work as not-started.

Test	What it asserts
test_status_code_is_integer_type	http.response.status_code is an OTLP intValue, explicitly not a doubleValue
test_server_port_is_integer_type	server.port is an intValue
test_http_endpoint_retained_otlp	http.endpoint survives the OTLP export, an OTLP-side filter that strips Datadog-only keys must not take it
test_otel_attributes_present_otlp	required server keys present on the OTLP span
test_legacy_attributes_absent_otlp	legacy server keys absent on the OTLP span

Test_OtelSemantics_OTLP_Client (4): OTEL_SEMANTICS_OTLP only

Same four checks as above, on the client span: two integer types, required keys present, legacy keys absent.

Test_OtelSemantics_OTLP_Spans_Http_ErrorStatusConfiguration (2): OTEL_SEMANTICS_OTLP_CUSTOM_ERROR_STATUSES

Server spans only. DD_TRACE_HTTP_CLIENT_ERROR_STATUSES exists in Java, .NET, Go, Ruby and PHP but not in Python or Node.js, so there is nothing to configure on the client side in the two tracers under test.

Test	Request	What it asserts
test_server_error_statuses_config_overrides	GET / (200)	with the range configured to 200, a 200 is an error and error.type is "200"
test_server_error_statuses_config_excludes_500	GET /status?code=500	a 500 is outside the configured range, so it is not an error. This is the half that catches a tracer that unions the configured range with a hardcoded 5xx rule instead of replacing it

Test_OtelSemantics_Stats_Consistency (2): OTEL_SEMANTICS_STATS

Three requests each, then waits for a client-side stats payload.

Test	What it asserts
test_stats_agree_with_span_error	3x /status?code=500: the span is an error, a stats bucket exists keyed on the span name with HTTPStatusCode=500, and Hits == Errors
test_stats_agree_with_span_success	3x /status?code=200: the span is not an error and the matching bucket records zero errors

Why this scenario exists: an export-time rename keeps in-process consumers working but lets the span and the stats be computed from two different rules. It also catches the second failure mode, that http.status_code is the key the agent's aggregation reads and the flag suppresses it, so HTTPStatusCode silently becomes 0 unless the tracer routes the numeric status to stats another way.

Tag changes under test, for review

Every assertion below is gated on the flag. With it off, nothing in this suite runs.

Server spans

Datadog name	asserted OTel name
http.method	http.request.method, _OTHER for an unaccepted verb with http.request.method_original carrying the raw token
http.url	url.path, url.query (obfuscated), url.scheme
http.status_code	http.response.status_code
http.useragent	user_agent.original
http.client_ip	client.address
network.destination.ip	network.peer.address
http.host	server.address, server.port
http.route	http.route, unchanged
http.endpoint	http.endpoint, unchanged, no OTel equivalent

Client spans

Datadog name	asserted OTel name
http.method	http.request.method, same _OTHER rule
http.url	url.full, no url.path on a client span
http.status_code	http.response.status_code
out.host / out.port	server.address / server.port

Each rename is asserted twice: the new name present, and the legacy name absent. The legacy tuples are shared constants so a reviewer can see the whole removal list in one place.

Typing. On OTLP exactly two attributes are ints, server.port and http.response.status_code. That is asserted in the OTLP-only classes. On the Agent/MsgPack path every attribute is a string in meta, so the same tests read from meta there. This two-representation rule is why the int helper exists.

Span naming. {method} {http.route} for server, {method} {url.template} for client, bare {method} with no target. This is the OpenTelemetry Collector's set_semconv_span_name (processor/transformprocessor/internal/traces), agreed with the PM and the OTel team. The collector does no substitution, so an unaccepted method names the span _OTHER /users, not HTTP /users. Worth flagging in review: the HTTP semconv prose says HTTP there, and every implementation that normalizes the attribute correctly (dd-trace-js among them) currently emits HTTP. The collector is the divergent one, and it is what we agreed to follow.

Notable fixes inside the branch

• Client correlation. _server_span correlated the request on user_agent.original only, so for a tracer that has converted its client side but not its server side (dotnet) the lookup missed, fell through to a "first SERVER span in the batch" fallback, and handed every client assertion an unrelated trace id. Reported as 0/11 for a tracer that scores 10/11 on the same span via the Agent path. Fixed by matching whichever user-agent key the tracer emits; correlation is plumbing, not the thing under test. Worth keeping in mind for any shared suite: correlate on something the flag cannot rename.
• http.endpoint tests were asking for a routed path with DD_TRACE_RESOURCE_RENAMING_ENABLED unset, so the attribute could never exist and all four combinations failed for a reason unrelated to the flag.
• Removed test_method_original_on_case_variant. It sent a lowercase verb and expected http.request.method_original to preserve it, but no weblog can ever see a lowercase token: requests uppercases in prepare_method, Node's http server uppercases req.method, WSGI uppercases REQUEST_METHOD. It failed identically on every weblog, and the RFC states no case-sensitivity rule. If a language whose server does preserve the token needs checking later, the test has to come back with a raw-socket client.

[datadog-prod-us1-5]

 

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 11 Pipeline jobs failed

DataDog/system-tests | K8S_LIB_INJECTION_PROFILING_ENABLED: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/cluster-agent:7.81.1, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/apm-inject:latest, 3.231.5, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/dd-lib-java-init:latest]     

DataDog/system-tests | K8S_LIB_INJECTION_PROFILING_ENABLED: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/cluster-agent:7.81.1, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/apm-inject:latest, 3.231.5, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/dd-lib-java-init:latest_snapshot]     

DataDog/system-tests | K8S_LIB_INJECTION_PROFILING_ENABLED: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/cluster-agent:7.81.1, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/apm-inject:latest_snapshot, 3.231.5, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/dd-lib-java-init:latest]     

View all 11 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 90b21fb | Docs | Datadog PR Page | Give us feedback!}