Repository navigation
Expand file tree
/
Copy pathautter_tracing.py
More file actions
996 lines (870 loc) · 44 KB
/
Copy pathautter_tracing.py
File metadata and controls
996 lines (870 loc) · 44 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
"""Always-on error spans for Python services on plain OpenTelemetry.
Head sampling keeps a fraction of traces: ``ParentBased(TraceIdRatioBased(0.1))``
drops 90% of them, and every exception recorded on a dropped trace goes with
it. The Python SDK cannot export a span it did not sample (``BatchSpanProcessor``
skips spans without the sampled flag), so keeping errors takes two pieces:
* :class:`RecordUnsampled` wraps the head sampler. A trace it would drop is
recorded instead (``RECORD_ONLY``): its spans run through the span
processors but keep the "not sampled" flag, so nothing exports them by
default and downstream services see the same sampling decision.
* :class:`RetainErrorTraces` wraps the exporting span processor. Sampled spans
pass straight through. Unsampled spans are held per trace until the trace
shows an error (a span with an ``exception`` event or ERROR status); then
the whole local trace is exported: the spans that already ended (the child
that raised, the database calls before it) and every span of the trace that
ends later (the request span, which ends last). A trace whose local root
ends without an error is discarded. Memory is bounded (256 spans per trace,
5,000 spans and traces in all, 30 s per trace); past that it degrades to
plain head sampling for the evicted traces.
Spans exported by retention carry ``autter.tail_retained = true``, as
``@autter/runtime-node``'s error-linked trace retention marks its spans. The
ingester keeps them out of the span-fed request counts (they are not a sample
of the traffic) and counts each error on them once, as one event.
:func:`init_tracing` wires both, with :class:`RedactingSpanExporter` (secret
and PII scrubbing through ``redact.py``) and the OTLP/HTTP exporter, from the
standard ``OTEL_*`` environment variables::
from autter_tracing import init_tracing
init_tracing() # before the framework instrumentor runs
Request counts (on by default; off with ``init_tracing(count_requests=False)``
or ``AUTTER_COUNT_REQUESTS=false``). Without them the ingester counts the
requests of the traces head sampling kept, so at a 1% ratio request counts
read 1% of the traffic while every error is kept, and error rates read 100
times too high. :class:`CountRequests` sees every HTTP server span (sampled or not:
:class:`RecordUnsampled` records them all) and records each request once, in
a delta ``http.server.duration`` histogram (ms; ``http.route``,
``http.request.method``, ``http.response.status_code`` from the span) that
the ingester counts every request from. It marks the HTTP server spans it
counted ``autter.metric_has_route = true`` and the resource
``autter.metrics_wired`` / ``autter.metrics_routes``, so the ingester (1.8.0+)
adds no request from those spans: each request is counted once, whether it
was sampled, retained for an error, or not exported. The framework
instrumentations' own ``http.server.duration`` / ``http.server.request.duration``
are dropped from that meter provider (they would count every request a
second time). Needs ``opentelemetry-exporter-otlp-proto-http`` for metrics
(without it, counting stays off with a warning). An ingester older than 1.8.0
ignores the marks and counts the sampled requests a second time, from their
spans. :func:`init_tracing` registers its meter provider as the global one:
an app that registers its own afterwards should turn counting off.
Test runs, commit and release. :func:`init_tracing` fills in three resource
attributes the service did not set itself (an explicit value always wins):
* ``deployment.environment.name = test`` when the process runs under a test
runner (pytest, or Django's ``manage.py test``) and the resource has no
environment at all (neither ``deployment.environment.name`` nor
``deployment.environment``). Off with ``AUTTER_DETECT_TEST_ENV=false`` or
``init_tracing(detect_test_env=False)``. A request made through
Starlette's or FastAPI's ``TestClient`` (user agent ``testclient``, host
``testserver``), or through Django's test ``Client`` where the
instrumentation records its host ``testserver`` (old HTTP semantic
conventions), is marked ``autter.test_client = true`` on its server span,
whatever the environment.
* ``deployment.commit``: the running commit, from the first of
:data:`COMMIT_ENV_VARS` holding 7 to 40 hexadecimal characters
(environment variables only: no git, no subprocess).
* ``service.version`` (the release) when it is absent: ``AUTTER_RELEASE``,
else the detected commit, in full. Off with ``AUTTER_DETECT_RELEASE=false``
or ``init_tracing(detect_release=False)``.
Copy this file and ``redact.py`` into your project. Needs ``opentelemetry-sdk``
(tested with 1.20, 1.27 and 1.41 on Python 3.9, 1.45 on Python 3.14) and, for
:func:`init_tracing`'s default exporter, ``opentelemetry-exporter-otlp-proto-http``.
"""
from __future__ import annotations
import logging
import os
import re
import sys
import threading
import time
import weakref
from collections import OrderedDict
from typing import Any, Optional, Sequence
from opentelemetry import metrics, trace
from opentelemetry.sdk.metrics import Histogram, MeterProvider
from opentelemetry.sdk.metrics.export import AggregationTemporality, MetricExporter, PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import DropAggregation, View
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Event, ReadableSpan, SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult
from opentelemetry.sdk.trace.sampling import (
ALWAYS_OFF,
ALWAYS_ON,
Decision,
ParentBased,
Sampler,
SamplingResult,
TraceIdRatioBased,
)
from opentelemetry.trace import Link, SpanContext, SpanKind, StatusCode, TraceFlags
from opentelemetry.trace.status import Status
try: # imported as part of a package
from .redact import (
CLIENT_IP_HASH_ATTRIBUTE,
CLIENT_IP_SALT_ENV,
CLIENT_IP_SALT_MIN_LENGTH,
DROP_ATTRIBUTES_ENV,
HASH_CLIENT_IP_ENV,
client_ip_of,
hash_client_ip,
usable_client_ip_salt,
KEEP_IDENTIFYING_ATTRIBUTES_ENV,
drop_attributes_from_env,
drop_identifying_attributes,
identifying_keys_to_drop,
keep_identifying_from_env,
redact_attributes,
redact_text,
)
except ImportError: # copied next to the app as a plain module
from redact import (
CLIENT_IP_HASH_ATTRIBUTE,
CLIENT_IP_SALT_ENV,
CLIENT_IP_SALT_MIN_LENGTH,
DROP_ATTRIBUTES_ENV,
HASH_CLIENT_IP_ENV,
client_ip_of,
hash_client_ip,
usable_client_ip_salt,
KEEP_IDENTIFYING_ATTRIBUTES_ENV,
drop_attributes_from_env,
drop_identifying_attributes,
identifying_keys_to_drop,
keep_identifying_from_env,
redact_attributes,
redact_text,
)
__all__ = [
"TAIL_RETAINED_ATTRIBUTE",
"METRIC_COUNTED_ATTRIBUTE",
"TEST_CLIENT_ATTRIBUTE",
"COMMIT_ENV_VARS",
"REQUEST_COUNT_RESOURCE_ATTRIBUTES",
"detect_commit",
"detect_test_runner",
"CountRequests",
"request_meter_provider",
"RecordUnsampled",
"RetainErrorTraces",
"RedactingSpanExporter",
"head_sampler_from_env",
"init_tracing",
]
LOGGER = logging.getLogger("autter.runtime")
#: Marks spans exported by error retention rather than head sampling. Same
#: key and meaning as @autter/runtime-node's tail retention.
TAIL_RETAINED_ATTRIBUTE = "autter.tail_retained"
#: On an HTTP server span whose request :class:`CountRequests` counted in the
#: delta histogram: the ingester counts that request from the metric only.
#: Same key as @autter/runtime-node's (whose metric carries the route).
METRIC_COUNTED_ATTRIBUTE = "autter.metric_has_route"
#: Resource attributes of a service whose delta HTTP duration histogram counts
#: its requests; the ingester honours :data:`METRIC_COUNTED_ATTRIBUTE` only
#: with both. Never set them without that histogram: marked requests would
#: then be counted by nothing.
REQUEST_COUNT_RESOURCE_ATTRIBUTES = {"autter.metrics_wired": True, "autter.metrics_routes": True}
#: The histogram :class:`CountRequests` records into, exported (renamed by a
#: view) as ``http.server.duration``, the instrument the ingester counts.
_REQUEST_INSTRUMENT = "autter.http.server.duration"
_HTTP_SERVER_DURATIONS = ("http.server.duration", "http.server.request.duration")
#: On a server span of a request made by a test client: Starlette's and
#: FastAPI's ``TestClient`` (user agent ``testclient``, host ``testserver``)
#: or Django's test ``Client`` (host ``testserver``). Read from the raw
#: attributes, before the user agent is dropped. A hint for people reading
#: the trace, nothing more: a caller can send any user agent or host, so
#: nothing may act on it (alerting, paging, the environment). The
#: environment comes from the process alone (:func:`detect_test_runner`).
TEST_CLIENT_ATTRIBUTE = "autter.test_client"
_TEST_CLIENT_USER_AGENT = "testclient"
_TEST_CLIENT_HOST = "testserver"
_USER_AGENT_KEYS = ("http.user_agent", "user_agent.original")
_HOST_KEYS = ("server.address", "net.host.name", "http.host", "http.server_name")
_ENVIRONMENT_KEYS = ("deployment.environment.name", "deployment.environment")
_TEST_ENVIRONMENT = "test"
DETECT_TEST_ENV_ENV = "AUTTER_DETECT_TEST_ENV"
#: Environment variables that carry the running commit, in the order they are
#: tried (the order @autter/runtime-node and the source-map CLI use). The first one holding 7 to 40
#: hexadecimal characters is the commit; other values are ignored.
COMMIT_ENV_VARS = (
"AUTTER_COMMIT",
"GITHUB_SHA",
"VERCEL_GIT_COMMIT_SHA",
"CF_PAGES_COMMIT_SHA",
"GIT_SHA",
"GIT_COMMIT",
"COMMIT_SHA",
"SOURCE_COMMIT",
"RENDER_GIT_COMMIT",
"RAILWAY_GIT_COMMIT_SHA",
"HEROKU_SLUG_COMMIT",
"SOURCE_VERSION",
)
_COMMIT_RE = re.compile(r"[0-9a-fA-F]{7,40}")
_COMMIT_ATTRIBUTE = "deployment.commit"
_RELEASE_ATTRIBUTE = "service.version"
RELEASE_ENV = "AUTTER_RELEASE"
DETECT_RELEASE_ENV = "AUTTER_DETECT_RELEASE"
MAX_SPANS_PER_TRACE = 256
MAX_BUFFERED_SPANS = 5_000
TRACE_TTL_SECONDS = 30.0
SWEEP_INTERVAL_SECONDS = 10.0
class RecordUnsampled(Sampler):
"""Records what ``delegate`` would drop: DROP becomes RECORD_ONLY.
RECORD_ONLY spans carry no sampled flag, so they reach span processors but
no exporter, and propagate "not sampled" downstream. Unlike the SDK's own
``AlwaysRecordSampler`` (1.45+), the span keeps the attributes it was
started with: a dropping sampler hands back none.
"""
def __init__(self, delegate: Sampler):
if delegate is None:
raise ValueError("delegate sampler is required")
self._delegate = delegate
def should_sample(
self,
parent_context,
trace_id,
name,
kind=None,
attributes=None,
links=None,
trace_state=None,
) -> SamplingResult:
# The SDK calls samplers with six arguments; pass trace_state on only
# when given, so delegates written against that call keep working.
if trace_state is None:
result = self._delegate.should_sample(parent_context, trace_id, name, kind, attributes, links)
else:
result = self._delegate.should_sample(
parent_context, trace_id, name, kind, attributes, links, trace_state
)
if result.decision is Decision.DROP:
return SamplingResult(Decision.RECORD_ONLY, attributes, result.trace_state)
return result
def get_description(self) -> str:
return f"RecordUnsampled{{{self._delegate.get_description()}}}"
def _is_error(span: ReadableSpan) -> bool:
if span.status is not None and span.status.status_code is StatusCode.ERROR:
return True
return any(event.name == "exception" for event in span.events or ())
def _is_local_root(span: ReadableSpan) -> bool:
# The first span of this process in the trace: no parent, or a parent
# that arrived from another service.
return span.parent is None or span.parent.is_remote
class _CopiedSpan(ReadableSpan):
"""A finished span with some fields replaced (spans are immutable once
ended); keeps the original's dropped-item counts for the exporter."""
def __init__(self, span: ReadableSpan, **overrides: Any):
super().__init__(
name=overrides.get("name", span.name),
context=overrides.get("context", span.context),
parent=span.parent,
resource=overrides.get("resource", span.resource),
attributes=overrides.get("attributes", span.attributes),
events=overrides.get("events", span.events),
links=overrides.get("links", span.links),
kind=span.kind,
# The deprecated property warns; the OTLP exporter reads the scope.
instrumentation_info=getattr(span, "_instrumentation_info", None),
status=overrides.get("status", span.status),
start_time=span.start_time,
end_time=span.end_time,
instrumentation_scope=span.instrumentation_scope,
)
self._dropped_counts = (
getattr(span, "dropped_attributes", 0),
getattr(span, "dropped_events", 0),
getattr(span, "dropped_links", 0),
)
@property
def dropped_attributes(self) -> int:
return self._dropped_counts[0]
@property
def dropped_events(self) -> int:
return self._dropped_counts[1]
@property
def dropped_links(self) -> int:
return self._dropped_counts[2]
def _retained(span: ReadableSpan) -> ReadableSpan:
"""The span with the sampled flag (what the export processor checks, and
the truth: retention sampled it) and the tail-retention marker."""
ctx = span.context
sampled = SpanContext(
ctx.trace_id,
ctx.span_id,
is_remote=ctx.is_remote,
trace_flags=TraceFlags(ctx.trace_flags | TraceFlags.SAMPLED),
trace_state=ctx.trace_state,
)
attributes = dict(span.attributes or {})
attributes[TAIL_RETAINED_ATTRIBUTE] = True
return _CopiedSpan(span, context=sampled, attributes=attributes)
class _TraceBuffer:
__slots__ = ("spans", "retained", "first_seen")
def __init__(self, now: float):
self.spans: list = []
self.retained = False
self.first_seen = now
class RetainErrorTraces(SpanProcessor):
"""Exports unsampled traces that error, through ``processor``.
Use with :class:`RecordUnsampled`, wrapping the processor that exports
(``BatchSpanProcessor``): sampled spans go to it unchanged; an unsampled
trace goes to it, whole, the moment one of its spans ends with an
``exception`` event or ERROR status, each span marked
``autter.tail_retained``. Errors recorded after the trace's local root
ended start a new buffer for the stragglers, so those still export.
"""
def __init__(
self,
processor: SpanProcessor,
*,
max_spans_per_trace: int = MAX_SPANS_PER_TRACE,
max_buffered_spans: int = MAX_BUFFERED_SPANS,
trace_ttl_seconds: float = TRACE_TTL_SECONDS,
):
if max_spans_per_trace < 1 or max_buffered_spans < 1 or trace_ttl_seconds <= 0:
raise ValueError("invalid error retention limits")
self._processor = processor
self._max_spans_per_trace = max_spans_per_trace
self._max_buffered = max_buffered_spans
self._ttl = trace_ttl_seconds
self._at_fork_reinit()
if hasattr(os, "register_at_fork"):
# A forked worker (gunicorn, Celery prefork) gets a fresh lock and
# buffer; the BatchSpanProcessor restarts its own thread.
reinit = weakref.WeakMethod(self._at_fork_reinit)
def after_in_child() -> None:
method = reinit()
if method is not None:
method()
os.register_at_fork(after_in_child=after_in_child)
def _at_fork_reinit(self) -> None:
self._lock = threading.Lock()
self._traces: "OrderedDict[int, _TraceBuffer]" = OrderedDict()
self._buffered = 0
self._next_sweep = time.monotonic() + SWEEP_INTERVAL_SECONDS
def on_start(self, span, parent_context=None) -> None:
self._processor.on_start(span, parent_context=parent_context)
def _on_ending(self, span) -> None: # SDK 1.39+ hook
on_ending = getattr(self._processor, "_on_ending", None)
if on_ending is not None:
on_ending(span)
def on_end(self, span: ReadableSpan) -> None:
ctx = span.context
if ctx is None:
return
if ctx.trace_flags.sampled:
# Head-sampled: exported as usual, never marked.
self._processor.on_end(span)
return
forward: list = []
now = time.monotonic()
with self._lock:
if now >= self._next_sweep:
self._sweep(now)
entry = self._entry(ctx.trace_id, now)
if entry.retained:
forward.append(span)
elif _is_error(span):
entry.retained = True
forward.extend(entry.spans)
forward.append(span)
self._buffered -= len(entry.spans)
entry.spans = []
elif len(entry.spans) < self._max_spans_per_trace:
entry.spans.append(span)
self._buffered += 1
self._evict_over_budget()
# The local root ended healthy: the request is over and an error
# in it would have shown by now. A retained entry stays until the
# sweep, so late spans of the trace still export.
if not entry.retained and _is_local_root(span):
self._drop(ctx.trace_id)
for item in forward:
self._processor.on_end(_retained(item))
def force_flush(self, timeout_millis: int = 30_000) -> bool:
return self._processor.force_flush(timeout_millis)
def shutdown(self) -> None:
with self._lock:
self._traces.clear()
self._buffered = 0
self._processor.shutdown()
def _entry(self, trace_id: int, now: float) -> _TraceBuffer:
entry = self._traces.get(trace_id)
if entry is None:
if len(self._traces) >= self._max_buffered:
self._drop(next(iter(self._traces)))
entry = _TraceBuffer(now)
self._traces[trace_id] = entry
return entry
def _drop(self, trace_id: int) -> None:
entry = self._traces.pop(trace_id, None)
if entry is not None:
self._buffered -= len(entry.spans)
def _evict_over_budget(self) -> None:
if self._buffered <= self._max_buffered:
return
# Oldest traces first, whole traces at a time so what stays is
# coherent. Entries without buffered spans keep their rescue state.
for trace_id in [tid for tid, entry in self._traces.items() if entry.spans]:
self._drop(trace_id)
if self._buffered <= self._max_buffered:
return
def _sweep(self, now: float) -> None:
self._next_sweep = now + SWEEP_INTERVAL_SECONDS
cutoff = now - self._ttl
for trace_id in [tid for tid, entry in self._traces.items() if entry.first_seen <= cutoff]:
self._drop(trace_id)
def _http_method(span: ReadableSpan) -> Optional[str]:
attributes = span.attributes or {}
method = attributes.get("http.request.method") or attributes.get("http.method")
return str(method) if method else None
def _counted_request(span: ReadableSpan) -> bool:
"""An HTTP server span: what the HTTP instrumentations open per request."""
return span.kind is SpanKind.SERVER and _http_method(span) is not None
def _request_point(span: ReadableSpan) -> dict:
attributes = span.attributes or {}
point = {"http.request.method": _http_method(span)}
route = attributes.get("http.route")
if isinstance(route, str) and route:
point["http.route"] = route
status = attributes.get("http.response.status_code", attributes.get("http.status_code"))
try:
if status is not None:
point["http.response.status_code"] = int(status)
except (TypeError, ValueError):
pass
return point
class CountRequests(SpanProcessor):
"""Counts every HTTP request once, sampled or not, in a delta histogram.
Wraps the exporting processor (``RetainErrorTraces(...)``) under a
provider whose sampler is :class:`RecordUnsampled`, so it sees every span.
For each HTTP server span (kind SERVER with ``http.request.method`` or
``http.method``) that is its process's local root it records one point in
``meter_provider``'s request histogram (see :func:`request_meter_provider`),
and it passes every HTTP server span on marked
:data:`METRIC_COUNTED_ATTRIBUTE`, nested ones included. The ingester then
counts each such request from the metric and none from its spans: one
count per request, with a 5xx counted failed by the metric, and an error
recorded in a request that answered below 500 counted once from its span.
The resource must carry :data:`REQUEST_COUNT_RESOURCE_ATTRIBUTES`.
``force_flush`` and ``shutdown`` reach the meter provider too.
"""
def __init__(self, processor: SpanProcessor, meter_provider: MeterProvider):
self._processor = processor
self._meter_provider = meter_provider
self._histogram = meter_provider.get_meter("autter.runtime").create_histogram(
_REQUEST_INSTRUMENT,
unit="ms",
description="Duration of HTTP server requests, one point per request",
)
def on_start(self, span, parent_context=None) -> None:
self._processor.on_start(span, parent_context=parent_context)
def _on_ending(self, span) -> None: # SDK 1.39+ hook
on_ending = getattr(self._processor, "_on_ending", None)
if on_ending is not None:
on_ending(span)
def on_end(self, span: ReadableSpan) -> None:
if _counted_request(span) and span.start_time is not None and span.end_time is not None:
if _is_local_root(span):
try:
self._histogram.record(max(span.end_time - span.start_time, 0) / 1e6, _request_point(span))
except Exception: # never fail the request over a metric
LOGGER.debug("[autter-runtime] request count not recorded", exc_info=True)
attributes = dict(span.attributes or {})
attributes[METRIC_COUNTED_ATTRIBUTE] = True
span = _CopiedSpan(span, attributes=attributes)
self._processor.on_end(span)
def force_flush(self, timeout_millis: int = 30_000) -> bool:
spans = self._processor.force_flush(timeout_millis)
return self._meter_provider.force_flush(timeout_millis) and spans
def shutdown(self) -> None:
self._processor.shutdown()
self._meter_provider.shutdown()
def _prefers_delta_histograms(exporter: MetricExporter) -> bool:
preferred = getattr(exporter, "_preferred_temporality", None) or {}
return preferred.get(Histogram) is AggregationTemporality.DELTA
def request_meter_provider(
resource: Resource, exporter: Optional[MetricExporter] = None
) -> Optional[MeterProvider]:
"""The meter provider :class:`CountRequests` records into: histograms
exported with delta temporality (the ingester sums deltas; it ignores
cumulative points), :class:`CountRequests`' histogram exported as
``http.server.duration``, and the framework instrumentations'
``http.server.duration`` / ``http.server.request.duration`` dropped. Other
instruments export as usual.
``exporter`` defaults to ``OTLPMetricExporter`` (``OTEL_EXPORTER_OTLP_*``)
preferring delta histograms; without that package installed, or with an
exporter that does not prefer delta for histograms, the result is
``None`` (with a warning): counting then stays off.
The export interval is ``OTEL_METRIC_EXPORT_INTERVAL`` (default 60 s; the
ingester's latency histograms need two minutes or less).
"""
if exporter is None:
try:
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
except ImportError:
LOGGER.warning(
"[autter-runtime] request counts are off: install opentelemetry-exporter-otlp-proto-http "
"to send them, or set AUTTER_COUNT_REQUESTS=false to silence this."
)
return None
exporter = OTLPMetricExporter(preferred_temporality={Histogram: AggregationTemporality.DELTA})
if not _prefers_delta_histograms(exporter):
LOGGER.warning(
"[autter-runtime] request counts are off: the metric exporter does not export histograms "
"with delta temporality, and the ingester does not count cumulative ones."
)
return None
return MeterProvider(
resource=resource,
metric_readers=[PeriodicExportingMetricReader(exporter)],
views=[
View(instrument_name=_REQUEST_INSTRUMENT, name="http.server.duration"),
*(View(instrument_name=name, aggregation=DropAggregation()) for name in _HTTP_SERVER_DURATIONS),
],
# CountRequests.shutdown (run by the TracerProvider's) shuts it down.
shutdown_on_exit=False,
)
def _env_flag(name: str, default: bool = False) -> bool:
raw = (os.environ.get(name) or "").strip().lower()
if raw in ("1", "true", "yes", "on"):
return True
if raw in ("0", "false", "no", "off"):
return False
if raw:
LOGGER.warning("[autter-runtime] invalid %s %r, using %s", name, raw, "true" if default else "false")
return default
def _from_test_client(span: ReadableSpan) -> bool:
"""A server span whose raw user agent is exactly ``testclient`` or whose
host is exactly ``testserver``: a request a test client made."""
if span.kind is not SpanKind.SERVER:
return False
attributes = span.attributes or {}
return any(attributes.get(key) == _TEST_CLIENT_USER_AGENT for key in _USER_AGENT_KEYS) or any(
attributes.get(key) == _TEST_CLIENT_HOST for key in _HOST_KEYS
)
def detect_test_runner(
environ: Optional[Any] = None, modules: Optional[Any] = None, argv: Optional[Sequence[str]] = None
) -> Optional[str]:
"""The test runner this process runs under, or ``None``: ``"pytest"``
(``PYTEST_VERSION`` or ``PYTEST_CURRENT_TEST`` set, or ``pytest``
imported) or ``"manage.py test"`` (Django's test command). Defaults to
``os.environ``, ``sys.modules`` and ``sys.argv``."""
environ = os.environ if environ is None else environ
modules = sys.modules if modules is None else modules
argv = sys.argv if argv is None else argv
if environ.get("PYTEST_VERSION") or environ.get("PYTEST_CURRENT_TEST") or "pytest" in modules:
return "pytest"
if len(argv) >= 2 and os.path.basename(str(argv[0])) == "manage.py" and argv[1] == "test":
return "manage.py test"
return None
def detect_commit(environ: Optional[Any] = None) -> Optional[str]:
"""The running commit (lower case): the first of :data:`COMMIT_ENV_VARS`
whose value is 7 to 40 hexadecimal characters, or ``None``. Reads
environment variables only (default ``os.environ``): never git."""
environ = os.environ if environ is None else environ
for name in COMMIT_ENV_VARS:
value = (environ.get(name) or "").strip()
if _COMMIT_RE.fullmatch(value):
return value.lower()
return None
def _has_value(attributes: Any, key: str) -> bool:
value = attributes.get(key)
return value is not None and value != ""
def _with_deployment(resource: Resource, detect_test_env: Optional[bool], detect_release: Optional[bool]) -> Resource:
"""``resource`` with the test environment, the running commit and the
release added where it has none (an explicit value always wins)."""
attributes = resource.attributes
added: dict = {}
if not any(_has_value(attributes, key) for key in _ENVIRONMENT_KEYS):
if detect_test_env if detect_test_env is not None else _env_flag(DETECT_TEST_ENV_ENV, True):
runner = detect_test_runner()
if runner is not None:
added["deployment.environment.name"] = _TEST_ENVIRONMENT
LOGGER.info(
"[autter-runtime] test run detected (%s): reporting deployment.environment.name=test. Set "
"deployment.environment.name, or AUTTER_DETECT_TEST_ENV=false, to report another environment.",
runner,
)
if _has_value(attributes, _COMMIT_ATTRIBUTE):
# Set explicitly (OTEL_RESOURCE_ATTRIBUTES): kept as it is, and the
# release defaults to it (when it is a commit), never to a variable
# that may disagree.
explicit = str(attributes[_COMMIT_ATTRIBUTE]).strip()
commit = explicit.lower() if _COMMIT_RE.fullmatch(explicit) else None
else:
commit = detect_commit()
if commit is not None:
added[_COMMIT_ATTRIBUTE] = commit
if not _has_value(attributes, _RELEASE_ATTRIBUTE):
if detect_release if detect_release is not None else _env_flag(DETECT_RELEASE_ENV, True):
release = (os.environ.get(RELEASE_ENV) or "").strip() or commit
if release:
added[_RELEASE_ATTRIBUTE] = release
if not added:
return resource
return Resource({**attributes, **added}, getattr(resource, "schema_url", None))
def _keys_to_drop(keep: Any, drop: Optional[Sequence[str]]) -> frozenset:
"""``keep`` / ``drop`` as given, else from the environment."""
if keep is None:
keep = keep_identifying_from_env(os.environ.get(KEEP_IDENTIFYING_ATTRIBUTES_ENV))
if drop is None:
drop = drop_attributes_from_env(os.environ.get(DROP_ATTRIBUTES_ENV))
return identifying_keys_to_drop(keep, drop)
_resource_cache: tuple = (None, None, None)
def _without_identifying_resource(resource: Resource, dropped: frozenset) -> Resource:
"""The resource without the dropped keys; one copy per resource (every
span of a provider shares it), so the exporter's grouping stays cheap."""
global _resource_cache
cached_resource, cached_dropped, cached = _resource_cache
if cached_resource is resource and cached_dropped is dropped:
return cached
attributes = drop_identifying_attributes(resource.attributes, dropped)
result = resource if attributes is resource.attributes else Resource(attributes, getattr(resource, "schema_url", None))
_resource_cache = (resource, dropped, result)
return result
def _client_ip_hasher(enabled: Optional[bool], salt: Optional[str]) -> Optional[Any]:
"""The hasher for ``hash_client_ip`` (default ``AUTTER_HASH_CLIENT_IP``)
with ``salt`` (default ``AUTTER_CLIENT_IP_SALT``), or ``None`` when off.
On without a usable salt it stays off with one warning: never an
unsalted hash. The salt is never logged."""
if enabled is None:
enabled = _env_flag(HASH_CLIENT_IP_ENV)
if not enabled:
return None
usable = usable_client_ip_salt(salt if salt is not None else os.environ.get(CLIENT_IP_SALT_ENV))
if usable is None:
LOGGER.warning(
"[autter-runtime] hash_client_ip is on but no salt of at least %d characters is set "
"(client_ip_salt= or %s): client IP addresses are dropped and not hashed.",
CLIENT_IP_SALT_MIN_LENGTH,
CLIENT_IP_SALT_ENV,
)
return None
return lambda ip: hash_client_ip(usable, ip)
def _redacted(
span: ReadableSpan, redact: bool = True, dropped: Optional[frozenset] = None, hasher: Optional[Any] = None
) -> ReadableSpan:
def clean(attributes: Any) -> Any:
kept = drop_identifying_attributes(attributes, dropped)
return redact_attributes(kept) if redact else kept
attributes = clean(span.attributes)
if _from_test_client(span):
# From the raw user agent and host, before the user agent is dropped.
attributes = {**(attributes or {}), TEST_CLIENT_ATTRIBUTE: True}
if hasher is not None and (span.attributes or {}).get(CLIENT_IP_HASH_ATTRIBUTE) is None:
# From the raw address, before it was dropped; a hash already on the
# span (set upstream) is left alone.
ip = client_ip_of(span.attributes, span.kind == SpanKind.SERVER)
if ip:
attributes = {**(attributes or {}), CLIENT_IP_HASH_ATTRIBUTE: hasher(ip)}
events = [Event(event.name, clean(event.attributes), event.timestamp) for event in span.events]
links = [Link(link.context, clean(link.attributes)) for link in span.links]
status = span.status
if redact and status is not None and status.description:
status = Status(status.status_code, redact_text(status.description))
return _CopiedSpan(
span,
name=redact_text(span.name) if redact else span.name,
attributes=attributes,
events=events,
links=links,
status=status,
resource=_without_identifying_resource(span.resource, _DEFAULT_KEYS if dropped is None else dropped),
)
_DEFAULT_KEYS = identifying_keys_to_drop()
class RedactingSpanExporter(SpanExporter):
"""Scrubs secrets and PII before spans leave the process.
Span names, attributes (sensitive keys masked at any depth), event
attributes (``exception.message`` and ``exception.stacktrace`` included:
``str(exc)`` often carries a connection string or an API key) and status
descriptions go through ``redact.py``, the patterns @autter/runtime-node
uses. The ingester scrubs again on arrival.
Identifying attributes (``redact.IDENTIFYING_ATTRIBUTE_KEYS``: the client's
IP address and port, the raw user agent, ``host.id``, the command line and
its owner) are dropped from span, event and link attributes and from the
resource, also with ``redact=False``. ``keep_identifying_attributes``
(``True`` or a list of keys; default ``AUTTER_KEEP_IDENTIFYING_ATTRIBUTES``)
keeps them; ``drop_attributes`` (default ``AUTTER_DROP_ATTRIBUTES``, a
comma-separated list) drops more keys, such as ``host.name``.
``hash_client_ip`` (default ``AUTTER_HASH_CLIENT_IP``, off) adds
``client.address_hash``, HMAC-SHA256 of the client's normalised address
under ``client_ip_salt`` (default ``AUTTER_CLIENT_IP_SALT``, at least 16
characters; without one nothing is hashed). Same hash as
@autter/runtime-node and the ingester for the same salt.
"""
def __init__(
self,
exporter: SpanExporter,
*,
redact: bool = True,
keep_identifying_attributes: Any = None,
drop_attributes: Optional[Sequence[str]] = None,
hash_client_ip: Optional[bool] = None,
client_ip_salt: Optional[str] = None,
):
self._exporter = exporter
self._redact = redact
self._dropped = _keys_to_drop(keep_identifying_attributes, drop_attributes)
self._hasher = _client_ip_hasher(hash_client_ip, client_ip_salt)
def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
return self._exporter.export([_redacted(span, self._redact, self._dropped, self._hasher) for span in spans])
def shutdown(self) -> None:
self._exporter.shutdown()
def force_flush(self, timeout_millis: int = 30_000) -> bool:
flush = getattr(self._exporter, "force_flush", None)
return flush(timeout_millis) if flush is not None else True
_ENV_SAMPLERS = {
"always_on": lambda ratio: ALWAYS_ON,
"always_off": lambda ratio: ALWAYS_OFF,
"traceidratio": lambda ratio: TraceIdRatioBased(ratio),
"parentbased_always_on": lambda ratio: ParentBased(ALWAYS_ON),
"parentbased_always_off": lambda ratio: ParentBased(ALWAYS_OFF),
"parentbased_traceidratio": lambda ratio: ParentBased(TraceIdRatioBased(ratio)),
}
def head_sampler_from_env() -> Sampler:
"""The head sampler ``OTEL_TRACES_SAMPLER`` / ``OTEL_TRACES_SAMPLER_ARG``
name (the six standard samplers; default ``parentbased_always_on``)."""
name = (os.environ.get("OTEL_TRACES_SAMPLER") or "parentbased_always_on").strip().lower()
if name not in _ENV_SAMPLERS:
LOGGER.warning("[autter-runtime] unknown OTEL_TRACES_SAMPLER %r, using parentbased_always_on", name)
name = "parentbased_always_on"
raw = (os.environ.get("OTEL_TRACES_SAMPLER_ARG") or "").strip()
try:
ratio = float(raw) if raw else 1.0
except ValueError:
LOGGER.warning("[autter-runtime] invalid OTEL_TRACES_SAMPLER_ARG %r, using 1.0", raw)
ratio = 1.0
return _ENV_SAMPLERS[name](min(max(ratio, 0.0), 1.0))
def init_tracing(
*,
sampler: Optional[Sampler] = None,
exporter: Optional[SpanExporter] = None,
resource: Optional[Resource] = None,
redact: bool = True,
set_global: bool = True,
count_requests: Optional[bool] = None,
metric_exporter: Optional[MetricExporter] = None,
keep_identifying_attributes: Any = None,
drop_attributes: Optional[Sequence[str]] = None,
hash_client_ip: Optional[bool] = None,
client_ip_salt: Optional[str] = None,
detect_test_env: Optional[bool] = None,
detect_release: Optional[bool] = None,
) -> TracerProvider:
"""A TracerProvider that head-samples as configured and always exports
errors, registered as the global provider.
* ``sampler``: the head sampler (default: :func:`head_sampler_from_env`),
wrapped in :class:`RecordUnsampled`.
* ``exporter``: default ``OTLPSpanExporter()``, which reads
``OTEL_EXPORTER_OTLP_ENDPOINT`` (or ``…_TRACES_ENDPOINT``) and
``OTEL_EXPORTER_OTLP_HEADERS``.
* ``resource``: default ``Resource.create()``, which reads
``OTEL_SERVICE_NAME`` and ``OTEL_RESOURCE_ATTRIBUTES``. What it does
not set, the three below fill in; its own values always win.
* ``detect_test_env`` (default: on, unless ``AUTTER_DETECT_TEST_ENV=false``;
``False`` turns it off): under a test runner (:func:`detect_test_runner`:
pytest, Django's ``manage.py test``), a resource with neither
``deployment.environment.name`` nor ``deployment.environment`` gets
``deployment.environment.name=test``, logged once at info level.
* ``deployment.commit``: the running commit, :func:`detect_commit` (the
first of :data:`COMMIT_ENV_VARS` holding 7 to 40 hexadecimal
characters; environment variables only), when the resource has none.
* ``detect_release`` (default: on, unless ``AUTTER_DETECT_RELEASE=false``;
``False`` turns it off): a resource without ``service.version`` gets
``AUTTER_RELEASE``, else the commit (the full SHA, which the platform
resolves to the commit and its pull request).
* ``redact``: scrub secrets and PII in :class:`RedactingSpanExporter`.
* ``keep_identifying_attributes`` (default
``AUTTER_KEEP_IDENTIFYING_ATTRIBUTES``: ``true`` or a comma-separated
list): identifying attributes (client IP address and port, raw user
agent, ``host.id``, command line; ``redact.IDENTIFYING_ATTRIBUTE_KEYS``)
are dropped from spans and the resource, also with ``redact=False``;
``True`` keeps them all, a list keeps those keys. The ingester drops
them too unless it is given the same setting.
* ``drop_attributes`` (default ``AUTTER_DROP_ATTRIBUTES``): more keys to
drop the same way, such as ``["host.name"]``.
* ``hash_client_ip`` (default ``AUTTER_HASH_CLIENT_IP``, off) and
``client_ip_salt`` (default ``AUTTER_CLIENT_IP_SALT``): add
``client.address_hash``, a keyed hash of the client's address (see
:class:`RedactingSpanExporter`).
* ``count_requests`` (default: on, unless ``AUTTER_COUNT_REQUESTS=false``;
``False`` turns it off): count every HTTP request, sampled or not
(:class:`CountRequests`), in a delta histogram sent through
``metric_exporter`` (default ``OTLPMetricExporter`` preferring delta
histograms, which reads the same ``OTEL_EXPORTER_OTLP_*`` variables;
see :func:`request_meter_provider`), so error rates are errors over
all requests rather than over the sampled ones. With ``set_global`` its
meter provider is registered as the global one, so add your own
instruments (a memory gauge) to it rather than registering another; an
app that registers its own meter provider after this call should pass
``count_requests=False`` (OpenTelemetry keeps the first one). It stays
off, with a warning, when a meter provider is already registered (its
own HTTP duration histograms, if delta, would count every request a
second time) or ``opentelemetry-exporter-otlp-proto-http`` is not
installed. Needs an ingester from 1.8.0: an older one counts the
sampled requests twice.
Call it once per process before the framework instrumentor runs (after
fork in prefork servers: gunicorn without ``--preload`` imports the WSGI
module in each worker; Celery's ``worker_process_init``).
With ``set_global``, a process that already registered an SDK provider
(a second call, or the app started under ``opentelemetry-instrument``)
keeps it: it is returned unchanged, with a warning when it was not ours.
"""
if set_global:
existing = trace.get_tracer_provider()
if isinstance(existing, TracerProvider):
description = existing.sampler.get_description()
samples_all = description == "AlwaysOnSampler" or description.startswith("ParentBased{root:AlwaysOnSampler")
if not isinstance(existing.sampler, RecordUnsampled) and not samples_all:
LOGGER.warning(
"[autter-runtime] init_tracing(): an OpenTelemetry TracerProvider is already registered "
"(opentelemetry-instrument?) and is left as it is. Its sampler (%s) drops the errors of "
"unsampled traces: run it with OTEL_TRACES_SAMPLER=parentbased_always_on, or start the "
"app without opentelemetry-instrument.",
description,
)
return existing
if resource is None:
resource = Resource.create()
# Before the meter provider's resource is derived from it, so request
# counts carry the same environment, commit and release as the spans.
resource = _with_deployment(resource, detect_test_env, detect_release)
dropped = _keys_to_drop(keep_identifying_attributes, drop_attributes)
resource = _without_identifying_resource(resource, dropped)
meter_provider = None
if count_requests if count_requests is not None else _env_flag("AUTTER_COUNT_REQUESTS", True):
if set_global and isinstance(metrics.get_meter_provider(), MeterProvider):
LOGGER.warning(
"[autter-runtime] init_tracing(): a MeterProvider is already registered, so request counts are "
"off and error rates count only sampled requests. Register none before init_tracing() (its meter "
"provider replaces it), or set AUTTER_COUNT_REQUESTS=false to silence this."
)
else:
counted = resource.merge(Resource(REQUEST_COUNT_RESOURCE_ATTRIBUTES))
meter_provider = request_meter_provider(counted, metric_exporter)
if meter_provider is not None:
# Only with the histogram: these tell the ingester that marked
# requests are counted by the metric, not by their spans.
resource = counted
if exporter is None:
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
exporter = OTLPSpanExporter()
exporter = RedactingSpanExporter(
exporter,
redact=redact,
keep_identifying_attributes=keep_identifying_attributes,
drop_attributes=drop_attributes,
hash_client_ip=hash_client_ip,
client_ip_salt=client_ip_salt,
)
provider = TracerProvider(
sampler=RecordUnsampled(sampler if sampler is not None else head_sampler_from_env()),
resource=resource,
)
processor: SpanProcessor = RetainErrorTraces(BatchSpanProcessor(exporter))
if meter_provider is not None:
processor = CountRequests(processor, meter_provider)
provider.add_span_processor(processor)
if set_global:
if meter_provider is not None:
metrics.set_meter_provider(meter_provider)
trace.set_tracer_provider(provider)
return provider