fix(sdk): count logs dropped by the recursion guard - #5676
Open
RichardoMrMu wants to merge 10 commits into
Open
RichardoMrMu wants to merge 10 commits into
RichardoMrMu wants to merge 10 commits into
Conversation
|
|
Pull request dashboard statusWaiting on the author · refreshed 2026-09-27 05:48 UTC Respond to 1 review item (e.g. link a commit, explain why not, ask a follow-up):
Status above doesn't look right?
|
SimpleLogRecordProcessor.on_emit discards a record when it detects a
recursive emit loop, and says so in the log, but does not record the drop
on otel.sdk.processor.log.processed. The already-shutdown early return nine
lines below it does, with error.type=already_shutdown, so the two discard
paths in the same method disagree about whether a dropped record is worth
counting.
Measured on main, driving a real recursion with an exporter that logs while
exporting:
recursion drop: {'<none>': 4} -- 1 record dropped, 0 counted
shutdown drop: {'already_shutdown': 1} -- 1 record dropped, 1 counted
So the counter under-reports: a consumer summing otel.sdk.processor.log
.processed cannot tell that a record was lost, even though the SDK knew it
was dropping one.
Note that drop_items selects between prebuilt attribute dicts rather than
building them from its argument, so passing a new error_type without adding
the matching dict would silently attribute the drop to error.type=queue_full.
Hence the small change to _processor_metrics.py alongside it.
The recursion guard was added in open-telemetry#4799 (merged 2025-12) and the processor
accounting in open-telemetry#5472 (merged 2026-08); open-telemetry#5472's diff on this file starts after
the try: block, so this path was simply never reached by that work.
Assisted-by: Doubao
RichardoMrMu
force-pushed
the
fix/count-logs-dropped-by-recursion-guard
branch
from
September 20, 2026 09:49
87f5dbe to
f4f222f
Compare
lzchen
approved these changes
Sep 21, 2026
herin049
approved these changes
Sep 21, 2026
herin049
left a comment
Contributor
There was a problem hiding this comment.
Reminder to run pre-commit to fix the failing formatting checks in CI.
| @@ -0,0 +1 @@ | |||
| Count logs dropped by `SimpleLogRecordProcessor`'s recursion guard on `otel.sdk.processor.log.processed` with `error.type=recursion` | |||
Contributor
There was a problem hiding this comment.
Please include the package name (i.e. opentelemetry-sdk) in the changelog fragment.
Author
- add opentelemetry-sdk package name to .changelog/5676.fixed - remove extra blank line in test_export.py - run pre-commit to fix formatting
Assisted-by: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
SimpleLogRecordProcessor.on_emitdiscards a record when it detects a recursive emit loopand logs that it is doing so, but does not record the drop on
otel.sdk.processor.log.processed. The already-shutdown early return nine lines below itdoes record its drop, with
error.type=already_shutdown:Two discard paths in the same method, with the same
self._metricsin hand, disagree aboutwhether a dropped record is worth counting.
Measured on
main, driving a real recursion with an exporter that logs while exporting:{'<none>': 4}{'already_shutdown': 1}A consumer summing
otel.sdk.processor.log.processedtherefore cannot tell that a recordwas lost, even though the SDK knew it was dropping one and already has the vocabulary to
say so.
Why this touches
_processor_metrics.pytoodrop_itemsselects between prebuilt attribute dicts rather than building them from itsargument:
So adding the call alone would silently attribute a recursion drop to
error.type=queue_full, which would be worse than not counting it. The change adds oneprebuilt dict and one branch, mirroring the existing
already_shutdownshape; there is nosignature or public API change, and
NoOpProcessorMetricsis unaffected.History
The recursion guard came from #4799 (merged 2025-12) and the processor accounting from
#5472 (merged 2026-08). #5472's diff on this file begins after the
try:— it movedfinish_itemsand reworked error handling — so the guard above it was never in scope forthat work. This fills that gap rather than revisiting a decision.
Tests
test_metrics_recursive_loopinopentelemetry-sdk/tests/logs/test_export.py, placedbeside
test_simple_log_record_processor_doesnt_enter_recursive_loopwhose gap it fills.It is the intersection of two tests this suite already has: the recursion trigger from that
test, and the counter assertions from
test_metrics_already_shutdown.mainit fails with0 != 1 : the log dropped by the recursion guard was not counted.test_export.pypasses: 25 passed.Type of change
Does This PR Require a Core Repo Change?