Skip to content

fix(tracing): capture span body exceptions and export SGP status=ERROR#460

Merged
NiteshDhanpal merged 1 commit into
nextfrom
fix/tracing-capture-span-error-status
Jul 13, 2026
Merged

fix(tracing): capture span body exceptions and export SGP status=ERROR#460
NiteshDhanpal merged 1 commit into
nextfrom
fix/tracing-capture-span-error-status

Conversation

@NiteshDhanpal

@NiteshDhanpal NiteshDhanpal commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Spans exported to SGP always show status=SUCCESS, even when the operation they represent failed.

Root cause, in three parts:

  • The SGP span (scale_gp_beta) defaults status="SUCCESS" and only flips to "ERROR" inside its own __exit__ context manager. The agentex processor builds SGP spans via create_span(...) and flushes them directly, so __exit__ never runs.
  • The agentex Span type (generated from the OpenAPI spec) has no status/error field.
  • Trace.span() / AsyncTrace.span() ended spans in a bare finally, so a body exception was never recorded before the span was flushed.

Net effect: a failing operation is indistinguishable from a successful one in SGP.

Fix (SDK-only, no schema/backend change)

Capture → carry → map:

  1. Capture — both context managers now except Exception as exc: set_span_error(span, exc); raise.
  2. Carryspan_error.py stores the failure under the reserved span.data["__error__"] key, following the existing __span_type__/__source__ convention. data is a real field, so it survives model_copy(deep=True) and round-trips to both the SGP and agentex-native stores. (A first-class status/error field would require the OpenAPI → Stainless → backend → migration chain; this achieves the same SGP outcome without it.)
  3. Map_build_sgp_span() sets sgp_span.status = "ERROR" + error/error.type/error.message metadata when an error is present, matching SGP's native __exit__ shape.

Exceptions still propagate unchanged. asyncio.CancelledError / KeyboardInterrupt are intentionally not flagged (control flow, not failures) — broaden the except to BaseException if that changes.

Tests

tests/lib/core/tracing/test_span_error.py:

  • set_span_error/get_span_error helpers (incl. list-shaped data no-op).
  • Sync + async context managers record the error and re-raise; success path stays clean.
  • _build_sgp_span maps error → status=ERROR + metadata; no-error path stays SUCCESS.

ruff check clean; new + existing tracing tests pass (40 passed).

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a bug where all spans exported to SGP always showed status=SUCCESS, even when the operation they represented failed. The root cause was that the agentex processor builds SGP spans via create_span and flushes them directly, bypassing the __exit__ context manager that normally flips status to ERROR.

  • New span_error.py module — introduces set_span_error/get_span_error helpers that store exception info under the reserved span.data["__error__"] key, following the existing __span_type__/__source__ convention. Handles None, dict, and list-shaped data correctly.
  • trace.py — both Trace.span() and AsyncTrace.span() now catch Exception, record it on the span via set_span_error, then re-raise before the finally block flushes the span; asyncio.CancelledError/KeyboardInterrupt are intentionally excluded.
  • sgp_tracing_processor.py_build_sgp_span calls get_span_error after building the span and invokes sgp_span.set_error(error_type, error_message) to set status="ERROR" on the SGP side, matching SGP's native __exit__ shape.

Confidence Score: 5/5

Safe to merge. The change only touches the tracing layer, exceptions always re-propagate unchanged, and the success path is unaffected.

The fix is well-scoped: error capture happens before the span is flushed, the error key survives model_copy(deep=True) and recursive_model_dump, and the SGP status mapping is exercised by both the new unit tests and the existing tracing test suite (40 pass). No mutation of span identity fields, no change to public API surfaces, and no risk to callers that don't catch exceptions themselves.

No files require special attention.

Important Files Changed

Filename Overview
src/agentex/lib/core/tracing/span_error.py New module introducing set_span_error/get_span_error helpers that store exception info in span.data["error"]; correctly handles None/list/dict data shapes and follows the existing reserved-key convention.
src/agentex/lib/core/tracing/trace.py Adds except Exception as exc: set_span_error(span, exc); raise before the existing finally in both Trace.span() and AsyncTrace.span(), ensuring errors are captured before end_span flushes the span; no-op path unchanged.
src/agentex/lib/core/tracing/processors/sgp_tracing_processor.py Reads error from span.data via get_span_error and calls sgp_span.set_error(error_type, error_message) to flip the SGP span status to ERROR; the error key is also forwarded raw in metadata=span.data, resulting in slight redundancy but no incorrect behaviour.
tests/lib/core/tracing/test_span_error.py Good coverage of helpers, sync/async context managers, and SGP span mapping; async success path (no error) is not explicitly tested but the logic is symmetric with the sync success test.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant UC as User Code
    participant CM as Trace.span() / AsyncTrace.span()
    participant SE as span_error.py
    participant ES as end_span()
    participant SGP as _build_sgp_span()

    UC->>CM: enter context manager
    CM->>CM: start_span()
    CM-->>UC: yield span

    alt Exception raised in body
        UC-->>CM: raise Exception
        CM->>SE: set_span_error(span, exc)
        Note over SE: span.data["__error__"] = {type, message}
        CM->>CM: re-raise
        CM->>ES: finally: end_span(span)
        ES->>SGP: on_span_end(span)
        SGP->>SE: get_span_error(span)
        SE-->>SGP: "{type, message}"
        SGP->>SGP: sgp_span.set_error(error_type, error_message)
        Note over SGP: sgp_span.status = ERROR
        SGP->>SGP: sgp_span.flush()
    else Success
        UC-->>CM: normal return
        CM->>ES: finally: end_span(span)
        ES->>SGP: on_span_end(span)
        SGP->>SE: get_span_error(span) returns None
        Note over SGP: sgp_span.status stays SUCCESS
        SGP->>SGP: sgp_span.flush()
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant UC as User Code
    participant CM as Trace.span() / AsyncTrace.span()
    participant SE as span_error.py
    participant ES as end_span()
    participant SGP as _build_sgp_span()

    UC->>CM: enter context manager
    CM->>CM: start_span()
    CM-->>UC: yield span

    alt Exception raised in body
        UC-->>CM: raise Exception
        CM->>SE: set_span_error(span, exc)
        Note over SE: span.data["__error__"] = {type, message}
        CM->>CM: re-raise
        CM->>ES: finally: end_span(span)
        ES->>SGP: on_span_end(span)
        SGP->>SE: get_span_error(span)
        SE-->>SGP: "{type, message}"
        SGP->>SGP: sgp_span.set_error(error_type, error_message)
        Note over SGP: sgp_span.status = ERROR
        SGP->>SGP: sgp_span.flush()
    else Success
        UC-->>CM: normal return
        CM->>ES: finally: end_span(span)
        ES->>SGP: on_span_end(span)
        SGP->>SE: get_span_error(span) returns None
        Note over SGP: sgp_span.status stays SUCCESS
        SGP->>SGP: sgp_span.flush()
    end
Loading

Reviews (3): Last reviewed commit: "fix(tracing): capture span body exceptio..." | Re-trigger Greptile

@smoreinis

Copy link
Copy Markdown
Contributor

can we target next instead of main here and also fix the diff to pass lint?

@NiteshDhanpal NiteshDhanpal force-pushed the fix/tracing-capture-span-error-status branch from a2fa8d5 to 76af59f Compare July 10, 2026 17:00
@NiteshDhanpal NiteshDhanpal changed the base branch from main to next July 10, 2026 17:00
Spans exported to SGP always showed status=SUCCESS even when the operation
they represent failed. The SGP span defaults status to "SUCCESS" and only
flips to "ERROR" inside its own __exit__ context manager, but the agentex
processor builds SGP spans via create_span(...) and flushes them directly,
so __exit__ never runs. The Trace.span()/AsyncTrace.span() context managers
also ended spans in a bare finally, so a body exception was never recorded.

Capture the exception in both context managers and carry it on the span so
the SGP processor can map it:

- add span_error.py with set_span_error/get_span_error, storing the failure
  under the reserved span.data["__error__"] key (the Span model is generated
  from the OpenAPI spec and has no status/error field; data is a real field
  that survives model_copy(deep=True) and round-trips to both stores)
- Trace.span()/AsyncTrace.span(): except -> set_span_error(span, exc); raise
- _build_sgp_span(): when an error is present, set sgp_span.status = "ERROR"
  plus error/error.type/error.message metadata (matching SGP's native
  __exit__ shape)

Exceptions still propagate; asyncio.CancelledError/KeyboardInterrupt are not
flagged (control flow, not failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NiteshDhanpal NiteshDhanpal force-pushed the fix/tracing-capture-span-error-status branch from 76af59f to 3f6fb40 Compare July 12, 2026 16:40
@NiteshDhanpal NiteshDhanpal merged commit 6c23d76 into next Jul 13, 2026
48 checks passed
@NiteshDhanpal NiteshDhanpal deleted the fix/tracing-capture-span-error-status branch July 13, 2026 01:01
@stainless-app stainless-app Bot mentioned this pull request Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants