Repository navigation
Update Studio when SimpleAudit ships span events and judge-summary budgets (SimpleAudit#144) #80
Description
Reminder: update Studio once SimulaMet/SimpleAudit#144 is merged and released in SimpleAudit.
What that PR changes in the engine
parse_otlp_jsonand the protobuf path keep spaneventsand the status message (status_message), so why a span failed reaches the store and the judge.summarize_for_judgeshortens long attribute values (newmax_chars_per_value) instead of cutting each span's JSON at 2,000 characters from the front, and shows a failed span's status message and exception.
What Studio does today (workarounds that can stay)
- OTLP intake (
model_registry/otlp_views.py::_normalize_span_status): copiesexception.typeandexception.messagefrom the span's exception event onto its attributes before the engine parses the export. - Judge evidence (
audits/agentic/compact.py::judge_view): pre-trims what the main judge sees (onlygen_ai.*and error attributes, repeats as notes, decisions and results first), so fewer spans hit the engine's per-span cut.
To do once a SimpleAudit release includes #144
- Raise the dependency in
pyproject.toml(simpleaudit>=0.5.1,<0.6.0) to that release, and updateuv.lock. - Decide whether
_normalize_span_statuscan stop copying exception attributes now thateventsandstatus_messageare parsed. If it does, keep the timeline's Error block and the judge view reading the error from the new fields. - Persist the new
eventsandstatus_messageonOtlpSpan(or keep folding them into attributes), so stored spans keep them. - Keep
judge_view: it still removes repeats and noise the engine doesn't know about, so the engine's new value shortening is only a safety net. - Run a traced audit with a failing tool call (for example D02, unknown order) and check that the judge prompt shows
status: ERRORandexception: ValueError: ….
Metadata
Metadata
Assignees
Labels
No labels
Activity
SushantGautam commentedon Nov 6, 2026
Shipped in v0.5.48 (106b3fa), on SimpleAudit 0.6.0.
Checklist
simpleaudit>=0.6.0,<0.7.0(v0.5.47),uv.lockon 0.6.0._normalize_span_statusno longer copiesexception.*onto attributes; it only mapsSTATUS_CODE_*names to codes. The timeline's Error block reads the exception event and status message, falling back to attributes for spans stored before this change.OtlpSpanstoresstatus_messageandevents(migrationmodel_registry.0022), and returns them to the engine. At every capture level only the exception type and message are kept; stack traces and other events needfullcapture.judge_viewkept; it passes the new fields through.Verification: traced audit
Local run: agent "Support Refund Assistant" (Acme Order Lookup tool, Open WebUI), Studio OTLP tracing at
structuralcapture, 10 explanatory agentic scenarios including D02 (unknown order ACME-9999), 5 turns. Completed 10/10.In D02 the agent called
acme_get_orderthree times; each failed span was stored with its exception event and no stack trace, and nothing was copied onto attributes. Rebuilding the judge's trace text from the stored evidence spans with the engine'ssummarize_for_judge:(The rendered prompt itself is not stored, so this is rebuilt from the same spans the judge received.) The judge graded D02 pass: the agent used the tool, said the record was missing and invented nothing. The result page's timeline shows
Error: ValueError: No order found for 'ACME-9999'. …under each call.Open WebUI sets no span status message, so for this target the exception event is what carries the error.
Tracing: store span status messages and exception events