Skip to content

refactor: run detection on the kernel and remove the old graph SDK - #326

Merged
binaryaaron merged 13 commits into
aagonzales/graph-sdk/feat/mainfrom
aagonzales/chore/graph-sdk-remove-old-sdk
Oct 29, 2026
Merged

binaryaaron merged 13 commits into
aagonzales/graph-sdk/feat/mainfrom
aagonzales/chore/graph-sdk-remove-old-sdk

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Detection now runs main's stages on the private kernel from #324, and the old graph SDK is deleted:
engine/graph_sdk, engine/harness, their adapters and their test suite. anonymizer.graph now exports the
kernel's 23 public names. Apart from the model bridge timeout, which again behaves as on main (below), users of
Anonymizer see no change from this branch's base: outputs, errors and failed-record accounting are the same on
pandas 2 and 3, and the root anonymizer exports are unchanged.

This branch stacks on #325 and is rebased onto its base branch when #324 and #325 merge.

What changes

  • Detection (engine/detection/detection_workflow.py). EntityDetectionWorkflow.run starts from main's body and
    runs it as up to three kernel stages: detect, tag latent entities (only when enabled) and finalize. Each
    DataDesigner call is one opaque operation. Row correspondence for inputs with duplicated rows is kept:
    source_positions from the NDD adapter are composed across the stages, so each surviving row maps to its input row.
  • run_stages moves to the public module engine/graph_core/stages.py, used by detection and both Rewrite
    workflows.
  • anonymizer.graph re-exports exactly 23 names from engine/graph_core: Assessment, CancelToken,
    ContextScope, ContextSource, ContextView, Datum, Dependency, GraphRejection, GraphScope,
    InspectionSummary, Limits, Mention, Operation, OutputGroup, PreparedProtection, ProtectionGraph,
    ProtectionResult, Requests, TargetOutcome, Workflow, prepare, protect, register_kind. The previous 228
    names are removed. tests/graph/test_public_api.py pins the list and checks that importing anonymizer.graph loads
    only the kernel.
  • Example and docs. examples/graph_sdk/local_workflows.py and docs/concepts/graph-sdk.md are rewritten against
    the 23 names. docs/development/graph-sdk-cutover.md is left as is and still names deleted modules.
  • Model bridge timeout. In the detection column plugin, a bridged model call that times out is cancelled and
    raises ModelTimeoutError, as on main. Before this change the handler waited for the call to finish with no bound.
  • NDD adapter. The caller-supplied source-frame path, which only the graph SDK used, is removed. The adapter
    always captures its own frame from the input.

Deleted

  • engine/graph_sdk/ (47 modules) and engine/harness/ (4 modules).
  • Detection's graph route: detection/_semantic_detection.py, _opaque_redact.py, _stage_binding.py, artifacts.py,
    run_arguments.py, ndd/_route.py, engine/source_text.py, and the graph wiring in
    workflow_columns/detection/impl.py (now identical to main).
  • tests/graph_sdk/ and the test files main does not have, outside tests/spec_programs/. Test files main has keep
    their current content, minus the individual tests that imported deleted code. The compat pytest marker goes with
    them; pyproject.toml matches main.

Net non-blank, non-comment code lines under src/, tools/ and examples/: −22,407 (49,955 to 27,548). Test code
lines: 69,421 to 24,696.

Known differences from main that this branch keeps

These come from the base branches, not from this change:

  • Library log messages no longer include payload text, paths or exception details.
  • The root package loads its public names lazily.
  • Caller detectors with the older signature are not passed entity_label_examples.
  • The adapter does not apply main's DataDesigner scheduler-wait patch.
  • Chunked validation waits for sibling tasks to finish after one fails.

Checks

  • make check passes.
  • make test: 1,557 passed. test_wandb_snapshot_rejects_fifo_socket_and_device fails in the test environment
    with AF_UNIX path too long, as on the base.
  • uv run --group dev python -m tests.spec_programs.run_all: 59 pass, 8 retired.
  • Parity against main (f4f2b73d) on pandas 2 and 3, over all cases: no blocking difference beyond the
    differences the base already has.
  • Executed lines per case against main: median 1.20×, at most 1.22× on cases whose status matches main.

Main merged (5155d6d)

This branch now includes main at 5155d6d. Rewrite's _join_new_columns now uses main's record-id join: surviving rows keep input order. One difference from main remains: when record ids cannot match each surviving row to one input row, the join raises AnonymizerWorkflowError ("Unassociated workflow result has ambiguous source correspondence") where main's raises a pandas ValueError. The test test_run_in_memory_records_tracks_dropped_caller_id_and_survivor_order, added on main, is adapted on purpose: its mocked DataDesigner now returns the source-key column that this branch's adapter adds to each submitted row, for the rows that survive. Its assertions are unchanged.

Main merged (f294dc1)

This branch now also includes main at f294dc1 (#328, recovery from incomplete replacements in Rewrite) through its base branch.

🤖 Generated with Claude Code

binaryaaron and others added 10 commits October 23, 2026 20:00
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
EntityDetectionWorkflow.run() runs the main-line detection body (detect,
tag latent entities, finalize) through run_stages, each step one opaque
operation. The per-stage source positions the adapter returns are composed
across stages so the result still maps every surviving row to its input row.

run_stages moves to engine/graph_core/stages.py so detection and both
rewrite workflows import it from one public module. The detection graph
lowering, the opaque redact bridge, the stage binding, the route probe and
the caller-signature shim are removed with the graph wiring in the
detection plugin columns; nothing reached them any more. The candidate-span
artifacts move next to the graph SDK modules that import them, so nothing
under engine/detection imports the graph SDK or harness.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Drop the tests that exercised the removed detection graph route (borrowed
leases, canonical frame publication, route resolution, the retained-execution
helper) and the assertions that read a retained execution off a raised error.
Keep the tests that pin public behaviour: row-position composition, failure
merging, injected callbacks, caller detectors with older signatures, and the
error messages of empty and all-failed runs.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Re-export the 23 public names from engine/graph_core, rewrite the local
workflow example and the concept page against them, and reword the two
dogfood comments that named the old SDK modules.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Remove engine/graph_sdk, engine/harness and engine/source_text, which only
the old anonymizer.graph surface imported, along with the tests that
f4f2b73 does not have, the compat marker they used and the one test in a
shared file that imported a removed graph name. Add a test that pins the
public names and the import footprint of anonymizer.graph.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
binaryaaron and others added 3 commits October 27, 2026 15:32
…ore/graph-sdk-remove-old-sdk

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Rewrite's `_join_new_columns` takes main's body (aligns surviving rows on the record id column, in input order; `source_positions` is still accepted and unused). When record ids cannot associate each surviving row with exactly one input row, the join raises AnonymizerWorkflowError as before. The in-memory records test's mocked DataDesigner now echoes the adapter's source-key column for the surviving rows; its assertions are unchanged.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…ore/graph-sdk-remove-old-sdk

Keeps the deletion of the three test files this branch removed.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Base automatically changed from aagonzales/chore/graph-sdk-rewrite-on-kernel to aagonzales/graph-sdk/feat/main October 29, 2026 21:15
@binaryaaron
binaryaaron merged commit e8ea648 into aagonzales/graph-sdk/feat/main Oct 29, 2026
8 checks passed
@binaryaaron
binaryaaron deleted the aagonzales/chore/graph-sdk-remove-old-sdk branch October 29, 2026 21:15
@binaryaaron

Copy link
Copy Markdown
Collaborator Author

Folded into #319: aagonzales/graph-sdk/feat/main was fast-forwarded to 44924e2e, which contains this branch unchanged, so this step (detection on the kernel and removal of the old graph SDK) is now reviewed in #319. Nothing was rewritten; this description stays as the record of the step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant