Skip to content

chore: run Redact, Annotate, Hash and Substitute on a private kernel - #324

Merged
binaryaaron merged 8 commits into
aagonzales/graph-sdk/feat/mainfrom
aagonzales/graph-sdk/chore/kernel-replace-routes
Oct 29, 2026
Merged

binaryaaron merged 8 commits into
aagonzales/graph-sdk/feat/mainfrom
aagonzales/graph-sdk/chore/kernel-replace-routes

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Redact, Annotate, Hash and Substitute now run on a small private kernel, engine/graph_core/, instead of the
graph SDK's replacement dispatch, and the graph SDK code that only those routes used is deleted. Users see no change
from this branch's base: outputs, errors and failed-record accounting are the same on pandas 2 and 3, and the
public anonymizer and anonymizer.graph exports are untouched. Rewrite and detection keep their current paths for
now.

What changes

  • Kernel (src/anonymizer/engine/graph_core/). A domain-free kernel (kernel.py: kind registry, static plan,
    reducer over an immutable state, cancel token), the authoring types (authoring.py) and prepare/protect
    (execution.py). Everything runs on the caller's thread with no event loop of its own. A raising callable closes
    its target as failed, and BaseException (Ctrl-C) propagates unchanged. The package exports no names.
  • Routes (engine/replace/route.py). run_local_route runs Redact, Annotate and Hash with one target per row.
    run_substitute_route runs the LLM map step (LlmReplaceWorkflow.generate_map_only, whose body is 0.4.0's again)
    as one opaque operation, then applies the returned rows through the same route. ReplacementWorkflow.run and
    apply_local_replace_strategy call them, so a caller's replace_runner subclass still works.
  • Facade. For a replace config, _run_internal_impl calls the detection workflow and then the replace runner
    directly. The existing entity_label_examples withholding for 0.4.0-signature detectors is kept.
  • Context sources. Two targets that declare a context source under the same name but with a different provider or
    bounds are rejected at build time (duplicate_context_source), so one target can never read another's context.
  • Null and non-string text cells. The standalone detection call now publishes the text port only when every cell
    is a str, so these cells behave as in 0.4.0.

Deleted

The following files are deleted:

  • engine/replace/_semantic_replace.py
  • engine/replace/_semantic_map_profile.py
  • engine/replace/_graph_redact.py
  • engine/detection/_graph_profile.py
  • engine/detection/_graph_invocation.py

These code paths are deleted from files that stay:

  • run_map in _semantic_map.py;
  • the Redact/Substitute branches of _opaque_redact.py;
  • native_map and builtin_replacement in ndd/_route.py;
  • apply_replacement_map in strategies.py;
  • DelegatedRequestContract.publication_sources, which is not exported;
  • the entity-span ports in harness/frames.py.

Tests that exercised only deleted code or pinned the old dispatch topology are deleted. Tests that asserted on the old
route's internal executions keep their behaviour assertions:

  • Deleted files: test_all_replace_graph.py, test_detection_redact_config.py, test_substitute_semantic_map.py
    and test_publication_source_admission.py, plus individual functions in the route-resolution, detection-control and
    replace-strategy tests.
  • Rewritten: test_injected_redact_graph.py and parts of test_detection_source_frame.py.

The kernel's behavioural checks live in tests/spec_programs/.

Net non-blank, non-comment code lines under src/, tools/ and examples/: +58 (1,228 added, 1,170 removed; the
new kernel package is 869 of the added lines). Most of the Substitute map path stays for now because Rewrite still
reaches it.

Checks

  • make check passes.
  • make test: 5,442 passed. One test, test_wandb_snapshot_rejects_fifo_socket_and_device, fails in the test
    environment with AF_UNIX path too long, and fails the same way on the base.
  • uv run --group dev python -m tests.spec_programs.run_all: 59 pass, 8 retired.
  • Parity against main (f4f2b73d) on pandas 2 and 3, over all cases: no blocking difference beyond the differences the
    base already has.
  • Executed lines per case are at most 3.7% above the base on any replace case. Substitute runs with no detected
    entities execute about a third fewer lines.
  • Replace cost is linear in row count. It is still several times 0.4.0's plain loop: Redact takes about 7.8 s against
    1.3 s at 40,000 rows. The kernel applies state events for every row and operation, which a plain loop does not.

Main merged (5155d6d)

This branch now includes main at 5155d6d. Two tests that main added in #322 fail on this branch: test_join_new_columns_preserves_surviving_target_order and test_run_in_memory_records_tracks_dropped_caller_id_and_survivor_order. This slice keeps a positional Rewrite join and rejects a record that DataDesigner drops without reporting it; the remove-old-sdk PR in this stack takes main's record-id join and drops that check, and both tests pass from there on.

Main merged (f294dc1)

This branch now also includes main at f294dc1 (#328, recovery from incomplete replacements in Rewrite) through its base branch, with #328 ported as described on the base PR.

🤖 Generated with Claude Code

binaryaaron and others added 6 commits October 22, 2026 22:32
…sh on it

Adds the private graph kernel (engine/graph_core: kernel, authoring, execution) and routes the
local replace strategies through one graph invocation per frame. Detection is called directly
by the facade, as in 0.4.0. The spec programs become the kernel's tests. No deletions yet.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…s it replaces

Substitute's replacement map is one LlmReplaceWorkflow call, driven as the single operation of its
own kernel invocation; the rows it returns take the same route as the local strategies.
generate_map_only is the plain 0.4.0 body again (it also reports source positions), so the graph SDK's
direct-map graph, the Redact/Substitute detection invocation and the local-strategy graph freeze are
unreachable and removed, with the tests that only exercised them.

One target never receives another target's context: a second context source under an existing
name is rejected when the protection graph is built.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Removes the delegated publication-source contract (only the Substitute map graph declared it), the
authenticated span publication ports of frames, the apply branch of the map graph, apply_replacement_map
(Substitute applies rows through the kernel route now) and the private kernel re-export module. The
spec programs and the route import the kernel modules directly.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
State.rows was a tuple copied by every kernel event, so Redact and
Substitute's protect pass grew quadratically with the frame. Restore the
private persistent vector so each event copies O(log n) slots, and delete
the unreferenced _apply_replacement_map_to_text.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Rewrite the comments and docstrings this branch adds so each states its
behaviour or reason directly, without citing documents outside the repository
or naming internal process steps. Comment meaning is unchanged.

String literals that carried the same citations also change: the
PreparedProtection serialization TypeError message and the RETIRED detail
strings in the spec programs. run_all.py is reflowed by the formatter.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
binaryaaron and others added 2 commits October 27, 2026 15:32
…kernel-replace-routes

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…kernel-replace-routes

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron
binaryaaron merged commit b68adb8 into aagonzales/graph-sdk/feat/main Oct 29, 2026
8 checks passed
@binaryaaron
binaryaaron deleted the aagonzales/graph-sdk/chore/kernel-replace-routes branch October 29, 2026 21:15
@binaryaaron

Copy link
Copy Markdown
Collaborator Author

Folded into #319: aagonzales/graph-sdk/feat/main was fast-forwarded to 44924e2e, which contains this branch unchanged, so this step (Redact, Annotate, Hash and Substitute on the private kernel) is now reviewed in #319. Nothing was rewritten; this description stays as the record of the step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant