Repository navigation
chore: run Redact, Annotate, Hash and Substitute on a private kernel - #324
Merged
binaryaaron merged 8 commits intoOct 29, 2026
Conversation
…sh on it Adds the private graph kernel (engine/graph_core: kernel, authoring, execution) and routes the local replace strategies through one graph invocation per frame. Detection is called directly by the facade, as in 0.4.0. The spec programs become the kernel's tests. No deletions yet. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…s it replaces Substitute's replacement map is one LlmReplaceWorkflow call, driven as the single operation of its own kernel invocation; the rows it returns take the same route as the local strategies. generate_map_only is the plain 0.4.0 body again (it also reports source positions), so the graph SDK's direct-map graph, the Redact/Substitute detection invocation and the local-strategy graph freeze are unreachable and removed, with the tests that only exercised them. One target never receives another target's context: a second context source under an existing name is rejected when the protection graph is built. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Removes the delegated publication-source contract (only the Substitute map graph declared it), the authenticated span publication ports of frames, the apply branch of the map graph, apply_replacement_map (Substitute applies rows through the kernel route now) and the private kernel re-export module. The spec programs and the route import the kernel modules directly. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
State.rows was a tuple copied by every kernel event, so Redact and Substitute's protect pass grew quadratically with the frame. Restore the private persistent vector so each event copies O(log n) slots, and delete the unreferenced _apply_replacement_map_to_text. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Rewrite the comments and docstrings this branch adds so each states its behaviour or reason directly, without citing documents outside the repository or naming internal process steps. Comment meaning is unchanged. String literals that carried the same citations also change: the PreparedProtection serialization TypeError message and the RETIRED detail strings in the spec programs. run_all.py is reflowed by the formatter. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
This was referenced Oct 23, 2026
…kernel-replace-routes 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…kernel-replace-routes 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
binaryaaron
deleted the
aagonzales/graph-sdk/chore/kernel-replace-routes
branch
October 29, 2026 21:15
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Redact, Annotate, Hash and Substitute now run on a small private kernel,
engine/graph_core/, instead of thegraph SDK's replacement dispatch, and the graph SDK code that only those routes used is deleted. Users see no change
from this branch's base: outputs, errors and failed-record accounting are the same on pandas 2 and 3, and the
public
anonymizerandanonymizer.graphexports are untouched. Rewrite and detection keep their current paths fornow.
What changes
src/anonymizer/engine/graph_core/). A domain-free kernel (kernel.py: kind registry, static plan,reducer over an immutable state, cancel token), the authoring types (
authoring.py) andprepare/protect(
execution.py). Everything runs on the caller's thread with no event loop of its own. A raising callable closesits target as failed, and
BaseException(Ctrl-C) propagates unchanged. The package exports no names.engine/replace/route.py).run_local_routeruns Redact, Annotate and Hash with one target per row.run_substitute_routeruns the LLM map step (LlmReplaceWorkflow.generate_map_only, whose body is 0.4.0's again)as one opaque operation, then applies the returned rows through the same route.
ReplacementWorkflow.runandapply_local_replace_strategycall them, so a caller'sreplace_runnersubclass still works._run_internal_implcalls the detection workflow and then the replace runnerdirectly. The existing
entity_label_exampleswithholding for 0.4.0-signature detectors is kept.bounds are rejected at build time (
duplicate_context_source), so one target can never read another's context.is a
str, so these cells behave as in 0.4.0.Deleted
The following files are deleted:
engine/replace/_semantic_replace.pyengine/replace/_semantic_map_profile.pyengine/replace/_graph_redact.pyengine/detection/_graph_profile.pyengine/detection/_graph_invocation.pyThese code paths are deleted from files that stay:
run_mapin_semantic_map.py;_opaque_redact.py;native_mapandbuiltin_replacementinndd/_route.py;apply_replacement_mapinstrategies.py;DelegatedRequestContract.publication_sources, which is not exported;harness/frames.py.Tests that exercised only deleted code or pinned the old dispatch topology are deleted. Tests that asserted on the old
route's internal executions keep their behaviour assertions:
test_all_replace_graph.py,test_detection_redact_config.py,test_substitute_semantic_map.pyand
test_publication_source_admission.py, plus individual functions in the route-resolution, detection-control andreplace-strategy tests.
test_injected_redact_graph.pyand parts oftest_detection_source_frame.py.The kernel's behavioural checks live in
tests/spec_programs/.Net non-blank, non-comment code lines under
src/,tools/andexamples/: +58 (1,228 added, 1,170 removed; thenew kernel package is 869 of the added lines). Most of the Substitute map path stays for now because Rewrite still
reaches it.
Checks
make checkpasses.make test: 5,442 passed. One test,test_wandb_snapshot_rejects_fifo_socket_and_device, fails in the testenvironment with
AF_UNIX path too long, and fails the same way on the base.uv run --group dev python -m tests.spec_programs.run_all: 59 pass, 8 retired.f4f2b73d) on pandas 2 and 3, over all cases: no blocking difference beyond the differences thebase already has.
entities execute about a third fewer lines.
1.3 s at 40,000 rows. The kernel applies state events for every row and operation, which a plain loop does not.
Main merged (5155d6d)
This branch now includes main at 5155d6d. Two tests that main added in #322 fail on this branch:
test_join_new_columns_preserves_surviving_target_orderandtest_run_in_memory_records_tracks_dropped_caller_id_and_survivor_order. This slice keeps a positional Rewrite join and rejects a record that DataDesigner drops without reporting it; the remove-old-sdk PR in this stack takes main's record-id join and drops that check, and both tests pass from there on.Main merged (f294dc1)
This branch now also includes main at f294dc1 (#328, recovery from incomplete replacements in Rewrite) through its base branch, with #328 ported as described on the base PR.
🤖 Generated with Claude Code