Skip to content

refactor: run Rewrite as 0.4.0's stages on the graph kernel - #325

Draft
binaryaaron wants to merge 3 commits into
aagonzales/graph-sdk/chore/kernel-replace-routesfrom
aagonzales/chore/graph-sdk-rewrite-on-kernel
Draft

binaryaaron wants to merge 3 commits into
aagonzales/graph-sdk/chore/kernel-replace-routesfrom
aagonzales/chore/graph-sdk-rewrite-on-kernel

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Rewrite now runs 0.4.0's stages as one sequence on the private kernel from #324, and the graph SDK's own Rewrite
path is deleted. Users see no change from this branch's base: outputs, errors and failed-record accounting are the
same on pandas 2 and 3, and the public anonymizer and anonymizer.graph exports are untouched. Detection keeps its
current path for now.

This branch stacks on #324 and is rebased onto #324's base branch when #324 merges.

What changes

  • Rewrite (engine/rewrite/rewrite_workflow.py). RewriteWorkflow.run and evaluate start from 0.4.0's body.
    run_stages runs each step as one opaque kernel operation: the replacement map, the single pipeline call, the
    evaluate-repair loop and the judges. The repair loop is an unrolled sequence of repair-k and evaluate-k+1
    stages, so it runs at most max_repair_iterations repairs. The initial evaluate always runs, repairs touch failing
    rows only, and a repair stage does nothing once every row passes. The judges keep 0.4.0's catch-and-default.
  • Combined Rewrite. CombinedRewriteWorkflow is back to 0.4.0's body, with its single DataDesigner call run as one
    stage, because Rewrite.use_combined_graph is public.
  • Kept from the base. Position-based joins, RewriteResult.source_positions and the guard against masked
    assignment into an all-null Arrow column. Without the guard, a batch where every rewrite is withheld aborts inside
    pyarrow.
  • Facade. Anonymizer.evaluate runs the judges directly, as in 0.4.0. Results no longer retain a graph
    execution.
  • Moves. rewrite/artifacts.py moves to graph_sdk/rewrite_artifacts.py and ndd/_stage_binding.py moves to
    detection/_stage_binding.py, the only packages that still use them.

Deleted

  • engine/rewrite/_semantic_profile.py and engine/rewrite/_semantic_evaluation.py: the graph SDK's Rewrite
    lowering.
  • engine/replace/_semantic_map.py: the Substitute map lowering that chore: run Redact, Annotate, Hash and Substitute on a private kernel #324 kept for Rewrite.
  • run_detection_rewrite in _opaque_redact.py; builtin_map and builtin_rewrite_runner in ndd/_route.py; the
    rewrite_binding parameter of the NDD adapter; the retained-execution properties on results.

Tests that exercised only deleted code or pinned the old Rewrite topology are deleted
(test_semantic_rewrite_tail.py, test_rewrite_repair_decision.py, test_rewrite_object_dtypes.py,
test_rewrite_mixed_ingress.py, plus individual functions in the route-resolution, Rewrite, combined-Rewrite, NDD
adapter, Rewrite-graph and evaluate-graph tests). Tests that asserted on retained executions keep their behaviour
assertions.

Net non-blank, non-comment code lines under src/, tools/ and examples/: −2,240 (52,195 to 49,955). With #324,
the two branches remove 2,182 lines.

Checks

  • make check passes.
  • make test: 5,302 passed. One test, test_wandb_snapshot_rejects_fifo_socket_and_device, fails in the test
    environment with AF_UNIX path too long, and fails the same way on the base.
  • uv run --group dev python -m tests.spec_programs.run_all: 59 pass, 8 retired.
  • Parity against main (f4f2b73d) on pandas 2 and 3, over all cases, Rewrite included: no blocking difference beyond the
    differences the base already has.
  • Executed lines per case against the base: Rewrite cases run about 13% fewer lines (combined Rewrite about 11%);
    replace cases are within 2.1%.
  • A stub evaluate that fails every row runs exactly N repairs for max_repair_iterations=N and leaves the rows
    flagged for review; with N=0 only the initial evaluate runs.

Main merged (5155d6d)

This branch now includes main at 5155d6d. Two tests that main added in #322 fail on this branch: test_join_new_columns_preserves_surviving_target_order and test_run_in_memory_records_tracks_dropped_caller_id_and_survivor_order. This slice keeps a positional Rewrite join and rejects a record that DataDesigner drops without reporting it; the remove-old-sdk PR in this stack takes main's record-id join and drops that check, and both tests pass from there on.

Main merged (f294dc1)

This branch now also includes main at f294dc1, which adds #328 (an incomplete replacement in Rewrite keeps its generated rewrite, is eligible for repair, and is flagged for human review). This branch removes _semantic_evaluation.py, so #328 is ported into rewrite_workflow.py: _withhold_rewritten_text no longer clears the text, repair no longer skips unavailable replacements, and flag_for_review includes them.

🤖 Generated with Claude Code

RewriteWorkflow.run and evaluate now run 0.4.0's stages (replacement map, the
single pipeline call, the evaluate-repair loop, the judges) as one kernel
sequence of opaque operations. The repair loop is an unrolled sequence bounded
by max_repair_iterations; the initial evaluate always runs and a repair stage
does nothing once no row needs repair.

Delete the graph SDK's Rewrite lowering (_semantic_profile,
_semantic_evaluation), the Substitute map lowering kept for it
(_semantic_map), and the facade wiring (run_detection_rewrite, retained
executions on results, rewrite_binding on the NDD adapter). Move rewrite
artifacts into graph_sdk and the stage binding into detection, the only
packages that still use them. CombinedRewriteWorkflow is restored to 0.4.0
because Rewrite.use_combined_graph is public.

Keep the position-based joins, RewriteResult.source_positions and the guard
against masked assignment into an all-null Arrow column, which the test suite
requires. Remove the tests of the deleted code.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
binaryaaron and others added 2 commits October 27, 2026 15:32
…s/chore/graph-sdk-rewrite-on-kernel

Conflict in src/anonymizer/interface/anonymizer.py (two hunks in evaluate and
_run_internal_impl). Kept this branch's lines (failed_records=all_failed,
return AnonymizerResult(...)) and the parent's id_column threading.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…s/chore/graph-sdk-rewrite-on-kernel

Keeps the deletion of _semantic_evaluation.py and tests/engine/test_rewrite_repair_decision.py;
ports main's incomplete-replacement recovery into rewrite_workflow.py.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant