Repository navigation
refactor: run Rewrite as 0.4.0's stages on the graph kernel - #325
Draft
binaryaaron wants to merge 3 commits into
Draft
binaryaaron wants to merge 3 commits into
binaryaaron wants to merge 3 commits into
Conversation
RewriteWorkflow.run and evaluate now run 0.4.0's stages (replacement map, the single pipeline call, the evaluate-repair loop, the judges) as one kernel sequence of opaque operations. The repair loop is an unrolled sequence bounded by max_repair_iterations; the initial evaluate always runs and a repair stage does nothing once no row needs repair. Delete the graph SDK's Rewrite lowering (_semantic_profile, _semantic_evaluation), the Substitute map lowering kept for it (_semantic_map), and the facade wiring (run_detection_rewrite, retained executions on results, rewrite_binding on the NDD adapter). Move rewrite artifacts into graph_sdk and the stage binding into detection, the only packages that still use them. CombinedRewriteWorkflow is restored to 0.4.0 because Rewrite.use_combined_graph is public. Keep the position-based joins, RewriteResult.source_positions and the guard against masked assignment into an all-null Arrow column, which the test suite requires. Remove the tests of the deleted code. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…s/chore/graph-sdk-rewrite-on-kernel Conflict in src/anonymizer/interface/anonymizer.py (two hunks in evaluate and _run_internal_impl). Kept this branch's lines (failed_records=all_failed, return AnonymizerResult(...)) and the parent's id_column threading. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…s/chore/graph-sdk-rewrite-on-kernel Keeps the deletion of _semantic_evaluation.py and tests/engine/test_rewrite_repair_decision.py; ports main's incomplete-replacement recovery into rewrite_workflow.py. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrite now runs 0.4.0's stages as one sequence on the private kernel from #324, and the graph SDK's own Rewrite
path is deleted. Users see no change from this branch's base: outputs, errors and failed-record accounting are the
same on pandas 2 and 3, and the public
anonymizerandanonymizer.graphexports are untouched. Detection keeps itscurrent path for now.
This branch stacks on #324 and is rebased onto #324's base branch when #324 merges.
What changes
engine/rewrite/rewrite_workflow.py).RewriteWorkflow.runandevaluatestart from 0.4.0's body.run_stagesruns each step as one opaque kernel operation: the replacement map, the single pipeline call, theevaluate-repair loop and the judges. The repair loop is an unrolled sequence of
repair-kandevaluate-k+1stages, so it runs at most
max_repair_iterationsrepairs. The initial evaluate always runs, repairs touch failingrows only, and a repair stage does nothing once every row passes. The judges keep 0.4.0's catch-and-default.
CombinedRewriteWorkflowis back to 0.4.0's body, with its single DataDesigner call run as onestage, because
Rewrite.use_combined_graphis public.RewriteResult.source_positionsand the guard against maskedassignment into an all-null Arrow column. Without the guard, a batch where every rewrite is withheld aborts inside
pyarrow.
Anonymizer.evaluateruns the judges directly, as in 0.4.0. Results no longer retain a graphexecution.
rewrite/artifacts.pymoves tograph_sdk/rewrite_artifacts.pyandndd/_stage_binding.pymoves todetection/_stage_binding.py, the only packages that still use them.Deleted
engine/rewrite/_semantic_profile.pyandengine/rewrite/_semantic_evaluation.py: the graph SDK's Rewritelowering.
engine/replace/_semantic_map.py: the Substitute map lowering that chore: run Redact, Annotate, Hash and Substitute on a private kernel #324 kept for Rewrite.run_detection_rewritein_opaque_redact.py;builtin_mapandbuiltin_rewrite_runnerinndd/_route.py; therewrite_bindingparameter of the NDD adapter; the retained-execution properties on results.Tests that exercised only deleted code or pinned the old Rewrite topology are deleted
(
test_semantic_rewrite_tail.py,test_rewrite_repair_decision.py,test_rewrite_object_dtypes.py,test_rewrite_mixed_ingress.py, plus individual functions in the route-resolution, Rewrite, combined-Rewrite, NDDadapter, Rewrite-graph and evaluate-graph tests). Tests that asserted on retained executions keep their behaviour
assertions.
Net non-blank, non-comment code lines under
src/,tools/andexamples/: −2,240 (52,195 to 49,955). With #324,the two branches remove 2,182 lines.
Checks
make checkpasses.make test: 5,302 passed. One test,test_wandb_snapshot_rejects_fifo_socket_and_device, fails in the testenvironment with
AF_UNIX path too long, and fails the same way on the base.uv run --group dev python -m tests.spec_programs.run_all: 59 pass, 8 retired.f4f2b73d) on pandas 2 and 3, over all cases, Rewrite included: no blocking difference beyond thedifferences the base already has.
replace cases are within 2.1%.
max_repair_iterations=Nand leaves the rowsflagged for review; with N=0 only the initial evaluate runs.
Main merged (5155d6d)
This branch now includes main at 5155d6d. Two tests that main added in #322 fail on this branch:
test_join_new_columns_preserves_surviving_target_orderandtest_run_in_memory_records_tracks_dropped_caller_id_and_survivor_order. This slice keeps a positional Rewrite join and rejects a record that DataDesigner drops without reporting it; the remove-old-sdk PR in this stack takes main's record-id join and drops that check, and both tests pass from there on.Main merged (f294dc1)
This branch now also includes main at f294dc1, which adds #328 (an incomplete replacement in Rewrite keeps its generated rewrite, is eligible for repair, and is flagged for human review). This branch removes
_semantic_evaluation.py, so #328 is ported intorewrite_workflow.py:_withhold_rewritten_textno longer clears the text,repairno longer skips unavailable replacements, andflag_for_reviewincludes them.🤖 Generated with Claude Code