Repository navigation
feat(graph): invocation-scoped operations in the kernel - #332
Draft
binaryaaron wants to merge 6 commits into
Draft
binaryaaron wants to merge 6 commits into
binaryaaron wants to merge 6 commits into
Conversation
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Add SP-14 and SP-15, register them in run_all, add Recorder.invocation_transform, and document invocation-scoped operations in the graph SDK concepts page. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…unknown scopes per target A wrong-shape return from an invocation-scoped operation now fails every dispatched target with invalid_output and a TypeError naming what was returned and how many entries were expected, so a caller that re-raises recorded errors no longer loses the rows silently. The request forecast now counts every operation whose scope is not invocation once per target, matching dispatch, so a misspelt scope can no longer slip past a hard request budget. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…/graph-sdk-invocation-operation 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…/graph-sdk-invocation-operation 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The graph kernel can now run an operation once over every target in a call to
protect()instead of once pertarget, and still record a separate outcome for each target. This lets a batch backend, which handles all rows in one
request, take part in the protection record without one call per row. Nothing in
Anonymizer, the routes or thepublic names changes: no existing workflow declares such an operation yet.
This branch stacks on #326 and is rebased onto its base branch when the stack moves.
What changes
Declaration (
engine/graph_core/authoring.py).Operationgainsscope,"target"by default, andOperation.per_invocation(name, fn, inputs=None, outputs=None, requests=Requests()), which builds an operation withscope="invocation". Ports and kinds are declared per target, exactly as for other operations. An invocation-scopedoperation whose output kind is
assessmentis a verifier and can cover the final text, under the existing rule.Dispatch (
engine/graph_core/kernel.py). When the kernel reaches an invocation-scoped step, it calls theoperation once with every target that is waiting there and whose earlier steps succeeded. Each input port arrives
as a tuple with one value per target, in target order. The return must be a sequence with one entry per target:
Nonefails that target withmissing_output;Exceptioninstance fails that target withraised.If the operation raises, every target it was given fails with
raised. If it returns anything other than a sequenceof the right length, they all fail with
invalid_output, and aTypeErrornaming what was returned is recorded.BaseExceptionstill propagates to the caller. Targets that failed an earlier step are not passed in and stayblocked. Calling an operation once per target works as before.
Request forecast (
engine/graph_core/execution.py). An invocation-scoped operation's declared requests arecounted once per call to
protect(), not once per target. Every other operation's requests are counted once pertarget, as they are dispatched.
Docs.
docs/concepts/graph-sdk.mddescribes invocation-scoped operations after theOperationparagraph.Spec programs.
sp14_invocation_operation.pychecks the contract above (one call, per-target fan-out, eachfailure form, blocked targets, the forecast and budget rejection, and an invocation-scoped verifier including
findings attributed to the wrong target).
sp15_stage_sequence.pycoversrun_stages, which detection and Rewriteuse and no program covered: stage order, a raised error reaching the caller, and
KeyboardInterruptpropagating.Net non-blank, non-comment code lines under
src/: +67 (kernel +44, authoring +22, execution +1).Known limits
scopeis not checked at run time. A misspelt value is caught by the type checker; at run time it behaves as aper-target operation, in dispatch and in the forecast.
Checks
make checkpasses.make test: 1,557 passed.test_wandb_snapshot_rejects_fifo_socket_and_devicefails in the test environment withAF_UNIX path too long, as on the base.uv run --group dev python -m tests.spec_programs.run_all: 70 pass, 8 retired, 0 fail.f4f2b73d) on pandas 2 and 3, over all cases: no blocking difference beyond the differencesthe base already has.
and 1.22×).
Main merged (5155d6d)
This branch now includes main at 5155d6d through its base branch, including the Rewrite join change described on the remove-old-sdk PR.
Main merged (f294dc1)
This branch now also includes main at f294dc1 (#328, recovery from incomplete replacements in Rewrite) through its base branch.
🤖 Generated with Claude Code