feat(core): workflows with job dependencies (M8) - #9
Merged
Merged
Conversation
giraffesyo
force-pushed
the
feat/hoppersql
branch
from
September 26, 2026 03:24
03ac878 to
12642d7
Compare
giraffesyo
force-pushed
the
feat/workflows
branch
from
September 26, 2026 03:25
7c21931 to
e4788ea
Compare
giraffesyo
force-pushed
the
feat/hoppersql
branch
from
September 26, 2026 14:49
12642d7 to
d41dba2
Compare
giraffesyo
force-pushed
the
feat/workflows
branch
from
September 26, 2026 14:49
e4788ea to
0e23dff
Compare
giraffesyo
force-pushed
the
feat/workflows
branch
from
September 26, 2026 14:51
0e23dff to
eede8e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of the M8 PRs: DAG workflows. Streams and
hopperuifollow separately.What changed
hopper.NewWorkflow(name, opts),wf.Add(args, opts) Step,hopper.After(steps...),client.InsertWorkflow/InsertWorkflowTx,client.WorkflowGet, andhopper workflows get <id>. A step can only depend on steps added before it, so a workflow is a DAG by construction.StepOpts.OnDependencyFailureisDependencyCancel(default) orDependencyIgnore.OnSuccess/OnFailure/OnComplete), a name, and its graph on the batch row. Steps with dependencies are insertedpending; roots are available at once.hopper_job_deps(the working set of unsatisfied edges),hopper_batches.nameand.edges, and abatch_idindex on history for inspection.finishedCTE in one shape and ends with the same tail: cascade-cancel the pending dependents of a failed job (recursive CTE,canceledges only), archive, count batches down and insert callbacks, promote dependents whose dependencies have all left the live table, notify their queues, drop the satisfied edges. This replaced the separate history-insert and batch-accounting helpers.WorkflowInsert(IDs generated up front by the database, batch + jobs + edges in one transaction, no unique keys allowed) andWorkflowGet;DependencyFailure,JobDependency,JobEdge,WorkflowRow.drivertestWorkflows(diamond promotion, transitive cascade with anignorebranch, cancelling a pending step, validation) runs on both drivers; a client-level end-to-end test; a CLI case.hopperbench
Local, PG 17,
-jobs 60000, five rounds alternating base (feat/hoppersql) and head, median run:The finalize tail adds index probes on
hopper_job_depsper finalized job, which is empty for plain jobs; no measurable cost.