Skip to content

feat(core): workflows with job dependencies (M8) - #9

Merged
giraffesyo merged 1 commit into
canaryfrom
feat/workflows
Sep 26, 2026
Merged

giraffesyo merged 1 commit into
canaryfrom
feat/workflows

Conversation

@giraffesyo

Copy link
Copy Markdown
Member

First of the M8 PRs: DAG workflows. Streams and hopperui follow separately.

What changed

  • API: hopper.NewWorkflow(name, opts), wf.Add(args, opts) Step, hopper.After(steps...), client.InsertWorkflow/InsertWorkflowTx, client.WorkflowGet, and hopper workflows get <id>. A step can only depend on steps added before it, so a workflow is a DAG by construction. StepOpts.OnDependencyFailure is DependencyCancel (default) or DependencyIgnore.
  • Model: a workflow is a batch whose jobs depend on each other, so it gets the batch's progress and callbacks for free (OnSuccess/OnFailure/OnComplete), a name, and its graph on the batch row. Steps with dependencies are inserted pending; roots are available at once.
  • Schema v4: hopper_job_deps (the working set of unsatisfied edges), hopper_batches.name and .edges, and a batch_id index on history for inspection.
  • Finalize: every statement that finalizes jobs (batched finalize, cancelling a waiting job, expiring) now produces a finished CTE in one shape and ends with the same tail: cascade-cancel the pending dependents of a failed job (recursive CTE, cancel edges only), archive, count batches down and insert callbacks, promote dependents whose dependencies have all left the live table, notify their queues, drop the satisfied edges. This replaced the separate history-insert and batch-accounting helpers.
  • Driver: WorkflowInsert (IDs generated up front by the database, batch + jobs + edges in one transaction, no unique keys allowed) and WorkflowGet; DependencyFailure, JobDependency, JobEdge, WorkflowRow.
  • Tests: drivertest Workflows (diamond promotion, transitive cascade with an ignore branch, cancelling a pending step, validation) runs on both drivers; a client-level end-to-end test; a CLI case.
  • Docs: plan §4.6, §6, §11, §13, §16; changelog; getting-started section on batches and workflows.

hopperbench

Local, PG 17, -jobs 60000, five rounds alternating base (feat/hoppersql) and head, median run:

insert       base 185729 jobs/s -> head 194540 jobs/s (+4.7%) ok
latency      base p50 1.00ms p99 2.36ms -> head p50 1.01ms p99 2.46ms (-4.0%) ok
retry        base 31578 jobs/s -> head 32292 jobs/s (+2.3%) ok
throughput   base 68895 jobs/s -> head 69180 jobs/s (+0.4%) ok

The finalize tail adds index probes on hopper_job_deps per finalized job, which is empty for plain jobs; no measurable cost.

Base automatically changed from feat/hoppersql to canary September 26, 2026 14:51
@giraffesyo
giraffesyo merged commit 9473b66 into canary Sep 26, 2026
8 checks passed
@giraffesyo
giraffesyo deleted the feat/workflows branch September 26, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant