Repository navigation
feat(report): a runFlow's steps are in the report as they run - #276
Open
luka-vladika-happening-xyz wants to merge 2 commits into
Open
luka-vladika-happening-xyz wants to merge 2 commits into
luka-vladika-happening-xyz wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
|
Thanks, the live view is a nice touch. One problem: each nested step's start and end now rewrites the whole flow file, so the cost grows with the square of the step count. A |
Contributor
Author
|
Thanks, that's right: every nested start and end rewrote the whole file and copied every step so far. Pushed
|
A runFlow's, repeat's or retry's steps were written to the flow's file only when it ended, so a reader of the report (a live view of the run) saw a one-line runFlow for its whole run and then all its steps at once. Each nested step is now written as it starts (running) and as it ends, with the compound steps running around it, through FlowWriter.CommandSubs; the list written at the step's end (CommandEndWithSubs) is unchanged.
Each nested step's start and end rewrote the whole flow file (and copied every step so far into a new list), so the cost grew with the square of the steps: a repeat of 2 taps x 2000 took about 48 s and wrote the file 8,004 times, against 0.4 s before the live steps. FlowWriter.CommandSubs now writes at most once every LiveWriteInterval (250 ms). A call within that time leaves its list to a timer, which writes the last list given when the time is up, unless another write took it first, so steps that end quickly before a long one are in the file within the interval and not only once the long one ends. A nested runFlow's, repeat's or retry's end goes through the same limit (a repeat of a runFlow would otherwise write at each of its ends); the top-level step's start and end write at once, as before, and take a list left waiting; nothing is written for it after the flow's end. The interval starts once a write and its index update are done, so a write slower than the interval (a large flow on a slow machine, or under -race) does not let every step write again. The list is built only when it is written, on the timer's goroutine too: liveSubs passes the steps that ended as they are (they never change, and later ones are appended past their length) instead of copying them, and the FlowWriter's methods hold a lock, since the timer writes the flow too. The same repeat now takes 0.37 s and writes the file 4 times.
luka-vladika-happening-xyz
force-pushed
the
pr/live-substeps
branch
from
November 19, 2026 14:58
6c7fcc2 to
2c124d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
runFlow's,retry's orrepeat's steps were written to the flow's report file only when it ended (CommandEndWithSubs), so a reader following the report while the flow runs (a live view of the run, whichupdateSeqallows) saw the step as onerunningline for its whole run, then all its steps at once. A flow made mostly ofrunFlows showed almost nothing while it ran.Maestro shows a subflow's steps as they run:
Orchestra.executeSubflowCommandscallsonCommandStartandonCommandCompletefor each of them,MaestroCommandRunnerredraws its view at each, andAnsiResultViewexpands a running or failedrunFlowdown to its steps. This gives the report the same.Each nested step is now written to the flow's file as it starts (
running) and as it ends, with the compound steps around it shown asrunningtoo, through a newFlowWriter.CommandSubs, which writes the command's steps so far and updates the index, as a top-level step's start and end do. The list written at the command's end is unchanged, so a finished report is the same as before. The file is written at most every 250 ms for them (report.LiveWriteInterval): a start or end within that time of the last write is left to a timer, which writes the latest list when the time is up, so a step that runs long after short ones still finds them in the file.The cost: at most one more write of the flow's file and the index every 250 ms while a
runFlow,repeatorretryruns, whatever its number of steps. The list is built only when it is written. Arepeatof 2 taps × 2000 takes about as long as on main (0.37 s against 0.39 s with the mock driver) and writes the file 4 times.TestRunFlowStepsAreReportedAsTheyRunruns arunFlowholding a tap, a nestedrunFlowwith a tap, and a tap, and waits while each tap runs for the flow's file to show it: firsttapOn: running; then the first tap passed and the nestedrunFlowrunning with its tap running; then the nestedrunFlowpassed and the last tap running. It fails on main.TestQuickStepsReachTheFileWhileALongStepRunschecks that steps that ended quickly are in the file while a long one runs, andTestLongRepeatWritesTheFlowFileAtMostEveryIntervalthat arepeatof 2 taps × 1000, or of arunFlow× 1000, writes the file at most once every 250 ms. Thepkg/reporttests coverCommandSubs' interval, its timer, the end and a slow write.go test -race ./pkg/executor ./pkg/reportpasses; golangci-lint v2.13.2: 0 issues.