Skip to content

feat(report): a runFlow's steps are in the report as they run - #276

Open
luka-vladika-happening-xyz wants to merge 2 commits into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:pr/live-substeps
Open

luka-vladika-happening-xyz wants to merge 2 commits into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:pr/live-substeps

Conversation

@luka-vladika-happening-xyz

@luka-vladika-happening-xyz luka-vladika-happening-xyz commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

A runFlow's, retry's or repeat's steps were written to the flow's report file only when it ended (CommandEndWithSubs), so a reader following the report while the flow runs (a live view of the run, which updateSeq allows) saw the step as one running line for its whole run, then all its steps at once. A flow made mostly of runFlows showed almost nothing while it ran.

Maestro shows a subflow's steps as they run: Orchestra.executeSubflowCommands calls onCommandStart and onCommandComplete for each of them, MaestroCommandRunner redraws its view at each, and AnsiResultView expands a running or failed runFlow down to its steps. This gives the report the same.

Each nested step is now written to the flow's file as it starts (running) and as it ends, with the compound steps around it shown as running too, through a new FlowWriter.CommandSubs, which writes the command's steps so far and updates the index, as a top-level step's start and end do. The list written at the command's end is unchanged, so a finished report is the same as before. The file is written at most every 250 ms for them (report.LiveWriteInterval): a start or end within that time of the last write is left to a timer, which writes the latest list when the time is up, so a step that runs long after short ones still finds them in the file.

The cost: at most one more write of the flow's file and the index every 250 ms while a runFlow, repeat or retry runs, whatever its number of steps. The list is built only when it is written. A repeat of 2 taps × 2000 takes about as long as on main (0.37 s against 0.39 s with the mock driver) and writes the file 4 times.

TestRunFlowStepsAreReportedAsTheyRun runs a runFlow holding a tap, a nested runFlow with a tap, and a tap, and waits while each tap runs for the flow's file to show it: first tapOn: running; then the first tap passed and the nested runFlow running with its tap running; then the nested runFlow passed and the last tap running. It fails on main. TestQuickStepsReachTheFileWhileALongStepRuns checks that steps that ended quickly are in the file while a long one runs, and TestLongRepeatWritesTheFlowFileAtMostEveryInterval that a repeat of 2 taps × 1000, or of a runFlow × 1000, writes the file at most once every 250 ms. The pkg/report tests cover CommandSubs' interval, its timer, the end and a slow write. go test -race ./pkg/executor ./pkg/report passes; golangci-lint v2.13.2: 0 issues.

@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.98246% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/report/flow.go 87.50% 7 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@omnarayan

Copy link
Copy Markdown
Contributor

Thanks, the live view is a nice touch. One problem: each nested step's start and end now rewrites the whole flow file, so the cost grows with the square of the step count. A repeat of 2 taps × 2000 takes 26.8 s with this change against 0.19 s without it, and the file is rewritten about 8,000 times at up to 1 MB. Could you write at most every ~250 ms, and always when a runFlow ends, and add a test with a long repeat that limits the number of writes? The rest looks good.

@luka-vladika-happening-xyz

Copy link
Copy Markdown
Contributor Author

Thanks, that's right: every nested start and end rewrote the whole file and copied every step so far. Pushed 6c7fcc2:

  • At most one write every 250 ms for a running runFlow's, repeat's or retry's steps (report.LiveWriteInterval). A call within that time leaves its list to a timer, which writes the last list given when the 250 ms are up, so steps that end quickly before a long one (a tap, then a 30 s wait) are in the file within 250 ms while the long one runs, not only when it ends.
  • The list is built only when it is written. liveSubs hands over the steps that ended as they are (they never change, and later ones are appended past their length) instead of copying them at each start and end. The FlowWriter now holds a lock, since the timer writes from its own goroutine.
  • The top-level step's start and end write at once, as before, and take a list left waiting, so a finished report is the same as before. Nothing is written for it after the flow's end.
  • A nested runFlow's end goes through the same 250 ms, with the timer after it, rather than being written at once: a repeat of a runFlow 2000 times would otherwise write at each of its 2000 ends, which brings the square back. Happy to write each runFlow's end at once instead if you prefer.
  • The 250 ms start once a write and its index update are done. Counted from the write's start, a write slower than the interval let the next step write at once: a 1.3 MB flow under -race, where the index's flush (which regenerates the HTML, JUnit and Allure reports) took 620 ms, ran for minutes.
  • Your case, with the mock driver: a repeat of 2 taps × 2000 takes 0.37 s and writes the file 4 times (0.39 s on main; 48.8 s and 8,004 writes before this commit, on this machine). Under -race: 5.2 s, against 4.0 s on main.
  • Tests: TestLongRepeatWritesTheFlowFileAtMostEveryInterval runs a repeat of 2 taps × 1000 and a repeat of a one-tap runFlow × 1000 and checks that the file was written at most once per 250 ms of the run, plus the flow's own writes (4,004 writes without the limit); TestQuickStepsReachTheFileWhileALongStepRuns checks that a nested runFlow and a tap that ended quickly are in the file while the next step runs, and that the top-level runFlow's end is there at once for the step after it; TestRunFlowStepsAreReportedAsTheyRun now waits for each state to reach the file. In pkg/report, the CommandSubs tests cover: no write within the interval, the last list written when it is up, nothing after the end, and the interval counted after a slow write. go test -race ./pkg/executor ./pkg/report passes (the new tests add about 5 s); golangci-lint v2.13.2: 0 issues.

A runFlow's, repeat's or retry's steps were written to the flow's file only
when it ended, so a reader of the report (a live view of the run) saw a
one-line runFlow for its whole run and then all its steps at once. Each
nested step is now written as it starts (running) and as it ends, with the
compound steps running around it, through FlowWriter.CommandSubs; the list
written at the step's end (CommandEndWithSubs) is unchanged.
Each nested step's start and end rewrote the whole flow file (and copied
every step so far into a new list), so the cost grew with the square of the
steps: a repeat of 2 taps x 2000 took about 48 s and wrote the file 8,004
times, against 0.4 s before the live steps.

FlowWriter.CommandSubs now writes at most once every LiveWriteInterval
(250 ms). A call within that time leaves its list to a timer, which writes
the last list given when the time is up, unless another write took it
first, so steps that end quickly before a long one are in the file within
the interval and not only once the long one ends. A nested runFlow's,
repeat's or retry's end goes through the same limit (a repeat of a runFlow
would otherwise write at each of its ends); the top-level step's start and
end write at once, as before, and take a list left waiting; nothing is
written for it after the flow's end. The interval starts once a write and
its index update are done, so a write slower than the interval (a large
flow on a slow machine, or under -race) does not let every step write again.

The list is built only when it is written, on the timer's goroutine too:
liveSubs passes the steps that ended as they are (they never change, and
later ones are appended past their length) instead of copying them, and the
FlowWriter's methods hold a lock, since the timer writes the flow too.

The same repeat now takes 0.37 s and writes the file 4 times.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants