Skip to content

fix(executor): a flow file made only of runFlows is one flow, as in Maestro; suite: true splits it - #284

Open
luka-vladika-happening-xyz wants to merge 2 commits into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:fix/runflow-only-file-is-one-flow-2
Open

luka-vladika-happening-xyz wants to merge 2 commits into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:fix/runflow-only-file-is-one-flow-2

Conversation

@luka-vladika-happening-xyz

Copy link
Copy Markdown
Contributor

Summary

A test file whose steps are all runFlows of files was split into a flow per runFlow by the sequential runner (not by the parallel one, so the same file gave a different result with --parallel). Maestro runs it as one flow: a runFlow is a sub-flow, output carries from one to the next, a failure ends the test, and the test is reported under its own name. As agreed in #278, such a file now runs as one flow in both runners, and the old split is asked for in the file's header with suite: true. Stacked on #277 (a runFlow's file runs its own hooks), since a suite's parts run inside one flow now; until #277 is in, this PR shows its commit too.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Changes Made

  • flow.Config.Suite (suite: true in a flow's header). Flow.IsSuite is true only when the header says so and the steps can be split; Flow.SuiteProblem says why they cannot (another step, a runFlow with inline commands, when: or an else, a single runFlow), naming the step.
  • Runner.Run and now ParallelRunner.Run call expandSuites, which splits only such a file. Each part takes the suite's appId and env and its runFlow's env, as before.
  • The validator refuses a top-level file that says suite: true and cannot be split, naming the step: suite: true, but step 3 is tapOn, not a runFlow; a suite has only runFlow steps of files, two or more, without when:, else or inline commands. The key's only effect is the split, so a file that cannot honour it would otherwise run as one flow against what its header says, and the validator already refuses a file before the run for other mistakes in it (a missing runFlow file, a parse error); as with those, no flow of the run starts, so one such file in a folder holds back the rest until its header or its steps are fixed. If you'd rather it warned and ran the file as one flow, that is a two-line change. Reached by a runFlow, a file's suite: true changes nothing: it runs nested, as any runFlow's file. A library caller that skips the validator gets such a file run as one flow, with a warning in the log.
  • The run's log (maestro-runner.log) says how each runFlow-only file runs: checkout.yaml has suite: true: running its 2 runFlow files as separate flows; without the key, checkout.yaml is made only of runFlow steps: running it as one flow (suite: true in its header runs each as a flow of its own); with the key on a file that cannot be split (library callers only), [WARN] checkout.yaml has suite: true, but step 3 is tapOn, not a runFlow: running it as one flow.
  • README's Flow Config section shows the key.
  • Breaking: a runFlow-only file that relied on the split is one test now unless its header says suite: true. Maestro keeps an unknown header key (YamlConfig's @JsonAnySetter into ext; 2.12.1's check-syntax passes such a file), so the file still runs there, as one flow.

Related Issues

Fixes #278

Testing

  • Tests pass locally (go test -race of pkg/flow, pkg/executor, pkg/validator and pkg/cli)
  • Linting passes (golangci-lint v2.13.2: 0 issues)
  • Added tests for new functionality
  • Tested manually with sample flows

pkg/executor/suite_file_test.go, reading the report files: a runFlow-only file is one flow under its name and path in report.json, with its runFlows and their steps nested in the flow's file, one JUnit testcase, and the same JSON keys as a file with a step of its own beside the runFlows; what the first part keeps in output the second reads; a failing runFlow fails the file and the next one does not run; with suite: true the file is split as before; the parallel runner splits it only then; suite: true on a file that cannot be split runs it as written; the three log lines. TestValidate_SuiteKey: the validator takes a suite: true file that can be split, refuses one that cannot with the step named, and leaves it alone when it is reached by a runFlow. TestFlow_IsSuite checks every case with and without the key, and TestFlow_SuiteProblem each reason. Without the change, all but two of the new executor tests and the validator test fail; the two that pass guard the opt-in and a mixed file.

The default was run on an Android emulator (API 36) with Maestro 2.12.1 for reference while we discussed the shape, with the same one-flow code: the repro of #278 runs as one flow, checkout, which passes, as on Maestro (main: login passes, cart fails); with #277, a file run twice by runFlow with its own hooks, then a check of what the hooks counted in output, runs as one flow and passes, as on Maestro (main: three flows, the check fails). On this branch, on a real Android phone (Android 16, the devicelab driver): the repro as written is one flow, checkout, which passes (one JUnit testcase, checkout), and the log says checkout.yaml is made only of runFlow steps: running it as one flow (suite: true in its header runs each as a flow of its own); the same file with suite: true in its header is split as before, login passed and cart failed (assertTrue failed: ${output.user == 'alice'}), with checkout_suite.yaml has suite: true: running its 2 runFlow files as separate flows in the log; a suite: true file with an evalScript between its runFlows stops at validation (suite: true, but step 2 is evalScript, not a runFlow; …). Maestro 2.12.1 on the same phone runs the suite: true file as one flow, checkout_suite, which passes.

Checklist

… and onFlowComplete, as in Maestro

A file reached by runFlow or retry applied only its env:; its header's
onFlowStart and onFlowComplete never ran. Maestro runs them around the
file's commands (Orchestra.kt runSubFlow, for runFlowCommand and
retryCommand alike; the command's config is the file's header): the
steps after onFlowStart, onFlowComplete in a finally.

executeSubFlow now runs onFlowStart, then the steps unless it failed,
then onFlowComplete, also after a failed step; the sub-flow fails when a
hook fails, with the first failure, as a flow's own hooks report it. A
retry runs them on each attempt. Once the run is cancelled or a runFlow's
timeout: has passed, nothing more runs, onFlowComplete included (Maestro
skips it on a cancel too). Unlike Maestro, the steps run when onFlowStart
changed nothing on screen: runSubFlow takes executeSubflowCommands'
"screen changed" result as onFlowStart's success, so a hook of only
assertions skips the file's steps there.

The pre-session scan for launchApp reads a sub-file's onFlowStart before
its steps and its onFlowComplete after them, so a launchApp with
permissions in a sub-file's hook arms the driver as one in its steps does.
…aestro; suite: true splits it

A test file whose steps are all runFlows of files was split into a flow
per runFlow (IsSuite, expandSuites), in a sequential run only: the
parallel runner never split one, so the same file gave a different result
with --parallel. Maestro has no such rule. A test file is one flow and a
runFlow a sub-flow of it (Orchestra.kt runFlowCommand, runSubFlow,
executeSubflowCommands): what one sub-flow keeps in output the next one
reads, since the output binding lives as long as the flow's JS engine; a
failing sub-flow ends the test; and the test is reported under its own
name. Split, the next part read output as undefined, the parts after a
failure still ran, and the report and the JUnit listed the parts but not
the test.

The split is now asked for in the file's header, suite: true, and both
runners honour it. The file says what it is, so it means the same on
every run, from the command line, CI or a library caller. Maestro keeps
an unknown header key (YamlConfig's @JsonAnySetter into ext), so such a
file still runs there, as one flow.

A file that says suite: true and cannot be split (another step, a
runFlow with when:, an else or inline commands, a single runFlow) fails
validation, naming the step, rather than running as one flow against what
its header says. Reached by a runFlow, a file's suite: true changes
nothing. A library caller that skips the validator gets such a file run
as one flow, with a warning in the log.

The run's log says how each runFlow-only file runs: split by its key, or
one flow, with the key that would split it.

This changes a default: a file that relied on the split is one test now
unless its header says suite: true. The README's Flow Config section
shows the key.
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A flow file made only of runFlow steps is split into flows, where Maestro runs it as one

2 participants