Repository navigation
Description
Following up on #208, where you suggested an issue if the difference from Maestro bites: it does for us, and here is a proposal. It would change a default you chose on purpose (#18), so I'd like to agree on the shape first.
What happens
A test file whose steps are all runFlows of other files is run as a suite: each runFlow becomes a flow of its own (IsSuite, expandSuites, since 5b30cdb), as you explained on #18. Maestro has no such rule: a test file is one flow, whatever its steps, and a runFlow is a sub-flow of it (Orchestra.runFlowCommand → runSubFlow → executeSubflowCommands).
Writing a test as a list of shared parts is common in suites written for Maestro, and those tests behave differently here:
outputdoes not carry over. What one part keeps inoutput(a test user from a script, a value read from the screen) is gone in the next part. In Maestro theoutputbinding lives as long as the flow's JS engine, so the next sub-flow reads it.- A failure does not end the test. The parts after a failed one still run. In Maestro a failing sub-flow fails the test there.
- The test is never reported under its own name. The report and the JUnit list the parts, not the test. A CI step that maps test files to results sees the test as not run.
- Sequential and parallel runs disagree.
ParallelRunner.Rundoes not callexpandSuites, so with--parallelor several devices the same file is one flow.
Minimal repro:
# checkout.yaml
appId: com.example
---
- runFlow: login.yaml
- runFlow: cart.yaml# login.yaml
appId: com.example
---
- evalScript: ${output.user = 'alice'}# cart.yaml
appId: com.example
---
- assertTrue: ${output.user == 'alice'}On an Android emulator (API 36), Maestro 2.12.1 runs one flow, checkout, which passes (its JUnit has one testcase, checkout). maestro-runner at main (deddc58) runs two: login passes and cart fails with assertTrue failed: ${output.user == 'alice'} (cause: assertion failed).
Proposal
Run such a file as one flow by default, as Maestro does, and keep today's split behind an opt-in for those who group test cases in a file. Two shapes come to mind:
- a
testflag,--expand-suites, applied by both the sequential and the parallel runner, or - a key in the suite file's header (for example
suite: true), so the file says it is a suite. Maestro keeps unknown header keys (YamlConfig's@JsonAnySetter), so such a file still runs there, as one flow.
Either way it is a breaking change for anyone who relies on the split today: their suite files become one test each unless they opt in.
Run as one flow, such a file also needs its parts to run their own onFlowStart and onFlowComplete, which they ran as flows of their own. A file reached by runFlow (or retry) applies only its env: today, while Maestro's runSubFlow runs its hooks around its commands; #277 (a separate PR) fixes that, and is worth having whatever you decide here.
We have the first shape ready as a PR on top of #277 (a RunnerConfig.ExpandSuites read by Runner.Run and ParallelRunner.Run, the flag, docs/go-library.md, and tests that read report.json, the flow's file and the JUnit), and are happy to do it the other way if you prefer.
Activity
omnarayan commentedon Nov 18, 2026
Thanks, Luka. This is the issue I asked for on #208, and the four points make the case. Point 4 is a bug on its own: the same file shouldn't give a different result with
--parallel.Let's match Maestro. A file made only of
runFlowsteps runs as one flow, sooutputcarries over, a failing part ends the test, and the report shows the test under its own name, in both the sequential and the parallel runner.For the opt-in, let's use the header key,
suite: true, rather than a CLI flag. The file then says what it is, and it means the same thing on every run, from the command line or CI or a tool that starts a run. And since Maestro keeps unknown header keys, the same file still runs there, as one flow.For anyone who wants several test cases run together without a suite file, a folder of flows, or
flows:inconfig.yaml, already does that, both here and in Maestro.This changes a default people may rely on, so the changelog will say so plainly and show the one-line opt-in. #277 goes in first, since the parts need their own
onFlowStartandonFlowCompleteonce they run inside one flow.Go ahead with the PR. One request: please have the runner log a line when it treats a file as a suite, so a run that splits (or doesn't) never surprises anyone.
luka-vladika-happening-xyz commentedon Nov 18, 2026
Thanks! PR #284 does it this way:
suite: truein the header splits the file in both runners, without it a runFlow-only file is one flow, and the run's log says which. Asuite: truefile that cannot be split fails validation, naming the step. It is stacked on #277 until that one is in.