Skip to content

A flow file made only of runFlow steps is split into flows, where Maestro runs it as one #278

Open

Description

Following up on #208, where you suggested an issue if the difference from Maestro bites: it does for us, and here is a proposal. It would change a default you chose on purpose (#18), so I'd like to agree on the shape first.

What happens

A test file whose steps are all runFlows of other files is run as a suite: each runFlow becomes a flow of its own (IsSuite, expandSuites, since 5b30cdb), as you explained on #18. Maestro has no such rule: a test file is one flow, whatever its steps, and a runFlow is a sub-flow of it (Orchestra.runFlowCommand → runSubFlow → executeSubflowCommands).

Writing a test as a list of shared parts is common in suites written for Maestro, and those tests behave differently here:

  1. output does not carry over. What one part keeps in output (a test user from a script, a value read from the screen) is gone in the next part. In Maestro the output binding lives as long as the flow's JS engine, so the next sub-flow reads it.
  2. A failure does not end the test. The parts after a failed one still run. In Maestro a failing sub-flow fails the test there.
  3. The test is never reported under its own name. The report and the JUnit list the parts, not the test. A CI step that maps test files to results sees the test as not run.
  4. Sequential and parallel runs disagree. ParallelRunner.Run does not call expandSuites, so with --parallel or several devices the same file is one flow.

Minimal repro:

# checkout.yaml
appId: com.example
---
- runFlow: login.yaml
- runFlow: cart.yaml
# login.yaml
appId: com.example
---
- evalScript: ${output.user = 'alice'}
# cart.yaml
appId: com.example
---
- assertTrue: ${output.user == 'alice'}

On an Android emulator (API 36), Maestro 2.12.1 runs one flow, checkout, which passes (its JUnit has one testcase, checkout). maestro-runner at main (deddc58) runs two: login passes and cart fails with assertTrue failed: ${output.user == 'alice'} (cause: assertion failed).

Proposal

Run such a file as one flow by default, as Maestro does, and keep today's split behind an opt-in for those who group test cases in a file. Two shapes come to mind:

  • a test flag, --expand-suites, applied by both the sequential and the parallel runner, or
  • a key in the suite file's header (for example suite: true), so the file says it is a suite. Maestro keeps unknown header keys (YamlConfig's @JsonAnySetter), so such a file still runs there, as one flow.

Either way it is a breaking change for anyone who relies on the split today: their suite files become one test each unless they opt in.

Run as one flow, such a file also needs its parts to run their own onFlowStart and onFlowComplete, which they ran as flows of their own. A file reached by runFlow (or retry) applies only its env: today, while Maestro's runSubFlow runs its hooks around its commands; #277 (a separate PR) fixes that, and is worth having whatever you decide here.

We have the first shape ready as a PR on top of #277 (a RunnerConfig.ExpandSuites read by Runner.Run and ParallelRunner.Run, the flag, docs/go-library.md, and tests that read report.json, the flow's file and the JUnit), and are happy to do it the other way if you prefer.

Activity

  1. omnarayan commented on Nov 18, 2026

    @omnarayan
    Contributor

    Thanks, Luka. This is the issue I asked for on #208, and the four points make the case. Point 4 is a bug on its own: the same file shouldn't give a different result with --parallel.

    Let's match Maestro. A file made only of runFlow steps runs as one flow, so output carries over, a failing part ends the test, and the report shows the test under its own name, in both the sequential and the parallel runner.

    For the opt-in, let's use the header key, suite: true, rather than a CLI flag. The file then says what it is, and it means the same thing on every run, from the command line or CI or a tool that starts a run. And since Maestro keeps unknown header keys, the same file still runs there, as one flow.

    For anyone who wants several test cases run together without a suite file, a folder of flows, or flows: in config.yaml, already does that, both here and in Maestro.

    This changes a default people may rely on, so the changelog will say so plainly and show the one-line opt-in. #277 goes in first, since the parts need their own onFlowStart and onFlowComplete once they run inside one flow.

    Go ahead with the PR. One request: please have the runner log a line when it treats a file as a suite, so a run that splits (or doesn't) never surprises anyone.

  2. luka-vladika-happening-xyz commented on Nov 18, 2026

    @luka-vladika-happening-xyz
    ContributorAuthor

    Thanks! PR #284 does it this way: suite: true in the header splits the file in both runners, without it a runFlow-only file is one flow, and the run's log says which. A suite: true file that cannot be split fails validation, naming the step. It is stacked on #277 until that one is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions