Repository navigation
fix(executor): a flow file made only of runFlows is one flow, as in Maestro; suite: true splits it - #284
Open
luka-vladika-happening-xyz wants to merge 2 commits into
Conversation
… and onFlowComplete, as in Maestro A file reached by runFlow or retry applied only its env:; its header's onFlowStart and onFlowComplete never ran. Maestro runs them around the file's commands (Orchestra.kt runSubFlow, for runFlowCommand and retryCommand alike; the command's config is the file's header): the steps after onFlowStart, onFlowComplete in a finally. executeSubFlow now runs onFlowStart, then the steps unless it failed, then onFlowComplete, also after a failed step; the sub-flow fails when a hook fails, with the first failure, as a flow's own hooks report it. A retry runs them on each attempt. Once the run is cancelled or a runFlow's timeout: has passed, nothing more runs, onFlowComplete included (Maestro skips it on a cancel too). Unlike Maestro, the steps run when onFlowStart changed nothing on screen: runSubFlow takes executeSubflowCommands' "screen changed" result as onFlowStart's success, so a hook of only assertions skips the file's steps there. The pre-session scan for launchApp reads a sub-file's onFlowStart before its steps and its onFlowComplete after them, so a launchApp with permissions in a sub-file's hook arms the driver as one in its steps does.
…aestro; suite: true splits it A test file whose steps are all runFlows of files was split into a flow per runFlow (IsSuite, expandSuites), in a sequential run only: the parallel runner never split one, so the same file gave a different result with --parallel. Maestro has no such rule. A test file is one flow and a runFlow a sub-flow of it (Orchestra.kt runFlowCommand, runSubFlow, executeSubflowCommands): what one sub-flow keeps in output the next one reads, since the output binding lives as long as the flow's JS engine; a failing sub-flow ends the test; and the test is reported under its own name. Split, the next part read output as undefined, the parts after a failure still ran, and the report and the JUnit listed the parts but not the test. The split is now asked for in the file's header, suite: true, and both runners honour it. The file says what it is, so it means the same on every run, from the command line, CI or a library caller. Maestro keeps an unknown header key (YamlConfig's @JsonAnySetter into ext), so such a file still runs there, as one flow. A file that says suite: true and cannot be split (another step, a runFlow with when:, an else or inline commands, a single runFlow) fails validation, naming the step, rather than running as one flow against what its header says. Reached by a runFlow, a file's suite: true changes nothing. A library caller that skips the validator gets such a file run as one flow, with a warning in the log. The run's log says how each runFlow-only file runs: split by its key, or one flow, with the key that would split it. This changes a default: a file that relied on the split is one test now unless its header says suite: true. The README's Flow Config section shows the key.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A test file whose steps are all
runFlows of files was split into a flow perrunFlowby the sequential runner (not by the parallel one, so the same file gave a different result with--parallel). Maestro runs it as one flow: arunFlowis a sub-flow,outputcarries from one to the next, a failure ends the test, and the test is reported under its own name. As agreed in #278, such a file now runs as one flow in both runners, and the old split is asked for in the file's header withsuite: true. Stacked on #277 (arunFlow's file runs its own hooks), since a suite's parts run inside one flow now; until #277 is in, this PR shows its commit too.Type of Change
Changes Made
flow.Config.Suite(suite: truein a flow's header).Flow.IsSuiteis true only when the header says so and the steps can be split;Flow.SuiteProblemsays why they cannot (another step, arunFlowwith inline commands,when:or an else, a singlerunFlow), naming the step.Runner.Runand nowParallelRunner.RuncallexpandSuites, which splits only such a file. Each part takes the suite'sappIdandenvand itsrunFlow'senv, as before.suite: trueand cannot be split, naming the step:suite: true, but step 3 is tapOn, not a runFlow; a suite has only runFlow steps of files, two or more, without when:, else or inline commands. The key's only effect is the split, so a file that cannot honour it would otherwise run as one flow against what its header says, and the validator already refuses a file before the run for other mistakes in it (a missingrunFlowfile, a parse error); as with those, no flow of the run starts, so one such file in a folder holds back the rest until its header or its steps are fixed. If you'd rather it warned and ran the file as one flow, that is a two-line change. Reached by arunFlow, a file'ssuite: truechanges nothing: it runs nested, as anyrunFlow's file. A library caller that skips the validator gets such a file run as one flow, with a warning in the log.maestro-runner.log) says how each runFlow-only file runs:checkout.yaml has suite: true: running its 2 runFlow files as separate flows; without the key,checkout.yaml is made only of runFlow steps: running it as one flow (suite: true in its header runs each as a flow of its own); with the key on a file that cannot be split (library callers only),[WARN] checkout.yaml has suite: true, but step 3 is tapOn, not a runFlow: running it as one flow.suite: true. Maestro keeps an unknown header key (YamlConfig's@JsonAnySetterintoext; 2.12.1'scheck-syntaxpasses such a file), so the file still runs there, as one flow.Related Issues
Fixes #278
Testing
go test -raceofpkg/flow,pkg/executor,pkg/validatorandpkg/cli)pkg/executor/suite_file_test.go, reading the report files: a runFlow-only file is one flow under its name and path inreport.json, with itsrunFlows and their steps nested in the flow's file, one JUnit testcase, and the same JSON keys as a file with a step of its own beside therunFlows; what the first part keeps inoutputthe second reads; a failingrunFlowfails the file and the next one does not run; withsuite: truethe file is split as before; the parallel runner splits it only then;suite: trueon a file that cannot be split runs it as written; the three log lines.TestValidate_SuiteKey: the validator takes asuite: truefile that can be split, refuses one that cannot with the step named, and leaves it alone when it is reached by arunFlow.TestFlow_IsSuitechecks every case with and without the key, andTestFlow_SuiteProblemeach reason. Without the change, all but two of the new executor tests and the validator test fail; the two that pass guard the opt-in and a mixed file.The default was run on an Android emulator (API 36) with Maestro 2.12.1 for reference while we discussed the shape, with the same one-flow code: the repro of #278 runs as one flow,
checkout, which passes, as on Maestro (main:loginpasses,cartfails); with #277, a file run twice byrunFlowwith its own hooks, then a check of what the hooks counted inoutput, runs as one flow and passes, as on Maestro (main: three flows, the check fails). On this branch, on a real Android phone (Android 16, the devicelab driver): the repro as written is one flow,checkout, which passes (one JUnit testcase,checkout), and the log sayscheckout.yaml is made only of runFlow steps: running it as one flow (suite: true in its header runs each as a flow of its own); the same file withsuite: truein its header is split as before,loginpassed andcartfailed (assertTrue failed: ${output.user == 'alice'}), withcheckout_suite.yaml has suite: true: running its 2 runFlow files as separate flowsin the log; asuite: truefile with anevalScriptbetween itsrunFlows stops at validation (suite: true, but step 2 is evalScript, not a runFlow; …). Maestro 2.12.1 on the same phone runs thesuite: truefile as one flow,checkout_suite, which passes.Checklist