Repository navigation
ci: run the instrumented tests on an emulator, every Compose test on the v2 rules - #40
Merged
Merged
Conversation
Until now the 501 instrumented tests ran on phones only, and nothing ran them: the long press that closed 3.1.0 (#36) and the "Translate" that sent a note's text out were both visible only there. New job, API 29 (the S9's Android 10): the app and its tests are built before the emulator boots; a screen lock is set first (without one, every PIN-vault test fails for that reason alone). The emulator still lacks a hardware Keystore, a fingerprint and the dictation model, so the Gradle task fails whatever happens: tools/check-instrumented-results.py judges each test instead. AGP reports an assumption as a failure, so causes are read one by one; only three named reasons may skip a test; the hardware Keystore witness must fail with KeystoreSoftwareOnlyException (if it passed, the vault would have accepted a software key); fewer than 495 cases means the suite did not run. Its own self-test, seven doctored results, runs first in the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first emulator run in CI failed on this test only: it read the message in the device's language and looked for the French "supprimé"; the CI emulator is in English. It now reads the message in each of the five languages, each in its own configuration, and still checks it is displayed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AppLockLifecycleTest failed once on the CI emulator, never locally (10/10): "addObserver must be called on the main thread", the NavHost created on an IO thread. The cause, read in the Compose 1.12.1 test sources and the CI stack trace: the v1 rule composes through an UnconfinedTestDispatcher. AppLockManager publishes the unlocked state from its IO dispatcher (on purpose, under its mutex); the collector resumed there, and a race with the main thread's clock let the frame run on that IO thread. The app is not affected: its Recomposer runs on the main thread. The v2 rule (androidx.compose.ui.test.junit4.v2) queues composition work on a StandardTestDispatcher that the test clock runs only through runOnUiThread. Used by the two tests that drive the real MainActivity, where real app code publishes from IO; the screen tests are unchanged. Checked with a throwaway probe (state emitted from IO, composition thread recorded) under both rules, and the whole ui package on the emulator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit moved the two real-activity tests only. The next CI
run met the same race in a third test: EcranAProposEtLegalTest laid the
legal page out off the main thread ("performMeasureAndLayout called
during measure layout"), its Markdown reading resuming from
Dispatchers.Default, and the crash took the rest of the suite with it
(232 cases of 501).
Any v1 rule can meet it, in whichever test a background resume happens
to hit a frame: the 24 other rules move to androidx.compose.ui.test.
junit4.v2 (one import each; the types are the same).
ComposeTestRulesAreV2Test (JVM) fails on any v1 factory left in
src/androidTest, with its negative control. Its first sabotage passed:
the unit test task did not take src/androidTest as an input and stayed
up to date. It now does (app/build.gradle.kts); sabotage red, restored
green.
Whole suite on the local emulator under v2: 501 cases, 486 passed,
3 skipped; the 12 failures are the Keystore tests on an emulator with
no screen lock (the CI job sets one) and the hardware-key witness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The instrumented tests (501 cases) never ran in CI: the gate builds, tests on the JVM, lints and runs detekt, and the suite on a device lived only on the maintainer's emulator and phone. The 3.1.1 crash (#36, long press closing the app) was the kind of defect only that suite sees.
What this adds
Instrumented tests (emulator, API 29)inandroid.yml: an emulator with a screen lock set (locksettings set-pin), so that the vault's Keystore tests can create their keys, and the whole suite on it.tools/check-instrumented-results.py, which judges the XML rather than the exit code: a case may be skipped for three reasons only (what an emulator cannot provide), the hardware-key witness must FAIL withKeystoreSoftwareOnlyException(an emulator has no secure hardware — a pass would mean the check measures nothing), and at least 495 cases must have run.--self-testruns 7 scenarios of its own.EcranDInstallationVocaleTestchecks its message in every language instead of the device's.What the first runs found, and fixed at the cause
AppLockLifecycleTest, thenEcranAProposEtLegalTest, failed once each on the CI emulator, never locally (10/10):addObserver must be called on the main thread, thenperformMeasureAndLayout called during measure layout, the crash taking the rest of the suite with it (232 cases of 501).Read in the CI stack traces and in the Compose 1.12.1 test sources: the v1 test rules compose through an
UnconfinedTestDispatcher, so a screen resuming from another thread — the app lock's state published from IO, a Markdown reading back fromDispatchers.Default— runs the next frame right there, composition and layout, while the main thread draws. The app is not affected: its Recomposer runs on the main thread.androidx.compose.ui.test.junit4.v2, one import each), which queue that work on aStandardTestDispatcherthe test clock runs only throughrunOnUiThread.ComposeTestRulesAreV2Test(JVM) fails on any v1 rule left insrc/androidTest, with its negative control;src/androidTestis now an input of the unit test task (the first sabotage passed: the task stayed up to date).Checked
cd030ec: the four jobs green; 501 cases, 489 passed, 11 skipped (the admitted reasons), 1 failure: the witness, as required.🤖 Generated with Claude Code