From e02294df641c7678b84c37f1df9358a3a504dd55 Mon Sep 17 00:00:00 2001 From: Robert DeLuca Date: Thu, 12 Nov 2026 07:50:16 -0600 Subject: [PATCH 1/3] =?UTF-8?q?=E2=9C=A8=20Repeat=20browser=20workflows=20?= =?UTF-8?q?with=20reproducible=20seeds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exercise complete selections within one managed session, retiring final documents before project draining and advancing only after persistence. Keep failures visible across rounds and save bounded replay evidence, aggregate reports, and source-generated guides. --- docs/architecture.md | 6 + docs/guide.md | 31 +++- src/browser/shell.js | 2 +- src/cli-output.js | 18 +++ src/cli-run.js | 104 ++++++++++--- src/cli.js | 28 ++++ src/coordinator.js | 17 ++- src/repeat-reports.js | 142 ++++++++++++++++++ src/repeat.js | 67 +++++++++ src/reports.js | 66 +++++--- src/server.js | 8 +- tests/core/cli.test.js | 6 + tests/core/repeat.test.js | 79 ++++++++++ tests/execution/repeat.test.js | 266 +++++++++++++++++++++++++++++++++ tests/reporter/repeat.test.js | 32 ++++ 15 files changed, 827 insertions(+), 45 deletions(-) create mode 100644 src/repeat-reports.js create mode 100644 src/repeat.js create mode 100644 tests/core/repeat.test.js create mode 100644 tests/execution/repeat.test.js create mode 100644 tests/reporter/repeat.test.js diff --git a/docs/architecture.md b/docs/architecture.md index c54b413..c4421d8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -147,6 +147,12 @@ The CLI, live reporter and saved report expose the seed. `schedule.json` records ## Shared coordination +Managed headless `--repeat` composes ordinary runs inside one CLI session. `repeat.js` folds accepted round outcomes into immutable per-test counts; the existing run model still accepts one execution per stable test ID. Each seeded round uses a versioned root-derived seed, with the literal root preserved for round one. Queue replay stays fixed. The coordinator's common replacement path receives the next seed and reuses the pinned test bundle, worker contexts, servers and project resources. + +Automatic repeated executions park their final app document before project draining and release, including when there is no later real test. The next round waits for terminal persistence and the current serialized operation to settle. Each round has its own browser-result deadline; lifecycle and storage failures stop advancement. Ordinary interactive completion keeps the final app inspectable as before. Storage and backend reset remain project-owned. + +`repeat-reports.js` saves compact round outcomes and schedules independently of the coordinator's 20-run cache. Failed/interrupted and final rounds use the ordinary portable formatter. Successful intermediate captures are removed after evidence persistence. A 256 MiB evidence limit stops further rounds explicitly; accepted counts and aggregate/session metadata remain available outside that budget. The test fingerprint does not identify the running app or backend, and rerunning a batch prefix does not replay asynchronous interleaving. + `coordinator.js` owns run snapshots, processes results in order, and coordinates assignment waiting, message replays, draining and final saves. `run.js` contains pure ownership/resource transitions. Private gateways bind requests to their worker ID; posted worker IDs cannot borrow a peer execution. Cross-gateway image reads resolve accepted UUIDs from the same canonical outcomes. Project setup runs once per CLI session and optionally returns `createWorker`. Both receive one frozen session identity and immediate cleanup registration. Registration seals when acquisition settles; partial failures use the same cleanup stack as returned teardown. Every started acquisition settles before worker resources or shared parents are destroyed. The factory supplies private server/env/fixtures/beforeExecution/drain/teardown descriptors. Startup joins every worker task before cleanup, including on partial failure. Cleanup finishes the coordinator and gateways before stopping app servers, worker projects and shared setup. Browser shutdown follows. diff --git a/docs/guide.md b/docs/guide.md index 4178a9b..e58be31 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -431,6 +431,27 @@ Turn on **Pause on failure** in Run options to inspect a failing app. Continue h The CLI also prints a separate reporter URL. Open it in another tab if you'd rather keep the app uncovered. A completed interactive run keeps each worker's last app available; rerun an earlier test to inspect it again. +### Repeat while developing + +Exercise a workflow repeatedly, or vary the order of a whole selection: + +```sh +dyno --test 'checkout' --repeat 100 +dyno --suite Checkout --repeat 100 --random +dyno --suite Checkout --repeat 100 --seed checkout-race +dyno --repeat 100 --replay .dyno/reports//schedule.json +``` + +`--repeat N` runs the complete selection N times, including the first run. It supports managed headless runs; combining it with `--ui`, `--manual`, or `--watch` fails before startup. Tests keep their normal hooks and per-execution fixture preparation. Setup runs once, and browser storage, servers, and project worker resources survive between rounds. Keep scenario reset and backend cleanup in your project. + +Each automatic execution removes its test runtime and parks the app before project draining, including the final test. A round must finish cleanup and save its results before the next starts. Each round gets its own `runTimeout`. A failed test stays in the batch result while later rounds continue; cleanup, browser, preparation, persistence, or deadline failures stop repetition. Ctrl-C saves accepted partial results and stops further rounds. + +Without a seed, rounds use declared priority. With `--seed`, round one uses that literal seed and later rounds derive repeatable seeds from it. `--random` generates and prints a root seed; pass it to `--seed` to reproduce the series. The count does not affect seed derivation, so rerunning through the first failing round preserves the prefix. Repeated `--replay` keeps the recorded worker queues fixed. + +The summary prints commands to replay the first failing round and rerun the original selection through it. A focused rerun can remove the predecessor that caused a failure. A round-only replay also omits state accumulated in earlier rounds. Seeds reproduce priority, and schedules reproduce queues; neither controls app randomness, backend data, or asynchronous timing. Keep app code and data stable while collecting results. Dyno pins the test bundle, not the running application. + +See [saved reports](guide.md#saved-reports) for batch counts, schedules, images, and evidence limits. Passing many times is evidence under those conditions; it does not prove a test can never fail. + ### Bring your own browser ```sh @@ -501,10 +522,18 @@ Each folder includes `run.json`, `junit.xml`, `schedule.json`, and `session.json Runs are `completed` or `interrupted`; tests without accepted results are `not-run`. Failures retain their phase and source context. JUnit adds an error case for interruption, so an unfinished run can't look green. -Dyno saves results before reporting completion, including accepted outcomes on signals, browser loss, and app failure. A force kill or machine failure can't flush results. Storage errors fail the CLI, and partially written folders aren't published as reports. A source image removed before saving appears as **Screenshot unavailable**. +Dyno saves results before reporting completion, including accepted outcomes on signals, browser loss, and app failure. A force kill or machine failure can't flush results. Storage errors fail the CLI, and partially written run folders aren't published as reports. A source image removed before saving appears as **Screenshot unavailable**. Reports contain test output and source excerpts. Dyno doesn't save config, fixtures, connection keys, or launch tokens in them. +### Repeated-run reports + +`--repeat` writes `.dyno/reports/repeat-/`. Open its `index.html` for aggregate counts and the first failure. `batch.json` records the root seed, test-bundle fingerprint, browser identity, original selection, and observed outcomes. Skipped and unfinished tests stay separate from passes. The batch's `junit.xml` gives each test execution its round number, so a later pass cannot replace an earlier failure. + +`rounds.jsonl` records completed and interrupted rounds, per-test outcomes, and paths to their worker schedules under `schedules/`. Full portable reports and images are retained under `runs/` for failed or interrupted rounds and the final round. Successful intermediate rounds keep compact evidence; their screenshots are discarded. Session timing lives once in the batch folder. + +Round evidence and retained reports share a 256 MiB budget. Reaching it stops repetition, marks unavailable evidence explicitly, and preserves accepted counts in `batch.json`. The aggregate summary, final compact outcome record, and session timing are saved outside that budget so a limit cannot hide a failure or stop reason. Batch metadata updates after each saved round; HTML and JUnit are finalized on shutdown. Storage failures also fail the command. Copy the whole batch folder to keep its relative report and image links; printed commands refer to the original local paths. + ## Understand where time goes Open **Slowest tests** in the reporter, then a test's **Timings** tab. Dyno always records document startup, hooks, the test body, queued errors, and operations such as navigation, requests, screenshots, and native calls. diff --git a/src/browser/shell.js b/src/browser/shell.js index c95c09a..e80ab86 100644 --- a/src/browser/shell.js +++ b/src/browser/shell.js @@ -833,7 +833,7 @@ try { (state.phase === 'pausing' || (state.pauseOnFailure && status === 'failed')) ) await paused(); - if (reported.workRemaining) await parkApp(runner.id); + if (reported.workRemaining || settings.releaseFinal) await parkApp(runner.id); } let failed = state.tests.filter((test) => test.status === 'failed').length; let skipped = state.tests.filter((test) => test.status === 'skipped').length; diff --git a/src/cli-output.js b/src/cli-output.js index 5af1613..de7fc82 100644 --- a/src/cli-output.js +++ b/src/cli-output.js @@ -65,6 +65,24 @@ export function createCLIOutput({ json, stdout = process.stdout, stderr = proces break; case 'fatal': log(`Browser run failed: ${event.error?.message ?? 'Unknown error'}`); break; case 'watch': log(`Watch: ${event.message ?? event.status}`); break; + case 'repeat-start': + line(`Repeating ${event.requested} rounds; browser storage and project worker resources are reused.`); + if (event.seed !== null) line(`Root seed: ${event.seed}`); + line(`Reproduce: ${event.command}`); + break; + case 'repeat-round': line(`\nRound ${event.round}/${event.requested}`); break; + case 'repeat-complete': + line(`\nRepeated runs: ${event.status} · ${event.completed}/${event.requested} rounds completed · ${event.failedRounds} with failures`); + for (let test of event.tests) + line(`${testTitle(test)}: ${test.passed} passed, ${test.failed} failed, ${test.skipped} skipped, ${test.notRun} not run`); + if (event.firstFailure) { + line(`First failure: round ${event.firstFailure.round} · ${event.firstFailure.error?.message ?? 'Run interrupted'}`); + if (event.firstFailureReplay) line(`Replay this round: ${event.firstFailureReplay}`); + line(`Rerun through failure: ${event.prefixCommand}`); + } + if (event.reason) log(`Repeated runs stopped: ${event.reason}`); + line(`Repeated-run report: ${event.html}`); + break; case 'report-saved': line(`Report: ${event.html}`); break; case 'browser-stopped': if (!event.graceful) log('Browser required forced cleanup. See session.json for timings.'); diff --git a/src/cli-run.js b/src/cli-run.js index 2d8f1b3..c32ee97 100644 --- a/src/cli-run.js +++ b/src/cli-run.js @@ -1,6 +1,8 @@ import { randomUUID } from 'node:crypto'; import { pathToFileURL } from 'node:url'; import { networkInterfaces } from 'node:os'; +import { join } from 'node:path'; +import { rm } from 'node:fs/promises'; import { configOverrides } from './cli.js'; import { loadConfig } from './config.js'; import { buildSuite } from './build.js'; @@ -15,6 +17,8 @@ import { createWatchSession } from './watch-session.js'; import { createReportWriter } from './reports.js'; import { createTimings } from './browser/timings.js'; import { readReplay } from './schedule.js'; +import { createRepeat, repeatSeed, acceptRepeatRound, finishRepeat, repeatCommand } from './repeat.js'; +import { createRepeatWriter } from './repeat-reports.js'; async function duringStartup(signal, start) { let controller = new AbortController(); @@ -63,9 +67,15 @@ export async function runTests(options, { signal, output }) { let browserPlan = options.manual ? null : await measure('Resolve browser', () => resolveSession(config.browser, { headless: !options.ui })); let cleanups = createCleanup(); let cleanup = (name, work) => cleanups.add(() => sessionTimings.measure('lifecycle', name, work)); - let { promise: finished, resolve: complete, reject: failed } = Promise.withResolvers(); - finished.catch(() => {}); + let pending = Promise.withResolvers(); + pending.promise.catch(() => {}); + let failed = error => { + pending.reject(error); + controller.abort(error); + }; let endingError; + let finalizationErrors = []; + let batch, batchWriter, batchReason; let watcher, coordinator; let instances = []; let launchKeys = []; @@ -74,6 +84,22 @@ export async function runTests(options, { signal, output }) { }); let savedRuns = 0; let saveRun = async (run) => { + if (batch && coordinator.isPrimary(run.runId)) { + let next = acceptRepeatRound(batch, run); + if (next === batch) return; + batch = next; + let result = await batchWriter.save(batch, run); + // Portable failure/final reports own retained images. Successful + // intermediate captures must not accumulate outside the evidence budget. + for (let image of run.outcomes.flatMap(outcome => outcome.attachments ?? [])) + if (image.status === 'captured') await rm(image.path, { force: true }); + if (result.saved) { savedRuns++; output.event({ type: 'report-saved', ...result.saved }); } + if (result.limited) { + batchReason = 'evidence-limit'; + throw new Error('Repeated-run evidence reached 256 MiB. Accepted counts were saved; no further rounds will start.'); + } + return; + } let result = await writer.save(run, sessionSnapshot('running')); savedRuns++; output.event({ type: 'report-saved', ...result }); @@ -87,11 +113,11 @@ export async function runTests(options, { signal, output }) { if (coordinator?.isPrimary(event.runId)) watcher?.observe(event); output.event(event, { persistent }); if (coordinator?.isPrimary(event.runId) && !persistent) { - if (event.type === 'complete') complete(event); + if (event.type === 'complete') pending.resolve(event); if (event.type === 'fatal') failed(new Error(event.error?.message ?? 'Browser runtime failed.')); } }; - let aborted = () => failed(runSignal.reason); + let aborted = () => pending.reject(runSignal.reason); runSignal.addEventListener('abort', aborted, { once: true }); if (runSignal.aborted) aborted(); try { @@ -180,7 +206,7 @@ export async function runTests(options, { signal, output }) { onDrain: project.drain ? context => sessionTimings.measure('lifecycle', ('Project cleanup: ' + [...context.test.suites, context.test.name].join(' › ')).slice(0, 200), () => project.drain(context), { signal: context.signal }) : undefined, - interactive: Boolean(options.ui || options.manual), onEvent: report, + interactive: Boolean(options.ui || options.manual), releaseFinal: options.repeat !== undefined, onEvent: report, })); let instance = { id, gateway, browser, launchKey, config: project.config }; instances = [...instances, instance]; @@ -225,6 +251,18 @@ export async function runTests(options, { signal, output }) { let browserInfo = browserPlan ? { ...first.browser.identity, path: browserPlan.path, isolated: browserPlan.isolated } : null; + if (options.repeat !== undefined) { + batch = createRepeat({ id: sessionId, count: options.repeat, seed: config.seed, source: bundle.fingerprint, + workers: config.workers, replay: Boolean(replay), browser: browserInfo, + selection: Object.freeze(replay ? { ids: replay.selected } : + { test: options.test, ids: options.id, files: options.file, suites: options.suite }), + command: repeatCommand(options, { seed: config.seed, count: options.repeat, workers: config.workers }) }); + batchWriter = await createRepeatWriter(join(config.reportDir, 'repeat-' + sessionId), { + redact: value => launchKeys.reduce((text, key) => text.replaceAll(key, '[private launch]'), value) + }); + output.event({ type: 'repeat-start', batchId: batch.id, requested: batch.requested, seed: batch.seed, + resources: batch.resources, command: batch.command, browser: browserInfo }); + } output.event({ type: 'ready', url: captureURL(first).href, appURL: first.config.webServer.url, workers: instances.map(instance => ({ id: instance.id, url: captureURL(instance).href, appURL: instance.config.webServer.url })), @@ -238,23 +276,34 @@ export async function runTests(options, { signal, output }) { output.line(`Device: ${gateway.url.replace(/127\.0\.0\.1|\[::1\]/, entry.address)}`); } } - let deadline = persistent ? null : AbortSignal.timeout(config.runTimeout); let expired = () => { let message = `No complete browser result within ${config.runTimeout} ms. The browser may be disconnected, blocked, or stuck.`; let output = instances.map(instance => instance.browser?.logs?.trim()).filter(Boolean).join('\n'); controller.abort(new Error(output ? `${message}\nBrowser output:\n${output}` : message)); }; - deadline?.addEventListener('abort', expired, { once: true }); - try { - if (browserPlan) await measure('Open browsers', () => Promise.all(instances.map(instance => { - let url = captureURL(instance); - url.hash = new URLSearchParams({ launch: instance.launchKey }).toString(); - return instance.browser.open(url.href); - }))); - let summary = await sessionTimings.measure('lifecycle', 'Browser results', () => finished, { signal: runSignal }); - exitCode = summary.failed ? 1 : 0; - } finally { - deadline?.removeEventListener('abort', expired); + let previous; + for (let number = 1; number <= (options.repeat ?? 1); number++) { + runSignal.throwIfAborted(); + if (number > 1) { + pending = Promise.withResolvers(); + pending.promise.catch(() => {}); + } + let deadline = persistent ? null : AbortSignal.timeout(config.runTimeout); + deadline?.addEventListener('abort', expired, { once: true }); + try { + if (batch) output.event({ type: 'repeat-round', batchId: batch.id, round: number, + requested: batch.requested, seed: repeatSeed(batch, number) }); + if (number > 1) await coordinator.advance(previous.runId, repeatSeed(batch, number)); + else if (browserPlan) await measure('Open browsers', () => Promise.all(instances.map(instance => { + let url = captureURL(instance); + url.hash = new URLSearchParams({ launch: instance.launchKey }).toString(); + return instance.browser.open(url.href); + }))); + previous = await sessionTimings.measure('lifecycle', 'Browser results', () => pending.promise, { signal: runSignal }); + if (previous.failed) exitCode = 1; + } finally { + deadline?.removeEventListener('abort', expired); + } } } catch (error) { let reason = runSignal.aborted ? runSignal.reason : error; @@ -264,18 +313,37 @@ export async function runTests(options, { signal, output }) { controller.abort(new Error('Dyno is shutting down.')); for (let error of await sessionTimings.measure('lifecycle', 'Cleanup', () => cleanups.run())) { output.event({ type: 'error', phase: 'cleanup', message: error.message }); + finalizationErrors.push(error.message); exitCode = 1; } let session = sessionSnapshot(exitCode ? 'failed' : signal.aborted ? 'interrupted' : endingError ? 'failed' : 'completed', sessionTimings.end()); try { await writer.finishSession(session); } catch (error) { exitCode = 1; + finalizationErrors.push(error.message); session = Object.freeze({ ...session, status: 'failed' }); output.event({ type: 'error', phase: 'report', message: error.message }); } - output.event({ type: 'session-timing', ...session }, { savedRuns }); + if (batchWriter) { + let reason = [batchReason ?? (signal.aborted ? signal.reason.message : endingError?.message), ...finalizationErrors] + .filter(Boolean).join(' ') || undefined; + batch = finishRepeat(batch, reason); + try { + let prefixCommand = batch.firstFailure ? repeatCommand(options, { seed: batch.seed, + count: batch.firstFailure.round, workers: config.workers }) : undefined; + let saved = await batchWriter.finish(batch, session, { prefixCommand }); + output.event({ type: 'repeat-complete', ...batch, ...saved, + ...(prefixCommand ? { prefixCommand } : {}) }); + } catch (error) { + exitCode = 1; + finalizationErrors.push(error.message); + output.event({ type: 'error', phase: 'report', message: error.message }); + } + } + output.event({ type: 'session-timing', ...session }, { savedRuns: batchWriter ? 0 : savedRuns }); runSignal.removeEventListener('abort', aborted); } + if (signal.aborted && !finalizationErrors.length) signal.throwIfAborted(); if (endingError && !signal.aborted) throw endingError; if (exitCode) return exitCode; signal.throwIfAborted(); diff --git a/src/cli.js b/src/cli.js index 3d91cf3..1858ab7 100644 --- a/src/cli.js +++ b/src/cli.js @@ -27,6 +27,27 @@ * * The CLI also prints a separate reporter URL. Open it in another tab if you'd rather keep the app uncovered. A completed interactive run keeps each worker's last app available; rerun an earlier test to inspect it again. * + * ## Repeat while developing + * + * Exercise a workflow repeatedly, or vary the order of a whole selection: + * + * ```sh + * dyno --test 'checkout' --repeat 100 + * dyno --suite Checkout --repeat 100 --random + * dyno --suite Checkout --repeat 100 --seed checkout-race + * dyno --repeat 100 --replay .dyno/reports//schedule.json + * ``` + * + * `--repeat N` runs the complete selection N times, including the first run. It supports managed headless runs; combining it with `--ui`, `--manual`, or `--watch` fails before startup. Tests keep their normal hooks and per-execution fixture preparation. Setup runs once, and browser storage, servers, and project worker resources survive between rounds. Keep scenario reset and backend cleanup in your project. + * + * Each automatic execution removes its test runtime and parks the app before project draining, including the final test. A round must finish cleanup and save its results before the next starts. Each round gets its own `runTimeout`. A failed test stays in the batch result while later rounds continue; cleanup, browser, preparation, persistence, or deadline failures stop repetition. Ctrl-C saves accepted partial results and stops further rounds. + * + * Without a seed, rounds use declared priority. With `--seed`, round one uses that literal seed and later rounds derive repeatable seeds from it. `--random` generates and prints a root seed; pass it to `--seed` to reproduce the series. The count does not affect seed derivation, so rerunning through the first failing round preserves the prefix. Repeated `--replay` keeps the recorded worker queues fixed. + * + * The summary prints commands to replay the first failing round and rerun the original selection through it. A focused rerun can remove the predecessor that caused a failure. A round-only replay also omits state accumulated in earlier rounds. Seeds reproduce priority, and schedules reproduce queues; neither controls app randomness, backend data, or asynchronous timing. Keep app code and data stable while collecting results. Dyno pins the test bundle, not the running application. + * + * See [saved reports](guide.md#saved-reports) for batch counts, schedules, images, and evidence limits. Passing many times is evidence under those conditions; it does not prove a test can never fail. + * * ## Bring your own browser * * ```sh @@ -64,6 +85,11 @@ function workers(value) { throw new InvalidArgumentError('must be a whole number between 1 and 32'); return Number(value); } +function repeat(value) { + if (!/^\d+$/.test(value) || !Number.isSafeInteger(Number(value)) || Number(value) < 1) + throw new InvalidArgumentError('must be a positive safe integer'); + return Number(value); +} function appOptions(command) { return command .addOption(valueOption('-c, --config ', 'Configuration file (dyno.config.js)')) @@ -102,6 +128,8 @@ Tests, actions, and assertions execute in the browser. Use --manual for Safari a .map(([key, value]) => [key, Array.isArray(value) ? Object.freeze([...value]) : value]))); }); appOptions(command('run', 'Run tests (the default command)', { isDefault: true })) + .addOption(new Option('--repeat ', 'Repeat the complete selection; seeded rounds vary reproducibly (headless only)') + .argParser(repeat).conflicts(['ui', 'manual', 'watch'])) .addOption(new Option('--workers ', 'Isolated browser workers (1); concurrent runs require project worker setup').argParser(workers)) .addOption(valueOption('--replay ', 'Replay recorded worker queues from schedule.json').conflicts(['seed', 'random', 'test', 'id', 'file', 'suite', 'watch'])) .addOption(valueOption('--seed ', 'Run in a repeatable randomized order')) diff --git a/src/coordinator.js b/src/coordinator.js index 776a477..689ba92 100644 --- a/src/coordinator.js +++ b/src/coordinator.js @@ -240,7 +240,7 @@ export function createCoordinator(config, bundle, { }); return Object.freeze(Object.fromEntries(entries)); }; - let create = (snapshot, ids, managed, selected, runId = randomUUID()) => { + let create = (snapshot, ids, managed, selected, runId = randomUUID(), seed = config.seed) => { if (config.replay && JSON.stringify(snapshot.fingerprint) !== JSON.stringify(config.replay.source)) throw new Error('The replay source fingerprint changed. Record a new schedule for these tests.'); while (runs.size >= 20) { @@ -250,7 +250,7 @@ export function createCoordinator(config, bundle, { for (let map of [runs, bundles, receipts, drains, preparations, progress, views, versions, retirements]) map.delete(finished.runId); } if (config.replay) selected = selection({ ids: config.replay.selected }); - let run = Object.freeze({ ...createRun(runId, { seed: config.seed, workers: ids, replay: config.replay }), + let run = Object.freeze({ ...createRun(runId, { seed, workers: ids, replay: config.replay }), managed, selection: selected, source: snapshot.fingerprint, startedAt: new Date().toISOString() }); bundles.set(runId, snapshot); if (managed) { @@ -266,7 +266,7 @@ export function createCoordinator(config, bundle, { publish(run); return run; }; - let restart = (runId, selected) => { + let restart = (runId, selected, seed = config.seed) => { requireOpen(); if (restarting) throw problem(409, 'This browser run is already restarting.'); if (runId !== managedRun) throw problem(409, 'This browser run was replaced.'); @@ -281,9 +281,10 @@ export function createCoordinator(config, bundle, { requireOpen(); let snapshot = rebuild ? await rebuild() : bundle; requireOpen(); - let run = create(snapshot, workerIds, true, selected); + let run = create(snapshot, workerIds, true, selected, randomUUID(), seed); primaryRun = run.runId; await Promise.all(run.workers.map(worker => workers.get(worker.id).lifecycle.open())); + return run.runId; })().catch(async error => { try { await serialize(runId, () => stopRun(runs.get(runId), { phase: 'runtime', message: error.message })); @@ -295,6 +296,14 @@ export function createCoordinator(config, bundle, { }; return Object.freeze({ register, requireOpen, serialize, + async advance(runId, seed) { + // The completion event fires inside serialization. Let its request settle + // before replacing the run and reopening its workers. + await serialize(runId, () => { + if (runs.get(runId)?.status !== 'completed') throw problem(409, 'Only a completed run can advance.'); + }); + return restart(runId, runs.get(runId).selection, seed); + }, lifecycle(workerId, lifecycle) { workers.set(workerId, { ...workers.get(workerId), lifecycle }); }, diff --git a/src/repeat-reports.js b/src/repeat-reports.js new file mode 100644 index 0000000..49256cb --- /dev/null +++ b/src/repeat-reports.js @@ -0,0 +1,142 @@ +import { mkdir, writeFile, appendFile, rename, rm, open } from 'node:fs/promises'; +import { join } from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { createReport } from './report-model.js'; +import { createReportWriter } from './reports.js'; +import { escapeMarkup } from './report-format.js'; + +export let repeatEvidenceLimit = 256 * 1024 * 1024; + +async function replaceFile(path, contents) { + let temporary = path + '.' + randomUUID(); + try { await writeFile(temporary, contents); await rename(temporary, path); } + finally { await rm(temporary, { force: true }); } +} + +function overview(batch) { + let x = escapeMarkup; + let rows = batch.tests.map(test => `${x([...test.suites, test.name].join(' › '))}${test.passed}${test.failed}${test.skipped}${test.notRun}`).join(''); + let failure = batch.firstFailure; + return `Dyno repeated runs + +

Repeated runs: ${x(batch.status)}

${batch.completed}/${batch.requested} rounds completed · ${batch.failedRounds} with failures

+

Browser storage and project worker resources were reused. Seeds reproduce priority; schedules reproduce worker queues, not asynchronous timing. The source fingerprint covers the test bundle, not the running app or backend.

+${batch.reason ? `

Stopped: ${x(batch.reason)}

` : ''} +

Root seed: ${x(batch.seed ?? 'declared order')}

+

Observed executions

${rows}
TestPassedFailedSkippedNot run
+${failure ? `

First failure: round ${failure.round}

${x(failure.error?.message)}

Run: ${x(failure.runId)}. Previous tests on this worker:

${x((failure.predecessors ?? []).join('\n') || 'None recorded')}

Rerun the original selection through this round to exercise state accumulated across earlier rounds.

` : ''} +${batch.firstFailureReplay ? `

Open the failed run

${x(batch.firstFailureReplay)}
` : ''} +${batch.prefixCommand ? `

Rerun through the first failure:

${x(batch.prefixCommand)}
` : ''} +${batch.status === 'passed' ? `

Open the final run

` : ''} +${batch.evidenceUnavailable ? `

${x(batch.evidenceUnavailable)}

` : ''} +

Reproduce the batch

${x(batch.command)}

Run this from the original project directory with the same app, backend data and environment.

+

Round outcomes and schedules · Batch metadata · JUnit results

\n`; +} + +let cases = entry => entry.tests.map(test => { + let x = escapeMarkup; + let result = test.status === 'failed' + ? `${x(test.error?.stack ?? test.error?.message)}` + : ['skipped', 'not-run'].includes(test.status) ? '' : ''; + return `${result}\n`; +}).join(''); + +export async function createRepeatWriter(directory, { redact = value => value, limit = repeatEvidenceLimit } = {}) { + await mkdir(directory, { recursive: true }); + let ledger = join(directory, 'rounds.jsonl'); + await writeFile(ledger, ''); + let writer = createReportWriter(join(directory, 'runs'), { redact }); + let used = 0; + let limited = false; + let firstFailureReplay; + return Object.freeze({ + async save(batch, run) { + let report = createReport(run); + let entry = { round: batch.rounds, runId: run.runId, seed: run.seed, status: report.status, + summary: report.summary, + tests: report.tests.map(({ id, name, file, suites, status, error, workerId }) => + ({ id, name, file, suites, status, ...(error ? { error } : {}), workerId })) }; + let retain = report.status === 'interrupted' || report.summary.failed > 0 || batch.rounds === batch.requested; + if (retain) entry.report = `runs/${run.runId}/index.html`; + entry.schedule = `schedules/${run.runId}.json`; + entry.replay = `dyno --replay '${join(directory, entry.schedule).replaceAll("'", "'\\''")}'`; + let schedule = redact(JSON.stringify(report.schedule) + '\n'); + let line = redact(JSON.stringify(entry) + '\n'); + let lineBytes = Buffer.byteLength(line) + Buffer.byteLength(schedule); + let saved; + if (used + lineBytes > limit) limited = true; + if (retain && !limited) { + try { + // Batch session timing lives once at the batch root. Per-run HTML + // and images keep the ordinary portable report format. + saved = await writer.save(run, undefined, { maxBytes: limit - used - lineBytes }); + used += saved.bytes; + } catch (error) { + if (error.code !== 'DYNO_EVIDENCE_LIMIT') throw error; + limited = true; + } + } + if (limited) { + // Always preserve accepted counts in batch.json, even when a report + // cannot fit. Never claim the missing evidence was saved. + delete entry.schedule; + delete entry.report; + delete entry.replay; + entry = { ...entry, tests: entry.tests.map(({ error, ...test }) => ({ ...test, + ...(error ? { error: { message: error.message, phase: error.phase } } : {}) })), + evidenceUnavailable: 'The batch evidence limit was reached.' }; + } + line = redact(JSON.stringify(entry) + '\n'); + if (!limited) { + await mkdir(join(directory, 'schedules'), { recursive: true }); + await writeFile(join(directory, entry.schedule), schedule); + used += Buffer.byteLength(line) + Buffer.byteLength(schedule); + if (saved && batch.firstFailure?.runId === run.runId) firstFailureReplay = entry.replay; + } + // One final compact outcome record is stop metadata, outside the budget, + // so accepted failures still have matching JUnit cases at the limit. + await appendFile(ledger, line); + await replaceFile(join(directory, 'batch.json'), redact(JSON.stringify({ ...batch, evidenceBytes: used, + evidenceLimit: limit, ...(limited ? { evidenceUnavailable: entry.evidenceUnavailable } : {}) }, null, 2) + '\n')); + return { saved, limited, evidenceBytes: used }; + }, + async finish(batch, session, { prefixCommand } = {}) { + let metadata = { ...batch, evidenceBytes: used, evidenceLimit: limit, + ...(prefixCommand ? { prefixCommand } : {}), ...(firstFailureReplay ? { firstFailureReplay } : {}), + ...(limited ? { evidenceUnavailable: 'The batch evidence limit was reached.' } : {}) }; + await replaceFile(join(directory, 'batch.json'), redact(JSON.stringify(metadata, null, 2) + '\n')); + await replaceFile(join(directory, 'session.json'), redact(JSON.stringify(session, null, 2) + '\n')); + await replaceFile(join(directory, 'index.html'), redact(overview(metadata))); + let temporary = join(directory, '.junit-' + randomUUID()); + let input; + let destination; + try { + input = await open(ledger); + destination = await open(temporary, 'w'); + let total = { tests: 0, failures: 0, skipped: 0 }; + for await (let line of input.readLines()) { + for (let test of JSON.parse(line).tests) { + total.tests++; + if (test.status === 'failed') total.failures++; + if (['skipped', 'not-run'].includes(test.status)) total.skipped++; + } + } + await input.close(); + input = await open(ledger); + let errors = Number(batch.status === 'interrupted'); + let counts = `tests="${total.tests + errors}" failures="${total.failures}" skipped="${total.skipped}" errors="${errors}"`; + await destination.write(`\n\n`); + for await (let line of input.readLines()) await destination.write(cases(JSON.parse(line))); + if (batch.status === 'interrupted') await destination.write(`\n`); + await destination.write('\n'); + await destination.close(); destination = undefined; + await rename(temporary, join(directory, 'junit.xml')); + } finally { + await input?.close(); await destination?.close(); + await rm(temporary, { force: true }); + } + return { directory, json: join(directory, 'batch.json'), html: join(directory, 'index.html'), + ledger, junit: join(directory, 'junit.xml'), ...(firstFailureReplay ? { firstFailureReplay } : {}) }; + } + }); +} diff --git a/src/repeat.js b/src/repeat.js new file mode 100644 index 0000000..69cf29b --- /dev/null +++ b/src/repeat.js @@ -0,0 +1,67 @@ +import { createHash } from 'node:crypto'; + +let freeze = value => Object.freeze(value); + +export function createRepeat({ id, count, seed, source, workers, selection, command, browser, replay = false }) { + if (!Number.isSafeInteger(count) || count < 1) throw new Error('Repeat count must be a positive safe integer.'); + return freeze({ schemaVersion: 1, id, requested: count, seed, source, workers, selection, command, browser, + algorithm: 'dyno-repeat-v1', replay, resources: 'reused', status: 'running', rounds: 0, + completed: 0, failedRounds: 0, tests: freeze([]), firstFailure: null, lastRunId: null }); +} + +export function repeatSeed(batch, number) { + if (batch.replay || batch.seed === null || number === 1) return batch.seed; + return createHash('sha256').update(JSON.stringify([batch.algorithm, batch.seed, number])).digest('hex'); +} + +export function acceptRepeatRound(batch, run) { + if (run.runId === batch.lastRunId) return batch; + if (batch.status !== 'running' || batch.rounds >= batch.requested) throw new Error('The repeated run has ended.'); + let number = batch.rounds + 1; + if (JSON.stringify(run.source) !== JSON.stringify(batch.source)) throw new Error('The repeated test source changed.'); + if (run.seed !== repeatSeed(batch, number)) throw new Error('The run does not match this repeat round.'); + if (batch.tests.length && JSON.stringify(run.tests.map(test => test.id)) !== JSON.stringify(batch.tests.map(test => test.id))) + throw new Error('The test selection changed between repeated runs.'); + let previousTests = new Map(batch.tests.map(test => [test.id, test])); + let outcomes = new Map(run.outcomes.map(outcome => [outcome.testId, outcome])); + let tests = run.tests.map(test => { + let previous = previousTests.get(test.id) ?? { id: test.id, file: test.file, suites: test.suites, name: test.name, + passed: 0, failed: 0, skipped: 0, notRun: 0 }; + let outcome = outcomes.get(test.id); + let status = outcome?.status ?? 'notRun'; + return freeze({ ...previous, [status]: previous[status] + 1 }); + }); + let failure = run.outcomes.find(outcome => outcome.status === 'failed'); + let failed = Boolean(failure || run.status !== 'completed'); + let position = failure ? run.assignments.findIndex(item => item.id === failure.executionId) : -1; + let firstFailure = batch.firstFailure; + if (!firstFailure && failed) firstFailure = freeze({ round: number, runId: run.runId, seed: run.seed, + ...(failure ? { testId: failure.testId, name: failure.name, workerId: failure.workerId, + predecessors: freeze(run.assignments.slice(0, position).filter(item => item.workerId === failure.workerId) + .map(item => item.testId)) } : {}), + error: failure?.error ?? run.error }); + return freeze({ ...batch, rounds: number, completed: batch.completed + Number(run.status === 'completed'), + failedRounds: batch.failedRounds + Number(failed), tests: freeze(tests), firstFailure, + lastRunId: run.runId }); +} + +export function finishRepeat(batch, reason) { + let status = batch.failedRounds ? 'failed' : 'passed'; + if (reason) status = 'interrupted'; + return freeze({ ...batch, status, + ...(reason ? { reason } : {}) }); +} + +let quote = value => `'${String(value).replaceAll("'", "'\\''")}'`; + +// Commands are display text. They are never executed by Dyno. +export function repeatCommand(options, { seed, count, workers }) { + let args = ['dyno']; + for (let key of ['config', 'browser', 'browserPath', 'test', 'replay', 'host', 'port']) + if (options[key] !== undefined) args.push('--' + key.replace(/[A-Z]/g, value => '-' + value.toLowerCase()), quote(options[key])); + for (let key of ['id', 'file', 'suite']) + for (let value of options[key] ?? []) args.push('--' + key, quote(value)); + args.push('--workers', String(workers), '--repeat', String(count)); + if (seed !== null && !options.replay) args.push('--seed', quote(seed)); + return args.join(' '); +} diff --git a/src/reports.js b/src/reports.js index e9b1d0e..f089e9c 100644 --- a/src/reports.js +++ b/src/reports.js @@ -10,12 +10,20 @@ * * Runs are `completed` or `interrupted`; tests without accepted results are `not-run`. Failures retain their phase and source context. JUnit adds an error case for interruption, so an unfinished run can't look green. * - * Dyno saves results before reporting completion, including accepted outcomes on signals, browser loss, and app failure. A force kill or machine failure can't flush results. Storage errors fail the CLI, and partially written folders aren't published as reports. A source image removed before saving appears as **Screenshot unavailable**. + * Dyno saves results before reporting completion, including accepted outcomes on signals, browser loss, and app failure. A force kill or machine failure can't flush results. Storage errors fail the CLI, and partially written run folders aren't published as reports. A source image removed before saving appears as **Screenshot unavailable**. * * Reports contain test output and source excerpts. Dyno doesn't save config, fixtures, connection keys, or launch tokens in them. + * + * ## Repeated-run reports + * + * `--repeat` writes `.dyno/reports/repeat-/`. Open its `index.html` for aggregate counts and the first failure. `batch.json` records the root seed, test-bundle fingerprint, browser identity, original selection, and observed outcomes. Skipped and unfinished tests stay separate from passes. The batch's `junit.xml` gives each test execution its round number, so a later pass cannot replace an earlier failure. + * + * `rounds.jsonl` records completed and interrupted rounds, per-test outcomes, and paths to their worker schedules under `schedules/`. Full portable reports and images are retained under `runs/` for failed or interrupted rounds and the final round. Successful intermediate rounds keep compact evidence; their screenshots are discarded. Session timing lives once in the batch folder. + * + * Round evidence and retained reports share a 256 MiB budget. Reaching it stops repetition, marks unavailable evidence explicitly, and preserves accepted counts in `batch.json`. The aggregate summary, final compact outcome record, and session timing are saved outside that budget so a limit cannot hide a failure or stop reason. Batch metadata updates after each saved round; HTML and JUnit are finalized on shutdown. Storage failures also fail the command. Copy the whole batch folder to keep its relative report and image links; printed commands refer to the original local paths. */ -import { mkdir, mkdtemp, readFile, writeFile, copyFile, rename, rm } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, writeFile, copyFile, rename, rm, stat } from 'node:fs/promises'; import { randomUUID } from 'node:crypto'; import { join } from 'node:path'; import { createReport, missingImages } from './report-model.js'; @@ -24,7 +32,7 @@ import { junitReport, htmlReport } from './report-format.js'; export function createReportWriter(directory, { redact = (value) => value } = {}) { let writes = new WeakMap(); let saved = new Set(); - let save = async (run, session) => { + let save = async (run, session, { maxBytes = Infinity } = {}) => { let report = createReport(run); let staging; let target = join(directory, run.runId); @@ -36,33 +44,50 @@ export function createReportWriter(directory, { redact = (value) => value } = {} .filter((image) => image.status === 'captured'); if (images.length) await mkdir(join(staging, 'images')); let missing = []; + let imageBytes = 0; for (let image of images) { + try { imageBytes += (await stat(image.path)).size; } + catch (error) { if (error.code === 'ENOENT') missing.push(image.id); else throw error; } + } + if (missing.length) report = missingImages(report, missing); + let styles = await Promise.all( + ['reporter.css', 'saved-report.css'].map(file => readFile(new URL(`./browser/${file}`, import.meta.url), 'utf8')) + ); + let css = styles.join('\n'); + let reportFiles = () => [ + ['schedule.json', JSON.stringify(report.schedule, null, 2) + '\n'], + ['run.json', JSON.stringify(report, null, 2) + '\n'], + ['junit.xml', junitReport(report)], + ['index.html', htmlReport(report, css, { session: Boolean(session) })], + ...(session ? [['session.json', JSON.stringify(session, null, 2) + '\n']] : []) + ].map(([name, contents]) => [name, redact(contents)]); + let files = reportFiles(); + let bytes = imageBytes + files.reduce((total, [, contents]) => total + Buffer.byteLength(contents), 0); + if (bytes > maxBytes) throw Object.assign(new Error('Repeated-run evidence reached its storage limit.'), { code: 'DYNO_EVIDENCE_LIMIT' }); + let copiedBytes = 0; + for (let image of images) { + if (missing.includes(image.id)) continue; try { await copyFile(image.path, join(staging, 'images', `${image.id}.png`)); + copiedBytes += (await stat(join(staging, 'images', `${image.id}.png`))).size; } catch (error) { if (error.code === 'ENOENT') missing = [...missing, image.id]; else throw error; } } - if (missing.length) report = missingImages(report, missing); - let styles = await Promise.all( - ['reporter.css', 'saved-report.css'].map((file) => - readFile(new URL(`./browser/${file}`, import.meta.url), 'utf8') - ) - ); - let css = styles.join('\n'); - await Promise.all([ - writeFile(join(staging, 'schedule.json'), redact(JSON.stringify(report.schedule, null, 2) + '\n')), - writeFile(join(staging, 'run.json'), redact(JSON.stringify(report, null, 2) + '\n')), - writeFile(join(staging, 'junit.xml'), redact(junitReport(report))), - writeFile(join(staging, 'index.html'), redact(htmlReport(report, css, { session: Boolean(session) }))), - ...(session ? [writeFile(join(staging, 'session.json'), redact(JSON.stringify(session, null, 2) + '\n'))] : []) - ]); + if (missing.length) { + report = missingImages(report, missing); + files = reportFiles(); + } + bytes = copiedBytes + files.reduce((total, [, contents]) => total + Buffer.byteLength(contents), 0); + if (bytes > maxBytes) throw Object.assign(new Error('Repeated-run evidence reached its storage limit.'), { code: 'DYNO_EVIDENCE_LIMIT' }); + await Promise.all(files.map(([name, contents]) => writeFile(join(staging, name), contents))); await rename(staging, target); if (session) saved.add(target); return Object.freeze({ ...(session ? { session: join(target, 'session.json') } : {}), runId: run.runId, + bytes, directory: target, json: join(target, 'run.json'), schedule: join(target, 'schedule.json'), @@ -82,7 +107,8 @@ export function createReportWriter(directory, { redact = (value) => value } = {} [error, cleanup], `Could not save Dyno report: ${error.message}. Temporary report cleanup also failed: ${cleanup.message}` ); - throw new Error(`Could not save Dyno report: ${error.message}`, { cause: error }); + throw Object.assign(new Error(`Could not save Dyno report: ${error.message}`, { cause: error }), + error.code === 'DYNO_EVIDENCE_LIMIT' ? { code: error.code } : {}); } }; return Object.freeze({ @@ -95,11 +121,11 @@ export function createReportWriter(directory, { redact = (value) => value } = {} } finally { await rm(temporary, { force: true }); } } }, - save(run, session) { + save(run, session, options) { if (!writes.has(run)) writes.set( run, - save(run, session).catch((error) => { + save(run, session, options).catch((error) => { writes.delete(run); throw error; }) diff --git a/src/server.js b/src/server.js index fc298dc..74014bb 100644 --- a/src/server.js +++ b/src/server.js @@ -43,6 +43,7 @@ export async function createGateway( beforeExecution, rebuild, interactive = false, + releaseFinal = false, coordinator, workerId = '0', } = {}, @@ -97,6 +98,7 @@ export async function createGateway( files: bundle.files, fixtures, interactive, + releaseFinal, appURL: config.webServer.url, viewport: config.viewport, managedViewport: defaultViewport, @@ -220,14 +222,18 @@ export async function createGateway( ); } if (route === 'bridge.js') return send('text/javascript', bundle.bridge); - if (route === 'settings') + if (route === 'settings') { + let run = coordinator.get(url.searchParams.get('run')); return send( 'application/json', JSON.stringify({ ...settings, + ...(run ? { seed: run.seed } : {}), + releaseFinal: releaseFinal && Boolean(run?.managed), files: coordinator.bundle(url.searchParams.get('run'))?.files ?? bundle.files, }), ); + } if (route === 'results' && request.method === 'GET') return send('application/json', JSON.stringify(coordinator.values())); if (['assignment', 'released'].includes(route) && request.method === 'POST') { diff --git a/tests/core/cli.test.js b/tests/core/cli.test.js index 91e1968..d2f2643 100644 --- a/tests/core/cli.test.js +++ b/tests/core/cli.test.js @@ -67,6 +67,12 @@ test('CLI mistakes are structured errors and never partially initialize a projec [['--workers', '0'], /between 1 and 32/], [['--workers', '1.5'], /between 1 and 32/], [['--workers', '33'], /between 1 and 32/], + [['--repeat', '0'], /positive safe integer/], + [['--repeat', '1.5'], /positive safe integer/], + [['--repeat', '9007199254740992'], /positive safe integer/], + [['--repeat', '2', '--ui'], /cannot be used/], + [['--repeat', '2', '--manual'], /cannot be used/], + [['--repeat', '2', '--watch'], /cannot be used/], [['--browser', 'netscape'], /Allowed choices/], [['--manual', '--browser', 'firefox'], /cannot be used/], [['--test', ' '], /non-empty/], diff --git a/tests/core/repeat.test.js b/tests/core/repeat.test.js new file mode 100644 index 0000000..68ac9d7 --- /dev/null +++ b/tests/core/repeat.test.js @@ -0,0 +1,79 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { createRepeat, repeatSeed, acceptRepeatRound, finishRepeat, repeatCommand } from '../../src/repeat.js'; +import { createRepeatWriter } from '../../src/repeat-reports.js'; +import { createRun, acceptEvent, claimTest, releaseTest } from '../../src/run.js'; +import { testId } from '../../src/suite.js'; + +let source = { algorithm: 'sha256-source-content-v1', hash: 'a'.repeat(64) }; +let descriptor = { id: testId('checkout.test.js', ['Shop'], 'checkout'), name: 'checkout', file: 'checkout.test.js', suites: ['Shop'], index: 0 }; +let batch = count => createRepeat({ id: 'batch', count, seed: 'root', source, workers: 1, selection: {}, command: 'dyno --repeat 3 --seed root' }); +function run(number, seed, { fail = false, interrupt = false } = {}) { + let value = { ...createRun('run-' + number, { seed }), source }; + value = acceptEvent(value, { runId: value.runId, type: 'start', tests: [descriptor] }).run; + value = claimTest(value, { executionId: 'execution-' + number }).run; + if (!interrupt) { + value = acceptEvent(value, { type: 'test', runId: value.runId, executionId: 'execution-' + number, + testId: descriptor.id, index: 0, name: descriptor.name, status: fail ? 'failed' : 'passed', + ...(fail ? { error: { message: 'Expected confirmation', phase: 'test' } } : {}) }).run; + value = releaseTest(value, { executionId: 'execution-' + number }); + } + return acceptEvent(value, { runId: value.runId, type: interrupt ? 'fatal' : 'complete', + ...(interrupt ? { error: { message: 'Lost browser', phase: 'runtime' } } : {}) }).run; +} + +test('repeat counts preserve failures, partial work and source identity without counting a delivered run twice', () => { + let state = batch(3); + let first = run(1, repeatSeed(state, 1), { fail: true }); + state = acceptRepeatRound(state, first); + assert.equal(acceptRepeatRound(state, first), state); + state = acceptRepeatRound(state, run(2, repeatSeed(state, 2))); + state = acceptRepeatRound(state, run(3, repeatSeed(state, 3), { interrupt: true })); + let result = finishRepeat(state, 'Lost browser'); + assert.equal(result.completed, 2); + assert.equal(result.tests[0].passed, 1); + assert.equal(result.tests[0].failed, 1); + assert.equal(result.tests[0].notRun, 1); + assert.equal(result.firstFailure.round, 1); + assert.equal(result.status, 'interrupted'); + assert.throws(() => acceptRepeatRound(result, run(4, 'root')), /ended/); + assert.throws(() => acceptRepeatRound(batch(3), { ...first, source: {} }), /source changed/); +}); + +test('round seed series preserve single-run input and prefixes, while replay repeats a fixed seed', () => { + let short = batch(2), long = batch(100); + assert.equal(repeatSeed(short, 1), 'root'); + assert.equal(repeatSeed(short, 2), repeatSeed(long, 2)); + assert.notEqual(repeatSeed(short, 1), repeatSeed(short, 2)); + assert.equal(repeatSeed({ ...long, replay: true }, 2), 'root'); + assert.equal(repeatSeed({ ...long, seed: null }, 2), null); + let command = repeatCommand({ config: "a'b/config.js", suite: ['Shop'] }, { seed: 'root', count: 2, workers: 4 }); + assert.ok(command.includes("--config 'a'\\''b/config.js'")); + assert.ok(command.includes("--suite 'Shop' --workers 4 --repeat 2 --seed 'root'")); +}); + +test('the evidence budget preserves accepted counts and publishes an explicit interruption without partial reports', async t => { + let directory = await mkdtemp(join(tmpdir(), 'dyno-repeat-report-')); + t.after(() => rm(directory, { recursive: true, force: true })); + let writer = await createRepeatWriter(directory, { limit: 1 }); + let state = batch(1); + let first = run(1, 'root', { fail: true }); + state = acceptRepeatRound(state, first); + let saved = await writer.save(state, first); + assert.equal(saved.limited, true); + assert.equal(saved.saved, undefined); + let paths = await writer.finish(finishRepeat(state, 'evidence-limit'), {}); + let report = JSON.parse(await readFile(paths.json, 'utf8')); + assert.equal(report.tests[0].failed, 1); + assert.equal(report.reason, 'evidence-limit'); + assert.match(report.evidenceUnavailable, /evidence limit/); + assert.match(await readFile(paths.junit, 'utf8'), / name.startsWith('.')), false); +}); diff --git a/tests/execution/repeat.test.js b/tests/execution/repeat.test.js new file mode 100644 index 0000000..f6eb4ab --- /dev/null +++ b/tests/execution/repeat.test.js @@ -0,0 +1,266 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFile, readdir } from 'node:fs/promises'; +import { join } from 'node:path'; +import { project } from '../helpers/project.js'; + +let json = async path => JSON.parse(await readFile(path, 'utf8')); +async function evidence(result) { + let event = result.events.find(event => event.type === 'repeat-complete'); + assert.ok(event, result.stdout + result.stderr); + let batch = await json(event.json); + let rounds = (await readFile(event.ledger, 'utf8')).trim().split('\n').filter(Boolean).map(line => JSON.parse(line)); + let schedules = await Promise.all(rounds.map(round => json(join(event.directory, round.schedule)))); + return { event, batch, rounds, schedules }; +} + +test('repeated selections reuse resources, prepare every execution and retain evidence beyond live history', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `import {appendFile} from 'node:fs/promises'; + export default async function setup({config}) { + let log=phase=>appendFile(config.root+'/lifecycle',phase+'\\n');await log('setup');let execution=0; + return {beforeExecution:()=>({fixtures:{execution:++execution}}),drain:()=>log('drain'),teardown:()=>log('teardown')}; + }` }, + source: `describe('Repeated workflow',()=>{ + test('opens the app',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible(); + let visits=Number(localStorage.getItem('visits')||0)+1;localStorage.setItem('visits',String(visits)); + await I.assert(()=>visits===fixtures.execution,'Storage and execution fixtures must survive each round'); + console.log('Visit '+visits); + }); + test('unselected failure',()=>{throw new Error('Selection was lost')}); + });` + }); + let result = await fixture.run(['--test', 'opens the app', '--repeat', '22']).completion; + assert.equal(result.code, 0, result.stdout + result.stderr); + let { event, batch, rounds } = await evidence(result); + assert.equal(batch.status, 'passed'); + assert.equal(batch.completed, 22); + assert.equal(batch.tests.length, 1); + assert.equal(batch.tests[0].passed, 22); + assert.equal(rounds.length, 22); + assert.equal(new Set(rounds.map(round => round.runId)).size, 22); + let lifecycle = (await readFile(join(fixture.directory, 'lifecycle'), 'utf8')).trim().split('\n'); + assert.equal(lifecycle.filter(item => item === 'setup').length, 1); + assert.equal(lifecycle.filter(item => item === 'drain').length, 22); + assert.equal(lifecycle.at(-1), 'teardown'); + let full = result.events.filter(event => event.type === 'report-saved'); + assert.equal(full.length, 1); + let report = await json(full[0].json); + assert.ok(report.tests[0].output.some(entry => entry.text.includes('Visit 22'))); + assert.equal((await readdir(join(event.directory, 'runs'))).length, 1); +}); + +test('a later passing round preserves the first failure, image and aggregate JUnit result', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default ()=>{let execution=0;return { + beforeExecution:()=>({fixtures:{execution:++execution}}) + }};` }, + source: `describe('Checkout',()=>{test('confirms the order',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible(); + await app.screenshot('Order'); + await I.assert(()=>fixtures.execution>1,'Order was not confirmed'); + });});` + }); + let result = await fixture.run(['--repeat', '3', '--seed', 'checkout-race']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { event, batch, rounds } = await evidence(result); + assert.equal(batch.status, 'failed'); + assert.equal(batch.completed, 3); + assert.equal(batch.failedRounds, 1); + assert.equal(batch.tests[0].failed, 1); + assert.equal(batch.tests[0].passed, 2); + assert.equal(batch.firstFailure.round, 1); + assert.match(event.prefixCommand, /--repeat 1 --seed 'checkout-race'/); + let failed = await json(join(event.directory, 'runs', rounds[0].runId, 'run.json')); + let image = failed.tests[0].attachments[0]; + assert.equal(image.status, 'captured'); + assert.ok((await readFile(join(event.directory, 'runs', rounds[0].runId, image.path))).length > 0); + assert.equal((await readdir(join(fixture.directory, '.dyno/artifacts'))).filter(name => name.endsWith('.png')).length, 0); + let junit = await readFile(event.junit, 'utf8'); + assert.equal((junit.match(/ event.type === 'report-saved').length, 2); +}); + +test('a generated root reproduces varying round priorities and queue replay stays fixed', { timeout: 90000 }, async t => { + let fixture = await project(t, { + source: `describe('Ordering',()=>{for(let name of ['Alpha','Bravo','Charlie','Delta']) + test(name,async({app,I})=>{await app.goto('/');await I.find('Fixture app').visible()});});` + }); + let first = await fixture.run(['--repeat', '4', '--random']).completion; + assert.equal(first.code, 0, first.stdout + first.stderr); + let original = await evidence(first); + assert.equal(new Set(original.schedules.map(schedule => schedule.seed)).size, 4); + let second = await fixture.run(['--repeat', '4', '--seed', original.batch.seed]).completion; + assert.equal(second.code, 0, second.stdout + second.stderr); + let reproduced = await evidence(second); + assert.deepEqual(reproduced.schedules.map(schedule => [schedule.seed, schedule.priority]), + original.schedules.map(schedule => [schedule.seed, schedule.priority])); + let prefix = await fixture.run(['--repeat', '1', '--seed', original.batch.seed]).completion; + assert.equal(prefix.code, 0, prefix.stdout + prefix.stderr); + assert.deepEqual((await evidence(prefix)).schedules[0].priority, original.schedules[0].priority); + let replay = await fixture.run(['--repeat', '2', '--replay', join(original.event.directory, original.rounds[1].schedule)]).completion; + assert.equal(replay.code, 0, replay.stdout + replay.stderr); + for (let schedule of (await evidence(replay)).schedules) + assert.deepEqual(schedule.assignments.map(item => [item.workerId, item.testId]), + original.schedules[1].assignments.map(item => [item.workerId, item.testId])); +}); + +test('interruption preserves accepted outcomes and admits no later round', { timeout: 90000 }, async t => { + let fixture = await project(t, { + source: `describe('Interrupted repetition',()=>{test('finishes',async({app,I})=>{ + await app.goto('/');await I.find('Fixture app').visible(); + });test('waits on a real request',async({app})=>{ + await app.request('/api/browser-waiting',{method:'POST'});await app.request('/api/release-browser'); + });});` + }); + let session = fixture.run(['--repeat', '10']); + let ready = await session.waitForEvent('ready'); + await fetch(new URL('/api/browser-waiting', ready.appURL), { signal: t.signal }); + let result = await session.stop(); + assert.equal(result.code, 143, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.status, 'interrupted'); + assert.equal(batch.completed, 0); + assert.equal(batch.tests.find(test => test.name === 'finishes').passed, 1); + assert.equal(batch.tests.find(test => test.name === 'waits on a real request').notRun, 1); + assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 1); +}); + +test('each automatic final document retires before project drain and the next round', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js', cleanupTimeout: 2000, + webServer: { command: [process.execPath, 'app.js'], ready: 'RETIRE_READY' } }, + files: { + 'setup.js': `import {appendFile} from 'node:fs/promises';export default({config})=>({drain:async({signal})=>{ + let response=await fetch(config.webServer.url+'/drain',{signal}); + await appendFile(config.root+'/drains',await response.text()+'\\n'); + }});`, + 'app.js': `import {createServer} from 'node:http'; + let entered=Promise.withResolvers(),retired=Promise.withResolvers(),held,visits=0,writes=0; + let server=createServer(async(request,response)=>{ + if(request.url==='/held'){held=response;response.once('close',()=>retired.resolve());entered.resolve();} + else if(request.url==='/entered'){await entered.promise;response.end('entered');} + else if(request.url==='/late-write'){writes++;response.end('written');} + else if(request.url==='/drain'){await retired.promise;held.end('released');response.end(JSON.stringify({retired:true,writes}));} + else if(request.url==='/state'){response.setHeader('Content-Type','application/json');response.end(JSON.stringify({writes}));} + else {response.setHeader('Content-Type','text/html');response.end('

Retirement app

'+(request.url==='/workflow'&&++visits===1?'':''));} + });server.listen(Number(process.env.PORT),'127.0.0.1',()=>console.log('RETIRE_READY'));` + }, + source: `describe('Safe reuse',()=>{test('leaves no document work behind',async({app,I})=>{ + await app.goto('/workflow');await I.find('Retirement app').visible();await app.request('/entered'); + let state=await(await app.request('/state')).json();await I.assert(()=>state.writes===0); + });});` + }); + let result = await fixture.run(['--repeat', '2']).completion; + assert.equal(result.code, 0, result.stdout + result.stderr); + let drains = (await readFile(join(fixture.directory, 'drains'), 'utf8')).trim().split('\n').map(line => JSON.parse(line)); + assert.deepEqual(drains, [{ retired: true, writes: 0 }, { retired: true, writes: 0 }]); + assert.equal((await evidence(result)).batch.completed, 2); +}); + +test('a failed project drain stops repetition without replacing accepted test outcomes', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>({drain:()=>{throw new Error('Could not join project work')}});` } + }); + let result = await fixture.run(['--repeat', '5']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.status, 'interrupted'); + assert.equal(batch.tests[0].passed, 1); + assert.equal(batch.tests[0].failed, 0); + assert.match(batch.firstFailure.error.message, /Could not join project work/); + assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 1); +}); + +test('a later round gets its own result deadline and cannot admit another round after expiry', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js', runTimeout: 3000 }, + files: { 'setup.js': `export default()=>{let execution=0;return {beforeExecution:()=>({fixtures:{execution:++execution}})}};` }, + source: `describe('Round deadlines',()=>{test('opens the app',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible(); + if(fixtures.execution>1)await app.request('/api/release-browser'); + });});` + }); + let result = await fixture.run(['--repeat', '5']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.completed, 1); + assert.equal(batch.status, 'interrupted'); + assert.match(batch.reason, /No complete browser result within 3000 ms/); + assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 2); +}); + +test('private workers keep their assignment ownership and drains across repeated runs', { timeout: 90000 }, async t => { + let setup = await readFile(new URL('../fixtures/concurrent-setup.js', import.meta.url), 'utf8'); + let fixture = await project(t, { + config: { setup: './setup.js', workers: 2, webServer: { env: { GATE: 'false', OVERLAP: 'false' } } }, + files: { 'setup.js': setup }, + source: `describe('Private workers',()=>{for(let name of ['Alpha','Bravo','Charlie','Delta'])test(name,async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Worker '+fixtures.worker).visible(); + await I.find('Name').then.fill(name);await I.click('Save');await I.find('$([role=status])').text('Saved '+name); + });});` + }); + let result = await fixture.run(['--repeat', '3', '--seed', 'private-workers']).completion; + assert.equal(result.code, 0, result.stdout + result.stderr); + let { batch, schedules } = await evidence(result); + assert.equal(batch.completed, 3); + assert.ok(batch.tests.every(test => test.passed === 3)); + for (let schedule of schedules) { + assert.deepEqual(schedule.workers, ['0', '1']); + assert.equal(new Set(schedule.assignments.map(assignment => assignment.testId)).size, 4); + assert.ok(schedule.assignments.every(assignment => assignment.status === 'released')); + } + let workers = await json(join(fixture.directory, 'workers.json')); + assert.equal(workers.drains.length, 12); + assert.equal(new Set(workers.arrivals.map(arrival => arrival.worker)).size, 2); +}); + +test('teardown failure remains visible even when an earlier round already failed', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>{let execution=0;return { + beforeExecution:()=>({fixtures:{execution:++execution}}), + teardown:()=>{throw new Error('Could not release project resources')} + }};` }, + source: `describe('Final cleanup',()=>{test('opens the app',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible();await I.assert(()=>fixtures.execution>1,'First-round failure'); + });});` + }); + let result = await fixture.run(['--repeat', '2']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.completed, 2); + assert.equal(batch.failedRounds, 1); + assert.equal(batch.status, 'interrupted'); + assert.equal(batch.tests[0].failed, 1); + assert.match(batch.reason, /Could not release project resources/); +}); + +test('interrupting after a failed round preserves the signal exit and the earlier failure', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>{let execution=0;return {beforeExecution:()=>({fixtures:{execution:++execution}})}};` }, + source: `describe('Interrupted failure',()=>{test('retains its failure',async({app,I,fixtures})=>{ + if(fixtures.execution===1)throw new Error('First-round failure'); + await app.goto('/');await I.find('Fixture app').visible(); + await app.request('/api/browser-waiting',{method:'POST'});await app.request('/api/release-browser'); + });});` + }); + let session = fixture.run(['--repeat', '10']); + let ready = await session.waitForEvent('ready'); + await fetch(new URL('/api/browser-waiting', ready.appURL), { signal: t.signal }); + let result = await session.stop(); + assert.equal(result.code, 143, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.status, 'interrupted'); + assert.equal(batch.completed, 1); + assert.equal(batch.tests[0].failed, 1); + assert.equal(batch.tests[0].notRun, 1); + assert.equal(batch.firstFailure.round, 1); + assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 2); +}); diff --git a/tests/reporter/repeat.test.js b/tests/reporter/repeat.test.js new file mode 100644 index 0000000..7070f89 --- /dev/null +++ b/tests/reporter/repeat.test.js @@ -0,0 +1,32 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { pathToFileURL } from 'node:url'; +import { project } from '../helpers/project.js'; +import { browserEngine } from '../../src/browsers.js'; + +let browser = { name: process.env.DYNO_TEST_BROWSER ?? 'chromium' }; + +test('repeated-run HTML opens offline and links to the preserved failure after later passes', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>{let execution=0;return {beforeExecution:()=>({fixtures:{execution:++execution}})}};` }, + source: `describe('Checkout',()=>{test('confirms ',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible(); + await I.assert(()=>fixtures.execution>1,'Confirmation '); + });});` + }); + let result = await fixture.run(['--repeat', '2', '--seed', 'checkout']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let saved = result.events.find(event => event.type === 'repeat-complete'); + let native = await browserEngine(browser.name).launch({ headless: true }); + t.after(() => native.close()); + let page = await native.newPage(); + await page.goto(pathToFileURL(saved.html).href); + assert.equal(await page.getByRole('heading', { name: 'Repeated runs: failed' }).count(), 1); + assert.match(await page.locator('body').innerText(), /2\/2 rounds completed · 1 with failures/); + let row = page.getByRole('row').filter({ hasText: 'Checkout › confirms ' }); + assert.deepEqual(await row.getByRole('cell').allTextContents(), ['Checkout › confirms ', '1', '1', '0', '0']); + assert.match(await page.locator('body').innerText(), /Confirmation /); + await page.getByRole('link', { name: 'Open the failed run' }).click(); + assert.match(await page.locator('body').innerText(), /Confirmation /); +}); From 40a771bec3bf164af60de47fe16767d95c381bc7 Mon Sep 17 00:00:00 2001 From: Robert DeLuca Date: Thu, 12 Nov 2026 14:52:25 -0600 Subject: [PATCH 2/3] =?UTF-8?q?=F0=9F=90=9B=20Harden=20repeated-run=20repo?= =?UTF-8?q?rts=20and=20replay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Preserve explicit execution options in replay commands, retain accepted outcomes through artifact write failures, and publish terminal batch metadata last. Keep automatic batch replacement under one owner and give repeated browser workflows their own CI budget without dropping engine coverage. --- .github/actions/test/action.yml | 2 +- .github/workflows/ci.yml | 4 +- docs/ci.md | 7 ++- package.json | 1 + scripts/test.js | 4 +- src/cli-run.js | 2 + src/coordinator.js | 4 +- src/repeat-reports.js | 44 ++++++++------ src/repeat.js | 13 ++++- tests/core/repeat.test.js | 35 ++++++++++- tests/{execution => repeat}/repeat.test.js | 67 +++++++++++++++++++++- 11 files changed, 153 insertions(+), 30 deletions(-) rename tests/{execution => repeat}/repeat.test.js (79%) diff --git a/.github/actions/test/action.yml b/.github/actions/test/action.yml index 137c915..f8646d5 100644 --- a/.github/actions/test/action.yml +++ b/.github/actions/test/action.yml @@ -2,7 +2,7 @@ name: Run Dyno tests description: Run one suite and retain its console and JUnit output. inputs: suite: - description: core, browser, execution, project, reporter, package, capture, or native. + description: core, browser, execution, repeat, project, reporter, package, capture, or native. required: true browser: description: Playwright engine for browser suites. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5a49a7a..dc4a85b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -74,7 +74,7 @@ jobs: fail-fast: false matrix: engine: [chromium, firefox, webkit] - suite: [browser, execution, project, reporter, capture, native] + suite: [browser, execution, repeat, project, reporter, capture, native] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -95,7 +95,7 @@ jobs: fail-fast: false matrix: engine: [chromium, firefox, webkit] - suite: [browser, execution, project, reporter, capture, native] + suite: [browser, execution, repeat, project, reporter, capture, native] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: diff --git a/docs/ci.md b/docs/ci.md index a114d68..f817329 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -10,6 +10,7 @@ | Package | `npm run test:package` | Fresh tarball and checkout installs, init, and a real Chromium test on Node 26 | | Browser | `npm run test:browser -- firefox` | Real app, gateway, viewport, and process workflows in Chromium, Firefox, and WebKit on Linux and macOS, using Node 24 | | Execution | `npm run test:execution -- firefox` | Native concurrency, ownership, queue replay, seeded order, suite hooks, cleanup, pause, watch and replacement workflows in all three engines on Linux and macOS | +| Repeat | `npm run test:repeat -- firefox` | Repeated selections, seeded batches, resource reuse, cancellation and cleanup in all three engines on Linux and macOS | | Project | `npm run test:project -- firefox` | Project resource acquisition, fixture transport and per-execution preparation in all three engines on Linux and macOS | | Reporter | `npm run test:reporter -- firefox` | Inspection, output, timings, reruns, and portable saved reports in all three engines on Linux and macOS | | Native | `npm run test:native -- firefox` | Native defaults, capabilities, state restoration, dialogs, diagnostics, and cancellation in all three engines on Linux and macOS | @@ -22,11 +23,11 @@ node bin/dyno.js install --browser firefox npm run test:browser -- firefox ``` -Tests live under `tests/core`, `tests/package`, `tests/browser`, `tests/execution`, `tests/project`, `tests/reporter`, `tests/capture`, and `tests/native`. The runner discovers every `.test.js` file in the selected directory and rejects test files outside these suites. Keep browser-free checks in core and browser workflows in the relevant feature suite. Tests use real CLI, server, and Playwright browser boundaries; these groups only decide where they run. +Tests live under `tests/core`, `tests/package`, `tests/browser`, `tests/execution`, `tests/repeat`, `tests/project`, `tests/reporter`, `tests/capture`, and `tests/native`. The runner discovers every `.test.js` file in the selected directory and rejects test files outside these suites. Keep browser-free checks in core and browser workflows in the relevant feature suite. Tests use real CLI, server, and Playwright browser boundaries; these groups only decide where they run. -Core checks and actionlint workflow validation run first. Browser and package suites run in parallel once core passes. Documentation runs separately. Browser, execution, project, reporter, native, and capture suites have separate jobs so each complete suite has its own budget. Each test job has a hard five-minute cap, including setup. Browser and execution checks run up to three independent files together; each owns its app, gateway, browser profiles, and fixture directory. Tests inside a file stay sequential. Project, reporter, native, capture, and package suites stay sequential. The macOS process-group check runs once in core instead of repeating for each browser. There are no retries or extra sleeps. +Core checks and actionlint workflow validation run first. Browser and package suites run in parallel once core passes. Documentation runs separately. Browser, execution, repeat, project, reporter, native, and capture suites have separate jobs so each complete suite has its own budget. Each test job has a hard five-minute cap, including setup. Browser and execution checks run up to three independent files together; each owns its app, gateway, browser profiles, and fixture directory. Tests inside a file stay sequential. Repeat, project, reporter, native, capture, and package suites stay sequential. The macOS process-group check runs once in core instead of repeating for each browser. There are no retries or extra sleeps. -Execution and reporter checks have their own suites because the combined macOS Firefox browser suite reached 288–294 seconds. Project setup and preparation now have their own suite after execution exhausted the same cap. Every test and engine/OS combination remains covered. Product suites and benchmarks keep their configured execution; this grouping only changes Dyno's own CI. +Execution and reporter checks have their own suites because the combined macOS Firefox browser suite reached 288–294 seconds. Project setup and preparation now have their own suite after execution exhausted the same cap. Repeated runs have a separate suite after the added real-browser rounds pushed macOS Firefox execution past that cap. Every test and engine/OS combination remains covered. Product suites and benchmarks keep their configured execution; this grouping only changes Dyno's own CI. ## Results and setup diff --git a/package.json b/package.json index 1266932..758bd74 100644 --- a/package.json +++ b/package.json @@ -40,6 +40,7 @@ "test:core": "node scripts/test.js core", "test:browser": "node scripts/test.js browser", "test:execution": "node scripts/test.js execution", + "test:repeat": "node scripts/test.js repeat", "test:project": "node scripts/test.js project", "test:reporter": "node scripts/test.js reporter", "test:package": "node scripts/test.js package", diff --git a/scripts/test.js b/scripts/test.js index 20feab0..783abbb 100644 --- a/scripts/test.js +++ b/scripts/test.js @@ -7,11 +7,11 @@ import { fileURLToPath } from 'node:url'; import { browserEngine } from '../src/browsers.js'; let [suite = 'all', browser = 'chromium', ...extra] = process.argv.slice(2); -let directories = ['core', 'browser', 'execution', 'project', 'reporter', 'package', 'capture', 'native']; +let directories = ['core', 'browser', 'execution', 'repeat', 'project', 'reporter', 'package', 'capture', 'native']; let suites = ['all', ...directories]; let needsBrowser = suite === 'all' || (directories.includes(suite) && suite !== 'core'); if (!suites.includes(suite) || extra.length || (!needsBrowser && process.argv[3])) - throw new Error('Usage: node scripts/test.js [all|core|browser|execution|project|reporter|package|capture|native] [chromium|firefox|webkit for browser suites]'); + throw new Error('Usage: node scripts/test.js [all|core|browser|execution|repeat|project|reporter|package|capture|native] [chromium|firefox|webkit for browser suites]'); let root = fileURLToPath(new URL('../', import.meta.url)); let files = []; diff --git a/src/cli-run.js b/src/cli-run.js index c32ee97..7bf931a 100644 --- a/src/cli-run.js +++ b/src/cli-run.js @@ -170,6 +170,7 @@ export async function runTests(options, { signal, output }) { }); let workerIds = Array.from({ length: config.workers }, (_, index) => String(index)); coordinator = createCoordinator(config, bundle, { + automatic: options.repeat !== undefined, workerIds, onEvent: report, onRun: saveRun, onError: error => { output.event({ type: 'error', phase: error.phase ?? 'report', message: error.message }); @@ -258,6 +259,7 @@ export async function runTests(options, { signal, output }) { { test: options.test, ids: options.id, files: options.file, suites: options.suite }), command: repeatCommand(options, { seed: config.seed, count: options.repeat, workers: config.workers }) }); batchWriter = await createRepeatWriter(join(config.reportDir, 'repeat-' + sessionId), { + options, redact: value => launchKeys.reduce((text, key) => text.replaceAll(key, '[private launch]'), value) }); output.event({ type: 'repeat-start', batchId: batch.id, requested: batch.requested, seed: batch.seed, diff --git a/src/coordinator.js b/src/coordinator.js index 689ba92..668041f 100644 --- a/src/coordinator.js +++ b/src/coordinator.js @@ -12,7 +12,7 @@ let problem = (status, message) => Object.assign(new Error(message), { status }) // Network/process handles stay here. The run value remains the only source of // catalog, assignment, resource and outcome state. export function createCoordinator(config, bundle, { - workerIds = ['0'], onEvent = () => {}, onRun = async () => {}, onError = () => {}, rebuild, + workerIds = ['0'], onEvent = () => {}, onRun = async () => {}, onError = () => {}, rebuild, automatic = false, } = {}) { let runs = new Map(), bundles = new Map(), workers = new Map(); let queues = new Map(), receipts = new Map(), drains = new Map(), preparations = new Map(); @@ -399,6 +399,8 @@ export function createCoordinator(config, bundle, { command(input, selected) { let run = runs.get(input.runId); if (!run) throw problem(404, 'Unknown test run.'); + if (automatic && run.managed && ['rerun', 'reload'].includes(input.command)) + throw problem(409, 'Repeated runs control their own selection. Stop the CLI command before rerunning tests.'); if (input.command === 'stop') abortPreparations(run.runId, new Error('Test run stopped.')); if (input.command === 'rerun' && run.replay) { if (input.ids.length && (input.ids.length !== run.replay.selected.length || input.ids.some(id => !run.replay.selected.includes(id)))) diff --git a/src/repeat-reports.js b/src/repeat-reports.js index 49256cb..bcffc97 100644 --- a/src/repeat-reports.js +++ b/src/repeat-reports.js @@ -4,6 +4,7 @@ import { randomUUID } from 'node:crypto'; import { createReport } from './report-model.js'; import { createReportWriter } from './reports.js'; import { escapeMarkup } from './report-format.js'; +import { replayCommand } from './repeat.js'; export let repeatEvidenceLimit = 256 * 1024 * 1024; @@ -41,13 +42,14 @@ let cases = entry => entry.tests.map(test => { return `${result}\n`; }).join(''); -export async function createRepeatWriter(directory, { redact = value => value, limit = repeatEvidenceLimit } = {}) { +export async function createRepeatWriter(directory, { redact = value => value, limit = repeatEvidenceLimit, options = {} } = {}) { await mkdir(directory, { recursive: true }); let ledger = join(directory, 'rounds.jsonl'); await writeFile(ledger, ''); let writer = createReportWriter(join(directory, 'runs'), { redact }); let used = 0; let limited = false; + let unavailable; let firstFailureReplay; return Object.freeze({ async save(batch, run) { @@ -59,11 +61,12 @@ export async function createRepeatWriter(directory, { redact = value => value, l let retain = report.status === 'interrupted' || report.summary.failed > 0 || batch.rounds === batch.requested; if (retain) entry.report = `runs/${run.runId}/index.html`; entry.schedule = `schedules/${run.runId}.json`; - entry.replay = `dyno --replay '${join(directory, entry.schedule).replaceAll("'", "'\\''")}'`; + entry.replay = replayCommand(options, join(directory, entry.schedule)); let schedule = redact(JSON.stringify(report.schedule) + '\n'); let line = redact(JSON.stringify(entry) + '\n'); let lineBytes = Buffer.byteLength(line) + Buffer.byteLength(schedule); let saved; + let storageError; if (used + lineBytes > limit) limited = true; if (retain && !limited) { try { @@ -72,41 +75,45 @@ export async function createRepeatWriter(directory, { redact = value => value, l saved = await writer.save(run, undefined, { maxBytes: limit - used - lineBytes }); used += saved.bytes; } catch (error) { - if (error.code !== 'DYNO_EVIDENCE_LIMIT') throw error; - limited = true; + if (error.code === 'DYNO_EVIDENCE_LIMIT') limited = true; + else storageError = error; } } - if (limited) { + if (!limited && !storageError) { + try { + await mkdir(join(directory, 'schedules'), { recursive: true }); + await replaceFile(join(directory, entry.schedule), schedule); + used += Buffer.byteLength(schedule); + } catch (error) { storageError = error; } + } + if (limited || storageError) { // Always preserve accepted counts in batch.json, even when a report // cannot fit. Never claim the missing evidence was saved. delete entry.schedule; - delete entry.report; + if (!saved) delete entry.report; delete entry.replay; entry = { ...entry, tests: entry.tests.map(({ error, ...test }) => ({ ...test, ...(error ? { error: { message: error.message, phase: error.phase } } : {}) })), - evidenceUnavailable: 'The batch evidence limit was reached.' }; + evidenceUnavailable: storageError?.message ?? 'The batch evidence limit was reached.' }; + unavailable = entry.evidenceUnavailable; } line = redact(JSON.stringify(entry) + '\n'); - if (!limited) { - await mkdir(join(directory, 'schedules'), { recursive: true }); - await writeFile(join(directory, entry.schedule), schedule); - used += Buffer.byteLength(line) + Buffer.byteLength(schedule); + if (!limited && !storageError) { + used += Buffer.byteLength(line); if (saved && batch.firstFailure?.runId === run.runId) firstFailureReplay = entry.replay; } // One final compact outcome record is stop metadata, outside the budget, // so accepted failures still have matching JUnit cases at the limit. await appendFile(ledger, line); await replaceFile(join(directory, 'batch.json'), redact(JSON.stringify({ ...batch, evidenceBytes: used, - evidenceLimit: limit, ...(limited ? { evidenceUnavailable: entry.evidenceUnavailable } : {}) }, null, 2) + '\n')); + evidenceLimit: limit, ...(entry.evidenceUnavailable ? { evidenceUnavailable: entry.evidenceUnavailable } : {}) }, null, 2) + '\n')); + if (storageError) throw storageError; return { saved, limited, evidenceBytes: used }; }, async finish(batch, session, { prefixCommand } = {}) { let metadata = { ...batch, evidenceBytes: used, evidenceLimit: limit, ...(prefixCommand ? { prefixCommand } : {}), ...(firstFailureReplay ? { firstFailureReplay } : {}), - ...(limited ? { evidenceUnavailable: 'The batch evidence limit was reached.' } : {}) }; - await replaceFile(join(directory, 'batch.json'), redact(JSON.stringify(metadata, null, 2) + '\n')); - await replaceFile(join(directory, 'session.json'), redact(JSON.stringify(session, null, 2) + '\n')); - await replaceFile(join(directory, 'index.html'), redact(overview(metadata))); + ...(unavailable ? { evidenceUnavailable: unavailable } : {}) }; let temporary = join(directory, '.junit-' + randomUUID()); let input; let destination; @@ -135,6 +142,11 @@ export async function createRepeatWriter(directory, { redact = value => value, l await input?.close(); await destination?.close(); await rm(temporary, { force: true }); } + await replaceFile(join(directory, 'session.json'), redact(JSON.stringify(session, null, 2) + '\n')); + await replaceFile(join(directory, 'index.html'), redact(overview(metadata))); + // Terminal batch metadata is the commit marker: no successful status + // becomes durable until all final artifacts have been published. + await replaceFile(join(directory, 'batch.json'), redact(JSON.stringify(metadata, null, 2) + '\n')); return { directory, json: join(directory, 'batch.json'), html: join(directory, 'index.html'), ledger, junit: join(directory, 'junit.xml'), ...(firstFailureReplay ? { firstFailureReplay } : {}) }; } diff --git a/src/repeat.js b/src/repeat.js index 69cf29b..58c62b6 100644 --- a/src/repeat.js +++ b/src/repeat.js @@ -55,13 +55,22 @@ export function finishRepeat(batch, reason) { let quote = value => `'${String(value).replaceAll("'", "'\\''")}'`; // Commands are display text. They are never executed by Dyno. -export function repeatCommand(options, { seed, count, workers }) { +function runArguments(options) { let args = ['dyno']; - for (let key of ['config', 'browser', 'browserPath', 'test', 'replay', 'host', 'port']) + for (let key of ['config', 'browser', 'browserPath', 'test', 'replay', 'host', 'port', 'reportDir']) if (options[key] !== undefined) args.push('--' + key.replace(/[A-Z]/g, value => '-' + value.toLowerCase()), quote(options[key])); for (let key of ['id', 'file', 'suite']) for (let value of options[key] ?? []) args.push('--' + key, quote(value)); + return args; +} + +export function repeatCommand(options, { seed, count, workers }) { + let args = runArguments(options); args.push('--workers', String(workers), '--repeat', String(count)); if (seed !== null && !options.replay) args.push('--seed', quote(seed)); return args.join(' '); } + +export function replayCommand(options, path) { + return runArguments({ ...options, replay: path, test: undefined, id: [], file: [], suite: [] }).join(' '); +} diff --git a/tests/core/repeat.test.js b/tests/core/repeat.test.js index 68ac9d7..0926614 100644 --- a/tests/core/repeat.test.js +++ b/tests/core/repeat.test.js @@ -1,6 +1,6 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises'; +import { mkdtemp, readFile, readdir, rm, writeFile, mkdir } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { createRepeat, repeatSeed, acceptRepeatRound, finishRepeat, repeatCommand } from '../../src/repeat.js'; @@ -77,3 +77,36 @@ test('the evidence budget preserves accepted counts and publishes an explicit in assert.equal((junit.match(/ name.startsWith('.')), false); }); + +test('a schedule storage failure preserves the accepted round and its JUnit failure', async t => { + let directory = await mkdtemp(join(tmpdir(), 'dyno-repeat-storage-')); + t.after(() => rm(directory, { recursive: true, force: true })); + let writer = await createRepeatWriter(directory); + await writeFile(join(directory, 'schedules'), 'This file blocks the schedule directory.'); + let first = run(1, 'root', { fail: true }); + let state = acceptRepeatRound(batch(1), first); + await assert.rejects(writer.save(state, first), /schedules/); + let paths = await writer.finish(finishRepeat(state, 'Schedule storage failed'), {}); + let entry = JSON.parse((await readFile(paths.ledger, 'utf8')).trim()); + assert.equal(entry.tests[0].status, 'failed'); + assert.match(entry.tests[0].error.message, /Expected confirmation/); + assert.equal(entry.schedule, undefined); + let junit = await readFile(paths.junit, 'utf8'); + assert.match(junit, /failures="1"/); + assert.equal((junit.match(/ { + let directory = await mkdtemp(join(tmpdir(), 'dyno-repeat-finalize-')); + t.after(() => rm(directory, { recursive: true, force: true })); + let writer = await createRepeatWriter(directory); + let first = run(1, 'root'); + let state = acceptRepeatRound(batch(1), first); + await writer.save(state, first); + await mkdir(join(directory, 'junit.xml')); + await assert.rejects(writer.finish(finishRepeat(state), {})); + let persisted = JSON.parse(await readFile(join(directory, 'batch.json'), 'utf8')); + assert.equal(persisted.status, 'running'); + assert.equal(persisted.tests[0].passed, 1); + await assert.rejects(readFile(join(directory, 'index.html')), { code: 'ENOENT' }); +}); diff --git a/tests/execution/repeat.test.js b/tests/repeat/repeat.test.js similarity index 79% rename from tests/execution/repeat.test.js rename to tests/repeat/repeat.test.js index f6eb4ab..91a9793 100644 --- a/tests/execution/repeat.test.js +++ b/tests/repeat/repeat.test.js @@ -1,8 +1,8 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { readFile, readdir } from 'node:fs/promises'; +import { readFile, readdir, rename } from 'node:fs/promises'; import { join } from 'node:path'; -import { project } from '../helpers/project.js'; +import { project, launch, repository } from '../helpers/project.js'; let json = async path => JSON.parse(await readFile(path, 'utf8')); async function evidence(result) { @@ -264,3 +264,66 @@ test('interrupting after a failed round preserves the signal exit and the earlie assert.equal(batch.firstFailure.round, 1); assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 2); }); + +test('printed replay commands preserve explicit config, browser and report overrides', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>{let execution=0;return {beforeExecution:()=>({fixtures:{execution:++execution}})}};` }, + source: `describe('Replay context',()=>{test('fails on a fresh project',async({app,I,fixtures})=>{ + await app.goto('/');await I.find('Fixture app').visible();await I.assert(()=>fixtures.execution>1,'Fresh project failure'); + });});` + }); + let config = "custom run's.config.js"; + await rename(join(fixture.directory, 'dyno.config.js'), join(fixture.directory, config)); + let result = await fixture.run(['--config', config, '--browser', process.env.DYNO_TEST_BROWSER ?? 'chromium', + '--report-dir', 'custom reports', '--repeat', '2']).completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { event } = await evidence(result); + assert.ok(event.prefixCommand.includes("--report-dir 'custom reports'")); + let command = 'dyno() { exec "$DYNO_TEST_NODE" "$DYNO_TEST_CLI" "$@"; }\n' + event.firstFailureReplay + ' --json'; + let replay = launch('/bin/sh', ['-c', command], fixture.directory, t.signal, + { env: { ...process.env, DYNO_TEST_NODE: process.execPath, DYNO_TEST_CLI: join(repository, 'bin/dyno.js') } }); + t.after(() => replay.stop()); + let reproduced = await replay.completion; + assert.equal(reproduced.code, 1, reproduced.stdout + reproduced.stderr); + assert.equal(reproduced.summary?.failed, 1, reproduced.stdout + reproduced.stderr); + assert.match(reproduced.events.find(event => event.type === 'test').error.message, /Fresh project failure/); + assert.ok(reproduced.events.find(event => event.type === 'report-saved').directory.includes('custom reports')); +}); + +test('automatic batches reject reporter replacements and stop when project draining expires', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js', cleanupTimeout: 1000 }, + files: { 'setup.js': `export default({config})=>({drain:async({signal})=>{ + await fetch(config.webServer.url+'/api/release-browser',{signal}); + }});` } + }); + let session = fixture.run(['--repeat', '5']); + let ready = await session.waitForEvent('ready'); + let captured = await session.waitForEvent('capture'); + let replacement = await fetch(new URL('control', ready.url), { method: 'POST', signal: t.signal, + body: JSON.stringify({ runId: captured.runId, command: 'rerun', ids: [] }) }); + assert.equal(replacement.status, 409); + assert.match(await replacement.text(), /Repeated runs control their own selection/); + let result = await session.completion; + assert.equal(result.code, 1, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.tests[0].passed, 1); + assert.equal(batch.status, 'interrupted'); + assert.match(batch.reason, /cleanup exceeded 1000 ms/); + assert.equal(result.events.filter(event => event.type === 'repeat-round').length, 1); +}); + +test('all-skipped rounds preserve skip counts without executing hooks or project preparation', { timeout: 90000 }, async t => { + let fixture = await project(t, { + config: { setup: './setup.js' }, + files: { 'setup.js': `export default()=>({beforeExecution:()=>{throw new Error('Skipped preparation ran')},drain:()=>{throw new Error('Skipped cleanup ran')}});` }, + source: `describe('Skipped selection',()=>{beforeEach(()=>{throw new Error('Skipped hook ran')});test.skip('later');});` + }); + let result = await fixture.run(['--repeat', '3']).completion; + assert.equal(result.code, 0, result.stdout + result.stderr); + let { batch } = await evidence(result); + assert.equal(batch.completed, 3); + assert.equal(batch.tests[0].skipped, 3); + assert.equal(batch.tests[0].passed, 0); +}); From 417210983c57791433a4dec77848561a494f7971 Mon Sep 17 00:00:00 2001 From: Robert DeLuca Date: Thu, 12 Nov 2026 15:16:45 -0600 Subject: [PATCH 3/3] =?UTF-8?q?=F0=9F=93=9D=20Explain=20pnpm=20source=20de?= =?UTF-8?q?pendency=20requirements?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the source and build policies needed by the pinned Interactor fixes so projects can make that installation choice explicitly. --- docs/guide.md | 10 ++++++++++ src/init.js | 10 ++++++++++ 2 files changed, 20 insertions(+) diff --git a/docs/guide.md b/docs/guide.md index e58be31..2298b75 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -21,6 +21,16 @@ npx dyno install `init` adds a config and an example test without overwriting existing files. Commit your lockfile so local runs and CI use the same Dyno revision. +Dyno currently pins Interactor fixes that are not yet on the registry. pnpm 11 blocks Git subdependencies and their prepare builds by default. Projects that permit this Git source need the following in `pnpm-workspace.yaml`, alongside their existing build entries: + +```yaml +blockExoticSubdeps: false +allowBuilds: + interactor.js: true +``` + +`blockExoticSubdeps` changes the project's whole dependency-source policy; the build permission above only allows Interactor. See [pnpm's source policy](https://pnpm.io/settings#blockexoticsubdeps) before choosing that setting. + ### Point it at your app In `dyno.config.js`, set the command that starts your app and the URL it serves: diff --git a/src/init.js b/src/init.js index 83db36b..2a591e4 100644 --- a/src/init.js +++ b/src/init.js @@ -14,6 +14,16 @@ * * `init` adds a config and an example test without overwriting existing files. Commit your lockfile so local runs and CI use the same Dyno revision. * + * Dyno currently pins Interactor fixes that are not yet on the registry. pnpm 11 blocks Git subdependencies and their prepare builds by default. Projects that permit this Git source need the following in `pnpm-workspace.yaml`, alongside their existing build entries: + * + * ```yaml + * blockExoticSubdeps: false + * allowBuilds: + * interactor.js: true + * ``` + * + * `blockExoticSubdeps` changes the project's whole dependency-source policy; the build permission above only allows Interactor. See [pnpm's source policy](https://pnpm.io/settings#blockexoticsubdeps) before choosing that setting. + * * ## Point it at your app * * In `dyno.config.js`, set the command that starts your app and the URL it serves: