diff --git a/.changeset/config.json b/.changeset/config.json index f6700320..4f96e85b 100644 --- a/.changeset/config.json +++ b/.changeset/config.json @@ -9,6 +9,7 @@ "@cleverbrush/deep", "@cleverbrush/async", "@cleverbrush/scheduler", + "@cleverbrush/scheduler-postgres", "@cleverbrush/mapper", "@cleverbrush/knex-clickhouse", "@cleverbrush/react-form", diff --git a/.changeset/durable-job-execution.md b/.changeset/durable-job-execution.md new file mode 100644 index 00000000..8fac4c55 --- /dev/null +++ b/.changeset/durable-job-execution.md @@ -0,0 +1,28 @@ +--- +"@cleverbrush/scheduler": major +"@cleverbrush/scheduler-postgres": major +--- + +Redesign the scheduler for versioned immediate, delayed and recurring jobs, +typed separate handlers, durable ordered progress and fenced worker leases. +Add an independently installed PostgreSQL adapter using Framework ORM and +knex-schema, explicit migrations, transactional enqueue and restart recovery. + +Retries are opt-in and rerun whole handlers. Calendar triggers use explicit UTC +or IANA zones with persisted cursors and missed/overlap policies. See the +scheduler v4.x-to-v5 migration guide for the breaking API and rollout steps. +The adapter joins the fixed Framework release group. + +Retain schema-driven minute/day/week/month/year definitions and their +discriminated Schedule type, exposing individual schemas and the Schemas facade. +Normalize recurrence defaults, dates and weekday order before fingerprinting; +unchanged registrations retain their cursor and start anchor. Preserve the +one-based calculator index and accept the deprecated maxOccurences spelling +while rejecting ambiguous dual spelling. Name the explicit persistence option +storageRepository. Derive PostgreSQL row and entity types from schema definitions +without parallel hand-written row types. + +Use native Date/Intl calendar calculations without an additional date-time +runtime dependency. Preserve DST-gap skipping and earlier-fold selection, +including non-hour transitions and skipped calendar dates, independently of +the host time zone. Bound minute schedules to the representable Date range. diff --git a/.github/pr-evidence/durable-scheduler.png b/.github/pr-evidence/durable-scheduler.png new file mode 100644 index 00000000..85197c4e Binary files /dev/null and b/.github/pr-evidence/durable-scheduler.png differ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 54d00131..2819da0b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,6 +63,7 @@ jobs: --health-retries 10 env: QUERY_TEST_DATABASE_URL: postgres://framework_test:framework_test@127.0.0.1:5432/framework_queries + SCHEDULER_TEST_DATABASE_URL: postgres://framework_test:framework_test@127.0.0.1:5432/framework_queries steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 @@ -72,3 +73,6 @@ jobs: - run: npm ci - run: npm run build - run: npm run test:queries:integration + - run: npm run test:scheduler:integration + - run: node demos/durable-jobs/demo.ts + - run: node demos/durable-jobs/periodic.ts --fast diff --git a/AGENTS.md b/AGENTS.md index d93978be..c9ac2154 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,7 +58,8 @@ scripts/ ← build/release helper scripts | `@cleverbrush/async` | Async utilities: Collector, debounce, throttle, retry | | `@cleverbrush/mapper` | Schema-driven object mapper | | `@cleverbrush/react-form` | React form library powered by schema PropertyDescriptors | -| `@cleverbrush/scheduler` | Cron-like job scheduler with schema-validated config | +| `@cleverbrush/scheduler` | Typed durable jobs, recurring triggers and progress | +| `@cleverbrush/scheduler-postgres` | PostgreSQL job repository and explicit migrations | | `@cleverbrush/server` | Schema-first HTTP server: DI, auto-validation, RFC 9457 errors | | `@cleverbrush/server-openapi` | OpenAPI 3.x generation from server endpoints | | `@cleverbrush/client` | Type-safe HTTP client for `@cleverbrush/server` endpoints | diff --git a/README.md b/README.md index c8f8a1b7..60bdae01 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,8 @@ JSON Schema, API contracts, and Standard Schema integrations. | [`@cleverbrush/otel`](./libs/otel) | OpenTelemetry setup and instrumentation helpers for apps and clients. | | [`@cleverbrush/async`](./libs/async) | Async utilities including collector, debounce, throttle, and retry. | | [`@cleverbrush/deep`](./libs/deep) | Deep equality, deep extension, flattening, and object utilities. | -| [`@cleverbrush/scheduler`](./libs/scheduler) | Cron-like job scheduler with schema-validated job configuration. | +| [`@cleverbrush/scheduler`](./libs/scheduler) | Typed durable jobs, recurring triggers and ordered progress. | +| [`@cleverbrush/scheduler-postgres`](./libs/scheduler-postgres) | PostgreSQL job persistence, transactional enqueue and fenced leases. | ## How The Pieces Fit diff --git a/demos/durable-jobs/README.md b/demos/durable-jobs/README.md new file mode 100644 index 00000000..ee217e39 --- /dev/null +++ b/demos/durable-jobs/README.md @@ -0,0 +1,22 @@ +# Durable job contracts and separated handlers + +After npm ci and npm run build, run `node demos/durable-jobs/demo.ts` +with Node 24. It prints queued, running, progress and succeeded events, then +the typed result. This small example deliberately uses process-local memory. + +The core scheduler README shows PostgreSQL setup for real durability. +The sibling crash-worker.mjs and thread-handler.mjs are process-level fixtures +exercised by the scheduler integration and packaged-thread tests, respectively. + +## Periodic execution + +Run `node demos/durable-jobs/periodic.ts` to execute two reports one minute +apart. The example validates a minute schedule, registers it with +`upsertSchedule`, starts both the dispatcher and worker, waits for both +completed runs, and stops dispatch before draining the worker. + +For a fast smoke test, run `node demos/durable-jobs/periodic.ts --fast`. +Only this example's in-memory repository clock advances by one minute after +the first completion; the periodic dispatcher and worker still execute normally. +CI runs this mode. This is a test/demo clock, not a production scheduling option. +Choose PostgreSQL via `storageRepository` for restart-safe recurring jobs. diff --git a/demos/durable-jobs/contracts.ts b/demos/durable-jobs/contracts.ts new file mode 100644 index 00000000..05f993ee --- /dev/null +++ b/demos/durable-jobs/contracts.ts @@ -0,0 +1,9 @@ +import { object, string, number } from '@cleverbrush/schema'; +import { defineJob } from '@cleverbrush/scheduler'; + +export const Report = defineJob({ + name: 'report', version: 1, + input: object({ reportId: string() }), + progress: object({ percent: number() }), + output: object({ downloadUrl: string() }) +}); diff --git a/demos/durable-jobs/crash-worker.mjs b/demos/durable-jobs/crash-worker.mjs new file mode 100644 index 00000000..40044482 --- /dev/null +++ b/demos/durable-jobs/crash-worker.mjs @@ -0,0 +1,14 @@ +// Integration fixture: deliberately never finishes after committing progress. +import knex from 'knex'; +import { object, string, number } from '@cleverbrush/schema'; +import { defineJob, JobScheduler } from '@cleverbrush/scheduler'; +import { PostgresJobRepository } from '@cleverbrush/scheduler-postgres'; +const db = knex({ client: 'pg', connection: process.env.SCHEDULER_TEST_DATABASE_URL }); +const job = defineJob({ name: 'report', version: 1, input: object({ id: string() }), progress: object({ percent: number() }), output: object({ url: string() }), retry: { maxAttempts: 2, initialDelayMs: 1 } }); +const scheduler = new JobScheduler({ storageRepository: new PostgresJobRepository(db, { tablePrefix: process.env.JOB_TABLE_PREFIX }), namespace: process.env.JOB_NAMESPACE }); +const worker = scheduler.createWorker({ jobs: [job.handle(async (_, context) => { + await context.report({ percent: 50 }); + process.send?.({ ready: true }); + await new Promise(() => {}); +})], pollIntervalMs: 10, leaseMs: 500, heartbeatMs: 100 }); +await worker.start(); diff --git a/demos/durable-jobs/demo.ts b/demos/durable-jobs/demo.ts new file mode 100644 index 00000000..5622f915 --- /dev/null +++ b/demos/durable-jobs/demo.ts @@ -0,0 +1,16 @@ +import { JobScheduler, InMemoryJobRepository } from '@cleverbrush/scheduler'; +import { Report } from './contracts.ts'; +import { handleReport } from './handler.ts'; + +const jobs = new JobScheduler({ storageRepository: new InMemoryJobRepository(), pollIntervalMs: 10 }); +const worker = jobs.createWorker({ jobs: [Report.handle(handleReport)], pollIntervalMs: 10 }); +const run = await jobs.enqueue(Report, { reportId: 'quarterly' }); +await worker.start(); +try { + for await (const event of jobs.events(Report, run.id)) { + console.log(event.sequence, event.type, event.data); + } + const snapshot = await jobs.getRun(Report, run.id); + console.log(snapshot?.status, snapshot?.output); + if (snapshot?.status !== 'succeeded') process.exitCode = 1; +} finally { await worker.stop(); } diff --git a/demos/durable-jobs/handler.ts b/demos/durable-jobs/handler.ts new file mode 100644 index 00000000..042b4a99 --- /dev/null +++ b/demos/durable-jobs/handler.ts @@ -0,0 +1,9 @@ +import type { JobHandler } from '@cleverbrush/scheduler'; +import type { Report } from './contracts.ts'; + +/** Handler in a separate module retains all contract-inferred types. */ +export const handleReport: JobHandler = async (input, context) => { + context.signal.throwIfAborted(); + await context.report({ percent: 50 }); + return { downloadUrl: '/reports/' + input.reportId }; +}; diff --git a/demos/durable-jobs/periodic.ts b/demos/durable-jobs/periodic.ts new file mode 100644 index 00000000..7c08460e --- /dev/null +++ b/demos/durable-jobs/periodic.ts @@ -0,0 +1,42 @@ +import { setTimeout as delay } from 'node:timers/promises'; +import { InMemoryJobRepository, JobScheduler, ScheduleSchema } from '@cleverbrush/scheduler'; +import { Report } from './contracts.ts'; +import { handleReport } from './handler.ts'; + +// --fast advances only this demo's in-memory clock after the first completion. +const fast = process.argv.includes('--fast'); +let clock = Date.now(); +const jobs = new JobScheduler({ + storageRepository: new InMemoryJobRepository({ now: () => fast ? clock : Date.now() }), + pollIntervalMs: 10 +}); +const worker = jobs.createWorker({ jobs: [Report.handle(handleReport)], pollIntervalMs: 10 }); +await jobs.upsertSchedule('minute-report', Report, { reportId: 'periodic' }, { + schedule: ScheduleSchema.parse({ every: 'minute', interval: 1, maxOccurrences: 2 }), + missed: 'coalesce', overlap: 'skip' +}); + +try { + await worker.start(); // Executes jobs accepted by the dispatcher. + await jobs.start(); // Materializes due occurrences, not handler execution. + const deadline = Date.now() + (fast ? 5000 : 65000); + let advanced = false; + for (;;) { + const completed = (await jobs.health()).counts.succeeded ?? 0; + if (completed === 2) { + console.log('Completed both periodic reports.'); + break; + } + if (jobs.lastError) throw jobs.lastError; + if (worker.lastError) throw worker.lastError; + if (Date.now() > deadline) throw new Error('Periodic demo timed out'); + if (fast && completed === 1 && !advanced) { + clock += 60000; + advanced = true; + } + await delay(10); + } +} finally { + await jobs.stop(); // Stop producing before draining the worker. + await worker.stop(); +} diff --git a/demos/durable-jobs/thread-handler.mjs b/demos/durable-jobs/thread-handler.mjs new file mode 100644 index 00000000..21537f05 --- /dev/null +++ b/demos/durable-jobs/thread-handler.mjs @@ -0,0 +1,12 @@ +// Trusted default-export handler for a worker-thread execution. +export default async function handler(input, context) { + if (input.mode === 'exit') process.exit(7); + if (input.mode === 'hang') await new Promise(() => {}); + if (input.mode === 'invalid') { await context.report({ percent: 'wrong' }); } + if (input.mode === 'accessor') { + await context.report({ get percent() { throw new Error('must not execute'); } }); + } + if (input.mode === 'unawaited-invalid') void context.report({ percent: 'wrong' }); + await context.report({ percent: 50 }); + return { url: '/reports/' + input.id }; +} diff --git a/demos/todo-backend/Dockerfile b/demos/todo-backend/Dockerfile index 3f11b1bd..848534b4 100644 --- a/demos/todo-backend/Dockerfile +++ b/demos/todo-backend/Dockerfile @@ -21,6 +21,7 @@ COPY libs/react-form/package.json ./libs/react-form/ COPY libs/schema/package.json ./libs/schema/ COPY libs/schema-json/package.json ./libs/schema-json/ COPY libs/scheduler/package.json ./libs/scheduler/ +COPY libs/scheduler-postgres/package.json ./libs/scheduler-postgres/ COPY libs/server/package.json ./libs/server/ COPY libs/server-openapi/package.json ./libs/server-openapi/ COPY libs/otel/package.json ./libs/otel/ diff --git a/demos/todo-frontend/Dockerfile b/demos/todo-frontend/Dockerfile index 9fd23c3c..d0dc2f8c 100644 --- a/demos/todo-frontend/Dockerfile +++ b/demos/todo-frontend/Dockerfile @@ -20,6 +20,7 @@ COPY libs/react-form/package.json ./libs/react-form/ COPY libs/schema/package.json ./libs/schema/ COPY libs/schema-json/package.json ./libs/schema-json/ COPY libs/scheduler/package.json ./libs/scheduler/ +COPY libs/scheduler-postgres/package.json ./libs/scheduler-postgres/ COPY libs/server/package.json ./libs/server/ COPY libs/server-openapi/package.json ./libs/server-openapi/ COPY libs/client/package.json ./libs/client/ diff --git a/libs/scheduler-postgres/README.md b/libs/scheduler-postgres/README.md new file mode 100644 index 00000000..7131e87e --- /dev/null +++ b/libs/scheduler-postgres/README.md @@ -0,0 +1,91 @@ +# @cleverbrush/scheduler-postgres + +PostgreSQL persistence for [@cleverbrush/scheduler](../scheduler/README.md). +Uses @cleverbrush/orm and @cleverbrush/knex-schema for schemas, migrations and +routine queries. Isolated native queries provide database-time leases, +SKIP LOCKED row claims and aggregate health checks. + +## Install and migrate + +```sh +npm install @cleverbrush/scheduler @cleverbrush/scheduler-postgres knex pg +``` + +```ts +// An explicit migration, invoked once by your existing migration runner. +import { createSchedulerTables, dropSchedulerTables } from '@cleverbrush/scheduler-postgres'; +import type { Knex } from 'knex'; + +export const up = (knex: Knex) => createSchedulerTables(knex); +export const down = (knex: Knex) => dropSchedulerTables(knex); +``` + +Down is destructive: it deletes runs, schedules, progress and attempts. +No migration runs implicitly when constructing a repository or starting workers. +A tablePrefix option (default cb_jobs) supports separate storage ownership. +Use exactly the same prefix for migrations and repositories. + +```ts +import knex from 'knex'; +import { JobScheduler } from '@cleverbrush/scheduler'; +import { PostgresJobRepository } from '@cleverbrush/scheduler-postgres'; + +const database = knex({ + client: 'pg', connection: process.env.DATABASE_URL, + pool: { min: 0, max: 10 }, acquireConnectionTimeout: 5000 +}); +const jobs = new JobScheduler({ + storageRepository: new PostgresJobRepository(database), + namespace: 'reporting' +}); +``` + +The adapter does not close the caller's pool. Stop workers and dispatchers first, +then database.destroy(). Use PostgreSQL's default READ COMMITTED isolation. +Transitions set local lock and statement timeouts (5 and 15 seconds). Configure +connection acquisition and network timeouts for your deployment as well. + +## Transactional enqueue + +```ts +await database.transaction(async transaction => { + // Write application data using this same transaction. + const producer = new JobScheduler({ + storageRepository: new PostgresJobRepository(transaction), + namespace: 'reporting' + }); + await producer.enqueue(Report, { reportId }, { idempotencyKey: reportId }); +}); +``` + +The enqueue resolves within a savepoint; durable acceptance occurs only when the +outer transaction commits. Rollback removes both the application write and job. +Do not create/start workers on a transaction-bound repository. Local timeout +settings also apply to the enclosing transaction after savepoint release. + +## Storage and guarantees + +Four library-owned tables store runs, attempts, events and recurring triggers. +Indexed scalar columns support claims, expiry, overlap and health queries. +Opaque strict-JSON record snapshots are stored as text so arbitrary validated +payloads do not need application-specific database schemas or lossy driver +decoding. Large artifacts belong in object storage, referenced by payload keys. + +Unique namespace/dedupe keys protect concurrent producers. Locked schedule +cursors and occurrence keys protect competing dispatchers. State transitions, +attempt history and ordered events commit together. Cleanup cascades dependent +events and attempts, but never removes active jobs or triggers. + +The repository uses PostgreSQL clock_timestamp for lease decisions, not worker +wall clocks. Expired owners cannot resurrect a lease. Physical job side effects +remain at-least-once: see the core guide's retry/idempotency rules. + +Integration tests require a disposable PostgreSQL database. They create unique +table prefixes and drop only those tables: + +```sh +SCHEDULER_TEST_DATABASE_URL=postgres://... npm run test:scheduler:integration +``` + +Tests exercise competing producers/workers/dispatchers, rollback, SKIP LOCKED, +lease expiry, retention and SIGKILL/restart recovery. CI runs them on PostgreSQL 16. diff --git a/libs/scheduler-postgres/integration/repository.test.ts b/libs/scheduler-postgres/integration/repository.test.ts new file mode 100644 index 00000000..e99d497a --- /dev/null +++ b/libs/scheduler-postgres/integration/repository.test.ts @@ -0,0 +1,209 @@ +import { fork } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { once } from 'node:events'; +import { JobScheduler } from '@cleverbrush/scheduler'; +import knex from 'knex'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { + repositoryContract, + testJob +} from '../../scheduler/testing/repository-contract.js'; +import { + createSchedulerTables, + dropSchedulerTables, + PostgresJobRepository +} from '../src/index.js'; + +const connection = process.env.SCHEDULER_TEST_DATABASE_URL; +if (!connection) + throw new Error( + 'SCHEDULER_TEST_DATABASE_URL is required for integration tests' + ); +const db = knex({ + client: 'pg', + connection, + pool: { min: 0, max: 10 }, + acquireConnectionTimeout: 5000 +}); +const options = { + tablePrefix: 'test_jobs_' + randomUUID().replaceAll('-', '').slice(0, 30) +}; +beforeAll(() => createSchedulerTables(db, options)); +afterAll(async () => { + await dropSchedulerTables(db, options); + await db.destroy(); +}); +const repository = new PostgresJobRepository(db, options); +repositoryContract(async () => ({ + repository, + scheduler: new JobScheduler({ + storageRepository: repository, + namespace: randomUUID() + }), + advance: ms => new Promise(resolve => setTimeout(resolve, ms)) +})); + +describe('PostgreSQL durability', () => { + it('executes recurring runs and retains the cursor across scheduler/repository replacement', async () => { + const namespace = randomUUID(); + const first = new JobScheduler({ + storageRepository: repository, + namespace + }); + const job = testJob(); + const spec = { + schedule: { + every: 'minute' as const, + startsOn: new Date(Date.now() - 120000), + maxOccurences: 3 + }, + missed: 'replay' as const + }; + await first.upsertSchedule('periodic', job, { id: 'periodic' }, spec); + const replacement = new JobScheduler({ + storageRepository: new PostgresJobRepository(db, options), + namespace + }); + const counts = await Promise.all([ + first.dispatch(), + replacement.dispatch() + ]); + expect(counts.reduce((a, b) => a + b)).toBe(3); + const worker = replacement.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async (input, context) => { + await context.report({ percent: 100 }); + return { url: '/' + input.id }; + }) + ] + }); + try { + await worker.start(); + await expect + .poll(async () => (await replacement.health()).counts.succeeded) + .toBe(3); + } finally { + await worker.stop(); + } + const same = await replacement.upsertSchedule( + 'periodic', + job, + { id: 'periodic' }, + { + ...spec, + schedule: { + ...spec.schedule, + maxOccurences: undefined, + maxOccurrences: 3, + interval: 1, + skipFirst: 0, + timeZone: 'UTC' + } + } + ); + expect(same).toMatchObject({ revision: 1, cursor: 3, nextAt: null }); + expect(await replacement.dispatch()).toBe(0); + }); + it('rolls back an enqueue with the enclosing business transaction', async () => { + let id = ''; + await expect( + db.transaction(async tx => { + const scheduler = new JobScheduler({ + storageRepository: new PostgresJobRepository(tx, options) + }); + id = (await scheduler.enqueue(testJob(), { id: 'rollback' })) + .id; + throw new Error('business rollback'); + }) + ).rejects.toThrow('business rollback'); + expect(await repository.get('default', id)).toBeUndefined(); + }); + it('skips a row held by another transaction instead of blocking claims', async () => { + const scheduler = new JobScheduler({ + storageRepository: repository, + namespace: randomUUID() + }); + const first = await scheduler.enqueue(testJob(), { id: 'one' }); + await scheduler.enqueue(testJob(), { id: 'two' }); + const tx = await db.transaction(); + try { + await tx(options.tablePrefix + '_runs') + .where({ id: first.id }) + .forUpdate(); + const run = await repository.claim( + scheduler.namespace, + [testJob()], + 10000 + ); + expect(run?.id).not.toBe(first.id); + expect(run).toBeDefined(); + } finally { + await tx.rollback(); + } + }); + it('recovers after SIGKILL and preserves committed progress across processes', async () => { + const namespace = randomUUID(); + const scheduler = new JobScheduler({ + storageRepository: repository, + namespace + }); + const job = testJob({ retry: { maxAttempts: 2, initialDelayMs: 1 } }); + const run = await scheduler.enqueue(job, { id: 'restart' }); + const child = fork( + new URL( + '../../../demos/durable-jobs/crash-worker.mjs', + import.meta.url + ), + [], + { + env: { + ...process.env, + SCHEDULER_TEST_DATABASE_URL: connection, + JOB_TABLE_PREFIX: options.tablePrefix, + JOB_NAMESPACE: namespace + }, + stdio: ['ignore', 'ignore', 'pipe', 'ipc'] + } + ); + try { + await Promise.race([ + once(child, 'message'), + new Promise((_, reject) => { + const timer = setTimeout( + () => + reject( + new Error( + 'Crash worker did not report progress' + ) + ), + 8000 + ); + timer.unref(); + }) + ]); + child.kill('SIGKILL'); + await once(child, 'exit'); + expect( + (await repository.events(namespace, run.id, 0)).map( + event => event.type + ) + ).toEqual(['queued', 'running', 'progress']); + await new Promise(resolve => setTimeout(resolve, 650)); + await repository.claim(namespace, [job], 10000); + await new Promise(resolve => setTimeout(resolve, 5)); + const claim = (await repository.claim(namespace, [job], 10000))!; + expect(claim.attempt).toBe(2); + await repository.complete(namespace, run.id, claim.leaseToken!, { + url: '/after-restart' + }); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'succeeded', + output: { url: '/after-restart' } + }); + } finally { + if (child.exitCode === null && child.signalCode === null) + child.kill('SIGKILL'); + } + }); +}); diff --git a/libs/scheduler-postgres/package.json b/libs/scheduler-postgres/package.json new file mode 100644 index 00000000..4ffdac09 --- /dev/null +++ b/libs/scheduler-postgres/package.json @@ -0,0 +1,36 @@ +{ + "name": "@cleverbrush/scheduler-postgres", + "version": "4.4.3", + "description": "Transactional PostgreSQL repository for the Cleverbrush durable scheduler", + "type": "module", + "sideEffects": false, + "license": "BSD-3-Clause", + "files": [ + "dist" + ], + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + } + }, + "dependencies": { + "@cleverbrush/scheduler": "^4.4.3", + "@cleverbrush/schema": "^4.4.3", + "@cleverbrush/orm": "^4.4.3", + "@cleverbrush/knex-schema": "^4.4.3" + }, + "peerDependencies": { + "knex": ">=3.1.0", + "pg": ">=8" + }, + "devDependencies": { + "@types/node": "^25.4.0" + }, + "scripts": { + "build": "tsup && tsc --project tsconfig.build.json --emitDeclarationOnly", + "clean": "rm -rf dist tsconfig.tsbuildinfo" + } +} diff --git a/libs/scheduler-postgres/src/index.ts b/libs/scheduler-postgres/src/index.ts new file mode 100644 index 00000000..74ecd406 --- /dev/null +++ b/libs/scheduler-postgres/src/index.ts @@ -0,0 +1,3 @@ +export { createSchedulerTables, dropSchedulerTables } from './migrations.js'; +export type { PostgresStorageOptions } from './schema.js'; +export { PostgresJobRepository, PostgresJobStorage } from './storage.js'; diff --git a/libs/scheduler-postgres/src/migrations.ts b/libs/scheduler-postgres/src/migrations.ts new file mode 100644 index 00000000..1ce0b742 --- /dev/null +++ b/libs/scheduler-postgres/src/migrations.ts @@ -0,0 +1,74 @@ +import { generateCreateTable, getTableName } from '@cleverbrush/knex-schema'; +import type { Knex } from 'knex'; +import { type PostgresStorageOptions, storageSchemas } from './schema.js'; + +/** + * Explicit initial migration. Call once through your migration runner, never on worker startup. + * Creates four tables, indexes and cascading foreign keys in one transaction. + */ +export async function createSchedulerTables( + knex: Knex, + options: PostgresStorageOptions = {} +): Promise { + const schemas = storageSchemas(options); + await knex.transaction(async tx => { + for (const schema of [ + schemas.runs, + schemas.schedules, + schemas.events, + schemas.attempts + ]) + await generateCreateTable(schema)(tx); + const runs = getTableName(schemas.runs); + await tx.schema.alterTable(runs, table => { + table.unique(['namespace', 'dedupeKey'], runs + '_dedupe'); + table.index( + ['namespace', 'status', 'availableAt'], + runs + '_ready' + ); + table.index( + ['namespace', 'status', 'leaseExpiresAt'], + runs + '_leases' + ); + table.index( + ['namespace', 'scheduleId', 'status'], + runs + '_overlap' + ); + table.index(['namespace', 'expiresAt'], runs + '_expiry'); + }); + await tx.schema.alterTable(getTableName(schemas.schedules), table => { + table.index( + ['namespace', 'active', 'nextAt'], + getTableName(schemas.schedules) + '_due' + ); + }); + for (const schema of [schemas.events, schemas.attempts]) { + await tx.schema.alterTable(getTableName(schema), table => { + table + .foreign('runId') + .references('id') + .inTable(runs) + .onDelete('CASCADE'); + }); + } + }); +} +/** + * Explicit destructive down migration. Deletes all scheduler data for this prefix. + * Stop all producers, dispatchers and workers before calling. + */ +export async function dropSchedulerTables( + knex: Knex, + options: PostgresStorageOptions = {} +): Promise { + const schemas = storageSchemas(options); + await knex.transaction(async tx => { + for (const schema of [ + schemas.events, + schemas.attempts, + schemas.schedules, + schemas.runs + ]) + await tx.schema.dropTable(getTableName(schema)); + }); +} diff --git a/libs/scheduler-postgres/src/schema.test-d.ts b/libs/scheduler-postgres/src/schema.test-d.ts new file mode 100644 index 00000000..f8fbb721 --- /dev/null +++ b/libs/scheduler-postgres/src/schema.test-d.ts @@ -0,0 +1,35 @@ +import { createDb } from '@cleverbrush/orm'; +import type { InferType } from '@cleverbrush/schema'; +import type { Knex } from 'knex'; +import { expectTypeOf } from 'vitest'; +import { storageSchemas } from './schema.js'; + +const schemas = storageSchemas(); +type Run = InferType; +expectTypeOf().toEqualTypeOf(); +expectTypeOf().toEqualTypeOf(); +expectTypeOf().toEqualTypeOf< + number | null | undefined +>(); +expectTypeOf< + InferType['active'] +>().toEqualTypeOf(); +expectTypeOf< + InferType['sequence'] +>().toEqualTypeOf(); +expectTypeOf< + InferType['attempt'] +>().toEqualTypeOf(); +declare const knex: Knex; +const db = createDb(knex, schemas.entities); +const rows = await db.runs + .select( + t => t.id, + t => t.version + ) + .execute(); +expectTypeOf(rows[0]).toEqualTypeOf<{ id: string; version: number }>(); +// @ts-expect-error Query properties come from the actual schema. +db.runs.select(t => t.missing); +// @ts-expect-error Insert values retain the column types. +db.events.insert({ runId: 'one', sequence: 'wrong', record: '{}' }); diff --git a/libs/scheduler-postgres/src/schema.test.ts b/libs/scheduler-postgres/src/schema.test.ts new file mode 100644 index 00000000..8ddd5e79 --- /dev/null +++ b/libs/scheduler-postgres/src/schema.test.ts @@ -0,0 +1,32 @@ +import { generateCreateTable, getTableName } from '@cleverbrush/knex-schema'; +import knex from 'knex'; +import { describe, expect, it } from 'vitest'; +import { storageSchemas } from './schema.js'; + +describe('PostgreSQL schemas', () => { + it('derives migration columns and composite keys with Framework schemas', () => { + const schemas = storageSchemas({ tablePrefix: 'test_jobs' }); + const db = knex({ client: 'pg' }); + const sql = generateCreateTable(schemas.events)(db).toQuery(); + expect(sql).toContain('primary key ("runId", "sequence")'); + expect(getTableName(schemas.runs)).toBe('test_jobs_runs'); + expect(generateCreateTable(schemas.runs)(db).toQuery()).toContain( + 'primary key ("id")' + ); + expect(generateCreateTable(schemas.schedules)(db).toQuery()).toContain( + 'primary key ("namespace", "id")' + ); + expect(generateCreateTable(schemas.attempts)(db).toQuery()).toContain( + 'primary key ("runId", "attempt")' + ); + const other = storageSchemas({ tablePrefix: 'other_jobs' }); + expect(getTableName(other.runs)).toBe('other_jobs_runs'); + expect(getTableName(schemas.runs)).toBe('test_jobs_runs'); + expect(getTableName(storageSchemas().runs)).toBe('cb_jobs_runs'); + }); + it('rejects untrusted identifiers', () => { + expect(() => + storageSchemas({ tablePrefix: 'jobs; drop table' }) + ).toThrow(); + }); +}); diff --git a/libs/scheduler-postgres/src/schema.ts b/libs/scheduler-postgres/src/schema.ts new file mode 100644 index 00000000..3ef7f12b --- /dev/null +++ b/libs/scheduler-postgres/src/schema.ts @@ -0,0 +1,98 @@ +import { + boolean, + defineEntity, + type Entity, + number, + object, + string +} from '@cleverbrush/orm'; + +const time = () => number().columnType('double precision'); +const runFields = { + id: string().primaryKey(), + namespace: string(), + name: string(), + version: number(), + status: string(), + availableAt: time(), + leaseExpiresAt: time().nullable().optional(), + expiresAt: time().nullable().optional(), + scheduleId: string().nullable().optional(), + dedupeKey: string().nullable().optional(), + record: string().columnType('text') +}; +const scheduleFields = { + namespace: string(), + id: string(), + active: boolean(), + nextAt: time().nullable().optional(), + record: string().columnType('text') +}; +const eventFields = { + runId: string(), + sequence: number(), + record: string().columnType('text') +}; +const attemptFields = { + runId: string(), + attempt: number(), + record: string().columnType('text') +}; + +// Named field maps keep declaration emission portable without duplicating row types. +const runs: ReturnType> = + object(runFields).hasTableName('cb_jobs_runs'); +const schedules: ReturnType> = object( + scheduleFields +) + .hasTableName('cb_jobs_schedules') + .hasPrimaryKey(['namespace', 'id']); +const events: ReturnType> = object( + eventFields +) + .hasTableName('cb_jobs_events') + .hasPrimaryKey(['runId', 'sequence']); +const attempts: ReturnType> = object( + attemptFields +) + .hasTableName('cb_jobs_attempts') + .hasPrimaryKey(['runId', 'attempt']); + +type StorageSchemas = { + runs: typeof runs; + schedules: typeof schedules; + events: typeof events; + attempts: typeof attempts; + entities: { + runs: Entity; + schedules: Entity; + events: Entity; + attempts: Entity; + }; +}; + +/** Storage table naming. Use separate prefixes for independent schema ownership. */ +export type PostgresStorageOptions = { tablePrefix?: string }; +/** @internal Indexed columns plus an opaque JSON text snapshot, decoded explicitly. */ +export function storageSchemas( + options: PostgresStorageOptions = {} +): StorageSchemas { + const prefix = options.tablePrefix ?? 'cb_jobs'; + if (!/^[a-z][a-z0-9_]{0,39}$/.test(prefix)) + throw new TypeError('Invalid scheduler table prefix'); + const tables = { + runs: runs.hasTableName(prefix + '_runs'), + schedules: schedules.hasTableName(prefix + '_schedules'), + events: events.hasTableName(prefix + '_events'), + attempts: attempts.hasTableName(prefix + '_attempts') + }; + return { + ...tables, + entities: { + runs: defineEntity(tables.runs), + schedules: defineEntity(tables.schedules), + events: defineEntity(tables.events), + attempts: defineEntity(tables.attempts) + } + }; +} diff --git a/libs/scheduler-postgres/src/storage.ts b/libs/scheduler-postgres/src/storage.ts new file mode 100644 index 00000000..0d581cdb --- /dev/null +++ b/libs/scheduler-postgres/src/storage.ts @@ -0,0 +1,328 @@ +import { + createDb, + getTableName, + number, + object, + rawQuery, + string +} from '@cleverbrush/orm'; +import { + type JobAttempt, + type JobEvent, + JobRepository, + type JobRepositoryHealth, + type JobStorage, + type JobStorageTransaction, + type RunRecord, + type ScheduleRecord +} from '@cleverbrush/scheduler'; +import type { Knex } from 'knex'; +import { type PostgresStorageOptions, storageSchemas } from './schema.js'; + +const clockSql = 'floor(extract(epoch from clock_timestamp()) * 1000)'; +function runRow(run: RunRecord) { + return { + id: run.id, + namespace: run.namespace, + name: run.name, + version: run.version, + status: run.status, + availableAt: run.availableAt, + leaseExpiresAt: run.leaseExpiresAt, + expiresAt: + run.completedAt === null + ? null + : run.completedAt + run.policy.retentionMs, + scheduleId: run.scheduleId, + dedupeKey: run.dedupeKey, + record: JSON.stringify(run) + }; +} +function scheduleRow(schedule: ScheduleRecord) { + return { + namespace: schedule.namespace, + id: schedule.id, + active: schedule.active, + nextAt: schedule.nextAt, + record: JSON.stringify(schedule) + }; +} +/** @internal Storage records are library-owned JSON, never arbitrary driver objects. */ +function unpack(row: { record: string } | undefined): T | undefined { + return row ? (JSON.parse(row.record) as T) : undefined; +} + +/** + * PostgreSQL transactional adapter. The supplied Knex instance/pool remains caller-owned. + * Binding to a transaction provides atomic business-write + enqueue; do not start a worker + * on a transaction-bound repository. Acceptance is durable only after the outer commit. + */ +export class PostgresJobStorage implements JobStorage { + private readonly schemas; + constructor( + private readonly knex: Knex, + options: PostgresStorageOptions = {} + ) { + this.schemas = storageSchemas(options); + } + /** Run an atomic transition; row locks, events and cursor writes share the commit. */ + async atomic( + action: (transaction: JobStorageTransaction) => Promise + ): Promise { + return this.knex.transaction(async tx => { + await tx.raw("set local lock_timeout = '5s'"); + await tx.raw("set local statement_timeout = '15s'"); + return action(this.transaction(tx)); + }); + } + private transaction(tx: Knex.Transaction): JobStorageTransaction { + const db = createDb(tx, this.schemas.entities); + const runs = getTableName(this.schemas.runs); + // Native Knex is confined to row locking and aggregates unsupported by the ORM. + const lockedRun = async (builder: Knex.QueryBuilder) => + unpack( + ( + await rawQuery( + tx, + db.runs.rowSchema, + builder.forUpdate().limit(1) + ) + )[0] + ); + const lockedSchedule = async (builder: Knex.QueryBuilder) => + unpack( + ( + await rawQuery( + tx, + db.schedules.rowSchema, + builder.forUpdate().limit(1) + ) + )[0] + ); + const runQuery = (namespace: string, id: string) => + db.runs.where(t => t.namespace, namespace).where(t => t.id, id); + const scheduleQuery = (namespace: string, id: string) => + db.schedules + .where(t => t.namespace, namespace) + .where(t => t.id, id); + return { + now: async () => + ( + await rawQuery( + tx, + object({ now: number() }), + 'select (' + clockSql + ')::double precision as now' + ) + )[0].now, + run: async (namespace, id, lock) => + lock + ? lockedRun(runQuery(namespace, id).toKnexQuery()) + : unpack(await runQuery(namespace, id).first()), + insertRun: async run => { + const inserted = await db.runs + .onConflict( + t => t.namespace, + t => t.dedupeKey + ) + .ignore(runRow(run)); + if (inserted) + return { + record: unpack(inserted)!, + inserted: true + }; + const record = await lockedRun( + db.runs + .where(t => t.namespace, run.namespace) + .where(t => t.dedupeKey, run.dedupeKey) + .toKnexQuery() + ); + if (!record) throw new Error('Conflicting run disappeared'); + return { record, inserted: false }; + }, + saveRun: async run => { + await runQuery(run.namespace, run.id).update(runRow(run)); + }, + runnable: async (namespace, supported) => { + const builder = db.runs + .where(t => t.namespace, namespace) + .toKnexQuery(); + builder.andWhere(group => { + group.where(expired => + expired + .where('status', 'running') + .whereRaw('?? <= ' + clockSql, ['leaseExpiresAt']) + ); + if (supported.length) + group.orWhere(ready => + ready + .whereIn('status', ['queued', 'retry_wait']) + .whereRaw('?? <= ' + clockSql, ['availableAt']) + .where(versions => { + for (const job of supported) + versions.orWhere({ + name: job.name, + version: job.version + }); + }) + ); + }); + return lockedRun( + builder + .orderBy('availableAt') + .orderBy('id') + .forUpdate() + .skipLocked() + ); + }, + appendEvent: async event => { + await db.events.insert({ + runId: event.runId, + sequence: event.sequence, + record: JSON.stringify(event) + }); + }, + events: async (runId, after, limit) => + ( + await db.events + .where(t => t.runId, runId) + .where(t => t.sequence, '>', after) + .orderBy(t => t.sequence) + .limit(limit) + .execute() + ).map(row => unpack(row)!), + saveAttempt: async attempt => { + await db.attempts + .onConflict( + t => t.runId, + t => t.attempt + ) + .merge({ + runId: attempt.runId, + attempt: attempt.attempt, + record: JSON.stringify(attempt) + }); + }, + attempts: async runId => + ( + await db.attempts + .where(t => t.runId, runId) + .orderBy(t => t.attempt) + .execute() + ).map(row => unpack(row)!), + schedule: async (namespace, id) => + lockedSchedule(scheduleQuery(namespace, id).toKnexQuery()), + insertSchedule: async schedule => { + await db.schedules + .onConflict( + t => t.namespace, + t => t.id + ) + .ignore(scheduleRow(schedule)); + return (await lockedSchedule( + scheduleQuery(schedule.namespace, schedule.id).toKnexQuery() + ))!; + }, + saveSchedule: async schedule => { + await scheduleQuery(schedule.namespace, schedule.id).update( + scheduleRow(schedule) + ); + }, + dueSchedules: async (namespace, limit) => { + const rows = await rawQuery( + tx, + db.schedules.rowSchema, + db.schedules + .where(t => t.namespace, namespace) + .where(t => t.active, true) + .toKnexQuery() + .whereRaw('?? <= ' + clockSql, ['nextAt']) + .orderBy('nextAt') + .orderBy('id') + .limit(limit) + .forUpdate() + .skipLocked() + ); + return rows.map(row => unpack(row)!); + }, + unfinishedScheduleRun: async (namespace, scheduleId) => + !!(await db.runs + .where(t => t.namespace, namespace) + .where(t => t.scheduleId, scheduleId) + .whereIn(t => t.status, ['queued', 'running', 'retry_wait']) + .first()), + cleanup: async (namespace, limit) => { + const rows = await rawQuery( + tx, + object({ id: string() }), + tx(runs) + .select('id') + .where({ namespace }) + .whereIn('status', ['succeeded', 'failed', 'cancelled']) + .whereRaw('?? <= ' + clockSql, ['expiresAt']) + .orderBy('expiresAt') + .limit(limit) + .forUpdate() + .skipLocked() + ); + if (!rows.length) return 0; + return db.runs + .whereIn( + t => t.id, + rows.map(row => row.id) + ) + .delete(); + }, + health: async namespace => { + const counts = await rawQuery( + tx, + object({ status: string(), count: number() }), + tx(runs) + .select('status') + .select(tx.raw('count(*)::float8 as count')) + .where({ namespace }) + .groupBy('status') + ); + const queued = tx(runs) + .where({ namespace }) + .whereIn('status', ['queued', 'retry_wait']); + const queuedDefinitions = await rawQuery( + tx, + object({ + name: string(), + version: number(), + count: number() + }), + queued + .clone() + .select('name', 'version') + .select(tx.raw('count(*)::float8 as count')) + .groupBy('name', 'version') + ); + const oldest = await rawQuery( + tx, + object({ at: number().nullable() }), + queued + .clone() + .whereRaw('?? <= ' + clockSql, ['availableAt']) + .select( + tx.raw('min(??)::float8 as at', ['availableAt']) + ) + ); + return { + counts: Object.fromEntries( + counts.map(row => [row.status, row.count]) + ), + oldestReadyAt: oldest[0].at, + queuedDefinitions + } satisfies JobRepositoryHealth; + } + }; + } +} + +/** Durable PostgreSQL repository with database-time leases and SKIP LOCKED claims. */ +export class PostgresJobRepository extends JobRepository { + constructor(knex: Knex, options: PostgresStorageOptions = {}) { + super(new PostgresJobStorage(knex, options)); + } +} diff --git a/libs/scheduler-postgres/tsconfig.build.json b/libs/scheduler-postgres/tsconfig.build.json new file mode 100644 index 00000000..e5c33a62 --- /dev/null +++ b/libs/scheduler-postgres/tsconfig.build.json @@ -0,0 +1,16 @@ +{ + "compilerOptions": { + "rootDir": "./src", + "outDir": "./dist", + "declaration": true, + "target": "ES2022", + "module": "ES2022", + "moduleResolution": "bundler", + "esModuleInterop": true, + "strict": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["src/**/*.ts"], + "exclude": ["src/**/*.test.ts", "src/**/*.test-d.ts"] +} diff --git a/libs/scheduler-postgres/tsconfig.json b/libs/scheduler-postgres/tsconfig.json new file mode 100644 index 00000000..8f2b78c8 --- /dev/null +++ b/libs/scheduler-postgres/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "./tsconfig.build.json", + "compilerOptions": { "types": ["node", "vitest/globals"] } +} diff --git a/libs/scheduler-postgres/tsconfig.typecheck.json b/libs/scheduler-postgres/tsconfig.typecheck.json new file mode 100644 index 00000000..c24a9fd6 --- /dev/null +++ b/libs/scheduler-postgres/tsconfig.typecheck.json @@ -0,0 +1,6 @@ +{ + "extends": "./tsconfig.build.json", + "compilerOptions": { "noEmit": true, "strict": true }, + "include": ["src/**/*.test-d.ts"], + "exclude": [] +} diff --git a/libs/scheduler-postgres/tsup.config.ts b/libs/scheduler-postgres/tsup.config.ts new file mode 100644 index 00000000..a79bfcde --- /dev/null +++ b/libs/scheduler-postgres/tsup.config.ts @@ -0,0 +1,9 @@ +import { defineConfig } from 'tsup'; +export default defineConfig({ + entry: ['src/index.ts'], + format: ['esm'], + tsconfig: './tsconfig.build.json', + sourcemap: true, + clean: true, + target: 'es2022' +}); diff --git a/libs/scheduler-postgres/typedoc.json b/libs/scheduler-postgres/typedoc.json new file mode 100644 index 00000000..f79e1cf6 --- /dev/null +++ b/libs/scheduler-postgres/typedoc.json @@ -0,0 +1,6 @@ +{ + "$schema": "https://typedoc.org/schema.json", + "entryPoints": ["src/index.ts"], + "name": "@cleverbrush/scheduler-postgres", + "excludePrivate": true +} diff --git a/libs/scheduler-postgres/vitest.config.mts b/libs/scheduler-postgres/vitest.config.mts new file mode 100644 index 00000000..e18b86e9 --- /dev/null +++ b/libs/scheduler-postgres/vitest.config.mts @@ -0,0 +1,11 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ + test: { + include: ['src/**/*.test.ts'], + typecheck: { + enabled: true, + include: ['src/**/*.test-d.ts'], + tsconfig: './tsconfig.typecheck.json' + } + } +}); diff --git a/libs/scheduler/MIGRATION-v5.md b/libs/scheduler/MIGRATION-v5.md new file mode 100644 index 00000000..bd4e9db1 --- /dev/null +++ b/libs/scheduler/MIGRATION-v5.md @@ -0,0 +1,77 @@ +# Scheduler migration: v4.x to v5 + +This is a breaking scheduler redesign in the Framework v5 release train. +Do not run v4 and v5 workers against the same work queue. + +## API replacements + +| v4.x | v5 | +| --- | --- | +| Implicit in-memory JobScheduler persistence | Explicit storageRepository option; PostgreSQL adapter for durability | +| rootFolder and file-oriented task registration | Versioned defineJob contract + handle(fn) or thread(fileURL) | +| IJobRepository task CRUD contract | JobRepository transition engine over transactional JobStorage | +| Auto-execution around scheduler registration | Producer enqueue, dispatcher start and worker start are separate | +| Process event listeners for progress | Durable events(definition, runId, { after, signal }) stream | +| maxOccurences | maxOccurrences (deprecated spelling still accepted; do not supply both) | + +Definitions require input, progress and output schemas. A no-progress or no-output +job can use null schemas and return null. Typed handlers can live in separate +modules using JobHandler. Thread modules default-export a +handler and must be built to executable ESM. + +## Schedule definitions are retained + +The minute/day/week/month/year objects, `Schedule` type and schedule members +of `Schemas` remain supported. Schemas are now also direct exports, and +`TaskSchedule` aliases the inferred `Schedule` union. Pass the schedule to +`upsertSchedule(id, Definition, input, { schedule })`; do not restore +`addJob`, `rootFolder` or the old file-based worker registration. + +```ts +const schedule = { + every: 'week' as const, interval: 2, dayOfWeek: [1, 5], + hour: 9, startsOn: new Date('2026-10-01T00:00:00Z'), + maxOccurences: 10 // accepted for migration; prefer maxOccurrences +}; +await jobs.upsertSchedule('weekly-report', Report, { reportId: 'weekly' }, { + schedule, missed: 'coalesce', overlap: 'skip' +}); +// Register Report.handle(handleReport), start the worker and jobs.start(). +``` + +`ScheduleCalculator(schedule)` still accepts one schedule and `next()` still +returns `{ date, index }` with a one-based index. Dates may be validated from +JSON strings through `ScheduleSchema.parse`. The deprecated `maxOccurences` +alias normalizes to `maxOccurrences`; supplying both throws, even if equal. +`interval` now defaults to 1 and `skipFirst` can be 0. Fractional calendar +components, duplicate weekdays, irrelevant variant fields and invalid dates +are rejected. Weekly, monthly and yearly identifying fields remain required. + +## Retry and calendar semantics + +Retries require an explicit maxAttempts greater than one, including crash recovery. +Audit application-side idempotency before enabling them. The first attempt counts +toward the limit. Jobs rerun from the beginning, not from a progress checkpoint. + +All calendar calculations default to UTC. Set an IANA zone explicitly for local +wall time. DST gaps are skipped and folds use the earlier instant only. Weekly intervals are anchored consistently to the week containing startsOn, +including when all selected weekdays in that first week have passed. Recurring +triggers persist schedule-slot cursors; occurrence limits count slots, not +successful executions. Missed triggers coalesce by default. + +## Rollout + +1. Stop/drain v4 workers. Inventory pending jobs and recurring registrations. + There is no universal migration from custom v4 repository data. +2. Install scheduler and scheduler-postgres from the same v5 release. +3. Run createSchedulerTables through your database migration runner. + Existing v4 tables are neither read nor modified. +4. Define versioned contracts and register handlers. Import pending jobs with + stable application idempotency keys; do not blindly rerun completed work. +5. Register schedules with explicit time zones and missed/overlap policies. +6. Start workers and dispatchers separately. Verify health and progress replay. +7. Keep needed old definition versions deployed until those v5 runs drain. + +Only install the PostgreSQL adapter in server processes. The application owns +the database pool, authorization, progress transport and shutdown hooks. +See the [current guide](./README.md) and [PostgreSQL setup](../scheduler-postgres). diff --git a/libs/scheduler/README.md b/libs/scheduler/README.md index bba3a28d..9cddfeda 100644 --- a/libs/scheduler/README.md +++ b/libs/scheduler/README.md @@ -1,174 +1,283 @@ # @cleverbrush/scheduler - -![Coverage](https://img.shields.io/badge/coverage-97.4%25-brightgreen) - -A job scheduler for Node.js that runs tasks in worker threads on configurable schedules. +Typed immediate, delayed and recurring jobs with durable progress. Producers, +dispatchers and workers share one execution model, with separate lifecycles. -## Installation +Use [@cleverbrush/scheduler-postgres](../scheduler-postgres/README.md) for durability. +`InMemoryJobRepository` is explicitly process-local and intended for development +and tests. Upgrading? Read [v4.x → v5 migration](./MIGRATION-v5.md). -```bash -npm install @cleverbrush/scheduler -``` - -This library uses the Node.js `worker_threads` module. It is tested on Node.js v16+. - -## Quick Start - -```typescript -import { JobScheduler } from '@cleverbrush/scheduler'; - -const scheduler = new JobScheduler({ - rootFolder: '/path/to/your/jobs' -}); +## Separate contracts and handlers -scheduler.addJob({ - id: 'my-job-1', - path: 'job1.js', - schedule: { - every: 'minute', - interval: 5 - } -}); +```ts +// contracts/report.ts — safe for producer processes to import +import { number, object, string } from '@cleverbrush/schema'; +import { defineJob } from '@cleverbrush/scheduler'; -scheduler.addJob({ - id: 'my-job-2', - path: 'job2.js', - schedule: { - every: 'week', - interval: 2, - dayOfWeek: [1, 3, 5], - hour: 9, - minute: 30 - }, - timeout: 1000 * 60 * 4, - maxRetries: 3 +export const Report = defineJob({ + name: 'report', version: 1, + input: object({ reportId: string() }), + progress: object({ percent: number() }), + output: object({ downloadUrl: string() }), + retry: { maxAttempts: 3 } // opt-in; default is ONE attempt }); - -scheduler.start(); ``` -## API - -### `JobScheduler` - -The main class for scheduling and running jobs. Extends `EventEmitter`. - -#### Constructor - -```typescript -const scheduler = new JobScheduler(props: JobSchedulerProps); +```ts +// handlers/report.ts — execution dependencies stay here +import type { JobHandler } from '@cleverbrush/scheduler'; +import { Report } from '../contracts/report.js'; + +export const handleReport: JobHandler = async (input, context) => { + context.signal.throwIfAborted(); + await context.report({ percent: 50 }); // committed before this resolves + // Write an idempotent result keyed by context.runId. + return { downloadUrl: '/reports/' + input.reportId }; +}; ``` -| Property | Type | Default | Description | -| --- | --- | --- | --- | -| `rootFolder` | `string` | — | Path to the folder containing job files | -| `defaultTimeZone` | `string` | `'UTC'` | Timezone used for scheduling | - -#### Methods - -| Method | Description | -| --- | --- | -| `start()` | Starts the scheduler | -| `stop()` | Stops the scheduler | -| `addJob(job)` | Adds a job to the scheduler | -| `removeJob(jobId)` | Removes a job by ID | -| `jobExists(jobId)` | Returns `true` if a job with the given ID exists | -| `status` | Current scheduler status: `'started'` or `'stopped'` | - -#### Events - -```typescript -scheduler.on('job:start', ({ jobId, instanceId, startDate }) => { - console.log(`Job ${jobId} started`); -}); - -scheduler.on('job:end', ({ jobId, instanceId, startDate, endDate, stdout, stderr }) => { - console.log(`Job ${jobId} finished`); -}); - -scheduler.on('job:error', ({ jobId, instanceId, startDate, endDate, stdout, stderr }) => { - console.error(`Job ${jobId} failed`); -}); +```ts +// infrastructure/jobs.ts +import knex from 'knex'; +import { JobScheduler } from '@cleverbrush/scheduler'; +import { PostgresJobRepository } from '@cleverbrush/scheduler-postgres'; -scheduler.on('job:timeout', ({ jobId, instanceId, startDate, endDate }) => { - console.warn(`Job ${jobId} timed out`); +export const database = knex({ + client: 'pg', connection: process.env.DATABASE_URL, + acquireConnectionTimeout: 5000 }); - -scheduler.on('job:message', ({ jobId, instanceId, message }) => { - console.log(`Message from ${jobId}:`, message); +export const jobs = new JobScheduler({ + storageRepository: new PostgresJobRepository(database), + namespace: 'reports' }); ``` -### Job Configuration - -```typescript -scheduler.addJob({ - id: 'unique-job-id', - path: 'relative/path/to/job.js', - schedule: { /* see Schedule Types */ }, - timeout: 60000, // timeout in milliseconds - maxRetries: 3, // retry on failure - maxConsequentFails: 10, // disable after N consecutive failures - noConcurrentRuns: true, // prevent overlapping executions - props: { key: 'value' } // data passed to the worker +```ts +// producer.ts — no start() required +import { jobs } from './infrastructure/jobs.js'; +import { Report } from './contracts/report.js'; + +const run = await jobs.enqueue(Report, { reportId: 'quarterly' }, { + idempotencyKey: 'quarterly:2026-Q4' }); +// Persist/send run.id to your client after acceptance commits. ``` -### Schedule Types - -All schedules share these optional properties: - -| Property | Type | Description | -| --- | --- | --- | -| `interval` | `number` | Number of periods between repeats (1–356) | -| `startsOn` | `Date` | Do not start before this date | -| `endsOn` | `Date` | Do not repeat after this date | -| `maxOccurences` | `number` | Maximum number of executions | -| `skipFirst` | `number` | Skip this many initial executions | +```ts +// worker.ts +import { jobs, database } from './infrastructure/jobs.js'; +import { Report } from './contracts/report.js'; +import { handleReport } from './handlers/report.js'; -#### Minute - -```typescript -{ every: 'minute', interval: 5 } +const worker = jobs.createWorker({ + jobs: [Report.handle(handleReport)], concurrency: 4 +}); +await worker.start(); +// On application shutdown: +await worker.stop({ drainTimeoutMs: 30000 }); +await jobs.stop(); +await database.destroy(); ``` -Runs every N minutes. - -#### Day - -```typescript -{ every: 'day', interval: 1, hour: 9, minute: 30 } +For isolated execution, register +`Report.thread(new URL('./handlers/report-thread.js', import.meta.url))`. +That module must default-export a compatible handler. URLs come only from +trusted deployment registration, never from job input. Build the handler for +Node ESM before starting the worker. A thread is not a security sandbox. + +## Progress, results and cancellation + +```ts +// Authorize access to runId in your application before either operation. +const snapshot = await jobs.getRun(Report, runId); +for await (const event of jobs.events(Report, runId, { + after: lastSeenSequence, signal: requestAbortSignal +})) { + // Persist event.sequence as the exclusive reconnect cursor. + // Use event.type === 'progress' to identify progress events. + sendToClient(event); +} +await jobs.cancel(runId); ``` -Runs every N days at the specified time. - -#### Week - -```typescript -{ every: 'week', interval: 2, dayOfWeek: [1, 3, 5], hour: 9, minute: 30 } +Events are persisted and ordered per run, including state transitions. +The stream replays after an **exclusive** sequence cursor and polls for updates +until terminal state or abort. Disconnecting a subscriber does not cancel work. +Missing or retention-deleted runs end the stream; use getRun to distinguish them +before opening it. Events/results never expose lease tokens. Namespaces partition +jobs but are **not authorization**. No HTTP, SSE, WebSocket or UI dependency is +required; the application owns transport and access control. + +Input, progress and output must validate against synchronous Framework schemas +and be strict JSON before and after validation. Undefined, sparse arrays, +accessors, dates, bigint, functions, cycles and non-finite numbers are rejected. +Represent files by object-storage keys and dates by strings. No arbitrary +functions, paths or credentials are serialized. Error messages are persisted: +handlers must avoid putting secrets in them. + +Schemas are checked at production, execution and observation boundaries. Keep +preprocessors deterministic and idempotent; put business transformations in +handlers instead of durable transport contracts. + +## Durability and retries + +Accepted PostgreSQL jobs survive process restarts. Workers claim with +transactional row locks and renewable leases. Every state/progress/output write +is fenced by the current unexpired lease. Database time controls ownership. +A worker without a matching name/version leaves the run queued; health exposes +queued definitions so deployments can detect unsupported work. + +Execution is **at-least-once when retries are enabled**, not exactly-once. +A crash or timeout can occur after a business side effect and before completion +is persisted. Handlers must use idempotent effects (often keyed by runId), and +cooperate with the AbortSignal. Fencing protects scheduler records, not external +services. Job versions identify persisted contracts: introduce a new version +when changing the contract and keep handlers for outstanding old versions. + +Retries are disabled by default (`maxAttempts: 1`). Lease expiry, timeout, +shutdown interruption and handler failure all consume an attempt. Explicit +retries rerun the **whole handler** with exponential capped delay. Throw +`NonRetryableJobError` for permanent failures; validation/size errors never retry. +There are no implicit checkpoints or workflow-step replay. + +Cancellation immediately fences the owner and marks the run cancelled. Ordinary +functions receive cancellation on their next heartbeat or failed progress write; +they cannot be forcibly stopped. Timed-out functions retain their local worker +slot until they actually settle, while another process may retry the run. +Worker threads can be terminated. Shutdown stops claims, drains, then requests +abort; it does not wait indefinitely for a non-cooperative function. Database +calls need bounded connection/statement timeouts too. Workers are single-use. + +## Recurring triggers + +Schedules are schema-driven discriminated objects. Import `Schedule` (also +available as `TaskSchedule`) for the inferred type, or `ScheduleSchema` to +validate configuration, including JSON date strings: + +```ts +import { + ScheduleSchema, ScheduleCalculator, type Schedule +} from '@cleverbrush/scheduler'; + +const examples: Schedule[] = [ + { every: 'minute', interval: 15 }, + { every: 'day', hour: 18, minute: 30 }, + { every: 'week', dayOfWeek: [1, 5], hour: 9 }, + { every: 'month', day: 'last' }, + { every: 'year', month: 2, day: 'last' } +]; +const schedule = ScheduleSchema.parse({ + every: 'week', dayOfWeek: [1, 5], + startsOn: '2026-10-01T00:00:00Z', maxOccurrences: 10 +}); +const preview = new ScheduleCalculator(schedule).next(); +// { date: Date, index: 1 } — public indexes are one-based. ``` -Runs every N weeks on the specified days (1 = Monday, 7 = Sunday). - -#### Month +`ScheduleMinuteSchema`, `ScheduleDaySchema`, `ScheduleWeekSchema`, +`ScheduleMonthSchema`, `ScheduleYearSchema` and `ScheduleSchemaBase` are +also direct exports and members of `Schemas`. Weekly schedules require +`dayOfWeek`; monthly schedules require `day`; yearly schedules require +`month` and `day`. Minute schedules reject local `hour`/`minute` fields. +The same schemas validate registration and calculator inputs. -```typescript -{ every: 'month', interval: 1, day: 15, hour: 0, minute: 0 } -// or on the last day of the month: -{ every: 'month', interval: 1, day: 'last', hour: 0, minute: 0 } +```ts +await jobs.upsertSchedule('weekday-reports', Report, { reportId: 'daily' }, { + schedule: { + every: 'week', dayOfWeek: [1, 2, 3, 4, 5], + hour: 9, minute: 0, timeZone: 'Europe/Berlin' + }, + missed: 'coalesce', // default; alternatives: 'skip', 'replay' + overlap: 'allow' // default; 'skip' includes queued and retry-wait runs +}); +const worker = jobs.createWorker({ jobs: [Report.handle(handleReport)] }); +await worker.start(); // execute accepted runs +await jobs.start(); // dispatch recurring occurrences + +// Keep the process alive until application shutdown, then: +await jobs.stop(); // stop producing occurrences first +await worker.stop({ drainTimeoutMs: 30000 }); +await database.destroy(); ``` -Runs every N months on the specified day (1–28 or `'last'`). - -#### Year - -```typescript -{ every: 'year', interval: 1, month: 6, day: 1, hour: 12, minute: 0 } +See the [runnable periodic demo](../../demos/durable-jobs/README.md) for a bounded +example. Use `pauseSchedule(id)`, `pauseSchedule(id, false)` and +`removeSchedule(id)` to manage a trigger without cancelling accepted runs. + +The dispatcher atomically enqueues occurrences and advances a persisted cursor. +Multiple dispatchers do not duplicate occurrences. Identical upserts retain +cursor, original start anchor and revision. Defaults, date representations, +weekday order and supported aliases are normalized before comparison; changes create a new revision for future dispatch, leaving +accepted runs intact. Pause/remove also leave accepted runs intact. Removed +triggers retain a small tombstone so recreating an ID cannot reuse old occurrence +identities. + +- `minute` uses elapsed minutes from startsOn, not local wall-clock rounding. +- `day/week/month/year` use calendar arithmetic in UTC by default or an IANA zone. + Calendar time defaults to 09:00. ISO weekdays are 1 (Monday) through 7. +- Nonexistent DST wall times are skipped. A repeated wall time executes only at + its earlier instant. +- Month/year days are 1–28 or `'last'`; months are 1–12; interval is 1–356 (default 1). +- startsOn defaults to registration time. endsOn is inclusive. maxOccurrences + and skipFirst count calendar slots, including skipped DST gaps, not successes. +- Coalesce enqueues the latest overdue occurrence. Skip drops an accumulated + backlog when more than one occurrence is due. Replay materializes every due + occurrence in bounded batches (100 per schedule/pass). +- Overlap skip consumes skipped occurrences; it does not defer them. + +Calendar calculations use built-in `Date` and `Intl.DateTimeFormat` APIs; +no additional date-time package is required. IANA rules come from the Node.js +runtime's ICU data. Keep runtime/tzdata versions aligned across dispatchers +so they agree on time-zone rule updates. Calculations do not depend on the +host process's `TZ` setting. + +`ScheduleCalculator` previews the same rules. Supply startsOn for reproducibility; +`next()` returns `{ date, index }` with a one-based slot index (including skipped +slots), and `hasNext()` checks exhaustion. Internal persisted cursors are zero-based. + +## Limits and operations + +| Setting | Default | +| --- | --- | +| Worker concurrency | 1 per worker, not a cluster quota | +| Poll interval | 1 second | +| Lease / heartbeat | 30 seconds / 10 seconds | +| Attempt timeout | 5 minutes | +| Drain timeout | 30 seconds, then up to 1 second abort grace | +| Input or output size | 1 MiB each | +| Progress event size / count | 64 KiB / 10,000 per run across attempts | +| Terminal retention | 7 days | + +Definition options customize limits and retry policy; each run snapshots them. +Payload nesting is limited to 100. Terminal cleanup removes the run, attempts +and events together. Active and queued runs never expire. Workers/dispatchers +perform bounded cleanup once a minute; producer-only installations should call +`jobs.cleanup()` themselves. + +Idempotency keys are scoped by namespace, job name and version. Reusing a key +with different input, runAt or policy throws `SubmissionConflictError`. +Deduplication lasts only while the run is retained. It is not permanent business +uniqueness. Identical schedule registration likewise does not act as resume; +use pauseSchedule(id, false). + +`jobs.health()` returns state counts, oldest ready timestamp and queued versions. +`worker.lastError`, `jobs.lastError`, dispatcher onError and worker onDiagnostic +expose infrastructure problems without swallowing them. Diagnostics contain +identifiers, not payloads; bridge them to your logger/metrics/tracing system. +No automatic OTEL dependency is introduced. + +## Adapter authors and tests + +`JobRepository` implements transitions over `JobStorage.atomic()`. Custom +storage must provide detached data, rollback, atomic deduplication, exclusive +claims, row ownership locks and ordered event writes as documented by +`JobStorageTransaction`. See the shared conformance tests in +`testing/repository-contract.ts`. Memory and PostgreSQL run those same tests. + +From the repository root after `npm ci && npm run build`: + +```sh +npx vitest run --typecheck libs/scheduler libs/scheduler-postgres/src +SCHEDULER_TEST_DATABASE_URL=postgres://... npm run test:scheduler:integration +node demos/durable-jobs/demo.ts ``` - -Runs every N years on the specified month (1–12) and day (1–28 or `'last'`). - -## License - -BSD-3-Clause diff --git a/libs/scheduler/src/ScheduleCalculator.ts b/libs/scheduler/src/ScheduleCalculator.ts deleted file mode 100644 index 3540ef8b..00000000 --- a/libs/scheduler/src/ScheduleCalculator.ts +++ /dev/null @@ -1,404 +0,0 @@ -import type { Schedule } from './types.js'; - -const MS_IN_DAY = 1000 * 60 * 60 * 24; -const MS_IN_WEEK = MS_IN_DAY * 7; - -const getDayOfWeek = (date: Date) => { - const res = date.getUTCDay(); - if (res === 0) return 7; - return res; -}; - -const getNumberOfDaysInMonth = (date: Date) => { - return new Date( - Date.UTC(date.getUTCFullYear(), date.getUTCMonth() + 1, 0, 0, 0, 0, 0) - ).getDate(); -}; - -/** - * Iterates over the dates defined by a {@link Schedule}. - * - * Given a schedule configuration (e.g. every 2 days, every week on Monday/Friday, - * every month on the 15th) the calculator produces the sequence of `Date` objects - * that match that schedule. Use {@link hasNext} / {@link next} to walk through - * the sequence. - * - * @example - * ```ts - * const calc = new ScheduleCalculator({ - * every: 'day', - * interval: 1, - * hour: 9, - * minute: 0, - * startsOn: new Date('2025-01-01T00:00:00Z'), - * maxOccurences: 5 - * }); - * - * while (calc.hasNext()) { - * const { date, index } = calc.next(); - * console.log(`Run #${index} at ${date.toISOString()}`); - * } - * ``` - */ -export class ScheduleCalculator { - #schedule: Schedule; - #currentDate = new Date(); - #hour = 9; - #minute = 0; - #maxRepeat = -1; - #repeatCount = 0; - - #hasNext = false; - #next: Date | undefined; - - /** - * @param schedule - The schedule definition to iterate over. - * @throws If `schedule` is falsy. - */ - constructor(schedule: Schedule) { - if (!schedule) throw new Error('schedule is required'); - this.#schedule = { ...schedule }; - - if (typeof schedule.startsOn !== 'undefined') { - this.#currentDate = schedule.startsOn; - } else { - this.#schedule.startsOn = new Date(); - this.#currentDate = this.#schedule.startsOn; - } - - if (schedule.every !== 'minute') { - if (typeof schedule.hour === 'number') { - this.#hour = schedule.hour; - } - - if (typeof schedule.minute === 'number') { - this.#minute = schedule.minute; - } - - if ( - schedule.every === 'day' && - new Date( - Date.UTC( - this.#currentDate.getUTCFullYear(), - this.#currentDate.getUTCMonth(), - this.#currentDate.getUTCDate(), - this.#hour, - this.#minute, - 0, - 0 - ) - ).getTime() < this.#currentDate.getTime() - ) { - const date = new Date( - Date.UTC( - this.#currentDate.getUTCFullYear(), - this.#currentDate.getUTCMonth(), - this.#currentDate.getUTCDate() + 1, - this.#hour, - this.#minute, - 0, - 0 - ) - ); - this.#currentDate = date; - } - } - - if (typeof schedule.maxOccurences === 'number') { - this.#maxRepeat = schedule.maxOccurences; - } - - const next = this.#getNext(); - - if (typeof next !== 'undefined') { - this.#next = next; - this.#hasNext = true; - } - - let leftToSkip = - typeof this.#schedule.skipFirst === 'number' - ? this.#schedule.skipFirst - : 0; - - while (leftToSkip-- > 0 && this.#hasNext) { - this.next(); - } - } - - #getNext(): Date | undefined { - let candidate: Date | null = null; - let dayOfWeek: number; - - switch (this.#schedule.every) { - case 'minute': - candidate = - this.#repeatCount === 0 - ? (this.#schedule.startsOn as Date) - : new Date( - Date.UTC( - this.#currentDate.getUTCFullYear(), - this.#currentDate.getUTCMonth(), - this.#currentDate.getUTCDate(), - this.#currentDate.getUTCHours(), - this.#currentDate.getUTCMinutes() + - this.#schedule.interval, - this.#currentDate.getUTCSeconds(), - this.#currentDate.getUTCMilliseconds() - ) - ); - break; - case 'day': - { - const date = new Date( - this.#currentDate.getTime() + - MS_IN_DAY * - (this.#repeatCount === 0 - ? 0 - : this.#schedule.interval - 1) - ); - - candidate = new Date( - Date.UTC( - date.getUTCFullYear(), - date.getUTCMonth(), - date.getUTCDate(), - this.#hour, - this.#minute, - 0, - 0 - ) - ); - } - break; - case 'week': - { - let date = this.#currentDate; - - do { - let found = false; - dayOfWeek = getDayOfWeek(date); - if (Number.isNaN(dayOfWeek)) return; - for (let i = 0; i < 7 - dayOfWeek + 1; i++) { - date = new Date( - date.getTime() + (i === 0 ? 0 : MS_IN_DAY) - ); - if ( - this.#schedule.endsOn && - date > this.#schedule.endsOn - ) { - return; - } - if ( - this.#schedule.dayOfWeek.includes( - getDayOfWeek(date) - ) - ) { - const dateWithTime = new Date( - Date.UTC( - date.getUTCFullYear(), - date.getUTCMonth(), - date.getUTCDate(), - this.#hour, - this.#minute, - 0, - 0 - ) - ); - if ( - dateWithTime > - (this.#schedule.startsOn as Date) - ) { - candidate = dateWithTime; - found = true; - break; - } - } - } - - if (found) break; - - date = new Date( - date.getTime() + - MS_IN_DAY + - (this.#repeatCount === 0 - ? 0 - : (this.#schedule.interval - 1) * - MS_IN_WEEK) - ); - } while ( - (this.#schedule.endsOn && - date <= this.#schedule.endsOn) || - !this.#schedule.endsOn - ); - } - break; - case 'month': - { - let dateTime; - const cDate = this.#currentDate; - let iteration = 0; - do { - const date = - this.#schedule.day === 'last' - ? getNumberOfDaysInMonth( - new Date( - Date.UTC( - cDate.getUTCFullYear(), - cDate.getUTCMonth() + - iteration * - (this.#repeatCount === 0 - ? 1 - : this.#schedule - .interval), - 1, - 0, - 0, - 0, - 0 - ) - ) - ) - : (this.#schedule.day as number); - dateTime = new Date( - Date.UTC( - cDate.getUTCFullYear(), - cDate.getUTCMonth() + - iteration * - (this.#repeatCount === 0 - ? 1 - : this.#schedule.interval), - date, - this.#hour, - this.#minute, - 0, - 0 - ) - ); - iteration++; - } while ( - dateTime < (this.#schedule.startsOn as Date) || - dateTime <= this.#currentDate - ); - candidate = dateTime; - } - break; - case 'year': - { - let dateTime; - const cDate = this.#currentDate; - let iteration = 0; - do { - const date = - this.#schedule.day === 'last' - ? getNumberOfDaysInMonth( - new Date( - Date.UTC( - cDate.getUTCFullYear() + - iteration * - (this.#repeatCount === 0 - ? 1 - : this.#schedule - .interval), - this.#schedule.month - 1, - 1, - 0, - 0, - 0, - 0 - ) - ) - ) - : (this.#schedule.day as number); - dateTime = new Date( - Date.UTC( - cDate.getUTCFullYear() + - iteration * - (this.#repeatCount === 0 - ? 1 - : this.#schedule.interval), - this.#schedule.month - 1, - date, - this.#hour, - this.#minute, - 0, - 0 - ) - ); - iteration++; - } while ( - dateTime < (this.#schedule.startsOn as Date) || - dateTime <= this.#currentDate - ); - candidate = dateTime; - } - break; - default: - throw new Error('unknown schedule type'); - } - - if (!candidate) return; - - if ( - typeof this.#schedule.endsOn !== 'undefined' && - candidate > this.#schedule.endsOn - ) { - return; - } - - return candidate; - } - - /** - * Returns `true` when the schedule has at least one more date. - * - * @param span - Optional millisecond window. When provided the method - * returns `true` only if the next date falls within `span` ms from now. - */ - public hasNext(span?: number): boolean { - if (!this.#hasNext) { - return false; - } - - if (typeof span !== 'number') return this.#hasNext; - - if (!this.#next) return false; - return this.#next.getTime() - Date.now() <= span; - } - - /** - * Advances to the next scheduled date and returns it together with - * its 1-based index in the sequence. - * - * @returns An object with the scheduled `date` and its `index`. - * @throws If the schedule has no more dates ({@link hasNext} is `false`). - */ - public next(): { - date: Date; - index: number; - } { - if (!this.#hasNext) throw new Error('schedule is over'); - - const result = this.#next as Date; - - this.#currentDate = new Date( - result.getTime() + - (['day', 'week'].includes(this.#schedule.every) ? MS_IN_DAY : 0) - ); - - this.#repeatCount++; - const next = this.#getNext(); - - this.#next = next as Date; - this.#hasNext = typeof next !== 'undefined'; - - if (this.#maxRepeat > 0 && this.#repeatCount >= this.#maxRepeat) { - this.#next = undefined; - this.#hasNext = false; - } - - return { - date: result, - index: this.#repeatCount - }; - } -} diff --git a/libs/scheduler/src/calendar.test.ts b/libs/scheduler/src/calendar.test.ts new file mode 100644 index 00000000..898ecf04 --- /dev/null +++ b/libs/scheduler/src/calendar.test.ts @@ -0,0 +1,120 @@ +import { describe, expect, it } from 'vitest'; +import { + addDays, + addMonths, + calendarDate, + localDate, + resolveLocal, + timeZoneFormatter +} from './calendar.js'; + +describe('native calendar and time zones', () => { + it.each([ + ['UTC', '2026-01-01T09:00:00Z', '2026-01-01T09:00:00Z', true], + [ + 'America/New_York', + '2026-01-01T00:00:00Z', + '2026-01-01T05:00:00Z', + true + ], + [ + 'America/New_York', + '2026-03-08T02:30:00Z', + '2026-03-08T06:30:00Z', + false + ], + [ + 'America/New_York', + '2026-11-01T01:30:00Z', + '2026-11-01T05:30:00Z', + true + ], + [ + 'Australia/Lord_Howe', + '2026-10-04T02:15:00Z', + '2026-10-03T15:15:00Z', + false + ], + [ + 'Australia/Lord_Howe', + '2026-04-05T01:45:00Z', + '2026-04-04T14:45:00Z', + true + ], + ['Pacific/Apia', '2011-12-30T09:00:00Z', '2011-12-29T19:00:00Z', false], + [ + 'Asia/Kathmandu', + '2026-01-01T09:00:00Z', + '2026-01-01T03:15:00Z', + true + ], + [ + 'America/St_Johns', + '2026-01-01T09:00:00Z', + '2026-01-01T12:30:00Z', + true + ], + ['Etc/GMT+5', '2026-01-01T09:00:00Z', '2026-01-01T14:00:00Z', true], + ['Europe/Paris', '1890-01-01T09:00:00Z', '1890-01-01T08:50:39Z', true] + ] as const)('resolves %s local %s', (zone, wall, instant, executable) => { + const result = resolveLocal(new Date(wall), zone); + expect(result).toEqual({ at: Date.parse(instant), executable }); + expect(localDate(result.at, zone).getTime() === Date.parse(wall)).toBe( + executable + ); + }); + it.each([ + 0, 1, 22, 99, -1 + ])('preserves ISO year %s, including eras', year => { + const wall = calendarDate(year, 1, 1, 0, 0, 0, 123); + expect(wall.getUTCFullYear()).toBe(year); + expect(localDate(wall.getTime(), 'Etc/UTC')).toEqual(wall); + expect(resolveLocal(wall, 'Etc/UTC')).toEqual({ + at: wall.getTime(), + executable: true + }); + }); + it('uses calendar arithmetic without changing the input date', () => { + const leap = calendarDate(2028, 2, 28, 9); + expect(addDays(leap, 1).toISOString()).toBe('2028-02-29T09:00:00.000Z'); + expect(addDays(leap, 2).toISOString()).toBe('2028-03-01T09:00:00.000Z'); + const first = calendarDate(2026, 12, 1, 9); + expect(addMonths(first, 2).toISOString()).toBe( + '2027-02-01T09:00:00.000Z' + ); + expect(first.toISOString()).toBe('2026-12-01T09:00:00.000Z'); + expect(leap.getUTCDate()).toBe(28); + }); + it('rejects invalid zones and fixed-offset identifiers, and caches valid formatters', () => { + for (const zone of ['Wrong/Zone', '+02:00', '-0500']) + expect(() => timeZoneFormatter(zone)).toThrow(); + const formatter = timeZoneFormatter('Europe/Berlin'); + expect(timeZoneFormatter('Europe/Berlin')).toBe(formatter); + expect(formatter.resolvedOptions()).toMatchObject({ + calendar: 'gregory', + numberingSystem: 'latn', + hourCycle: 'h23' + }); + }); + it('evicts old formatters without changing results', () => { + const formatter = timeZoneFormatter('UTC'); + for (const zone of Intl.supportedValuesOf('timeZone').slice(0, 65)) + timeZoneFormatter(zone); + expect(timeZoneFormatter('UTC')).not.toBe(formatter); + expect(localDate(0, 'Asia/Kathmandu').toISOString()).toBe( + '1970-01-01T05:30:00.000Z' + ); + }); + it('fails cleanly outside the Date range', () => { + const last = new Date(8640000000000000); + expect(resolveLocal(last, 'UTC')).toEqual({ + at: last.getTime(), + executable: true + }); + expect(() => addDays(last, 1)).toThrow(RangeError); + expect(() => calendarDate(999999, 1, 1)).toThrow(RangeError); + expect(() => resolveLocal(new Date(Number.NaN), 'UTC')).toThrow( + RangeError + ); + }); +}); diff --git a/libs/scheduler/src/calendar.ts b/libs/scheduler/src/calendar.ts new file mode 100644 index 00000000..18aaf463 --- /dev/null +++ b/libs/scheduler/src/calendar.ts @@ -0,0 +1,114 @@ +const DAY_MS = 86400000; +const DATE_LIMIT = 8640000000000000; +const formatters = new Map(); + +/** @internal Validate named zones and reuse a bounded set of ICU formatters. */ +export function timeZoneFormatter(zone: string): Intl.DateTimeFormat { + if (/^[+-]/.test(zone)) throw new RangeError('Use a named IANA time zone'); + let formatter = formatters.get(zone); + if (!formatter) { + formatter = new Intl.DateTimeFormat('en-US', { + timeZone: zone, + // Gregorian fields match the ISO calendar; requesting gregory + // also preserves the era on ICU versions that omit it for iso8601. + calendar: 'gregory', + numberingSystem: 'latn', + hourCycle: 'h23', + era: 'short', + year: 'numeric', + month: '2-digit', + day: '2-digit', + hour: '2-digit', + minute: '2-digit', + second: '2-digit', + fractionalSecondDigits: 3 + }); + if (formatters.size >= 64) + formatters.delete(formatters.keys().next().value!); + formatters.set(zone, formatter); + } + return formatter; +} + +/** @internal Construct a wall-clock date in UTC fields, preserving years 0–99. */ +export function calendarDate( + year: number, + month: number, + day: number, + hour = 0, + minute = 0, + second = 0, + millisecond = 0 +): Date { + const value = new Date(0); + value.setUTCFullYear(year, month - 1, day); + value.setUTCHours(hour, minute, second, millisecond); + return validDate(value); +} + +function validDate(date: Date): Date { + if (!Number.isFinite(date.getTime())) + throw new RangeError('Calendar exhausted'); + return date; +} + +/** @internal Add calendar days to a UTC-field wall-clock date, not an instant. */ +export function addDays(local: Date, days: number): Date { + return validDate(new Date(local.getTime() + days * DAY_MS)); +} + +/** @internal Advance a first-of-month wall-clock date without host-zone arithmetic. */ +export function addMonths(first: Date, months: number): Date { + const value = new Date(first); + value.setUTCMonth(value.getUTCMonth() + months); + return validDate(value); +} + +/** @internal Represent an instant's local ISO calendar fields using UTC Date fields. */ +export function localDate(at: number, zone: string): Date { + if (zone === 'UTC') return validDate(new Date(at)); + const fields = Object.fromEntries( + timeZoneFormatter(zone) + .formatToParts(at) + .map(part => [part.type, part.value]) + ); + const year = Number(fields.year); + return calendarDate( + fields.era === 'BC' ? 1 - year : year, + Number(fields.month), + Number(fields.day), + Number(fields.hour), + Number(fields.minute), + Number(fields.second), + Number(fields.fractionalSecond) + ); +} + +/** + * @internal Resolve a wall-clock date to the earlier instant of a fold. + * IANA offsets are shorter than a day. Probe either side of that window to + * find both offsets of a transition, then round-trip each candidate. This + * handles non-hour shifts and date-line jumps without assuming a 1-hour DST + * change. A gap keeps its earlier ordering instant but is never executable. + */ +export function resolveLocal( + local: Date, + zone: string +): { at: number; executable: boolean } { + const wall = validDate(local).getTime(); + if (zone === 'UTC') return { at: wall, executable: true }; + const offsets = new Set(); + for (const delta of [-DAY_MS, 0, DAY_MS]) { + const probe = Math.max(-DATE_LIMIT, Math.min(DATE_LIMIT, wall + delta)); + offsets.add(localDate(probe, zone).getTime() - probe); + } + const candidates = [...offsets] + .map(offset => wall - offset) + .filter(at => Math.abs(at) <= DATE_LIMIT) + .sort((a, b) => a - b); + for (const at of candidates) + if (localDate(at, zone).getTime() === wall) + return { at, executable: true }; + if (!candidates.length) throw new RangeError('Calendar exhausted'); + return { at: candidates[0], executable: false }; +} diff --git a/libs/scheduler/src/contracts.test-d.ts b/libs/scheduler/src/contracts.test-d.ts new file mode 100644 index 00000000..1b91292b --- /dev/null +++ b/libs/scheduler/src/contracts.test-d.ts @@ -0,0 +1,30 @@ +import { number, object, string } from '@cleverbrush/schema'; +import { expectTypeOf } from 'vitest'; +import { defineJob, type JobHandler, type JobScheduler } from './index.js'; + +const report = defineJob({ + name: 'report', + version: 1, + input: object({ id: string() }), + progress: object({ percent: number() }), + output: object({ url: string() }) +}); +const handler: JobHandler = async (input, context) => { + expectTypeOf(input.id).toEqualTypeOf(); + await context.report({ percent: 50 }); + // @ts-expect-error Progress is inferred from the definition. + await context.report({ percent: 'wrong' }); + return { url: input.id }; +}; +report.handle(handler); +// @ts-expect-error Output is inferred even with a separately declared handler. +report.handle(() => ({ other: 1 })); +declare const scheduler: JobScheduler; +// @ts-expect-error Input contract is required. +scheduler.enqueue(report, { id: 1 }); +const run = await scheduler.enqueue(report, { id: 'one' }); +expectTypeOf(run.output).toEqualTypeOf<{ url: string } | null>(); +for await (const event of scheduler.events(report, run.id)) { + if (event.type === 'progress') + expectTypeOf(event.data.percent).toEqualTypeOf(); +} diff --git a/libs/scheduler/src/contracts.ts b/libs/scheduler/src/contracts.ts new file mode 100644 index 00000000..3d89655e --- /dev/null +++ b/libs/scheduler/src/contracts.ts @@ -0,0 +1,158 @@ +import type { InferType, SchemaBuilder } from '@cleverbrush/schema'; + +/** Synchronous Framework schema accepted at a durable boundary. */ +export type JobSchema = SchemaBuilder; +/** Strict durable JSON transport; dates and files should be represented by strings. */ +export type JsonValue = + | null + | boolean + | number + | string + | JsonValue[] + | { [key: string]: JsonValue }; +/** Persisted lifecycle state. */ +export type RunStatus = + | 'queued' + | 'running' + | 'retry_wait' + | 'succeeded' + | 'failed' + | 'cancelled'; +/** Bounded persisted error, without stack or arbitrary exception properties. */ +export type JobError = { code: string; message: string }; +/** maxAttempts includes the first execution and lease-expiry recovery. */ +export type RetryPolicy = { + maxAttempts: number; + initialDelayMs: number; + maxDelayMs: number; +}; +/** Resource and retry policy snapshotted when a run is accepted. */ +export type RunPolicy = { + retry: RetryPolicy; + timeoutMs: number; + retentionMs: number; + maxPayloadBytes: number; + maxProgressBytes: number; + maxProgressEvents: number; +}; +/** Definition defaults. A retry requires an explicit maxAttempts greater than one. */ +export type JobOptions = Omit, 'retry'> & { + retry?: Partial; +}; +/** Handler context; cancellation is cooperative for ordinary functions. */ +export type JobContext

= { + readonly runId: string; + readonly attempt: number; + readonly signal: AbortSignal; + /** Resolves after the current lease holder commits this progress event. */ + report(progress: P): Promise; +}; +/** Strong handler typing without importing execution dependencies into producers. */ +export type JobHandler> = ( + input: InferType, + context: JobContext> +) => Promise> | InferType; +/** Trusted deployment registration; module URLs are never enqueued. */ +export type JobBinding = { + readonly definition: JobDefinition; + readonly handler?: JobHandler; + readonly moduleUrl?: string; +}; +/** Versioned contract shared by producers and workers. */ +export interface JobDefinition< + I extends JobSchema, + P extends JobSchema, + O extends JobSchema +> { + readonly name: string; + readonly version: number; + readonly input: I; + readonly progress: P; + readonly output: O; + readonly policy: Readonly; + /** Bind a handler imported from a separate module. */ + handle(handler: JobHandler>): JobBinding; + /** Bind a module whose default export is a compatible handler. */ + thread(moduleUrl: URL): JobBinding; +} +/** Public snapshot. Times are UTC Unix milliseconds; ownership secrets are excluded. */ +export type JobRun = { + id: string; + namespace: string; + name: string; + version: number; + status: RunStatus; + input: JsonValue; + output: O | null; + error: JobError | null; + attempt: number; + createdAt: number; + availableAt: number; + completedAt: number | null; + scheduleId: string | null; + sequence: number; +}; +/** Ordered event. Cursors are exclusive and scoped to a run. */ +export type JobEvent

= { + runId: string; + sequence: number; + attempt: number; + at: number; +} & ( + | { type: 'progress'; data: P } + | { type: RunStatus; data: JobError | null } +); +/** Audit entry for one acquisition of execution ownership. */ +export type JobAttempt = { + runId: string; + attempt: number; + startedAt: number; + endedAt: number | null; + status: 'running' | 'succeeded' | 'failed' | 'interrupted' | 'cancelled'; + error: JobError | null; +}; +/** Adapter record. Never return lease tokens through an application API. */ +export type RunRecord = JobRun & { + policy: RunPolicy; + leaseToken: string | null; + leaseExpiresAt: number | null; + dedupeKey: string | null; + fingerprint: string; + progressCount: number; +}; +/** Calendar recurrence. DST gaps are skipped; repeated wall times execute once. */ +export type TaskSchedule = import('./schedule-schemas.js').Schedule; +/** Serializable recurrence anchored once at registration, preserving each variant. */ +export type StoredSchedule = TaskSchedule extends infer S + ? S extends TaskSchedule + ? Omit & { + startsOn: number; + endsOn?: number; + } + : never + : never; +/** Recurring trigger, revision and persisted cursor. */ +export type ScheduleRecord = { + namespace: string; + id: string; + revision: number; + fingerprint: string; + active: boolean; + removed: boolean; + name: string; + version: number; + input: JsonValue; + policy: RunPolicy; + schedule: StoredSchedule; + missed: 'coalesce' | 'skip' | 'replay'; + overlap: 'allow' | 'skip'; + cursor: number; + nextAt: number | null; +}; +/** Operational hook deliberately excludes payloads and credentials. */ +export type SchedulerDiagnostic = { + type: 'started' | 'settled' | 'lease_lost' | 'infrastructure_error'; + runId?: string; + name?: string; + attempt?: number; +}; diff --git a/libs/scheduler/src/definition.test.ts b/libs/scheduler/src/definition.test.ts new file mode 100644 index 00000000..503c9cea --- /dev/null +++ b/libs/scheduler/src/definition.test.ts @@ -0,0 +1,60 @@ +import { any, number, object } from '@cleverbrush/schema'; +import { describe, expect, it } from 'vitest'; +import { testJob } from '../testing/repository-contract.js'; +import { jsonValue } from './definition.js'; +import { defineJob, InMemoryJobRepository, JobScheduler } from './index.js'; + +describe('durable boundaries', () => { + it.each([ + undefined, + NaN, + Infinity, + 1n, + new Date(), + () => 0, + // biome-ignore lint/suspicious/noSparseArray: deliberate invalid payload + [, 1], + { x: undefined } + ])('rejects lossy JSON %s', value => { + expect(() => jsonValue(value, 1024)).toThrow(); + }); + it('rejects cycles, accessors and oversized payloads', () => { + const array = [1]; + Object.defineProperty(array, '0', { + get() { + throw new Error('must not execute'); + } + }); + expect(() => jsonValue(array, 1024)).toThrow('Unsupported'); + const cyclic: any = {}; + cyclic.self = cyclic; + expect(() => jsonValue(cyclic, 1024)).toThrow('Cyclic'); + const getter = Object.defineProperty({}, 'value', { + get() { + throw new Error('must not execute'); + }, + enumerable: true + }); + expect(() => jsonValue(getter, 1024)).toThrow('Unsupported'); + expect(() => jsonValue('four', 3)).toThrow('size'); + }); + it('validates schema before persistence and rejects invalid policies', async () => { + const scheduler = new JobScheduler({ + storageRepository: new InMemoryJobRepository() + }); + await expect( + scheduler.enqueue(testJob(), { id: 1 } as any) + ).rejects.toThrow('validation'); + expect((await scheduler.health()).counts).toEqual({}); + expect(() => testJob({ retry: { maxAttempts: 0 } })).toThrow(); + expect(() => + defineJob({ + name: '', + version: 1, + input: any(), + progress: number(), + output: object({}) + }) + ).toThrow(); + }); +}); diff --git a/libs/scheduler/src/definition.ts b/libs/scheduler/src/definition.ts new file mode 100644 index 00000000..6d760d6d --- /dev/null +++ b/libs/scheduler/src/definition.ts @@ -0,0 +1,260 @@ +import { createHash } from 'node:crypto'; +import type { InferType } from '@cleverbrush/schema'; +import type { + JobDefinition, + JobError, + JobOptions, + JobSchema, + JsonValue, + RunPolicy +} from './contracts.js'; + +/** Explicitly prevent retries when another execution cannot help. */ +export class NonRetryableJobError extends Error { + constructor( + message: string, + readonly code = 'non_retryable' + ) { + super(message); + this.name = 'NonRetryableJobError'; + } +} +/** An expired, cancelled, or superseded attempt tried to write durable state. */ +export class LeaseLostError extends Error { + constructor() { + super('The job lease is no longer owned by this attempt'); + this.name = 'LeaseLostError'; + } +} +/** The same idempotency key was submitted with different data or policy. */ +export class SubmissionConflictError extends Error { + constructor() { + super('The idempotency key identifies a different submission'); + this.name = 'SubmissionConflictError'; + } +} +/** Validate integral configuration before creating timers. */ +export function positive( + value: number, + label: string, + maximum = Number.MAX_SAFE_INTEGER +): number { + if (!Number.isSafeInteger(value) || value < 1 || value > maximum) + throw new RangeError(label); + return value; +} +/** Validate identities without silently trimming them. */ +export function identity(value: string, label: string): string { + if (typeof value !== 'string' || !value.trim() || value.length > 200) + throw new TypeError(label); + return value; +} +/** Snapshot strict JSON, rejecting getters, cycles, holes, and lossy values. */ +export function jsonValue(value: unknown, maxBytes: number): JsonValue { + const ancestors = new Set(); + function visit(current: unknown): JsonValue { + if ( + current === null || + typeof current === 'string' || + typeof current === 'boolean' + ) + return current; + if (typeof current === 'number' && Number.isFinite(current)) + return current; + if (typeof current !== 'object' || current === null) + throw new NonRetryableJobError( + 'Only JSON values are accepted', + 'invalid_payload' + ); + if (ancestors.has(current)) + throw new NonRetryableJobError( + 'Cyclic job payload', + 'invalid_payload' + ); + if (ancestors.size >= 100) + throw new NonRetryableJobError( + 'Job payload nesting limit exceeded', + 'payload_limit' + ); + if ( + !Array.isArray(current) && + Object.getPrototypeOf(current) !== Object.prototype && + Object.getPrototypeOf(current) !== null + ) + throw new NonRetryableJobError( + 'Non-JSON object', + 'invalid_payload' + ); + ancestors.add(current); + let result: JsonValue; + if (Array.isArray(current)) { + const keys = Reflect.ownKeys(current); + if (keys.length !== current.length + 1) + throw new NonRetryableJobError( + 'Unsupported JSON array properties', + 'invalid_payload' + ); + result = Array.from({ length: current.length }, (_, i) => { + const descriptor = Object.getOwnPropertyDescriptor( + current, + String(i) + ); + if (!descriptor?.enumerable || !('value' in descriptor)) + throw new NonRetryableJobError( + 'Unsupported JSON array property', + 'invalid_payload' + ); + return visit(descriptor.value); + }); + } else { + result = {}; + for (const key of Reflect.ownKeys(current)) { + const descriptor = Object.getOwnPropertyDescriptor( + current, + key + )!; + if ( + typeof key !== 'string' || + !descriptor.enumerable || + !('value' in descriptor) + ) + throw new NonRetryableJobError( + 'Unsupported JSON property', + 'invalid_payload' + ); + Object.defineProperty(result, key, { + value: visit(descriptor.value), + enumerable: true, + writable: true, + configurable: true + }); + } + } + ancestors.delete(current); + return result; + } + const result = visit(value); + if (Buffer.byteLength(JSON.stringify(result)) > maxBytes) + throw new NonRetryableJobError( + 'Job payload size limit exceeded', + 'payload_limit' + ); + return result; +} +/** Parse a boundary and reject transformations into non-JSON values. */ +export function parsePayload( + schema: S, + value: unknown, + maxBytes: number +): InferType { + const parsed = schema.validate(jsonValue(value, maxBytes)); + if (!parsed.valid) + throw new NonRetryableJobError( + 'Job schema validation failed', + 'invalid_payload' + ); + return jsonValue(parsed.object, maxBytes) as InferType; +} +/** Canonical fingerprint independent of object property order. */ +export function fingerprint(value: unknown): string { + function ordered(item: any): any { + if (Array.isArray(item)) return item.map(ordered); + if (item !== null && typeof item === 'object') + return Object.fromEntries( + Object.keys(item) + .sort() + .map(key => [key, ordered(item[key])]) + ); + return item; + } + return createHash('sha256') + .update(JSON.stringify(ordered(value))) + .digest('hex'); +} +/** Materialize a policy so future deployments cannot change accepted runs. */ +export function jobPolicy(options: JobOptions = {}): RunPolicy { + const retry = { + maxAttempts: positive( + options.retry?.maxAttempts ?? 1, + 'maxAttempts', + 1000 + ), + initialDelayMs: positive( + options.retry?.initialDelayMs ?? 1000, + 'initialDelayMs' + ), + maxDelayMs: positive(options.retry?.maxDelayMs ?? 60000, 'maxDelayMs') + }; + if (retry.initialDelayMs > retry.maxDelayMs) + throw new RangeError('initialDelayMs exceeds maxDelayMs'); + return { + retry, + timeoutMs: positive( + options.timeoutMs ?? 300000, + 'timeoutMs', + 2147483647 + ), + retentionMs: positive( + options.retentionMs ?? 7 * 86400000, + 'retentionMs' + ), + maxPayloadBytes: positive( + options.maxPayloadBytes ?? 1048576, + 'maxPayloadBytes' + ), + maxProgressBytes: positive( + options.maxProgressBytes ?? 65536, + 'maxProgressBytes' + ), + maxProgressEvents: positive( + options.maxProgressEvents ?? 10000, + 'maxProgressEvents' + ) + }; +} +/** Create a producer-safe definition, bindable to either execution mode. */ +export function defineJob< + I extends JobSchema, + P extends JobSchema, + O extends JobSchema +>( + options: JobOptions & { + name: string; + version: number; + input: I; + progress: P; + output: O; + } +): JobDefinition { + const policy = jobPolicy(options); + Object.freeze(policy.retry); + Object.freeze(policy); + const definition: JobDefinition = { + name: identity(options.name, 'job name'), + version: positive(options.version, 'job version'), + input: options.input, + progress: options.progress, + output: options.output, + policy, + handle: handler => Object.freeze({ definition, handler }), + thread: moduleUrl => { + if (moduleUrl.protocol !== 'file:') + throw new TypeError('A trusted file URL is required'); + return Object.freeze({ definition, moduleUrl: moduleUrl.href }); + } + }; + return Object.freeze(definition); +} +/** Persist bounded diagnostics without stack traces or exception properties. */ +export function jobError(error: unknown): JobError { + return { + code: + error instanceof NonRetryableJobError + ? error.code + : 'handler_error', + message: (error instanceof Error + ? error.message + : 'Job execution failed' + ).slice(0, 2000) + }; +} diff --git a/libs/scheduler/src/index.ts b/libs/scheduler/src/index.ts index e04ac388..ac3a5691 100644 --- a/libs/scheduler/src/index.ts +++ b/libs/scheduler/src/index.ts @@ -1,567 +1,27 @@ -import { EventEmitter } from 'events'; -import fs from 'fs'; -import { access as fsAccess } from 'fs/promises'; -import { join as pathJoin } from 'path'; -import { PassThrough, type Readable } from 'stream'; -import { Worker } from 'worker_threads'; -import { type IJobRepository, InMemoryJobRepository } from './jobRepository.js'; - -import { ScheduleCalculator } from './ScheduleCalculator.js'; -import { - type CreateJobRequest, - type Job, - type JobInstance, - type JobInstanceStatus, - type JobSchedulerProps, - Schedule, - type SchedulerStatus, +export type * from './contracts.js'; +export { + defineJob, + LeaseLostError, + NonRetryableJobError, + SubmissionConflictError +} from './definition.js'; +export { InMemoryJobRepository, InMemoryJobStorage } from './memory.js'; +export { ScheduleCalculator } from './recurrence.js'; +export type { JobSubmission, ScheduleSubmission } from './repository.js'; +export { JobRepository } from './repository.js'; +export { + type Schedule, + ScheduleDaySchema, + ScheduleMinuteSchema, + ScheduleMonthSchema, + ScheduleSchema, + ScheduleSchemaBase, + ScheduleWeekSchema, + ScheduleYearSchema, Schemas -} from './types.js'; - -export { Schedule as TaskSchedule, ScheduleCalculator, Schemas }; - -type WorkerResult = { - status: JobInstanceStatus; - exitCode: number; - error?: Error; -}; - -const MAX_BUFFER_SIZE = 10 * 1024 * 1024; // 10MB -const CHECK_INTERVAL = 1000 * 10; // every 10 seconds -const SCHEDULE_JOB_SPAN = 1000 * 60; // 1 minute -const DEFAULT_JOB_TIMEOUT = 1000 * 20; // 20 seconds -const DEFAULT_MAX_CONSEQUENT_FAILS = 3; -const DEFAULT_MAX_RETRIES = 2; - -type JobStartItem = { - jobId: string; - instanceId: number; - stdout: Readable; - stderr: Readable; - startDate: Date; -}; - -type JobEndItem = JobStartItem & { - endDate: Date; -}; - -type JobErrorItem = JobStartItem & { - endDate: Date; - error?: Error; -}; - -type JobMessageItem = Omit & { - value: any; -}; - -type Events = { - 'job:start': (job: JobStartItem) => any; - 'job:end': (job: JobEndItem) => any; - 'job:error': (job: JobErrorItem) => any; - 'job:timeout': (job: JobErrorItem) => any; - 'job:message': (msg: JobMessageItem) => any; -}; - -interface IJobScheduler { - on(name: T, callback: Events[T]): this; - addJob(job: CreateJobRequest): Promise; - removeJob(id: string): Promise; -} - -/** - * Job scheduler class that handles all the scheduling and running of jobs. - */ -export class JobScheduler extends EventEmitter implements IJobScheduler { - protected _rootFolder: string; - protected _status: SchedulerStatus = 'stopped'; - protected _defaultTimezone!: string; - - protected _checkTimer: ReturnType | undefined; - - protected _jobsRepository: IJobRepository = new InMemoryJobRepository(); - - protected _jobProps: Map = new Map(); - - /** - * Status of the scheduler. - */ - public get status() { - return this._status; - } - - protected set status(val: SchedulerStatus) { - if (val === this._status) return; - this._status = val; - } - - private scheduleCalculatorCache = new Map(); - - /** - * Get the schedule calculator for a job by its id. - * @param {Job} job - * @returns {Promise} - */ - protected async getJobSchedule(job: Job) { - if (this.scheduleCalculatorCache.has(job.id)) { - return this.scheduleCalculatorCache.get(job.id); - } - - const schedule = { - ...job.schedule - }; - - if (typeof job.firstInstanceEndedAt !== 'undefined') { - schedule.startsOn = job.firstInstanceEndedAt; - } - - if (typeof job.successfullTimesRunned === 'number') { - schedule.skipFirst = job.successfullTimesRunned - 1; - } - - const res = new ScheduleCalculator(schedule); - this.scheduleCalculatorCache.set(job.id, res); - - return res; - } - - protected runWorkerWithTimeout(file: string, props: any, timeout: number) { - const worker = new Worker(file, { - workerData: props, - execArgv: ['--unhandled-rejections=strict'], - stderr: true, - stdout: true - }); - const promise = new Promise(resolve => { - let timedOut = false; - let isFinished = false; - let error: Error | undefined; - - const timeoutTimer = setTimeout(() => { - if (isFinished) return; - timedOut = true; - worker.terminate(); - resolve({ - exitCode: 1, - status: 'timedout' - }); - }, timeout); - - worker.on('error', (e: Error) => { - error = e; - }); - - worker.on('exit', (exitCode: number) => { - if (isFinished) return; - if (timedOut) return; - clearTimeout(timeoutTimer); - isFinished = true; - resolve({ - status: exitCode === 0 ? 'succeeded' : 'errored', - exitCode, - error - }); - }); - }); - - return { - promise, - worker - }; - } - - private readToEnd(source: Readable): Promise { - return new Promise((res, rej) => { - const chunks: Buffer[] = []; - source.on('data', (chunk: Buffer) => chunks.push(chunk)); - source.on('end', () => res(Buffer.concat(chunks))); - source.on('error', (err: Error) => rej(err)); - }); - } - - /** - * Start a job instance. This will run the job and update the status of the instance. - * @param {JobInstance} instance - The job instance to start. - */ - protected async startJobInstance(instance: JobInstance): Promise { - const startDate = new Date(); - instance = await this._jobsRepository.saveInstance({ - ...instance, - status: 'running', - startDate - }); - - let status: JobInstanceStatus = 'errored', - exitCode: number = 1; - - try { - const job = await this._jobsRepository.getJobById(instance.jobId); - - const fileName = pathJoin(this._rootFolder, job.path); - - const props = this._jobProps.get(job.id); - - let finalProps = typeof props === 'function' ? props() : props; - if (finalProps instanceof Promise) { - finalProps = await finalProps; - } - - instance = await this._jobsRepository.saveInstance({ - ...instance, - status: 'running' - }); - - const { promise, worker } = this.runWorkerWithTimeout( - fileName, - finalProps, - job.timeout - ); - - const stdOutPass = worker.stdout.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - const stdErrPass = worker.stderr.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - - const stdOutForJobStart = worker.stdout.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - const stdErrForJobStart = worker.stderr.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - - const stdOutForJobEnd = worker.stdout.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - const stdErrForJobEnd = worker.stderr.pipe( - new PassThrough({ - highWaterMark: MAX_BUFFER_SIZE - }) - ); - - worker.on('message', (value: any) => { - this.emit('job:message', { - instanceId: instance.id, - jobId: job.id, - startDate, - value - }); - }); - - this.emit('job:start', { - instanceId: instance.id, - jobId: job.id, - stderr: stdErrForJobStart, - stdout: stdOutForJobStart, - startDate - } as JobStartItem); - - const stdOutStr = (await this.readToEnd(stdOutPass)).toString(); - const stdErrStr = (await this.readToEnd(stdErrPass)).toString(); - - const result = await promise; - status = result.status; - exitCode = result.exitCode; - const { error } = result; - - const endDate = new Date(); - - switch (status) { - case 'errored': - this.emit('job:error', { - instanceId: instance.id, - jobId: job.id, - stderr: stdErrForJobEnd, - stdout: stdOutForJobEnd, - startDate, - endDate, - error - } as JobErrorItem); - break; - case 'timedout': - this.emit('job:timeout', { - instanceId: instance.id, - jobId: job.id, - stderr: stdErrForJobEnd, - stdout: stdOutForJobEnd, - startDate, - endDate, - error - } as JobErrorItem); - break; - default: - this.emit('job:end', { - instanceId: instance.id, - jobId: job.id, - stderr: stdErrForJobEnd, - stdout: stdOutForJobEnd, - startDate, - endDate - } as JobStartItem); - } - - instance = await this._jobsRepository.saveInstance({ - ...instance, - status, - exitCode, - stdErr: stdErrStr, - stdOut: stdOutStr, - endDate - }); - } finally { - const job = { - ...(await this._jobsRepository.getJobById(instance.jobId)) - }; - - job.timesRunned!++; - - let shouldRetry = false; - - const schedule = await this.getJobSchedule(job); - - if (status !== 'succeeded') { - if ( - job.consequentFailsCount + 1 >= job.maxConsequentFails && - job.maxConsequentFails > 0 - ) { - job.status = 'disabled'; - } else { - job.consequentFailsCount += 1; - shouldRetry = true; - } - } else { - job.successfullTimesRunned!++; - job.consequentFailsCount = 0; - if (!schedule!.hasNext()) { - job.status = 'finished'; - } - } - - await this._jobsRepository.saveJob(job); - - if (shouldRetry && instance.retryIndex < instance.maxRetries) { - await this.startJobInstance({ - ...instance, - retryIndex: instance.retryIndex + 1 - }); - } - } - } - - protected async scheduleJobTo( - job: Job, - date: Date, - index: number - ): Promise { - let timer; - - try { - const now = new Date(); - let interval = date.getTime() - now.getTime(); - if (interval < 0) { - interval = 0; - } - - const instance = await this._jobsRepository.addInstance(job.id, { - scheduledTo: date, - status: 'scheduled', - timeout: job.timeout, - index, - retryIndex: 0, - maxRetries: job.maxRetries - }); - - timer = setTimeout(async () => { - const actualJob = await this._jobsRepository.getJobById(job.id); - - if (!actualJob || actualJob.status !== 'active') { - instance.status = 'canceled'; - await this._jobsRepository.saveInstance(instance); - return; - } - - await this.startJobInstance(instance); - }, interval); - - return instance; - } catch (_e) { - clearTimeout(timer); - // console.log(e); - // console.log('task failed!'); - return null; - } - } - - protected async checkForUpcomingJobs(): Promise { - const jobs = await this._jobsRepository.getJobs(); - for (let i = 0; i < jobs.length; i++) { - if (jobs[i].status !== 'active') continue; - - const schedule = await this.getJobSchedule(jobs[i]); - - if (schedule!.hasNext()) { - const scheduledInstances = - await this._jobsRepository.getInstancesWithStatus( - jobs[i].id, - 'scheduled' - ); - - while (schedule!.hasNext(SCHEDULE_JOB_SPAN)) { - const { date: nextRun, index } = schedule!.next(); - if (nextRun < new Date()) continue; - - if (jobs[i].noConcurrentRuns) { - const runingInstances = - await this._jobsRepository.getInstancesWithStatus( - jobs[i].id, - 'running' - ); - - if (runingInstances.length > 0) { - return; - } - } - - const alreadyScheduled = scheduledInstances.find( - i => i.index === index - ); - if (alreadyScheduled) continue; - - await this.scheduleJobTo(jobs[i], nextRun, index); - } - } else { - jobs[i].status = 'finished'; - await this._jobsRepository.saveJob(jobs[i]); - } - } - } - - /** - * Starts the scheduler (all jobs will be scheduled and executed when it's time). - */ - public async start() { - if (this._status === 'started') { - throw new Error('Scheduler is already started'); - } - - this.status = 'started'; - - this._checkTimer = setInterval( - this.checkForUpcomingJobs.bind(this), - CHECK_INTERVAL - ); - - // TODO: add logic - } - - /** - * Stops the scheduler (all jobs will be stopped and no new jobs will be scheduled). - */ - public stop() { - if (this._status === 'stopped') { - throw new Error('Scheduler is already stopped'); - } - - clearInterval(this._checkTimer); - - this.status = 'stopped'; - // TODO: add logic - } - - /** - * Checks if job exists by id - * @param {string} jobId - id of the job - * @returns {Promise} - true if job exists, false otherwise - */ - public async jobExists(jobId: string): Promise { - return (await this._jobsRepository.getJobById(jobId)) !== null; - } - - /** - * Removes job by its id - * @param jobId {string} - id of the job - * @returns {Promise} - */ - public async removeJob(jobId: string): Promise { - if (typeof jobId !== 'string' || !jobId) { - throw new Error('id is required'); - } - - await this._jobsRepository.removeJob(jobId); - } - - /** - * Adds job to the scheduler - * @param job {CreateJobRequest} - job to add - */ - public async addJob(job: CreateJobRequest) { - const validationResult = Schemas.CreateJobRequestSchema.validate(job); - if (!validationResult.valid) { - throw new Error( - `Invalid CreateJobRequest: ${validationResult.errors?.join( - '; ' - )}` - ); - } - - const path = pathJoin(this._rootFolder, job.path); - await fsAccess(path, fs.constants.R_OK); - - this._jobProps.set(job.id, job.props); - - await this._jobsRepository.createJob({ - id: job.id, - createdAt: new Date(), - schedule: job.schedule, - timeout: job.timeout || DEFAULT_JOB_TIMEOUT, - path: job.path, - consequentFailsCount: 0, - timesRunned: 0, - successfullTimesRunned: 0, - maxConsequentFails: - typeof job.maxConsequentFails === 'number' - ? job.maxConsequentFails - : DEFAULT_MAX_CONSEQUENT_FAILS, - maxRetries: - typeof job.maxRetries === 'number' - ? job.maxRetries - : DEFAULT_MAX_RETRIES, - noConcurrentRuns: job.noConcurrentRuns || false - }); - } - - /** - * - * @param props {JobSchedulerProps} - scheduler properties - */ - constructor(props: JobSchedulerProps) { - super(); - if (typeof props.rootFolder !== 'string') { - throw new Error('rootFolder must be a string'); - } - if (typeof props.defaultTimeZone === 'string') { - this._defaultTimezone = props.defaultTimeZone; - } - - if (typeof props.persistRepository === 'object') { - this._jobsRepository = props.persistRepository; - } - - this._rootFolder = props.rootFolder; - } - - public on(name: T, callback: Events[T]): this { - super.on(name, callback); - return this; - } -} +} from './schedule-schemas.js'; +export type { JobSchedulerOptions } from './scheduler.js'; +export { JobScheduler } from './scheduler.js'; +export type * from './storage.js'; +export type { JobWorkerOptions } from './worker.js'; +export { JobWorker } from './worker.js'; diff --git a/libs/scheduler/src/jobRepository.test.ts b/libs/scheduler/src/jobRepository.test.ts deleted file mode 100644 index c9e8c181..00000000 --- a/libs/scheduler/src/jobRepository.test.ts +++ /dev/null @@ -1,162 +0,0 @@ -import { describe, expect, test } from 'vitest'; -import { InMemoryJobRepository } from './jobRepository.js'; - -describe('InMemoryJobRepository', () => { - test('getJobs returns empty array initially', async () => { - const repo = new InMemoryJobRepository(); - expect(await repo.getJobs()).toEqual([]); - }); - - test('createJob adds a job with status active', async () => { - const repo = new InMemoryJobRepository(); - const job = await repo.createJob({ id: 'j1', name: 'test job' } as any); - expect(job.id).toBe('j1'); - expect(job.status).toBe('active'); - - const jobs = await repo.getJobs(); - expect(jobs).toHaveLength(1); - expect(jobs[0]).toEqual(job); - }); - - test('getJobById returns the correct job', async () => { - const repo = new InMemoryJobRepository(); - await repo.createJob({ id: 'j1', name: 'a' } as any); - await repo.createJob({ id: 'j2', name: 'b' } as any); - - const found = await repo.getJobById('j2'); - expect(found.id).toBe('j2'); - }); - - test('getJobById throws for missing job', async () => { - const repo = new InMemoryJobRepository(); - await expect(repo.getJobById('nonexistent')).rejects.toThrow( - 'Job with id nonexistent not found' - ); - }); - - test('removeJob removes an existing job', async () => { - const repo = new InMemoryJobRepository(); - await repo.createJob({ id: 'j1', name: 'a' } as any); - await repo.removeJob('j1'); - expect(await repo.getJobs()).toHaveLength(0); - }); - - test('removeJob throws for non-existent job', async () => { - const repo = new InMemoryJobRepository(); - await expect(repo.removeJob('nope')).rejects.toThrow( - 'Job with id nope not found' - ); - }); - - test('saveJob updates an existing job', async () => { - const repo = new InMemoryJobRepository(); - const job = await repo.createJob({ - id: 'j1', - path: '/original' - } as any); - const updated = await repo.saveJob({ ...job, path: '/updated' } as any); - expect(updated.path).toBe('/updated'); - const stored = await repo.getJobById('j1'); - expect(stored.path).toBe('/updated'); - }); - - test('saveJob adds a job if not present', async () => { - const repo = new InMemoryJobRepository(); - const newJob = { id: 'j99', path: '/new', status: 'active' } as any; - const saved = await repo.saveJob(newJob); - expect(saved.id).toBe('j99'); - expect(await repo.getJobs()).toHaveLength(1); - }); - - test('addInstance creates instance with correct jobId', async () => { - const repo = new InMemoryJobRepository(); - const instance = await repo.addInstance('j1', { - status: 'running', - startedAt: new Date() - } as any); - expect(instance.jobId).toBe('j1'); - expect(instance.id).toBe(1); - }); - - test('addInstance increments id for each new instance', async () => { - const repo = new InMemoryJobRepository(); - const i1 = await repo.addInstance('j1', { status: 'running' } as any); - const i2 = await repo.addInstance('j1', { status: 'pending' } as any); - expect(i1.id).toBe(1); - expect(i2.id).toBe(2); - }); - - test('getInstancesWithStatus returns matching instances', async () => { - const repo = new InMemoryJobRepository(); - await repo.addInstance('j1', { status: 'running' } as any); - await repo.addInstance('j1', { status: 'failed' } as any); - await repo.addInstance('j1', { status: 'running' } as any); - - const running = await repo.getInstancesWithStatus('j1', 'running'); - expect(running).toHaveLength(2); - expect(running.every(i => i.status === 'running')).toBe(true); - }); - - test('getInstancesWithStatus returns empty array when none match', async () => { - const repo = new InMemoryJobRepository(); - await repo.addInstance('j1', { status: 'running' } as any); - const result = await repo.getInstancesWithStatus('j1', 'failed' as any); - expect(result).toHaveLength(0); - }); - - test('saveInstance updates an existing instance', async () => { - const repo = new InMemoryJobRepository(); - const inst = await repo.addInstance('j1', { status: 'running' } as any); - const updated = await repo.saveInstance({ - ...inst, - status: 'completed' - } as any); - expect(updated.status).toBe('completed'); - }); - - test('saveInstance adds instance if not present (assigns new id)', async () => { - const repo = new InMemoryJobRepository(); - // The implementation overwrites the id with the internal counter - const newInst = { id: 99, jobId: 'j1', status: 'pending' } as any; - const saved = await repo.saveInstance(newInst); - expect(saved.jobId).toBe('j1'); - expect(saved.status).toBe('pending'); - const running = await repo.getInstancesWithStatus( - 'j1', - 'pending' as any - ); - expect(running).toHaveLength(1); - }); - - test('setJobStatus updates status on an existing job', async () => { - const repo = new InMemoryJobRepository(); - await repo.createJob({ id: 'j1', path: '/job1' } as any); - const updated = await repo.setJobStatus('j1', 'failed' as any); - expect(updated.id).toBe('j1'); - expect(updated.status).toBe('failed'); - const stored = await repo.getJobById('j1'); - expect(stored.status).toBe('failed'); - }); - - test('setJobStatus throws for a nonexistent job', async () => { - const repo = new InMemoryJobRepository(); - await expect( - repo.setJobStatus('no-such-id', 'active' as any) - ).rejects.toThrow('Job with id no-such-id not found'); - }); - - test('getInstanceById returns the correct instance', async () => { - const repo = new InMemoryJobRepository(); - const inst = await repo.addInstance('j1', { status: 'running' } as any); - const found = await repo.getInstanceById(inst.id); - expect(found).toBeDefined(); - expect(found.id).toBe(inst.id); - expect(found.jobId).toBe('j1'); - }); - - test('getInstanceById returns undefined for missing instance', async () => { - const repo = new InMemoryJobRepository(); - const result = await repo.getInstanceById(9999); - expect(result).toBeUndefined(); - }); -}); diff --git a/libs/scheduler/src/jobRepository.ts b/libs/scheduler/src/jobRepository.ts deleted file mode 100644 index 4fe68652..00000000 --- a/libs/scheduler/src/jobRepository.ts +++ /dev/null @@ -1,147 +0,0 @@ -import type { - Job, - JobInstance, - JobInstanceStatus, - JobStatus -} from './types.js'; - -type AddJobRequest = Omit; -type AddJobInstanceRequest = Omit; - -/** - * Persistence contract for jobs and their running instances. - * - * Implement this interface to store jobs in a database, file system, or - * any other persistent backend. See {@link InMemoryJobRepository} for a - * reference implementation. - */ -export interface IJobRepository { - getJobs(): Promise; - getJobById(jobId: string): Promise; - createJob(item: AddJobRequest): Promise; - removeJob(jobId: string): Promise; - - saveJob(job: Job): Promise; - - getInstancesWithStatus( - jobId: string, - status: JobInstanceStatus - ): Promise; - - addInstance( - jobId: string, - instance: AddJobInstanceRequest - ): Promise; - - saveInstance(instance: JobInstance): Promise; -} - -/** - * In-memory {@link IJobRepository} implementation. - * - * Useful for tests and development. All data is lost when the process exits. - */ -export class InMemoryJobRepository implements IJobRepository { - private _jobs: Array = []; - private _jobInstances: Array = []; - private _instanceId = 1; - - async getJobs(): Promise { - return this._jobs; - } - - async removeJob(jobId: string) { - await this.getJobById(jobId); - this._jobs = this._jobs.filter(j => j.id !== jobId); - } - - async createJob(item: AddJobRequest): Promise { - const job: Job = { - ...item, - status: 'active' - }; - this._jobs.push(job); - return job; - } - - async getInstances(jobId: string): Promise { - return this._jobInstances.filter(ji => ji.jobId === jobId); - } - - async addInstance( - jobId: string, - instance: AddJobInstanceRequest - ): Promise { - const newInstance = { - ...instance, - id: this._instanceId++, - jobId - }; - - this._jobInstances.push(newInstance); - - return newInstance; - } - - async getJobById(jobId: string): Promise { - const job = this._jobs.find(j => j.id === jobId); - if (!job) throw new Error(`Job with id ${jobId} not found`); - return job; - } - - async setJobStatus(jobId: string, status: JobStatus): Promise { - const job = await this.getJobById(jobId); - job.status = status; - return job; - } - - async saveJob(job: Job): Promise { - const index = this._jobs.findIndex(j => j.id === job.id); - if (index !== -1) { - this._jobs[index] = { - ...job - }; - return this._jobs[index]; - } - - const result = { - ...job - }; - - this._jobs.push(result); - return result; - } - - async getInstancesWithStatus( - jobId: string, - status: JobInstanceStatus - ): Promise { - return (await this.getInstances(jobId)).filter( - i => i.status === status - ); - } - - async getInstanceById(id: number): Promise { - return this._jobInstances.find(ji => ji.id === id)!; - } - - async saveInstance(instance: JobInstance): Promise { - const oldIndex = this._jobInstances.findIndex( - ji => ji.id === instance.id - ); - if (oldIndex !== -1) { - this._jobInstances[oldIndex] = { - ...instance - }; - return this._jobInstances[oldIndex]; - } - - const result = { - ...instance, - id: this._instanceId++ - }; - - this._jobInstances.push(result); - return result; - } -} diff --git a/libs/scheduler/src/memory.ts b/libs/scheduler/src/memory.ts new file mode 100644 index 00000000..1399b4e3 --- /dev/null +++ b/libs/scheduler/src/memory.ts @@ -0,0 +1,228 @@ +import type { + JobAttempt, + JobEvent, + RunRecord, + ScheduleRecord +} from './contracts.js'; +import { isTerminal, JobRepository } from './repository.js'; +import type { JobStorage, JobStorageTransaction } from './storage.js'; + +/** Shared process-local state; reuse one instance to model multiple workers. */ +export class InMemoryJobStorage implements JobStorage { + private runs = new Map(); + private schedules = new Map(); + private eventLog = new Map(); + private attemptLog = new Map(); + private tail: Promise = Promise.resolve(); + /** Inject a deterministic clock for conformance/recovery tests. */ + constructor(private readonly clock: () => number = Date.now) {} + + /** Serialize transactions and restore snapshots on failure. Returned data is detached. */ + async atomic( + action: (transaction: JobStorageTransaction) => Promise + ): Promise { + const previous = this.tail; + let release!: () => void; + this.tail = new Promise(resolve => { + release = resolve; + }); + await previous; + const snapshot = structuredClone([ + this.runs, + this.schedules, + this.eventLog, + this.attemptLog + ]); + const scheduleKey = (namespace: string, id: string) => + JSON.stringify([namespace, id]); + const tx: JobStorageTransaction = { + now: async () => this.clock(), + run: async (namespace, id) => { + const run = this.runs.get(id); + return run?.namespace === namespace + ? structuredClone(run) + : undefined; + }, + insertRun: async run => { + const existing = + run.dedupeKey === null + ? undefined + : [...this.runs.values()].find( + item => + item.namespace === run.namespace && + item.dedupeKey === run.dedupeKey + ); + if (existing) + return { + record: structuredClone(existing), + inserted: false + }; + this.runs.set(run.id, structuredClone(run)); + return { record: structuredClone(run), inserted: true }; + }, + saveRun: async run => { + this.runs.set(run.id, structuredClone(run)); + }, + runnable: async (namespace, supported) => { + const now = this.clock(); + return structuredClone( + [...this.runs.values()] + .filter( + run => + run.namespace === namespace && + (run.status === 'running' + ? run.leaseExpiresAt! <= now + : (run.status === 'queued' || + run.status === 'retry_wait') && + run.availableAt <= now && + supported.some( + job => + job.name === run.name && + job.version === run.version + )) + ) + .sort( + (a, b) => + a.availableAt - b.availableAt || + a.id.localeCompare(b.id) + )[0] + ); + }, + appendEvent: async event => { + const events = this.eventLog.get(event.runId) ?? []; + if (events.some(item => item.sequence === event.sequence)) + throw new Error('Duplicate event sequence'); + events.push(structuredClone(event)); + this.eventLog.set(event.runId, events); + }, + events: async (id, after, limit) => + structuredClone( + (this.eventLog.get(id) ?? []) + .filter(event => event.sequence > after) + .slice(0, limit) + ), + saveAttempt: async attempt => { + const attempts = this.attemptLog.get(attempt.runId) ?? []; + const index = attempts.findIndex( + item => item.attempt === attempt.attempt + ); + if (index < 0) attempts.push(structuredClone(attempt)); + else attempts[index] = structuredClone(attempt); + this.attemptLog.set(attempt.runId, attempts); + }, + attempts: async id => + structuredClone(this.attemptLog.get(id) ?? []), + schedule: async (namespace, id) => + structuredClone(this.schedules.get(scheduleKey(namespace, id))), + insertSchedule: async schedule => { + const key = scheduleKey(schedule.namespace, schedule.id); + if (!this.schedules.has(key)) + this.schedules.set(key, structuredClone(schedule)); + return structuredClone(this.schedules.get(key)!); + }, + saveSchedule: async schedule => { + this.schedules.set( + scheduleKey(schedule.namespace, schedule.id), + structuredClone(schedule) + ); + }, + dueSchedules: async (namespace, limit) => + structuredClone( + [...this.schedules.values()] + .filter( + item => + item.namespace === namespace && + item.active && + !item.removed && + item.nextAt !== null && + item.nextAt <= this.clock() + ) + .sort((a, b) => a.nextAt! - b.nextAt!) + .slice(0, limit) + ), + unfinishedScheduleRun: async (namespace, id) => + [...this.runs.values()].some( + run => + run.namespace === namespace && + run.scheduleId === id && + !isTerminal(run.status) + ), + cleanup: async (namespace, limit) => { + let count = 0; + for (const run of this.runs.values()) { + if (count >= limit) break; + if ( + run.namespace === namespace && + isTerminal(run.status) && + run.completedAt! + run.policy.retentionMs <= + this.clock() + ) { + this.runs.delete(run.id); + this.eventLog.delete(run.id); + this.attemptLog.delete(run.id); + count++; + } + } + return count; + }, + health: async namespace => { + const counts: Record = {}; + let oldestReadyAt: number | null = null; + const definitions = new Map< + string, + { name: string; version: number; count: number } + >(); + for (const run of this.runs.values()) { + if (run.namespace !== namespace) continue; + counts[run.status] = (counts[run.status] ?? 0) + 1; + if ( + run.status === 'queued' || + run.status === 'retry_wait' + ) { + if (run.availableAt <= this.clock()) + oldestReadyAt = Math.min( + oldestReadyAt ?? Infinity, + run.availableAt + ); + const key = JSON.stringify([run.name, run.version]); + const entry = definitions.get(key) ?? { + name: run.name, + version: run.version, + count: 0 + }; + entry.count++; + definitions.set(key, entry); + } + } + return { + counts, + oldestReadyAt, + queuedDefinitions: [...definitions.values()] + }; + } + }; + try { + return structuredClone(await action(tx)); + } catch (error) { + [this.runs, this.schedules, this.eventLog, this.attemptLog] = + snapshot as [ + typeof this.runs, + typeof this.schedules, + typeof this.eventLog, + typeof this.attemptLog + ]; + throw error; + } finally { + release(); + } + } +} + +/** Test/development repository. Its state is not durable across process exits. */ +export class InMemoryJobRepository extends JobRepository { + constructor( + options: { storage?: InMemoryJobStorage; now?: () => number } = {} + ) { + super(options.storage ?? new InMemoryJobStorage(options.now)); + } +} diff --git a/libs/scheduler/src/ScheduleCalculator.test.ts b/libs/scheduler/src/recurrence-compatibility.test.ts similarity index 96% rename from libs/scheduler/src/ScheduleCalculator.test.ts rename to libs/scheduler/src/recurrence-compatibility.test.ts index f38e719c..8f2947cb 100644 --- a/libs/scheduler/src/ScheduleCalculator.test.ts +++ b/libs/scheduler/src/recurrence-compatibility.test.ts @@ -1,6 +1,7 @@ +// v4 schedule fixtures; weekly intervals now consistently anchor to the start week. import { expect, test } from 'vitest'; -import { ScheduleCalculator } from './ScheduleCalculator.js'; +import { ScheduleCalculator } from './recurrence.js'; test('day - 1', () => { const calculator = new ScheduleCalculator({ @@ -174,9 +175,9 @@ test('week - 2', () => { }); const results = [ - new Date(Date.UTC(2022, 8, 12, 9, 0, 0, 0)), - new Date(Date.UTC(2022, 8, 13, 9, 0, 0, 0)), - new Date(Date.UTC(2022, 8, 26, 9, 0, 0, 0)) + new Date(Date.UTC(2022, 8, 19, 9, 0, 0, 0)), + new Date(Date.UTC(2022, 8, 20, 9, 0, 0, 0)), + new Date(Date.UTC(2022, 9, 3, 9, 0, 0, 0)) ]; for (let i = 0; i < results.length; i++) { @@ -486,7 +487,6 @@ test('minute - 3', () => { }); test('constructor defaults startsOn to now when not provided', () => { - // Covers the else branch (lines 65-66) where startsOn is not given const before = Date.now(); const calc = new ScheduleCalculator({ every: 'minute', @@ -503,7 +503,6 @@ test('constructor defaults startsOn to now when not provided', () => { }); test('hasNext(span) returns true when next date is within the span', () => { - // Covers lines 364-365: hasNext called with a numeric span const farFutureStart = new Date(Date.now() + 500); const calc = new ScheduleCalculator({ every: 'minute', @@ -519,7 +518,6 @@ test('hasNext(span) returns true when next date is within the span', () => { }); test('throws for unknown schedule type', () => { - // Covers line 336: default case in #getNext switch expect( () => new ScheduleCalculator({ @@ -527,10 +525,10 @@ test('throws for unknown schedule type', () => { interval: 1, startsOn: new Date() }) - ).toThrow('unknown schedule type'); + ).toThrow(); }); -test('hasNext(span) returns false when schedule is exhausted (line 364)', () => { +test('hasNext(span) returns false when schedule is exhausted', () => { const past = new Date(Date.now() - 10_000); const calc = new ScheduleCalculator({ every: 'minute', @@ -544,7 +542,7 @@ test('hasNext(span) returns false when schedule is exhausted (line 364)', () => expect(calc.hasNext(10_000)).toBe(false); }); -test('next() throws when schedule is over (line 379)', () => { +test('next() throws when schedule is over', () => { const past = new Date(Date.now() - 10_000); const calc = new ScheduleCalculator({ every: 'minute', @@ -553,10 +551,10 @@ test('next() throws when schedule is over (line 379)', () => { maxOccurences: 1 }); calc.next(); - expect(() => calc.next()).toThrow('schedule is over'); + expect(() => calc.next()).toThrow('exhausted'); }); -test('year - day:last covers lines 292-298', () => { +test('year - day:last', () => { const calculator = new ScheduleCalculator({ every: 'year', day: 'last', diff --git a/libs/scheduler/src/recurrence.test.ts b/libs/scheduler/src/recurrence.test.ts new file mode 100644 index 00000000..a8bf2670 --- /dev/null +++ b/libs/scheduler/src/recurrence.test.ts @@ -0,0 +1,197 @@ +import { execFileSync } from 'node:child_process'; +import { describe, expect, it } from 'vitest'; +import type { TaskSchedule } from './contracts.js'; +import { + latestDueOccurrence, + nextOccurrence, + occurrenceAt, + ScheduleCalculator, + storeSchedule +} from './recurrence.js'; + +function dates(rule: TaskSchedule, count: number) { + const calc = new ScheduleCalculator(rule); + return Array.from({ length: count }, () => calc.next().date.toISOString()); +} +describe('calendar recurrence', () => { + it('keeps gaps as non-executable slots for limits and seeking', () => { + const rule = storeSchedule( + { + every: 'day', + hour: 2, + minute: 30, + timeZone: 'America/New_York', + startsOn: new Date('2026-03-07T00:00:00Z'), + maxOccurrences: 3 + }, + 0 + ); + expect(occurrenceAt(rule, 1)).toMatchObject({ + index: 1, + executable: false + }); + expect(nextOccurrence(rule, 1)).toMatchObject({ + index: 2, + at: Date.parse('2026-03-09T06:30:00Z'), + executable: true + }); + expect( + latestDueOccurrence(rule, 0, Date.parse('2026-03-08T12:00:00Z')) + ?.index + ).toBe(0); + expect( + nextOccurrence({ ...rule, maxOccurrences: 2 }, 1) + ).toBeUndefined(); + }); + it('skips a missing local calendar date without shifting following occurrences', () => { + expect( + dates( + { + every: 'day', + timeZone: 'Pacific/Apia', + startsOn: new Date('2011-12-29T10:00:00Z') + }, + 3 + ) + ).toEqual([ + '2011-12-29T19:00:00.000Z', + '2011-12-30T19:00:00.000Z', + '2011-12-31T19:00:00.000Z' + ]); + }); + it('exhausts minute schedules at the Date limit instead of returning Invalid Date', () => { + const calc = new ScheduleCalculator({ + every: 'minute', + startsOn: new Date(8640000000000000) + }); + expect(calc.next().date.getTime()).toBe(8640000000000000); + expect(calc.hasNext()).toBe(false); + }); + it.each([ + 'UTC', + 'America/Los_Angeles', + 'Asia/Tokyo' + ])('is independent of the host TZ=%s', tz => { + const moduleUrl = new URL('../dist/index.js', import.meta.url).href; + const program = `import { ScheduleCalculator } from ${JSON.stringify(moduleUrl)}; + const calc = new ScheduleCalculator({ every: 'day', hour: 9, + timeZone: 'Europe/Berlin', startsOn: new Date('2026-03-28T00:00:00Z') }); + process.stdout.write(JSON.stringify([calc.next().date, calc.next().date]));`; + const result = execFileSync( + process.execPath, + ['--input-type=module', '--eval', program], + { + env: { ...process.env, TZ: tz }, + encoding: 'utf8', + timeout: 5000 + } + ); + expect(JSON.parse(result)).toEqual([ + '2026-03-28T08:00:00.000Z', + '2026-03-29T07:00:00.000Z' + ]); + }); + it('skips DST gaps and selects the earlier repeated wall time once', () => { + expect( + dates( + { + every: 'day', + hour: 2, + minute: 30, + timeZone: 'America/New_York', + startsOn: new Date('2026-03-07T00:00:00Z') + }, + 3 + ) + ).toEqual([ + '2026-03-07T07:30:00.000Z', + '2026-03-09T06:30:00.000Z', + '2026-03-10T06:30:00.000Z' + ]); + expect( + dates( + { + every: 'day', + hour: 1, + minute: 30, + timeZone: 'America/New_York', + startsOn: new Date('2026-10-31T00:00:00Z') + }, + 3 + ) + ).toEqual([ + '2026-10-31T05:30:00.000Z', + '2026-11-01T05:30:00.000Z', + '2026-11-02T06:30:00.000Z' + ]); + }); + it('supports last day, leap years, weeks and anchored intervals', () => { + expect( + dates( + { + every: 'month', + day: 'last', + startsOn: new Date('2028-01-01T00:00:00Z') + }, + 3 + ) + ).toEqual([ + '2028-01-31T09:00:00.000Z', + '2028-02-29T09:00:00.000Z', + '2028-03-31T09:00:00.000Z' + ]); + expect( + dates( + { + every: 'year', + month: 2, + day: 'last', + startsOn: new Date('2028-01-01T00:00:00Z') + }, + 2 + ) + ).toEqual(['2028-02-29T09:00:00.000Z', '2029-02-28T09:00:00.000Z']); + expect( + dates( + { + every: 'week', + interval: 2, + dayOfWeek: [1, 5], + startsOn: new Date('2026-10-01T00:00:00Z') + }, + 3 + ) + ).toEqual([ + '2026-10-02T09:00:00.000Z', + '2026-10-12T09:00:00.000Z', + '2026-10-16T09:00:00.000Z' + ]); + }); + it('counts calendar slots for limits and seeks over long outages', () => { + const calc = new ScheduleCalculator({ + every: 'minute', + startsOn: new Date(0), + skipFirst: 2, + maxOccurrences: 3 + }); + expect(calc.next().index).toBe(3); + expect(calc.hasNext()).toBe(false); + expect(() => calc.next()).toThrow('exhausted'); + const rule = storeSchedule( + { every: 'minute', startsOn: new Date(0) }, + 0 + ); + expect(latestDueOccurrence(rule, 0, 600000000000)?.index).toBe( + 10000000 + ); + }); + it.each([ + { every: 'day', timeZone: 'Wrong/Zone' }, + { every: 'week', dayOfWeek: [] }, + { every: 'month', day: 31 }, + { every: 'minute', interval: 0 }, + { every: 'day', hour: 24 } + ])('rejects invalid rule %j', rule => { + expect(() => new ScheduleCalculator(rule as TaskSchedule)).toThrow(); + }); +}); diff --git a/libs/scheduler/src/recurrence.ts b/libs/scheduler/src/recurrence.ts new file mode 100644 index 00000000..291a9a72 --- /dev/null +++ b/libs/scheduler/src/recurrence.ts @@ -0,0 +1,186 @@ +import { + addDays, + addMonths, + calendarDate, + localDate, + resolveLocal +} from './calendar.js'; +import type { StoredSchedule, TaskSchedule } from './contracts.js'; +import { normalizeSchedule } from './schedule-schemas.js'; + +/** One recurrence slot. A DST gap has an ordering instant but is not executable. */ +export type Occurrence = { index: number; at: number; executable: boolean }; + +/** Validate and anchor a recurrence exactly once, using the repository clock. */ +export function storeSchedule( + input: TaskSchedule, + now: number +): StoredSchedule { + const { startsOn, endsOn, ...schedule } = normalizeSchedule(input); + const start = startsOn?.getTime() ?? now; + if (!Number.isFinite(start) || (endsOn && endsOn.getTime() < start)) + throw new RangeError('Invalid schedule date range'); + return { + ...schedule, + startsOn: start, + ...(endsOn === undefined ? {} : { endsOn: endsOn.getTime() }) + }; +} + +/** Calculate a slot without reference to the wall clock or successful run count. */ +export function occurrenceAt( + rule: StoredSchedule, + index: number +): Occurrence | undefined { + if ( + !Number.isSafeInteger(index) || + index < 0 || + (rule.maxOccurrences !== undefined && index >= rule.maxOccurrences) + ) + return undefined; + const interval = rule.interval ?? 1; + let value: { at: number; executable: boolean }; + try { + if (rule.every === 'minute') { + const at = rule.startsOn + index * interval * 60000; + if ( + !Number.isSafeInteger(at) || + !Number.isFinite(new Date(at).getTime()) + ) + return undefined; + value = { at, executable: true }; + } else { + const zone = rule.timeZone ?? 'UTC'; + const anchor = localDate(rule.startsOn, zone); + const hour = rule.hour ?? 9, + minute = rule.minute ?? 0; + const onDate = (day: number, month = anchor.getUTCMonth() + 1) => + calendarDate(anchor.getUTCFullYear(), month, day, hour, minute); + let local: Date; + if (rule.every === 'day') { + let first = onDate(anchor.getUTCDate()); + if (resolveLocal(first, zone).at < rule.startsOn) + first = addDays(first, 1); + local = addDays(first, index * interval); + } else if (rule.every === 'week') { + const monday = addDays( + onDate(anchor.getUTCDate()), + 1 - (anchor.getUTCDay() || 7) + ); + const days = rule.dayOfWeek; + const initial = days.filter( + day => + resolveLocal(addDays(monday, day - 1), zone).at >= + rule.startsOn + ); + if (index < initial.length) + local = addDays(monday, initial[index] - 1); + else { + const remaining = index - initial.length; + local = addDays( + monday, + (Math.floor(remaining / days.length) + 1) * + interval * + 7 + + days[remaining % days.length] - + 1 + ); + } + } else { + let first = onDate( + 1, + rule.every === 'year' + ? rule.month + : anchor.getUTCMonth() + 1 + ); + const period = rule.every === 'year' ? 12 : 1; + const onDay = (date: Date) => + calendarDate( + date.getUTCFullYear(), + date.getUTCMonth() + (rule.day === 'last' ? 2 : 1), + rule.day === 'last' ? 0 : rule.day, + hour, + minute + ); + if (resolveLocal(onDay(first), zone).at < rule.startsOn) + first = addMonths(first, period); + local = onDay(addMonths(first, index * interval * period)); + } + value = resolveLocal(local, zone); + } + } catch (error) { + if (error instanceof RangeError) return undefined; // End of JavaScript Date's supported calendar range. + throw error; + } + if (rule.endsOn !== undefined && value.at > rule.endsOn) return undefined; + return { index, ...value }; +} + +/** Find the next real occurrence, skipping nonexistent local wall-clock times. */ +export function nextOccurrence( + rule: StoredSchedule, + from: number +): Occurrence | undefined { + let index = Math.max(from, rule.skipFirst ?? 0); + for (;;) { + const value = occurrenceAt(rule, index++); + if (!value || value.executable) return value; + } +} + +/** Seek logarithmically across long outages instead of iterating every missed minute. */ +export function latestDueOccurrence( + rule: StoredSchedule, + from: number, + now: number +): Occurrence | undefined { + const first = nextOccurrence(rule, from); + if (!first || first.at > now) return undefined; + let low = first.index, + high = low + 1, + distance = 1; + for (;;) { + const candidate = occurrenceAt(rule, high); + if (!candidate || candidate.at > now) break; + low = high; + distance *= 2; + high = Math.min(Number.MAX_SAFE_INTEGER, high + distance); + if (low === high) break; + } + while (high - low > 1) { + const mid = low + Math.floor((high - low) / 2); + const candidate = occurrenceAt(rule, mid); + if (candidate && candidate.at <= now) low = mid; + else high = mid; + } + for (let index = low; index >= first.index; index--) { + const candidate = occurrenceAt(rule, index); + if (candidate?.executable) return candidate; + } + return first; +} + +/** Deterministic calendar iterator. Supply startsOn for reproducible sequences. */ +export class ScheduleCalculator { + private cursor: number; + private readonly rule: StoredSchedule; + constructor(schedule: TaskSchedule) { + this.rule = storeSchedule(schedule, Date.now()); + this.cursor = this.rule.skipFirst ?? 0; + } + /** Whether an occurrence remains, optionally within a look-ahead window. */ + hasNext(withinMs?: number): boolean { + const next = nextOccurrence(this.rule, this.cursor); + return ( + !!next && + (withinMs === undefined || next.at <= Date.now() + withinMs) + ); + } + /** Advance one occurrence; index is one-based and includes skipped calendar slots. */ + next(): { date: Date; index: number } { + const next = nextOccurrence(this.rule, this.cursor); + if (!next) throw new RangeError('Schedule exhausted'); + this.cursor = next.index + 1; + return { date: new Date(next.at), index: next.index + 1 }; + } +} diff --git a/libs/scheduler/src/repository.test.ts b/libs/scheduler/src/repository.test.ts new file mode 100644 index 00000000..a41f15a5 --- /dev/null +++ b/libs/scheduler/src/repository.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it, vi } from 'vitest'; +import { repositoryContract, testJob } from '../testing/repository-contract.js'; +import { + InMemoryJobRepository, + InMemoryJobStorage, + JobScheduler +} from './index.js'; + +repositoryContract(async () => { + let time = Date.now(); + const repository = new InMemoryJobRepository({ now: () => time }); + return { + repository, + scheduler: new JobScheduler({ storageRepository: repository }), + advance: async ms => { + time += ms; + } + }; +}); + +describe('memory repository', () => { + it('anchors an omitted start only once, even after the end bound has passed', async () => { + let now = 1000; + const jobs = new JobScheduler({ + storageRepository: new InMemoryJobRepository({ now: () => now }) + }); + const options = { + schedule: { every: 'minute' as const, endsOn: new Date(2000) } + }; + await jobs.upsertSchedule('once', testJob(), { id: 'one' }, options); + expect(await jobs.dispatch()).toBe(1); + now = 5000; + const current = await jobs.upsertSchedule( + 'once', + testJob(), + { id: 'one' }, + options + ); + expect(current).toMatchObject({ + revision: 1, + cursor: 1, + schedule: { startsOn: 1000 } + }); + expect(await jobs.dispatch()).toBe(0); + }); + it.each([ + { every: 'minute', next: '2026-01-01T09:01:00Z' }, + { every: 'day', next: '2026-01-02T09:00:00Z' }, + { every: 'week', dayOfWeek: [4], next: '2026-01-08T09:00:00Z' }, + { every: 'month', day: 1, next: '2026-02-01T09:00:00Z' }, + { every: 'year', day: 1, month: 1, next: '2027-01-01T09:00:00Z' } + ] as const)('executes periodic $every jobs with independent dispatcher and worker lifecycles', async ({ + next, + ...rule + }) => { + let now = Date.parse('2026-01-01T09:00:00Z'); + const jobs = new JobScheduler({ + storageRepository: new InMemoryJobRepository({ now: () => now }), + pollIntervalMs: 2 + }); + const job = testJob(); + const worker = jobs.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async (input, context) => { + await context.report({ percent: 100 }); + return { url: '/' + input.id }; + }) + ] + }); + await jobs.upsertSchedule( + 'periodic', + job, + { id: 'one' }, + { + schedule: { + ...rule, + ...('dayOfWeek' in rule + ? { dayOfWeek: [...rule.dayOfWeek] } + : {}), + maxOccurrences: 2 + } + } + ); + try { + await worker.start(); + expect((await jobs.health()).counts.succeeded ?? 0).toBe(0); + await jobs.start(); + await vi.waitFor(async () => + expect((await jobs.health()).counts.succeeded).toBe(1) + ); + now = Date.parse(next) - 1; + expect(await jobs.dispatch()).toBe(0); + now++; + await vi.waitFor(async () => + expect((await jobs.health()).counts.succeeded).toBe(2) + ); + expect(await jobs.dispatch()).toBe(0); + expect(worker.lastError).toBeUndefined(); + expect(jobs.lastError).toBeUndefined(); + } finally { + await jobs.stop(); + await worker.stop(); + } + }); + it('shares detached state across repository instances', async () => { + const storage = new InMemoryJobStorage(); + const a = new JobScheduler({ + storageRepository: new InMemoryJobRepository({ storage }) + }); + const b = new JobScheduler({ + storageRepository: new InMemoryJobRepository({ storage }) + }); + const run = await a.enqueue(testJob(), { id: 'one' }); + run.input = 'mutated'; + expect((await b.getRun(testJob(), run.id))!.input).toEqual({ + id: 'one' + }); + }); + it.each([ + 'coalesce', + 'skip', + 'replay' + ] as const)('applies missed policy %s', async missed => { + const repository = new InMemoryJobRepository(); + const scheduler = new JobScheduler({ storageRepository: repository }); + await scheduler.upsertSchedule( + 'one', + testJob(), + { id: 'one' }, + { + schedule: { + every: 'minute', + startsOn: new Date(Date.now() - 600000) + }, + missed + } + ); + expect(await scheduler.dispatch()).toBe( + missed === 'coalesce' ? 1 : missed === 'skip' ? 0 : 11 + ); + expect(await scheduler.dispatch()).toBe(0); + }); + it('skips overlap including already queued occurrences', async () => { + const scheduler = new JobScheduler({ + storageRepository: new InMemoryJobRepository() + }); + await scheduler.upsertSchedule( + 'one', + testJob(), + { id: 'one' }, + { + schedule: { + every: 'minute', + startsOn: new Date(Date.now() - 600000) + }, + missed: 'replay', + overlap: 'skip' + } + ); + expect(await scheduler.dispatch()).toBe(1); + }); +}); diff --git a/libs/scheduler/src/repository.ts b/libs/scheduler/src/repository.ts new file mode 100644 index 00000000..74b8bea2 --- /dev/null +++ b/libs/scheduler/src/repository.ts @@ -0,0 +1,490 @@ +import { randomUUID } from 'node:crypto'; +import type { + JobAttempt, + JobError, + JobEvent, + JsonValue, + RunRecord, + ScheduleRecord, + TaskSchedule +} from './contracts.js'; +import { + fingerprint, + jsonValue, + LeaseLostError, + NonRetryableJobError, + SubmissionConflictError +} from './definition.js'; +import { + latestDueOccurrence, + nextOccurrence, + storeSchedule +} from './recurrence.js'; +import type { + JobStorage, + JobStorageTransaction, + SupportedJob +} from './storage.js'; + +/** Fields accepted by a repository after schema validation by the producer. */ +export type JobSubmission = Pick< + RunRecord, + | 'namespace' + | 'name' + | 'version' + | 'input' + | 'policy' + | 'dedupeKey' + | 'fingerprint' +> & { availableAt?: number }; +/** Schedule registration, before its initial anchor/cursor is persisted. */ +export type ScheduleSubmission = Omit< + ScheduleRecord, + 'revision' | 'active' | 'removed' | 'cursor' | 'nextAt' | 'schedule' +> & { schedule: TaskSchedule }; +/** Terminal states are immutable and eligible for retention cleanup. */ +export function isTerminal(status: string): boolean { + return ( + status === 'succeeded' || status === 'failed' || status === 'cancelled' + ); +} + +/** + * Shared durable transition engine. Adapters supply locking/storage, not policy. + * Lease fencing protects scheduler state only, never a handler's external effects. + */ +export class JobRepository { + constructor(readonly storage: JobStorage) {} + + private async emit( + tx: JobStorageTransaction, + run: RunRecord, + type: JobEvent['type'], + data: JsonValue | JobError | null, + at: number + ) { + run.sequence++; + await tx.appendEvent({ + runId: run.id, + sequence: run.sequence, + attempt: run.attempt, + at, + ...(type === 'progress' + ? { type, data } + : { type, data: data as JobError | null }) + }); + await tx.saveRun(run); + } + + private async insert( + tx: JobStorageTransaction, + submission: JobSubmission, + now: number, + scheduleId: string | null = null + ): Promise { + const record: RunRecord = { + ...submission, + id: randomUUID(), + status: 'queued', + output: null, + error: null, + attempt: 0, + createdAt: now, + availableAt: submission.availableAt ?? now, + completedAt: null, + scheduleId, + sequence: 0, + leaseToken: null, + leaseExpiresAt: null, + progressCount: 0 + }; + const result = await tx.insertRun(record); + if (result.record.fingerprint !== submission.fingerprint) + throw new SubmissionConflictError(); + if (result.inserted) await this.emit(tx, record, 'queued', null, now); + return result.inserted ? record : result.record; + } + + /** Resolve only after persisted acceptance (or the enclosing transaction commit). */ + enqueue(submission: JobSubmission): Promise { + return this.storage.atomic(async tx => + this.insert(tx, submission, await tx.now()) + ); + } + + private async attemptEnd( + tx: JobStorageTransaction, + run: RunRecord, + status: JobAttempt['status'], + error: JobError | null, + now: number + ) { + const attempt = (await tx.attempts(run.id)).find( + item => item.attempt === run.attempt + ); + if (!attempt) throw new Error('Missing running attempt'); + await tx.saveAttempt({ ...attempt, endedAt: now, status, error }); + } + + private async endFailed( + tx: JobStorageTransaction, + run: RunRecord, + error: JobError, + retryable: boolean, + interrupted: boolean, + now: number + ) { + await this.attemptEnd( + tx, + run, + interrupted ? 'interrupted' : 'failed', + error, + now + ); + const retry = run.policy.retry; + run.error = error; + run.leaseToken = null; + run.leaseExpiresAt = null; + if (retryable && run.attempt < retry.maxAttempts) { + run.status = 'retry_wait'; + run.availableAt = + now + + Math.min( + retry.maxDelayMs, + retry.initialDelayMs * 2 ** (run.attempt - 1) + ); + } else { + run.status = 'failed'; + run.completedAt = now; + } + await this.emit(tx, run, run.status, error, now); + } + + /** Atomically acquire one run. Expired attempts are recorded before any retry. */ + claim( + namespace: string, + supported: readonly SupportedJob[], + leaseMs: number + ): Promise { + return this.storage.atomic(async tx => { + const run = await tx.runnable(namespace, supported); + if (!run) return undefined; + const now = await tx.now(); + if (run.status === 'running') { + await this.endFailed( + tx, + run, + { code: 'lease_expired', message: 'Worker lease expired' }, + true, + true, + now + ); + return undefined; + } + run.status = 'running'; + run.attempt++; + run.error = null; + run.leaseToken = randomUUID(); + run.leaseExpiresAt = now + leaseMs; + await tx.saveAttempt({ + runId: run.id, + attempt: run.attempt, + startedAt: now, + endedAt: null, + status: 'running', + error: null + }); + await this.emit(tx, run, 'running', null, now); + return run; + }); + } + + private owned( + namespace: string, + id: string, + token: string, + action: ( + tx: JobStorageTransaction, + run: RunRecord, + now: number + ) => Promise + ): Promise { + return this.storage.atomic(async tx => { + const run = await tx.run(namespace, id, true); + const now = await tx.now(); + if ( + !run || + run.status !== 'running' || + run.leaseToken !== token || + run.leaseExpiresAt! <= now + ) + throw new LeaseLostError(); + return action(tx, run, now); + }); + } + + /** Renew an unexpired lease; a late heartbeat cannot resurrect ownership. */ + heartbeat( + namespace: string, + id: string, + token: string, + leaseMs: number + ): Promise { + return this.owned(namespace, id, token, async (tx, run, now) => { + run.leaseExpiresAt = now + leaseMs; + await tx.saveRun(run); + }); + } + + /** Commit ordered progress within the ownership transaction. */ + report( + namespace: string, + id: string, + token: string, + progress: JsonValue + ): Promise { + return this.owned(namespace, id, token, async (tx, run, now) => { + const data = jsonValue(progress, run.policy.maxProgressBytes); + if (run.progressCount >= run.policy.maxProgressEvents) + throw new NonRetryableJobError( + 'Progress event limit exceeded', + 'progress_limit' + ); + run.progressCount++; + await this.emit(tx, run, 'progress', data, now); + }); + } + + /** Publish output and terminal event atomically, rejecting stale owners. */ + complete( + namespace: string, + id: string, + token: string, + output: JsonValue + ): Promise { + return this.owned(namespace, id, token, async (tx, run, now) => { + run.output = jsonValue(output, run.policy.maxPayloadBytes); + await this.attemptEnd(tx, run, 'succeeded', null, now); + run.status = 'succeeded'; + run.completedAt = now; + run.leaseToken = null; + run.leaseExpiresAt = null; + await this.emit(tx, run, 'succeeded', null, now); + }); + } + + /** Persist a failure or shutdown interruption and its retry decision. */ + fail( + namespace: string, + id: string, + token: string, + error: JobError, + retryable = true, + interrupted = false + ): Promise { + return this.owned(namespace, id, token, (tx, run, now) => + this.endFailed(tx, run, error, retryable, interrupted, now) + ); + } + + /** Fence a running owner immediately; physical cancellation is cooperative. */ + cancel(namespace: string, id: string): Promise { + return this.storage.atomic(async tx => { + const run = await tx.run(namespace, id, true); + if (!run || isTerminal(run.status)) return false; + const now = await tx.now(); + if (run.status === 'running') + await this.attemptEnd(tx, run, 'cancelled', null, now); + run.status = 'cancelled'; + run.completedAt = now; + run.leaseToken = null; + run.leaseExpiresAt = null; + await this.emit(tx, run, 'cancelled', null, now); + return true; + }); + } + + /** Read a detached internal record. Applications should use JobScheduler.getRun. */ + get(namespace: string, id: string): Promise { + return this.storage.atomic(tx => tx.run(namespace, id)); + } + /** Read a bounded page of committed events, scoped through the owning run. */ + events( + namespace: string, + id: string, + after: number, + limit = 100 + ): Promise { + return this.storage.atomic(async tx => + (await tx.run(namespace, id)) ? tx.events(id, after, limit) : [] + ); + } + /** Read attempt audit history, scoped through the owning run. */ + attempts(namespace: string, id: string): Promise { + return this.storage.atomic(async tx => + (await tx.run(namespace, id)) ? tx.attempts(id) : [] + ); + } + /** Purge terminal data only; deduplication ends when the retained run is removed. */ + cleanup(namespace: string, limit = 100): Promise { + return this.storage.atomic(tx => tx.cleanup(namespace, limit)); + } + /** Inspect aggregate queue health without reading payloads. */ + health(namespace: string) { + return this.storage.atomic(tx => tx.health(namespace)); + } + + /** Idempotent identical registration; changed specs create future-only revisions. */ + upsertSchedule(submission: ScheduleSubmission): Promise { + return this.storage.atomic(async tx => { + const now = await tx.now(); + const existing = await tx.schedule( + submission.namespace, + submission.id + ); + if ( + existing && + !existing.removed && + existing.fingerprint === submission.fingerprint + ) + return existing; + const rule = storeSchedule(submission.schedule, now); + const cursor = rule.skipFirst ?? 0; + const proposed: ScheduleRecord = { + ...submission, + schedule: rule, + revision: 1, + active: true, + removed: false, + cursor, + nextAt: nextOccurrence(rule, cursor)?.at ?? null + }; + const current = await tx.insertSchedule(proposed); + if ( + current.fingerprint === proposed.fingerprint && + !current.removed + ) + return current; + // Changed triggers only affect future slots, even when callers retain + // the original startsOn anchor. Never replay a previous revision's past. + const past = latestDueOccurrence( + rule, + cursor, + (await tx.now()) - 1 + ); + const revisedCursor = past ? past.index + 1 : cursor; + const next = { + ...proposed, + revision: current.revision + 1, + cursor: revisedCursor, + nextAt: nextOccurrence(rule, revisedCursor)?.at ?? null + }; + await tx.saveSchedule(next); + return next; + }); + } + /** Pausing stops new occurrences, not already accepted runs. */ + pauseSchedule( + namespace: string, + id: string, + paused = true + ): Promise { + return this.storage.atomic(async tx => { + const current = await tx.schedule(namespace, id); + if (!current || current.removed) return false; + await tx.saveSchedule({ ...current, active: !paused }); + return true; + }); + } + /** Tombstone a trigger while preserving identity/revision history. */ + removeSchedule(namespace: string, id: string): Promise { + return this.storage.atomic(async tx => { + const current = await tx.schedule(namespace, id); + if (!current || current.removed) return false; + await tx.saveSchedule({ ...current, active: false, removed: true }); + return true; + }); + } + + /** Atomically materialize due occurrences and advance locked schedule cursors. */ + dispatch( + namespace: string, + scheduleLimit = 10, + replayLimit = 100 + ): Promise { + return this.storage.atomic(async tx => { + const schedules = await tx.dueSchedules(namespace, scheduleLimit); + let inserted = 0; + for (const schedule of schedules) { + const now = await tx.now(); + const first = nextOccurrence( + schedule.schedule, + schedule.cursor + ); + if (!first || first.at > now) continue; + const due: (typeof first)[] = []; + if (schedule.missed === 'replay') { + let candidate: typeof first | undefined = first; + while ( + candidate && + candidate.at <= now && + due.length < replayLimit + ) { + due.push(candidate); + candidate = nextOccurrence( + schedule.schedule, + candidate.index + 1 + ); + } + schedule.cursor = due.at(-1)!.index + 1; + } else { + const last = latestDueOccurrence( + schedule.schedule, + first.index, + now + )!; + if ( + schedule.missed === 'coalesce' || + last.index === first.index + ) + due.push(last); + schedule.cursor = last.index + 1; + } + for (const occurrence of due) { + if ( + schedule.overlap === 'skip' && + (await tx.unfinishedScheduleRun(namespace, schedule.id)) + ) + continue; + const submission = { + namespace, + name: schedule.name, + version: schedule.version, + input: schedule.input, + policy: schedule.policy, + availableAt: occurrence.at, + dedupeKey: + 'occurrence:' + + fingerprint([ + schedule.id, + schedule.revision, + occurrence.index + ]), + fingerprint: fingerprint([ + schedule.fingerprint, + schedule.revision, + occurrence.index + ]) + }; + await this.insert(tx, submission, now, schedule.id); + inserted++; + } + schedule.nextAt = + nextOccurrence(schedule.schedule, schedule.cursor)?.at ?? + null; + await tx.saveSchedule(schedule); + } + return inserted; + }); + } +} diff --git a/libs/scheduler/src/schedule-schemas.test-d.ts b/libs/scheduler/src/schedule-schemas.test-d.ts new file mode 100644 index 00000000..12d65cda --- /dev/null +++ b/libs/scheduler/src/schedule-schemas.test-d.ts @@ -0,0 +1,32 @@ +import type { InferType } from '@cleverbrush/schema'; +import { expectTypeOf } from 'vitest'; +import type { Schedule, ScheduleSchema, TaskSchedule } from './index.js'; + +expectTypeOf().toEqualTypeOf>(); +expectTypeOf().toEqualTypeOf(); +const examples: Schedule[] = [ + { every: 'minute' }, + { every: 'day', hour: 12 }, + { every: 'week', dayOfWeek: [1, 5] }, + { every: 'month', day: 'last' }, + { every: 'year', day: 1, month: 1 } +]; +for (const value of examples) { + if (value.every === 'week') + expectTypeOf(value.dayOfWeek).toEqualTypeOf(); + if (value.every === 'year') + expectTypeOf(value.month).toEqualTypeOf(); + if (value.every === 'minute') { + // @ts-expect-error Elapsed-minute schedules do not have a local hour. + value.hour; + } +} +// @ts-expect-error Weekly schedules require weekdays. +const week: Schedule = { every: 'week' }; +// @ts-expect-error Monthly schedules require a day. +const month: Schedule = { every: 'month' }; +// @ts-expect-error Yearly schedules require a month. +const year: Schedule = { every: 'year', day: 1 }; +// @ts-expect-error Irrelevant local time is not part of the minute variant. +const minute: Schedule = { every: 'minute', hour: 9 }; +void [week, month, year, minute]; diff --git a/libs/scheduler/src/schedule-schemas.test.ts b/libs/scheduler/src/schedule-schemas.test.ts new file mode 100644 index 00000000..681a0900 --- /dev/null +++ b/libs/scheduler/src/schedule-schemas.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from 'vitest'; +import { testJob } from '../testing/repository-contract.js'; +import { + InMemoryJobRepository, + JobScheduler, + ScheduleCalculator, + ScheduleDaySchema, + ScheduleMinuteSchema, + ScheduleMonthSchema, + ScheduleSchema, + ScheduleSchemaBase, + ScheduleWeekSchema, + ScheduleYearSchema, + Schemas +} from './index.js'; +import { normalizeSchedule } from './schedule-schemas.js'; + +describe('schedule schemas', () => { + it('exports independently usable variants and the Schemas facade', () => { + expect(Schemas).toEqual({ + ScheduleSchemaBase, + ScheduleMinuteSchema, + ScheduleDaySchema, + ScheduleWeekSchema, + ScheduleMonthSchema, + ScheduleYearSchema, + ScheduleSchema + }); + for (const [schema, input] of [ + [ScheduleMinuteSchema, { every: 'minute', interval: 5 }], + [ScheduleDaySchema, { every: 'day', hour: 12 }], + [ScheduleWeekSchema, { every: 'week', dayOfWeek: [7, 1] }], + [ScheduleMonthSchema, { every: 'month', day: 'last' }], + [ScheduleYearSchema, { every: 'year', day: 28, month: 2 }] + ] as const) { + expect(schema.validate(input).valid).toBe(true); + expect(ScheduleSchema.validate(input).valid).toBe(true); + } + }); + it('parses JSON dates and normalizes defaults without mutating or anchoring input', () => { + const input = { + every: 'week', + dayOfWeek: [5, 1], + maxOccurences: 4, + endsOn: '2030-01-01T00:00:00Z' + }; + const normalized = normalizeSchedule(input); + expect(normalized).toEqual({ + every: 'week', + dayOfWeek: [1, 5], + maxOccurrences: 4, + endsOn: new Date(input.endsOn), + interval: 1, + timeZone: 'UTC', + skipFirst: 0, + hour: 9, + minute: 0 + }); + expect(normalized).not.toHaveProperty('startsOn'); + expect(input.dayOfWeek).toEqual([5, 1]); + const parsed = ScheduleSchema.parse({ + every: 'minute', + startsOn: '2026-01-01T00:00:00Z', + maxOccurrences: 1 + }); + expect(new ScheduleCalculator(parsed).next()).toEqual({ + date: new Date('2026-01-01T00:00:00Z'), + index: 1 + }); + }); + it.each([ + { every: 'hour' }, + { every: 'week' }, + { every: 'month' }, + { every: 'year', day: 1 }, + { every: 'year', month: 1 }, + { every: 'minute', hour: 9 }, + { every: 'day', dayOfWeek: [1] }, + { every: 'week', dayOfWeek: [] }, + { every: 'week', dayOfWeek: [1, 1] }, + { every: 'week', dayOfWeek: [0] }, + { every: 'week', dayOfWeek: [8] }, + { every: 'day', interval: 0 }, + { every: 'day', interval: 357 }, + { every: 'day', interval: 1.5 }, + { every: 'day', hour: 24 }, + { every: 'day', minute: 60 }, + { every: 'month', day: 29 }, + { every: 'year', day: 1, month: 13 }, + { every: 'day', startsOn: 'invalid' }, + { every: 'day', startsOn: new Date(Number.NaN) }, + { every: 'day', startsOn: new Date(1), endsOn: new Date(0) }, + { every: 'day', maxOccurrences: 1, maxOccurences: 1 }, + { every: 'day', maxOccurrences: 0 }, + { every: 'day', maxOccurences: 0 }, + { every: 'day', skipFirst: -1 }, + { every: 'day', skipFirst: Number.MAX_SAFE_INTEGER + 1 }, + { every: 'day', timeZone: '+02:00' }, + { every: 'day', timeZone: 'Wrong/Zone' } + ])('rejects %j at validation, calculation and registration boundaries', async input => { + expect(ScheduleSchema.validate(input).valid).toBe(false); + expect(() => new ScheduleCalculator(input as any)).toThrow(); + const jobs = new JobScheduler({ + storageRepository: new InMemoryJobRepository() + }); + await expect( + jobs.upsertSchedule( + 'bad', + testJob(), + { id: 'one' }, + { schedule: input as any } + ) + ).rejects.toThrow(); + expect((await jobs.health()).counts.queued ?? 0).toBe(0); + }); + it('keeps shared constraints after schema composition', () => { + for (const every of [ + 'minute', + 'day', + 'week', + 'month', + 'year' + ] as const) { + const variant = + every === 'week' + ? { dayOfWeek: [1] } + : every === 'month' + ? { day: 1 } + : every === 'year' + ? { day: 1, month: 1 } + : {}; + expect( + ScheduleSchema.validate({ + every, + ...variant, + maxOccurrences: 2, + maxOccurences: 2 + }).valid + ).toBe(false); + } + }); +}); diff --git a/libs/scheduler/src/schedule-schemas.ts b/libs/scheduler/src/schedule-schemas.ts new file mode 100644 index 00000000..d90009dd --- /dev/null +++ b/libs/scheduler/src/schedule-schemas.ts @@ -0,0 +1,159 @@ +import { + array, + date, + type InferType, + number, + object, + string, + union +} from '@cleverbrush/schema'; +import { timeZoneFormatter } from './calendar.js'; + +const count = () => number().isInteger().min(1).max(Number.MAX_SAFE_INTEGER); +const scheduleDate = date() + .coerce() + .addValidator(value => + Number.isFinite(value.getTime()) + ? { valid: true } + : { valid: false, errors: [{ message: 'Invalid schedule date' }] } + ); +const calendarDay = union(string('last')).or( + number().isInteger().min(1).max(28) +); + +/** Shared recurrence bounds and local time; omitted values are normalized at registration. */ +export const ScheduleSchemaBase = object({ + /** Number of periods between occurrences; defaults to 1. */ + interval: count().max(356).optional(), + /** Named IANA time zone; defaults to UTC. Fixed numeric offsets are rejected. */ + timeZone: string() + .addValidator(value => { + try { + timeZoneFormatter(value); + return { valid: true }; + } catch { + return { + valid: false, + errors: [{ message: 'Use a named IANA time zone' }] + }; + } + }) + .optional(), + /** Local hour, 0–23; defaults to 9 for calendar schedules. */ + hour: number().isInteger().min(0).max(23).optional(), + /** Local minute, 0–59; defaults to 0 for calendar schedules. */ + minute: number().isInteger().min(0).max(59).optional(), + /** Inclusive start; omitted starts are anchored once by the repository clock. */ + startsOn: scheduleDate.optional(), + /** Inclusive end; JSON date strings are accepted at validation boundaries. */ + endsOn: scheduleDate.optional(), + /** Maximum calendar slots, including skipped slots, not successful executions. */ + maxOccurrences: count().optional(), + /** @deprecated Use maxOccurrences. Supplying both spellings is invalid. */ + maxOccurences: count().optional(), + /** Number of initial calendar slots to skip; defaults to 0. */ + skipFirst: number() + .isInteger() + .min(0) + .max(Number.MAX_SAFE_INTEGER) + .optional() +}).addValidator(value => { + if (value.maxOccurrences !== undefined && value.maxOccurences !== undefined) + return { + valid: false, + errors: [ + { message: 'Supply only maxOccurrences, not both spellings' } + ] + }; + if ( + value.startsOn && + value.endsOn && + new Date(value.endsOn).getTime() < new Date(value.startsOn).getTime() + ) + return { + valid: false, + errors: [{ message: 'endsOn must not precede startsOn' }] + }; + return { valid: true }; +}); + +/** Elapsed-minute recurrence, including the start instant; no wall-clock hour/minute. */ +export const ScheduleMinuteSchema = ScheduleSchemaBase.omit('hour') + .omit('minute') + .addProps({ every: string('minute') }); +/** Daily calendar recurrence at a local wall-clock time. */ +export const ScheduleDaySchema = ScheduleSchemaBase.addProps({ + every: string('day') +}); +/** Weekly recurrence on unique ISO weekdays (Monday = 1, Sunday = 7). */ +export const ScheduleWeekSchema = ScheduleSchemaBase.addProps({ + every: string('week'), + dayOfWeek: array() + .of(number().isInteger().min(1).max(7)) + .minLength(1) + .maxLength(7) + .addValidator(value => + new Set(value).size === value.length + ? { valid: true } + : { + valid: false, + errors: [{ message: 'Weekdays must be unique' }] + } + ) +}); +/** Monthly recurrence on day 1–28 or the last day of the month. */ +export const ScheduleMonthSchema = ScheduleSchemaBase.addProps({ + every: string('month'), + day: calendarDay +}); +/** Yearly recurrence on an explicit month and day (or the month's last day). */ +export const ScheduleYearSchema = ScheduleSchemaBase.addProps({ + every: string('year'), + day: calendarDay, + month: number().isInteger().min(1).max(12) +}); +/** Discriminated recurrence contract shared by registration and calendar calculation. */ +export const ScheduleSchema = union(ScheduleMinuteSchema) + .or(ScheduleDaySchema) + .or(ScheduleWeekSchema) + .or(ScheduleMonthSchema) + .or(ScheduleYearSchema); + +/** Calendar recurrence inferred from the public validation schema. */ +export type Schedule = InferType; + +/** Reusable schema members for validating schedule configuration. */ +export const Schemas = { + ScheduleSchemaBase, + ScheduleMinuteSchema, + ScheduleDaySchema, + ScheduleWeekSchema, + ScheduleMonthSchema, + ScheduleYearSchema, + ScheduleSchema +}; + +/** @internal Canonicalize without inventing a start instant before fingerprinting. */ +export function normalizeSchedule(input: unknown): Schedule { + const parsed = ScheduleSchema.parse(input); + const { maxOccurences, maxOccurrences, startsOn, endsOn, ...rest } = parsed; + const limit = maxOccurrences ?? maxOccurences; + const common = { + interval: parsed.interval ?? 1, + timeZone: parsed.timeZone ?? 'UTC', + skipFirst: parsed.skipFirst ?? 0, + ...(startsOn === undefined ? {} : { startsOn: new Date(startsOn) }), + ...(endsOn === undefined ? {} : { endsOn: new Date(endsOn) }), + ...(limit === undefined ? {} : { maxOccurrences: limit }) + }; + if (rest.every === 'minute') return { ...rest, ...common }; + const time = { hour: rest.hour ?? 9, minute: rest.minute ?? 0 }; + if (rest.every === 'week') + return { + ...rest, + ...common, + ...time, + dayOfWeek: [...rest.dayOfWeek].sort((a, b) => a - b) + }; + return { ...rest, ...common, ...time }; +} diff --git a/libs/scheduler/src/scheduler.ts b/libs/scheduler/src/scheduler.ts new file mode 100644 index 00000000..f861bad7 --- /dev/null +++ b/libs/scheduler/src/scheduler.ts @@ -0,0 +1,274 @@ +import type { InferType } from '@cleverbrush/schema'; +import type { + JobDefinition, + JobEvent, + JobRun, + RunRecord, + TaskSchedule +} from './contracts.js'; +import { fingerprint, identity, parsePayload, positive } from './definition.js'; +import { isTerminal, type JobRepository } from './repository.js'; +import { normalizeSchedule } from './schedule-schemas.js'; +import { delay } from './wait.js'; +import { JobWorker, type JobWorkerOptions } from './worker.js'; + +/** Producer and schedule-dispatcher options; no implicit in-memory persistence. */ +export type JobSchedulerOptions = { + /** Persistence boundary used by producers, dispatchers and workers. */ + storageRepository: JobRepository; + namespace?: string; + pollIntervalMs?: number; + onError?: (error: unknown) => void; +}; +/** Durable scheduling and production API. Workers have a separate lifecycle. */ +export class JobScheduler { + readonly namespace: string; + private readonly pollMs: number; + private controller?: AbortController; + private loop?: Promise; + lastError: unknown; + constructor(private readonly options: JobSchedulerOptions) { + this.namespace = identity(options.namespace ?? 'default', 'namespace'); + this.pollMs = positive( + options.pollIntervalMs ?? 1000, + 'pollIntervalMs', + 2147483647 + ); + } + /** Accept a validated immediate/delayed job, or return its retained duplicate. */ + async enqueue>( + definition: D, + input: InferType, + options: { idempotencyKey?: string; runAt?: Date } = {} + ): Promise>> { + const parsed = parsePayload( + definition.input, + input, + definition.policy.maxPayloadBytes + ); + const at = options.runAt?.getTime(); + if (at !== undefined && !Number.isFinite(at)) + throw new RangeError('Invalid runAt'); + const key = + options.idempotencyKey === undefined + ? null + : 'submission:' + + fingerprint([ + definition.name, + definition.version, + identity(options.idempotencyKey, 'idempotencyKey') + ]); + const run = await this.options.storageRepository.enqueue({ + namespace: this.namespace, + name: definition.name, + version: definition.version, + input: parsed, + policy: structuredClone(definition.policy), + dedupeKey: key, + fingerprint: fingerprint([parsed, definition.policy, at ?? null]), + availableAt: at + }); + return this.snapshot(definition, run); + } + private snapshot>( + definition: D, + run: RunRecord + ): JobRun> { + if (run.name !== definition.name || run.version !== definition.version) + throw new TypeError('Job definition does not match run'); + const { + leaseToken: _token, + leaseExpiresAt: _expiry, + dedupeKey: _key, + fingerprint: _hash, + policy: _policy, + progressCount: _count, + ...snapshot + } = run; + return { + ...snapshot, + output: + run.status === 'succeeded' + ? parsePayload( + definition.output, + run.output, + run.policy.maxPayloadBytes + ) + : null + }; + } + /** Read a typed snapshot without leaking worker ownership credentials. */ + async getRun>( + definition: D, + id: string + ): Promise> | undefined> { + const run = await this.options.storageRepository.get( + this.namespace, + id + ); + return run ? this.snapshot(definition, run) : undefined; + } + /** Replay then follow committed events; disconnecting does not cancel the job. */ + async *events>( + definition: D, + id: string, + options: { after?: number; signal?: AbortSignal } = {} + ): AsyncGenerator>> { + let cursor = options.after ?? 0; + if (!Number.isSafeInteger(cursor) || cursor < 0) + throw new RangeError('Invalid event cursor'); + while (!options.signal?.aborted) { + const run = await this.options.storageRepository.get( + this.namespace, + id + ); + if (!run) return; + this.snapshot(definition, run); + const events = await this.options.storageRepository.events( + this.namespace, + id, + cursor + ); + for (const event of events) { + if (options.signal?.aborted) return; + yield event.type === 'progress' + ? { + ...event, + data: parsePayload( + definition.progress, + event.data, + run.policy.maxProgressBytes + ) + } + : event; + cursor = event.sequence; + } + if (isTerminal(run.status) && cursor >= run.sequence) return; + if (events.length < 100) await delay(this.pollMs, options.signal); + } + } + /** Fence cancellation; application authorization must happen before calling. */ + cancel(id: string): Promise { + return this.options.storageRepository.cancel(this.namespace, id); + } + /** Create a separately startable worker sharing this namespace/repository. */ + createWorker(options: JobWorkerOptions): JobWorker { + return new JobWorker( + this.options.storageRepository, + this.namespace, + options + ); + } + /** Upsert future recurrence; already accepted runs are never rewritten. */ + async upsertSchedule>( + id: string, + definition: D, + input: InferType, + options: { + schedule: TaskSchedule; + missed?: 'coalesce' | 'skip' | 'replay'; + overlap?: 'allow' | 'skip'; + } + ) { + identity(id, 'schedule id'); + const parsed = parsePayload( + definition.input, + input, + definition.policy.maxPayloadBytes + ); + const missed = options.missed ?? 'coalesce', + overlap = options.overlap ?? 'allow'; + if ( + !['coalesce', 'skip', 'replay'].includes(missed) || + !['allow', 'skip'].includes(overlap) + ) + throw new TypeError('Invalid schedule policy'); + const schedule = normalizeSchedule(options.schedule); + return this.options.storageRepository.upsertSchedule({ + namespace: this.namespace, + id, + name: definition.name, + version: definition.version, + input: parsed, + policy: structuredClone(definition.policy), + schedule, + missed, + overlap, + fingerprint: fingerprint([ + definition.name, + definition.version, + parsed, + definition.policy, + { + ...schedule, + startsOn: schedule.startsOn?.getTime() ?? null, + endsOn: schedule.endsOn?.getTime() ?? null + }, + missed, + overlap + ]) + }); + } + /** Pause/resume future occurrences. Resume applies the configured missed policy. */ + pauseSchedule(id: string, paused = true) { + return this.options.storageRepository.pauseSchedule( + this.namespace, + id, + paused + ); + } + /** Remove a recurring trigger, not its already accepted runs. */ + removeSchedule(id: string) { + return this.options.storageRepository.removeSchedule( + this.namespace, + id + ); + } + /** Run one bounded dispatcher pass, useful for externally managed lifecycles. */ + dispatch(): Promise { + return this.options.storageRepository.dispatch(this.namespace); + } + /** Inspect operational counts/queue age and queued definition versions. */ + health() { + return this.options.storageRepository.health(this.namespace); + } + /** Explicit bounded terminal-data cleanup. Workers/dispatcher also run this periodically. */ + cleanup(limit = 100) { + return this.options.storageRepository.cleanup( + this.namespace, + positive(limit, 'cleanup limit', 10000) + ); + } + /** Start only recurring dispatch; producer-only processes do not need start(). */ + async start(): Promise { + if (this.controller) throw new Error('Scheduler already started'); + this.controller = new AbortController(); + const signal = this.controller.signal; + this.loop = (async () => { + let cleanupAt = 0; + while (!signal.aborted) { + try { + await this.dispatch(); + if (Date.now() >= cleanupAt) { + await this.cleanup(); + cleanupAt = Date.now() + 60000; + } + this.lastError = undefined; + } catch (error) { + this.lastError = error; + try { + this.options.onError?.(error); + } catch { + /* Observer isolation. */ + } + } + await delay(this.pollMs, signal); + } + })(); + } + /** Stop dispatching. Workers must be stopped separately before closing database pools. */ + async stop(): Promise { + this.controller?.abort(); + await this.loop; + } +} diff --git a/libs/scheduler/src/storage.ts b/libs/scheduler/src/storage.ts new file mode 100644 index 00000000..708afc2d --- /dev/null +++ b/libs/scheduler/src/storage.ts @@ -0,0 +1,65 @@ +import type { + JobAttempt, + JobEvent, + RunRecord, + ScheduleRecord +} from './contracts.js'; + +/** Name/version pair a worker is deployed to execute. */ +export type SupportedJob = { name: string; version: number }; +/** + * Transactional storage primitives for adapter authors. + * Locked rows remain locked until atomic() commits. Reads/writes must be detached. + * Implementations must roll back every write on callback failure. + */ +export interface JobStorageTransaction { + /** Current repository time, obtained after any blocking lock acquisition. */ + now(): Promise; + run( + namespace: string, + id: string, + lock?: boolean + ): Promise; + /** Insert or return the conflicting dedupe record, atomically. */ + insertRun( + run: RunRecord + ): Promise<{ record: RunRecord; inserted: boolean }>; + saveRun(run: RunRecord): Promise; + /** Lock one ready supported run OR any expired owner; skip rows locked elsewhere. */ + runnable( + namespace: string, + supported: readonly SupportedJob[] + ): Promise; + appendEvent(event: JobEvent): Promise; + events(runId: string, after: number, limit: number): Promise; + saveAttempt(attempt: JobAttempt): Promise; + attempts(runId: string): Promise; + schedule( + namespace: string, + id: string + ): Promise; + /** Insert-if-absent; returns the actual locked record, including concurrent inserts. */ + insertSchedule(schedule: ScheduleRecord): Promise; + saveSchedule(schedule: ScheduleRecord): Promise; + dueSchedules(namespace: string, limit: number): Promise; + unfinishedScheduleRun( + namespace: string, + scheduleId: string + ): Promise; + /** Lock and remove expired terminal runs plus dependent events/attempts. */ + cleanup(namespace: string, limit: number): Promise; + /** Lightweight aggregate health; no job payloads. */ + health(namespace: string): Promise; +} +/** Aggregate health includes queued versions so deployments can detect unsupported work. */ +export type JobRepositoryHealth = { + counts: Record; + oldestReadyAt: number | null; + queuedDefinitions: Array; +}; +/** Adapter transaction boundary. No execution handler runs inside this callback. */ +export interface JobStorage { + atomic( + action: (transaction: JobStorageTransaction) => Promise + ): Promise; +} diff --git a/libs/scheduler/src/thread-entry.ts b/libs/scheduler/src/thread-entry.ts new file mode 100644 index 00000000..1aba517c --- /dev/null +++ b/libs/scheduler/src/thread-entry.ts @@ -0,0 +1,77 @@ +import { parentPort, workerData } from 'node:worker_threads'; +import type { JobContext } from './contracts.js'; +import { jobError, jsonValue, NonRetryableJobError } from './definition.js'; + +const port = parentPort!; +const abort = new AbortController(); +const pending = new Map< + number, + { resolve: () => void; reject: (error: Error) => void } +>(); +let sequence = 0; +const reports = new Set>(); +port.on('message', message => { + if (message.type === 'abort') abort.abort(); + if (message.type === 'ack') { + const waiter = pending.get(message.sequence); + pending.delete(message.sequence); + if (message.error) + waiter?.reject( + new NonRetryableJobError( + message.error.message, + message.error.code + ) + ); + else waiter?.resolve(); + } +}); +const context: JobContext = { + runId: workerData.runId, + attempt: workerData.attempt, + signal: abort.signal, + report: data => { + if (reports.size >= workerData.policy.maxProgressEvents) + throw new NonRetryableJobError( + 'Progress event limit exceeded', + 'progress_limit' + ); + const promise = new Promise((resolve, reject) => { + const id = ++sequence; + const snapshot = jsonValue( + data, + workerData.policy.maxProgressBytes + ); + pending.set(id, { resolve, reject }); + port.postMessage({ + type: 'progress', + sequence: id, + data: snapshot + }); + }); + reports.add(promise); + void promise.catch(() => undefined); + return promise; + } +}; +try { + const module = await import(workerData.moduleUrl); + if (typeof module.default !== 'function') + throw new NonRetryableJobError( + 'Thread module must default-export a handler', + 'invalid_handler' + ); + const result = await module.default(workerData.input, context); + await Promise.all(reports); + port.postMessage({ + type: 'result', + value: jsonValue(result, workerData.policy.maxPayloadBytes) + }); +} catch (error) { + port.postMessage({ + type: 'error', + error: jobError(error), + retryable: !(error instanceof NonRetryableJobError) + }); +} finally { + port.close(); +} diff --git a/libs/scheduler/src/thread.test.ts b/libs/scheduler/src/thread.test.ts new file mode 100644 index 00000000..ed11c9b5 --- /dev/null +++ b/libs/scheduler/src/thread.test.ts @@ -0,0 +1,69 @@ +import { number, object, string } from '@cleverbrush/schema'; +import { describe, expect, it, vi } from 'vitest'; +// Exercise the packed build: thread-entry.js must be emitted beside index.js. +import { + defineJob, + InMemoryJobRepository, + JobScheduler +} from '../dist/index.js'; + +describe('packaged worker threads', () => { + it.each([ + 'success', + 'exit', + 'hang', + 'invalid', + 'accessor', + 'unawaited-invalid' + ])('handles %s without leaking a worker', async mode => { + const job = defineJob({ + name: 'thread', + version: 1, + input: object({ id: string(), mode: string() }), + progress: object({ percent: number() }), + output: object({ url: string() }), + timeoutMs: mode === 'hang' ? 200 : 5000 + }); + const repository = new InMemoryJobRepository(); + const scheduler = new JobScheduler({ storageRepository: repository }); + const run = await scheduler.enqueue(job, { id: 'one', mode }); + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.thread( + new URL( + '../../../demos/durable-jobs/thread-handler.mjs', + import.meta.url + ) + ) + ] + }); + await worker.start(); + try { + await vi.waitFor( + async () => + expect(await scheduler.getRun(job, run.id)).toHaveProperty( + 'status', + mode === 'success' ? 'succeeded' : 'failed' + ), + { timeout: 6000 } + ); + if (mode === 'success') + expect( + (await repository.events('default', run.id, 0)).map( + event => event.type + ) + ).toEqual(['queued', 'running', 'progress', 'succeeded']); + if (mode === 'hang') + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + error: { code: 'timeout' } + }); + if (['invalid', 'accessor', 'unawaited-invalid'].includes(mode)) + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + error: { code: 'invalid_payload' } + }); + } finally { + await worker.stop({ drainTimeoutMs: 100 }); + } + }); +}); diff --git a/libs/scheduler/src/types.ts b/libs/scheduler/src/types.ts deleted file mode 100644 index a0465781..00000000 --- a/libs/scheduler/src/types.ts +++ /dev/null @@ -1,268 +0,0 @@ -import { - array, - boolean, - date, - func, - type InferType, - number, - object, - string, - union -} from '@cleverbrush/schema'; - -import type { IJobRepository } from './jobRepository.js'; - -const ScheduleSchemaBase = object({ - /** Number of intervals (days, months, minutes or weeks) - * between repeats. Interval type depends of `every` value */ - interval: number().min(1).max(356), - /** Hour (0-23) */ - hour: number().min(0).max(23).optional(), - /** Minute (0-59) */ - minute: number().min(0).max(59).optional(), - /** Do not start earlier than this date */ - startsOn: date().acceptJsonString().optional(), - /** Do not repeat after this date */ - endsOn: date().acceptJsonString().optional(), - /** Max number of repeats (min 1) */ - maxOccurences: number().min(1).optional(), - /** Skip this number of repeats. Min value is 1. */ - skipFirst: number().min(1).optional() -}).addValidator(val => { - if ( - 'endsOn' in val && - 'maxOccurences' in val && - typeof val.endsOn !== 'undefined' - ) { - return { - valid: false, - errors: [{ message: 'either endsOn or maxOccurences is required' }] - }; - } - return { valid: true }; -}); - -const ScheduleMinuteSchema = ScheduleSchemaBase.omit('hour') - .omit('minute') - .addProps({ - /** Repeat every minute */ - every: string('minute') - }); - -const ScheduleDaySchema = ScheduleSchemaBase.addProps({ - /** Repeat every day */ - every: string('day') -}); - -const ScheduleWeekSchema = ScheduleSchemaBase.addProps({ - /** Repeat every week */ - every: string('week'), - /** Days of week for schedule */ - dayOfWeek: array() - .of(number().min(1).max(7)) - .minLength(1) - .maxLength(7) - .addValidator(val => { - const map: Record = {}; - for (let i = 0; i < val.length; i++) { - if (map[val[i]]) { - return { - valid: false, - errors: [{ message: 'no duplicates allowed' }] - }; - } - map[val[i]] = true; - } - return { - valid: true - }; - }) -}); - -const ScheduleMonthSchema = ScheduleSchemaBase.addProps({ - /** Repeat every month */ - every: string('month'), - /** Day - 'last' or number from 1 to 28 */ - day: union(string('last')).or(number().min(1).max(28)) -}); - -const ScheduleYearSchema = ScheduleSchemaBase.addProps({ - /** Repeat every year */ - every: string('year'), - /** Day - 'last' or number from 1 to 28 */ - day: union(string('last')).or(number().min(1).max(28)), - /** Month - number from 1 to 12 */ - month: number().min(1).max(12) -}); - -const ScheduleSchema = union(ScheduleMinuteSchema) - .or(ScheduleDaySchema) - .or(ScheduleWeekSchema) - .or(ScheduleMonthSchema) - .or(ScheduleYearSchema); - -const CreateJobRequestSchema = object({ - /** Id of job, must be uniq */ - id: string(), - /** Path to js file (relative to root folder) */ - path: string().minLength(1), - /** Job's schedule */ - schedule: ScheduleSchema, - /** Timeout for job (in milliseconds) */ - timeout: number().min(0).optional(), - /** Arbitrary props for job (can be a callback returning props or Promise) */ - props: union(object().acceptUnknownProps()).or(func()).optional(), - /** Job will be considered as disabled when more than that count of runs fails consequently - * unlimited if negative - */ - maxConsequentFails: number().optional(), - /** - * Job will be retried right away this times. Job will be retried on next schedule run if this number is exceeded. - */ - maxRetries: number().optional().min(1), - /** - * If true, job will not be runned if previous run is not finished yet. - */ - noConcurrentRuns: boolean().optional() -}); - -/** - * Schedule for job. Can be one of: - * - every N minutes - * - every N days - * - every N weeks - * - every N months - * - every N years - * - every N days of week - * - every N months on N day - * - every N years on N day of N month - * - every N years on last day of N month - */ -export type Schedule = InferType; - -/** - * Object used to create new job. - */ -export type CreateJobRequest = InferType; - -export type Job = { - /** Id of job */ - id: string; - /** Job status */ - status: JobStatus; - /** Job's schedule */ - schedule: Schedule; - /** Path to the job's file (relative to the `rootFolder`) */ - path: string; - /** Timeout for job (in milliseconds) */ - timeout: number; - /** Date when job was created */ - createdAt: Date; - /** - * Date when job was started for the first time - */ - startedAt?: Date; - /** - * Date when job was ended for the first time - */ - firstInstanceEndedAt?: Date; - /** - * Number of times job was runned - */ - timesRunned?: number; - /** - * Number of times job was runned successfully - */ - successfullTimesRunned?: number; - /** - * Current number of consequent fails, resets to 0 when - * job is runned successfully. If this number is greater than - * `maxConsequentFails` job will be disabled. - */ - consequentFailsCount: number; - /** - * Job will be considered as disabled when more than that count of runs fails consequently - */ - maxConsequentFails: number; - /** - * Job will be retried right away this times. Job will be retried on - * next schedule run if this number is exceeded. - */ - maxRetries: number; - /** - * If true, job will not be runned if previous run is not finished yet. - */ - noConcurrentRuns?: boolean; -}; - -export type JobInstance = { - /** Id of job instance */ - id: number; - /** Id of job */ - jobId: string; - /** Index of job instance in the schedule sequence */ - index: number; - /** Status of the job instance */ - status: JobInstanceStatus; - /** Timeout value for job */ - timeout: number; - scheduledTo: Date; - /** Date when job instance was started */ - startDate?: Date; - /** Date when job instance was ended */ - endDate?: Date; - /** Stdout of the job saved to string */ - stdOut?: string; - /** Stderr of the job saved to string */ - stdErr?: string; - /** Exit code of the job */ - exitCode?: number; - /** Count of unsucessfull job retries in row*/ - retryIndex: number; - /** Max number of retries */ - maxRetries: number; -}; - -export type JobSchedulerProps = { - /** - * Path to the folder where job files are located. - */ - rootFolder: string; - /** - * Timezone for scheduling. Default is 'UTC'. - */ - defaultTimeZone?: string; - /** - * Repository for persisting jobs. - * @experimental - */ - persistRepository?: IJobRepository; -}; - -/** Lifecycle status of a job definition. */ -export type JobStatus = 'active' | 'disabled' | 'finished'; -/** Runtime status of the scheduler itself. */ -export type SchedulerStatus = 'started' | 'stopped'; -/** Status of a single job execution instance. */ -export type JobInstanceStatus = - | 'running' - | 'errored' - | 'succeeded' - | 'scheduled' - | 'timedout' - | 'canceled'; - -/** - * Pre-built `@cleverbrush/schema` schemas used internally by the scheduler. - * Exported for consumers who need to validate schedule/job payloads manually. - */ -export const Schemas = { - ScheduleSchemaBase, - ScheduleSchema, - CreateJobRequestSchema, - ScheduleMinuteSchema, - ScheduleDaySchema, - ScheduleWeekSchema, - ScheduleMonthSchema, - ScheduleYearSchema -}; diff --git a/libs/scheduler/src/wait.ts b/libs/scheduler/src/wait.ts new file mode 100644 index 00000000..00ff5757 --- /dev/null +++ b/libs/scheduler/src/wait.ts @@ -0,0 +1,13 @@ +/** Abortable polling delay that leaves no signal listener or timer behind. */ +export function delay(ms: number, signal?: AbortSignal): Promise { + if (signal?.aborted) return Promise.resolve(); + return new Promise(resolve => { + const finish = () => { + clearTimeout(timer); + signal?.removeEventListener('abort', finish); + resolve(); + }; + const timer = setTimeout(finish, ms); + signal?.addEventListener('abort', finish, { once: true }); + }); +} diff --git a/libs/scheduler/src/worker.test.ts b/libs/scheduler/src/worker.test.ts new file mode 100644 index 00000000..9502f8b7 --- /dev/null +++ b/libs/scheduler/src/worker.test.ts @@ -0,0 +1,215 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { testJob } from '../testing/repository-contract.js'; +import { + InMemoryJobRepository, + JobScheduler, + type JobWorker, + NonRetryableJobError +} from './index.js'; + +const workers: JobWorker[] = []; +afterEach(async () => { + await Promise.all( + workers.splice(0).map(worker => worker.stop({ drainTimeoutMs: 30 })) + ); +}); +function setup() { + const repository = new InMemoryJobRepository(); + return { + repository, + scheduler: new JobScheduler({ + storageRepository: repository, + pollIntervalMs: 2 + }) + }; +} +describe('function worker', () => { + it('stops over-limit progress without retrying or persisting extra events', async () => { + const { scheduler, repository } = setup(); + const job = testJob({ + maxProgressEvents: 1, + retry: { maxAttempts: 3 } + }); + const run = await scheduler.enqueue(job, { id: 'one' }); + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async (_, context) => { + await context.report({ percent: 10 }); + await context.report({ percent: 20 }); + return { url: '/report' }; + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'failed', + attempt: 1, + error: { code: 'progress_limit' } + }) + ); + expect( + (await repository.events('default', run.id, 0)).filter( + event => event.type === 'progress' + ) + ).toHaveLength(1); + }); + it('validates input/progress/output and publishes progress before success', async () => { + const { scheduler, repository } = setup(); + const job = testJob(); + const run = await scheduler.enqueue(job, { id: 'one' }); + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async (_, context) => { + await context.report({ percent: 50 }); + expect( + (await repository.events('default', run.id, 0)).at(-1) + ?.type + ).toBe('progress'); + return { url: '/report' }; + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, run.id)).toHaveProperty( + 'status', + 'succeeded' + ) + ); + }); + it('retries opt-in, but not explicit permanent failures', async () => { + const { scheduler } = setup(); + const job = testJob({ retry: { maxAttempts: 3, initialDelayMs: 1 } }); + const run = await scheduler.enqueue(job, { id: 'one' }); + let count = 0; + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(() => { + count++; + if (count === 1) throw new Error('retry'); + throw new NonRetryableJobError('permanent'); + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, run.id)).toHaveProperty( + 'status', + 'failed' + ) + ); + expect(count).toBe(2); + }); + it('fails invalid output and a forgotten invalid progress promise', async () => { + for (const mode of ['output', 'progress']) { + const { scheduler } = setup(); + const job = testJob(); + const run = await scheduler.enqueue(job, { id: mode }); + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle((_, context) => { + if (mode === 'progress') + void context.report({ percent: 'wrong' } as any); + return ( + mode === 'output' + ? { wrong: true } + : { url: '/report' } + ) as any; + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'failed', + error: { code: 'invalid_payload' } + }) + ); + } + }); + it('bounds progress and retains a local slot while a timed-out function is alive', async () => { + const { scheduler } = setup(); + const job = testJob({ timeoutMs: 20 }); + const first = await scheduler.enqueue( + job, + { id: 'one' }, + { runAt: new Date(0) } + ); + const second = await scheduler.enqueue(job, { id: 'two' }); + let release!: () => void; + let started = 0; + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async () => { + started++; + await new Promise(resolve => { + release = resolve; + }); + return { url: '/report' }; + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, first.id)).toMatchObject({ + status: 'failed', + error: { code: 'timeout' } + }) + ); + expect(started).toBe(1); + expect(await scheduler.getRun(job, second.id)).toHaveProperty( + 'status', + 'queued' + ); + await worker.stop({ drainTimeoutMs: 10 }); + release(); + await expect(worker.start()).rejects.toThrow('already'); + }); + it('records shutdown interruption and rejects progress after cancellation', async () => { + const { scheduler } = setup(); + const job = testJob(); + const run = await scheduler.enqueue(job, { id: 'one' }); + const worker = scheduler.createWorker({ + pollIntervalMs: 2, + jobs: [ + job.handle(async (_, context) => { + await new Promise(resolve => + context.signal.addEventListener( + 'abort', + () => resolve(), + { once: true } + ) + ); + await expect( + context.report({ percent: 100 }) + ).rejects.toThrow(); + return { url: '/report' }; + }) + ] + }); + workers.push(worker); + await worker.start(); + await vi.waitFor(async () => + expect(await scheduler.getRun(job, run.id)).toHaveProperty( + 'status', + 'running' + ) + ); + await worker.stop({ drainTimeoutMs: 5 }); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'failed', + error: { code: 'shutdown' } + }); + }); +}); diff --git a/libs/scheduler/src/worker.ts b/libs/scheduler/src/worker.ts new file mode 100644 index 00000000..d8469570 --- /dev/null +++ b/libs/scheduler/src/worker.ts @@ -0,0 +1,410 @@ +import { Worker } from 'node:worker_threads'; +import type { + JobBinding, + JobContext, + RunPolicy, + RunRecord, + SchedulerDiagnostic +} from './contracts.js'; +import { + jobError, + LeaseLostError, + NonRetryableJobError, + parsePayload, + positive +} from './definition.js'; +import type { JobRepository } from './repository.js'; +import { delay } from './wait.js'; + +/** Worker options are local capacity controls, not cluster-wide quotas. */ +export type JobWorkerOptions = { + jobs: readonly JobBinding[]; + concurrency?: number; + pollIntervalMs?: number; + leaseMs?: number; + heartbeatMs?: number; + onDiagnostic?: (event: SchedulerDiagnostic) => void; +}; + +/** Executes persisted work with bounded local concurrency and fenced ownership. */ +export class JobWorker { + private readonly bindings = new Map(); + private readonly active = new Map< + string, + { abort: AbortController; task: Promise } + >(); + private readonly concurrency: number; + private readonly pollMs: number; + private readonly leaseMs: number; + private readonly heartbeatMs: number; + private controller?: AbortController; + private loop?: Promise; + private stopPromise?: Promise; + /** Most recent infrastructure failure; cleared after a successful repository poll. */ + lastError: unknown; + + constructor( + private readonly repository: JobRepository, + private readonly namespace: string, + private readonly options: JobWorkerOptions + ) { + this.concurrency = positive( + options.concurrency ?? 1, + 'concurrency', + 10000 + ); + this.pollMs = positive( + options.pollIntervalMs ?? 1000, + 'pollIntervalMs', + 2147483647 + ); + this.leaseMs = positive(options.leaseMs ?? 30000, 'leaseMs'); + this.heartbeatMs = positive( + options.heartbeatMs ?? 10000, + 'heartbeatMs', + 2147483647 + ); + if (this.heartbeatMs >= this.leaseMs) + throw new RangeError('heartbeatMs must be shorter than leaseMs'); + for (const binding of options.jobs) { + const key = this.key( + binding.definition.name, + binding.definition.version + ); + if (this.bindings.has(key)) + throw new TypeError('Duplicate job name/version registration'); + if (!!binding.handler === !!binding.moduleUrl) + throw new TypeError('Register exactly one execution mode'); + this.bindings.set(key, binding); + } + if (!this.bindings.size) + throw new TypeError('At least one handler is required'); + } + private key(name: string, version: number) { + return JSON.stringify([name, version]); + } + private diagnostic(type: SchedulerDiagnostic['type'], run?: RunRecord) { + try { + this.options.onDiagnostic?.({ + type, + ...(run + ? { runId: run.id, name: run.name, attempt: run.attempt } + : {}) + }); + } catch { + /* Observers cannot change job execution semantics. */ + } + } + /** Start polling; resolves after startup, not after jobs finish. A worker is single-use. */ + async start(): Promise { + if (this.controller || this.stopPromise) + throw new Error('Worker already started or stopped'); + this.controller = new AbortController(); + this.loop = this.poll(this.controller.signal); + } + private async poll(signal: AbortSignal) { + let cleanupAt = 0; + const supported = [...this.bindings.values()].map(binding => ({ + name: binding.definition.name, + version: binding.definition.version + })); + while (!signal.aborted) { + try { + if (Date.now() >= cleanupAt) { + await this.repository.cleanup(this.namespace); + cleanupAt = Date.now() + 60000; + } + while (!signal.aborted && this.active.size < this.concurrency) { + const run = await this.repository.claim( + this.namespace, + supported, + this.leaseMs + ); + this.lastError = undefined; + if (!run) break; + if (signal.aborted) { + await this.repository.fail( + this.namespace, + run.id, + run.leaseToken!, + { + code: 'shutdown', + message: 'Worker stopped after claim' + }, + true, + true + ); + break; + } + const abort = new AbortController(); + const task = this.execute(run, abort) + .catch(error => { + this.lastError = error; + this.diagnostic('infrastructure_error', run); + }) + .finally(() => this.active.delete(run.id)); + this.active.set(run.id, { abort, task }); + } + } catch (error) { + this.lastError = error; + this.diagnostic('infrastructure_error'); + } + await delay(this.pollMs, signal); + } + } + private async inThread( + binding: JobBinding, + input: unknown, + context: JobContext, + policy: RunPolicy + ): Promise { + return new Promise((resolve, reject) => { + const thread = new Worker( + new URL('./thread-entry.js', import.meta.url), + { + workerData: { + moduleUrl: binding.moduleUrl, + input, + runId: context.runId, + attempt: context.attempt, + policy + }, + execArgv: [], + stdout: true, + stderr: true + } + ); + // Drain rather than buffer unbounded output or persist possible secrets. + thread.stdout.resume(); + thread.stderr.resume(); + let settled = false; + const finish = (error?: unknown, value?: unknown) => { + if (settled) return; + settled = true; + context.signal.removeEventListener('abort', abort); + void thread + .terminate() + .then( + () => (error ? reject(error) : resolve(value)), + reject + ); + }; + const abort = () => { + thread.postMessage({ type: 'abort' }); + finish(context.signal.reason ?? new Error('Aborted')); + }; + context.signal.addEventListener('abort', abort, { once: true }); + thread.on('message', message => { + if (message?.type === 'progress') { + void context.report(message.data).then( + () => { + if (!settled) + thread.postMessage({ + type: 'ack', + sequence: message.sequence + }); + }, + error => { + if (!settled) + thread.postMessage({ + type: 'ack', + sequence: message.sequence, + error: jobError(error) + }); + } + ); + } else if (message?.type === 'result') + finish(undefined, message.value); + else if (message?.type === 'error') + finish( + message.retryable + ? new Error(message.error.message) + : new NonRetryableJobError( + message.error.message, + message.error.code + ) + ); + }); + thread.on('error', error => finish(error)); + thread.on('exit', code => { + if (!settled) + finish( + new Error( + 'Worker thread exited without a result (' + + code + + ')' + ) + ); + }); + if (context.signal.aborted) abort(); + }); + } + private async execute( + run: RunRecord, + abort: AbortController + ): Promise { + const binding = this.bindings.get(this.key(run.name, run.version))!; + const owner = [this.namespace, run.id, run.leaseToken!] as const; + const heartbeatStop = new AbortController(); + let heartbeatError: unknown; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + abort.abort(new Error('Job attempt timed out')); + }, run.policy.timeoutMs); + const heartbeat = (async () => { + while (!heartbeatStop.signal.aborted) { + await delay(this.heartbeatMs, heartbeatStop.signal); + if (heartbeatStop.signal.aborted) break; + try { + await this.repository.heartbeat(...owner, this.leaseMs); + } catch (error) { + heartbeatError = error; + abort.abort(error); + break; + } + } + })(); + const reports = new Set>(); + const context: JobContext = { + runId: run.id, + attempt: run.attempt, + signal: abort.signal, + report: data => { + if (reports.size >= run.policy.maxProgressEvents) { + const error = new NonRetryableJobError( + 'Progress event limit exceeded', + 'progress_limit' + ); + abort.abort(error); + const rejected = Promise.reject(error); + void rejected.catch(() => undefined); + return rejected; + } + const promise = Promise.resolve().then(async () => { + abort.signal.throwIfAborted(); + const progress = parsePayload( + binding.definition.progress, + data, + run.policy.maxProgressBytes + ); + await this.repository.report(...owner, progress); + }); + reports.add(promise); + // Retain failures for completion even when the handler forgets to await. + void promise.catch(() => undefined); + return promise; + } + }; + this.diagnostic('started', run); + let removeAbort = () => {}; + const aborted = new Promise((_, reject) => { + const listener = () => + reject(abort.signal.reason ?? new Error('Aborted')); + removeAbort = () => + abort.signal.removeEventListener('abort', listener); + abort.signal.addEventListener('abort', listener, { once: true }); + if (abort.signal.aborted) listener(); + }); + const work = Promise.resolve().then(async () => { + const input = parsePayload( + binding.definition.input, + run.input, + run.policy.maxPayloadBytes + ); + const output = binding.handler + ? await binding.handler(input, context) + : await this.inThread(binding, input, context, run.policy); + await Promise.all(reports); + return parsePayload( + binding.definition.output, + output, + run.policy.maxPayloadBytes + ); + }); + try { + const output = await Promise.race([work, aborted]); + await this.repository.complete(...owner, output); + } catch (error) { + if (heartbeatError) { + this.diagnostic( + heartbeatError instanceof LeaseLostError + ? 'lease_lost' + : 'infrastructure_error', + run + ); + } else { + try { + const interrupted = + abort.signal.aborted && + !timedOut && + !(error instanceof NonRetryableJobError); + const failure = timedOut + ? { code: 'timeout', message: 'Job attempt timed out' } + : interrupted + ? { + code: 'shutdown', + message: 'Worker shutdown interrupted execution' + } + : jobError(error); + await this.repository.fail( + ...owner, + failure, + !(error instanceof NonRetryableJobError), + interrupted + ); + } catch (failure) { + this.lastError = failure; + this.diagnostic( + failure instanceof LeaseLostError + ? 'lease_lost' + : 'infrastructure_error', + run + ); + } + } + } finally { + clearTimeout(timeout); + removeAbort(); + heartbeatStop.abort(); + await heartbeat; + this.diagnostic('settled', run); + } + // A non-cooperative function still occupies its local slot. Never start an + // unbounded stream of overlapping functions just because timers expired. + await work.catch(() => undefined); + } + /** Stop claims, drain, then abort. Does not forcibly stop an ordinary function. */ + stop(options: { drainTimeoutMs?: number } = {}): Promise { + if (this.stopPromise) return this.stopPromise; + this.stopPromise = this.shutdown( + positive( + options.drainTimeoutMs ?? 30000, + 'drainTimeoutMs', + 2147483647 + ) + ); + return this.stopPromise; + } + private async shutdown(drainTimeoutMs: number): Promise { + this.controller?.abort(); + await this.loop; + const deadline = new AbortController(); + const all = Promise.all( + [...this.active.values()].map(entry => entry.task) + ); + await Promise.race([all, delay(drainTimeoutMs, deadline.signal)]); + deadline.abort(); + for (const entry of this.active.values()) + entry.abort.abort(new Error('Worker shutdown')); + // Give cooperative handlers/fenced failures a bounded chance to settle. + if (this.active.size) { + const cancellation = new AbortController(); + await Promise.race([ + all, + delay(Math.min(drainTimeoutMs, 1000), cancellation.signal) + ]); + cancellation.abort(); + } + } +} diff --git a/libs/scheduler/testing/repository-contract.ts b/libs/scheduler/testing/repository-contract.ts new file mode 100644 index 00000000..49c6ae17 --- /dev/null +++ b/libs/scheduler/testing/repository-contract.ts @@ -0,0 +1,343 @@ +import { number, object, string } from '@cleverbrush/schema'; +import { describe, expect, it } from 'vitest'; +import { defineJob, type JobRepository, JobScheduler } from '../src/index.js'; + +export const testJob = (options = {}) => + defineJob({ + name: 'report', + version: 1, + input: object({ id: string() }), + progress: object({ percent: number() }), + output: object({ url: string() }), + ...options + }); +export type Fixture = { + repository: JobRepository; + scheduler: JobScheduler; + advance(ms: number): Promise; +}; +/** Shared assertions: both adapters must preserve the same durable transition contract. */ +export function repositoryContract(create: () => Promise) { + describe('repository contract', () => { + it('deduplicates concurrent producers, rejects conflicts, isolates namespaces', async () => { + const { scheduler, repository } = await create(); + const job = testJob(); + const runs = await Promise.all( + Array.from({ length: 8 }, () => + scheduler.enqueue( + job, + { id: 'one' }, + { idempotencyKey: 'one' } + ) + ) + ); + expect(new Set(runs.map(run => run.id)).size).toBe(1); + await expect( + scheduler.enqueue(job, { id: 'two' }, { idempotencyKey: 'one' }) + ).rejects.toThrow('different submission'); + const other = new JobScheduler({ + storageRepository: repository, + namespace: 'other' + }); + expect(await other.getRun(job, runs[0].id)).toBeUndefined(); + expect(await other.cancel(runs[0].id)).toBe(false); + expect(await repository.events('other', runs[0].id, 0)).toEqual([]); + expect( + await other.enqueue( + job, + { id: 'one' }, + { idempotencyKey: 'one' } + ) + ).not.toHaveProperty('id', runs[0].id); + }); + it('claims once across competing workers and replays ordered durable progress', async () => { + const { scheduler, repository } = await create(); + const job = testJob(); + const run = await scheduler.enqueue(job, { id: 'one' }); + const claims = await Promise.all( + Array.from({ length: 4 }, () => + repository.claim(scheduler.namespace, [job], 10000) + ) + ); + expect(claims.filter(Boolean)).toHaveLength(1); + const owned = claims.find(Boolean)!; + const owner = [ + scheduler.namespace, + run.id, + owned.leaseToken! + ] as const; + await Promise.all([ + repository.report(...owner, { percent: 20 }), + repository.report(...owner, { percent: 80 }) + ]); + await repository.complete(...owner, { url: '/result' }); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'succeeded', + output: { url: '/result' }, + attempt: 1 + }); + expect(await scheduler.getRun(job, run.id)).not.toHaveProperty( + 'leaseToken' + ); + const events = []; + for await (const event of scheduler.events(job, run.id, { + after: 2 + })) + events.push(event); + expect(events.map(event => event.sequence)).toEqual([3, 4, 5]); + expect(events.map(event => event.type)).toEqual([ + 'progress', + 'progress', + 'succeeded' + ]); + expect( + await repository.attempts(scheduler.namespace, run.id) + ).toMatchObject([{ attempt: 1, status: 'succeeded' }]); + await expect(repository.complete(...owner, {})).rejects.toThrow( + 'lease' + ); + }); + it('fences expired owners; lease recovery consumes the configured attempt budget', async () => { + const { scheduler, repository, advance } = await create(); + const job = testJob({ + retry: { maxAttempts: 2, initialDelayMs: 1 } + }); + const run = await scheduler.enqueue(job, { id: 'one' }); + const first = (await repository.claim( + scheduler.namespace, + [job], + 500 + ))!; + await advance(550); + const oldOwner = [ + scheduler.namespace, + run.id, + first.leaseToken! + ] as const; + await expect( + repository.heartbeat(...oldOwner, 500) + ).rejects.toThrow('lease'); + await expect(repository.report(...oldOwner, {})).rejects.toThrow( + 'lease' + ); + expect( + await repository.claim(scheduler.namespace, [], 500) + ).toBeUndefined(); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'retry_wait', + error: { code: 'lease_expired' } + }); + await advance(5); + const second = (await repository.claim( + scheduler.namespace, + [job], + 500 + ))!; + expect(second.attempt).toBe(2); + expect(second.leaseToken).not.toBe(first.leaseToken); + await expect(repository.complete(...oldOwner, {})).rejects.toThrow( + 'lease' + ); + await repository.fail( + scheduler.namespace, + run.id, + second.leaseToken!, + { code: 'test', message: 'failed' } + ); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'failed', + attempt: 2 + }); + expect( + await repository.attempts(scheduler.namespace, run.id) + ).toMatchObject([{ status: 'interrupted' }, { status: 'failed' }]); + }); + it('does not retry by default, and respects delayed/unsupported work', async () => { + const { scheduler, repository, advance } = await create(); + const job = testJob(); + const delayed = await scheduler.enqueue( + job, + { id: 'later' }, + { runAt: new Date(Date.now() + 1000) } + ); + const run = await scheduler.enqueue(job, { id: 'now' }); + expect( + await repository.claim( + scheduler.namespace, + [{ name: job.name, version: 2 }], + 500 + ) + ).toBeUndefined(); + const claim = (await repository.claim( + scheduler.namespace, + [job], + 500 + ))!; + expect(claim.id).toBe(run.id); + await advance(550); + await repository.claim(scheduler.namespace, [job], 500); + expect(await scheduler.getRun(job, run.id)).toMatchObject({ + status: 'failed', + attempt: 1 + }); + expect(await scheduler.getRun(job, delayed.id)).toHaveProperty( + 'status', + 'queued' + ); + expect((await scheduler.health()).queuedDefinitions).toMatchObject([ + { name: 'report', version: 1, count: 1 } + ]); + }); + it('cancels queued/running jobs and cleanup removes only expired terminal data', async () => { + const { scheduler, repository, advance } = await create(); + const job = testJob({ retentionMs: 100 }); + const run = await scheduler.enqueue( + job, + { id: 'one' }, + { idempotencyKey: 'one' } + ); + const claim = (await repository.claim( + scheduler.namespace, + [job], + 10000 + ))!; + expect(await scheduler.cancel(run.id)).toBe(true); + expect(await scheduler.cancel(run.id)).toBe(false); + await expect( + repository.report( + scheduler.namespace, + run.id, + claim.leaseToken!, + {} + ) + ).rejects.toThrow('lease'); + const queued = await scheduler.enqueue(job, { id: 'keep' }); + await advance(150); + expect(await scheduler.cleanup()).toBe(1); + expect(await scheduler.getRun(job, run.id)).toBeUndefined(); + expect(await scheduler.getRun(job, queued.id)).toBeDefined(); + expect( + await repository.events(scheduler.namespace, run.id, 0) + ).toEqual([]); + const again = await scheduler.enqueue( + job, + { id: 'one' }, + { idempotencyKey: 'one' } + ); + expect(again.id).not.toBe(run.id); + }); + it('rolls back state and events together when a transition fails', async () => { + const { scheduler, repository } = await create(); + const run = await scheduler.enqueue(testJob(), { id: 'one' }); + await expect( + repository.storage.atomic(async tx => { + const record = (await tx.run( + scheduler.namespace, + run.id, + true + ))!; + record.status = 'failed'; + await tx.saveRun(record); + await tx.appendEvent({ + runId: run.id, + sequence: 2, + attempt: 0, + at: 0, + type: 'failed', + data: null + }); + throw new Error('rollback'); + }) + ).rejects.toThrow('rollback'); + expect(await scheduler.getRun(testJob(), run.id)).toHaveProperty( + 'status', + 'queued' + ); + expect( + await repository.events(scheduler.namespace, run.id, 0) + ).toHaveLength(1); + }); + it('preserves revisions for canonical defaults, weekday order and date input forms', async () => { + const { scheduler } = await create(); + const register = (schedule: any) => + scheduler.upsertSchedule( + 'equivalent', + testJob(), + { id: 'one' }, + { schedule } + ); + const startsOn = new Date('2030-01-01T00:00:00Z'); + const first = await register({ + every: 'week', + dayOfWeek: [5, 1], + startsOn, + maxOccurences: 3 + }); + const second = await register({ + every: 'week', + dayOfWeek: [1, 5], + startsOn: startsOn.toISOString(), + maxOccurrences: 3, + interval: 1, + timeZone: 'UTC', + hour: 9, + minute: 0, + skipFirst: 0 + }); + expect(second).toEqual(first); + const revised = await register({ + every: 'week', + dayOfWeek: [1, 5], + startsOn, + maxOccurrences: 3, + hour: 10 + }); + expect(revised.revision).toBe(2); + }); + it('dispatches each occurrence once, retains cursors and revisions', async () => { + const { scheduler, repository } = await create(); + const job = testJob(); + const options = { + schedule: { + every: 'minute' as const, + startsOn: new Date(Date.now() - 180100) + }, + missed: 'replay' as const + }; + const initial = await scheduler.upsertSchedule( + 'daily', + job, + { id: 'one' }, + options + ); + const counts = await Promise.all([ + scheduler.dispatch(), + scheduler.dispatch() + ]); + expect(counts.reduce((a, b) => a + b)).toBe(4); + expect(await scheduler.dispatch()).toBe(0); + const same = await scheduler.upsertSchedule( + 'daily', + job, + { id: 'one' }, + options + ); + expect(same.revision).toBe(initial.revision); + expect(same.cursor).toBe(4); + await scheduler.pauseSchedule('daily'); + expect(await scheduler.dispatch()).toBe(0); + await scheduler.removeSchedule('daily'); + const updated = await scheduler.upsertSchedule( + 'daily', + job, + { id: 'two' }, + options + ); + expect(updated.revision).toBe(2); + expect(await scheduler.dispatch()).toBe(0); + expect( + (await repository.health(scheduler.namespace)).counts.queued + ).toBe(4); + }); + }); +} diff --git a/libs/scheduler/tsconfig.build.json b/libs/scheduler/tsconfig.build.json index eff51d83..3a863e66 100644 --- a/libs/scheduler/tsconfig.build.json +++ b/libs/scheduler/tsconfig.build.json @@ -13,5 +13,5 @@ "types": ["node"] }, "include": ["src/**/*.ts"], - "exclude": ["src/**/*.test.ts"] + "exclude": ["src/**/*.test.ts", "src/**/*.test-d.ts"] } diff --git a/libs/scheduler/tsconfig.typecheck.json b/libs/scheduler/tsconfig.typecheck.json new file mode 100644 index 00000000..c24a9fd6 --- /dev/null +++ b/libs/scheduler/tsconfig.typecheck.json @@ -0,0 +1,6 @@ +{ + "extends": "./tsconfig.build.json", + "compilerOptions": { "noEmit": true, "strict": true }, + "include": ["src/**/*.test-d.ts"], + "exclude": [] +} diff --git a/libs/scheduler/tsup.config.ts b/libs/scheduler/tsup.config.ts index 715b5453..bf09f362 100644 --- a/libs/scheduler/tsup.config.ts +++ b/libs/scheduler/tsup.config.ts @@ -1,7 +1,7 @@ import { defineConfig } from 'tsup'; export default defineConfig({ - entry: ['src/index.ts'], + entry: ['src/index.ts', 'src/thread-entry.ts'], format: ['esm'], tsconfig: './tsconfig.build.json', minify: true, diff --git a/libs/scheduler/vitest.config.mts b/libs/scheduler/vitest.config.mts new file mode 100644 index 00000000..e18b86e9 --- /dev/null +++ b/libs/scheduler/vitest.config.mts @@ -0,0 +1,11 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ + test: { + include: ['src/**/*.test.ts'], + typecheck: { + enabled: true, + include: ['src/**/*.test-d.ts'], + tsconfig: './tsconfig.typecheck.json' + } + } +}); diff --git a/libs/schema/README.md b/libs/schema/README.md index 8ae25fa6..601743e6 100644 --- a/libs/schema/README.md +++ b/libs/schema/README.md @@ -714,7 +714,7 @@ const result = ShapeSchema.validate({ type: 'circle', radius: 5 }); ### Real-World Example: Job Scheduler -The `@cleverbrush/scheduler` library uses this exact pattern to validate job schedules. The `every` field acts as the discriminator, and each variant adds its own set of allowed properties: +An application's schedule-input form can use this pattern. The `every` field acts as the discriminator, and each variant adds its own set of allowed properties: ```typescript import { object, string, number, array, date, union, type InferType } from '@cleverbrush/schema'; diff --git a/package-lock.json b/package-lock.json index 128ec903..7ddc70a1 100644 --- a/package-lock.json +++ b/package-lock.json @@ -149,7 +149,7 @@ }, "libs/async": { "name": "@cleverbrush/async", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "devDependencies": { "@types/node": "^25.4.0" @@ -167,10 +167,10 @@ }, "libs/auth": { "name": "@cleverbrush/auth", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/schema": "^4.4.3" } }, "libs/benchmarks": { @@ -185,11 +185,11 @@ }, "libs/client": { "name": "@cleverbrush/client", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2", - "@cleverbrush/server": "^4.4.2" + "@cleverbrush/schema": "^4.4.3", + "@cleverbrush/server": "^4.4.3" }, "devDependencies": { "@tanstack/react-query": "^5.75.0", @@ -213,23 +213,23 @@ }, "libs/deep": { "name": "@cleverbrush/deep", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause" }, "libs/di": { "name": "@cleverbrush/di", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/schema": "^4.4.3" } }, "libs/env": { "name": "@cleverbrush/env", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/deep": "^4.4.2" + "@cleverbrush/deep": "^4.4.3" }, "devDependencies": { "@types/node": "^25.4.0" @@ -250,11 +250,11 @@ }, "libs/knex-clickhouse": { "name": "@cleverbrush/knex-clickhouse", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/async": "^4.4.2", - "@cleverbrush/deep": "^4.4.2", + "@cleverbrush/async": "^4.4.3", + "@cleverbrush/deep": "^4.4.3", "@clickhouse/client": "^1.18.2" }, "peerDependencies": { @@ -263,10 +263,10 @@ }, "libs/knex-schema": { "name": "@cleverbrush/knex-schema", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/schema": "^4.4.3" }, "peerDependencies": { "knex": ">=3.1.0" @@ -274,11 +274,11 @@ }, "libs/log": { "name": "@cleverbrush/log", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/async": "^4.4.2", - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/async": "^4.4.3", + "@cleverbrush/schema": "^4.4.3" }, "devDependencies": { "@types/node": "^25.4.0" @@ -312,19 +312,19 @@ }, "libs/mapper": { "name": "@cleverbrush/mapper", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/schema": "^4.4.3" } }, "libs/orm": { "name": "@cleverbrush/orm", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/knex-schema": "^4.4.2", - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/knex-schema": "^4.4.3", + "@cleverbrush/schema": "^4.4.3" }, "peerDependencies": { "knex": ">=3.1.0" @@ -332,7 +332,7 @@ }, "libs/orm-cli": { "name": "@cleverbrush/orm-cli", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { "@cleverbrush/knex-schema": "*", @@ -347,7 +347,7 @@ }, "libs/otel": { "name": "@cleverbrush/otel", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { "@opentelemetry/api": "^1.9.0", @@ -419,11 +419,11 @@ }, "libs/react-form": { "name": "@cleverbrush/react-form", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/deep": "^4.4.2", - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/deep": "^4.4.3", + "@cleverbrush/schema": "^4.4.3" }, "devDependencies": { "@types/react": "^19.0.0", @@ -435,15 +435,50 @@ }, "libs/scheduler": { "name": "@cleverbrush/scheduler", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/schema": "^4.4.2" + "@cleverbrush/schema": "^4.4.3" }, "devDependencies": { "@types/node": "^25.4.0" } }, + "libs/scheduler-postgres": { + "name": "@cleverbrush/scheduler-postgres", + "version": "4.4.3", + "license": "BSD-3-Clause", + "dependencies": { + "@cleverbrush/knex-schema": "^4.4.3", + "@cleverbrush/orm": "^4.4.3", + "@cleverbrush/scheduler": "^4.4.3", + "@cleverbrush/schema": "^4.4.3" + }, + "devDependencies": { + "@types/node": "^25.4.0" + }, + "peerDependencies": { + "knex": ">=3.1.0", + "pg": ">=8" + } + }, + "libs/scheduler-postgres/node_modules/@types/node": { + "version": "25.9.8", + "resolved": "https://registry.npmjs.org/@types/node/-/node-25.9.8.tgz", + "integrity": "sha512-VfMrScDmMhUJQmd5hArdQnFvK0OIeD36uN2Va1FcpYaLB/BgkgM8Ulc50XtYISPMz7APJ30+N0T5EM0jlcdfRw==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": ">=7.24.0 <7.24.7" + } + }, + "libs/scheduler-postgres/node_modules/undici-types": { + "version": "7.24.6", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz", + "integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==", + "dev": true, + "license": "MIT" + }, "libs/scheduler/node_modules/@types/node": { "version": "25.6.0", "resolved": "https://registry.npmjs.org/@types/node/-/node-25.6.0.tgz", @@ -456,10 +491,10 @@ }, "libs/schema": { "name": "@cleverbrush/schema", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "devDependencies": { - "@cleverbrush/deep": "^4.4.2" + "@cleverbrush/deep": "^4.4.3" }, "peerDependencies": { "@standard-schema/spec": "^1.1.0" @@ -467,7 +502,7 @@ }, "libs/schema-json": { "name": "@cleverbrush/schema-json", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "peerDependencies": { "@cleverbrush/schema": "^4.0.0", @@ -476,12 +511,12 @@ }, "libs/server": { "name": "@cleverbrush/server", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "dependencies": { - "@cleverbrush/auth": "^4.4.2", - "@cleverbrush/di": "^4.4.2", - "@cleverbrush/schema": "^4.4.2", + "@cleverbrush/auth": "^4.4.3", + "@cleverbrush/di": "^4.4.3", + "@cleverbrush/schema": "^4.4.3", "@fastify/busboy": "^3.2.0", "ws": "^8.20.0" }, @@ -502,7 +537,7 @@ }, "libs/server-openapi": { "name": "@cleverbrush/server-openapi", - "version": "4.4.2", + "version": "4.4.3", "license": "BSD 3-Clause", "peerDependencies": { "@cleverbrush/auth": "^4.0.0", @@ -1380,6 +1415,10 @@ "resolved": "libs/scheduler", "link": true }, + "node_modules/@cleverbrush/scheduler-postgres": { + "resolved": "libs/scheduler-postgres", + "link": true + }, "node_modules/@cleverbrush/schema": { "resolved": "libs/schema", "link": true diff --git a/package.json b/package.json index 8edf3da8..8fc2e49b 100644 --- a/package.json +++ b/package.json @@ -22,9 +22,10 @@ "version": "changeset version", "release": "npm run clean && npm run build && changeset publish", "publish:beta": "npm run clean && npm run build && changeset version --snapshot beta && changeset publish --tag beta --no-git-tag", - "run_scheduler": "node ./libs/scheduler/dist/index.js", + "run_scheduler": "node demos/durable-jobs/demo.ts", "test": "vitest --run --typecheck", "test:queries:integration": "vitest run --config vitest.queries.config.mts", + "test:scheduler:integration": "vitest run --config vitest.scheduler.config.mts", "test:coverage": "vitest --run --coverage && node scripts/update-coverage-badges.js", "bench": "vitest bench --run", "bench:json": "BENCH_JSON=bench-results.json vitest bench --run --project benchmarks && node scripts/relativize-bench-paths.js bench-results.json", diff --git a/typedoc.json b/typedoc.json index e3567863..57266890 100644 --- a/typedoc.json +++ b/typedoc.json @@ -17,6 +17,7 @@ "libs/otel", "libs/react-form", "libs/scheduler", + "libs/scheduler-postgres", "libs/schema", "libs/schema-json", "libs/server", @@ -34,4 +35,4 @@ "← Back to Website": "/", "All Versions": "/api-docs/" } -} \ No newline at end of file +} diff --git a/vitest.scheduler.config.mts b/vitest.scheduler.config.mts new file mode 100644 index 00000000..fdb4ffa0 --- /dev/null +++ b/vitest.scheduler.config.mts @@ -0,0 +1,5 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ test: { + include: ['libs/scheduler-postgres/integration/**/*.test.ts'], + testTimeout: 15000, hookTimeout: 30000 +} }); diff --git a/websites/docs/Dockerfile b/websites/docs/Dockerfile index 8f138441..c9b8f86e 100644 --- a/websites/docs/Dockerfile +++ b/websites/docs/Dockerfile @@ -15,6 +15,9 @@ COPY libs/react-form/ libs/react-form/ COPY libs/async/ libs/async/ COPY libs/deep/ libs/deep/ COPY libs/scheduler/ libs/scheduler/ +COPY libs/scheduler-postgres/ libs/scheduler-postgres/ +COPY libs/knex-schema/ libs/knex-schema/ +COPY libs/orm/ libs/orm/ COPY libs/schema-json/ libs/schema-json/ # Copy the shared package and the docs website diff --git a/websites/docs/app/scheduler/page.tsx b/websites/docs/app/scheduler/page.tsx index 21b5b06f..6eb42ca9 100644 --- a/websites/docs/app/scheduler/page.tsx +++ b/websites/docs/app/scheduler/page.tsx @@ -1,324 +1,266 @@ -/** biome-ignore-all lint/security/noDangerouslySetInnerHtml: needed for code examples */ +/** biome-ignore-all lint/security/noDangerouslySetInnerHtml: trusted highlighted examples */ import { highlightTS } from '@cleverbrush/website-shared/lib/highlight'; import { docsMetadata } from '../site'; export const metadata = docsMetadata('/scheduler'); +const contract = `// contracts/report.ts +import { object, string, number } from '@cleverbrush/schema'; +import { defineJob } from '@cleverbrush/scheduler'; + +export const Report = defineJob({ + name: 'report', version: 1, + input: object({ reportId: string() }), + progress: object({ percent: number() }), + output: object({ downloadUrl: string() }), + retry: { maxAttempts: 3 } // opt-in; default: one attempt +});`; + +const handler = `// handlers/report.ts — strong types across file boundaries +import type { JobHandler } from '@cleverbrush/scheduler'; +import { Report } from '../contracts/report.js'; + +export const handleReport: JobHandler = async (input, context) => { + context.signal.throwIfAborted(); + await context.report({ percent: 50 }); // persisted before resolving + return { downloadUrl: '/reports/' + input.reportId }; +};`; + +const runtime = `import knex from 'knex'; +import { JobScheduler } from '@cleverbrush/scheduler'; +import { PostgresJobRepository } from '@cleverbrush/scheduler-postgres'; +import { Report } from './contracts/report.js'; +import { handleReport } from './handlers/report.js'; + +const database = knex({ + client: 'pg', connection: process.env.DATABASE_URL, + acquireConnectionTimeout: 5000 +}); +const jobs = new JobScheduler({ + storageRepository: new PostgresJobRepository(database), + namespace: 'reports' +}); +const run = await jobs.enqueue(Report, { reportId: 'quarterly' }, { + idempotencyKey: 'quarterly:2026-Q4' +}); +const worker = jobs.createWorker({ + jobs: [Report.handle(handleReport)], concurrency: 4 +}); +await worker.start(); +for await (const event of jobs.events(Report, run.id)) { + console.log(event.sequence, event.type, event.data); +} +// On shutdown, stop workers before closing the caller-owned pool. +await worker.stop(); +await jobs.stop(); +await database.destroy();`; + +const schedules = `import { ScheduleSchema, type Schedule } from '@cleverbrush/scheduler'; + +const rules: Schedule[] = [ + { every: 'minute', interval: 15 }, + { every: 'day', hour: 18, minute: 30 }, + { every: 'week', dayOfWeek: [1, 5], hour: 9 }, + { every: 'month', day: 'last' }, + { every: 'year', month: 2, day: 'last' } +]; +const schedule = ScheduleSchema.parse({ + every: 'week', dayOfWeek: [1, 5], + startsOn: '2026-10-01T00:00:00Z', maxOccurrences: 10 +});`; + +const recurring = `await jobs.upsertSchedule('weekday-reports', Report, { reportId: 'daily' }, { + schedule: { + every: 'week', dayOfWeek: [1, 2, 3, 4, 5], + hour: 9, minute: 0, timeZone: 'Europe/Berlin' + }, + missed: 'coalesce', // or skip / replay + overlap: 'allow' // or skip +}); +const worker = jobs.createWorker({ jobs: [Report.handle(handleReport)] }); +await worker.start(); // executes accepted runs +await jobs.start(); // dispatches due occurrences +// Keep running until application shutdown, then: +await jobs.stop(); +await worker.stop({ drainTimeoutMs: 30000 }); +await database.destroy();`; + export default function SchedulerPage() { return (
-

Scheduler

+

Durable jobs and progress

- Schema-validated job scheduling with worker thread - isolation, automatic retries, and event streaming. + Typed immediate, delayed and recurring jobs with + PostgreSQL persistence and replayable progress.

- - {/* ── Install ─────────────────────────────────────── */}
-

Installation

+

Install and migrate

+ 
+                        
+                            npm install @cleverbrush/scheduler
+                            @cleverbrush/scheduler-postgres knex pg
+                        
+
+

+ Run createSchedulerTables(database) once through your + migration runner. Workers never create or alter tables + implicitly. For development, explicitly choose + InMemoryJobRepository; its state does not survive + process exits. +

+

+ + PostgreSQL setup and transactional enqueue + + {' · '} + + Migration from v4.x to v5 + +

+
+
+

Contracts and handlers in separate files
-

- Requires Node.js 16+ (uses worker_threads). -

-
- - {/* ── Features ────────────────────────────────────── */} -
-

Key features

-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- API reference table -
FeatureDescription
Worker thread isolation - Each job runs in its own{' '} - Worker thread — crashes - never affect the scheduler -
5 schedule types - Minute, day, week, month, year with - configurable intervals -
Automatic retries - maxRetries for immediate - retries; maxConsequentFails{' '} - to auto-disable flaky jobs -
Concurrency control - noConcurrentRuns: true{' '} - prevents overlapping executions -
Event-driven - job:start,{' '} - job:end,{' '} - job:error,{' '} - job:timeout,{' '} - job:message events with - stdout/stderr streams -
Timeout handling - Per-job timeout with automatic worker - termination -
Pluggable persistence - Implement IJobRepository{' '} - for database-backed storage (default: - in-memory) -
Schema-validated - All schedules validated with{' '} - @cleverbrush/schema — - export{' '} - Schemas.ScheduleSchema for - API validation -
-
-
- - {/* ── Basic usage ─────────────────────────────────── */} -
-

Basic usage
+

+ Input, progress and output use synchronous Framework + schemas and strict JSON values. Store file references + and date strings rather than binary files or Date + instances. Job versions identify persisted contracts; + keep handlers for versions still in the queue. +

- - {/* ── Events ──────────────────────────────────────── */}
-

Event handling

+

Produce, execute and observe

                          {
-    console.log(\`Job \${jobId} started at \${startDate}\`);
-    stdout.on('data', (chunk) => process.stdout.write(chunk));
-});
-
-scheduler.on('job:end', ({ jobId, endDate }) => {
-    console.log(\`Job \${jobId} completed at \${endDate}\`);
-});
-
-scheduler.on('job:error', ({ jobId, error }) => {
-    console.error(\`Job \${jobId} failed:\`, error);
-});
-
-scheduler.on('job:timeout', ({ jobId }) => {
-    console.warn(\`Job \${jobId} timed out — worker terminated\`);
-});
-
-scheduler.on('job:message', ({ jobId, value }) => {
-    console.log(\`Message from \${jobId}:\`, value);
-});`)
+                                __html: highlightTS(runtime)
                             }}
                         />
+

+ Producer processes need no start call. Workers and + recurring dispatchers have independent lifecycles and + may run in separate processes. Register a trusted + compiled module with Report.thread(new URL(...)) to use + worker threads. +

+

+ events accepts an exclusive after sequence cursor and an + AbortSignal. It replays committed events and follows + until terminal state or disconnect. Disconnecting does + not cancel work. Use cancel(runId) explicitly. + Applications own authentication, authorization and + SSE/WebSocket/HTTP transport; namespaces are not access + control. +

- - {/* ── Schedule types ──────────────────────────────── */} -
-

Schedule types

-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- API reference table -
TypeExampleRuns
- minute - - - { - '{ every: "minute", interval: 15 }' - } - - Every 15 minutes
- day - - - { - '{ every: "day", interval: 1, hour: 3, minute: 0 }' - } - - Daily at 03:00
- week - - - { - '{ every: "week", interval: 1, dayOfWeek: [1, 5], hour: 9, minute: 0 }' - } - - Mon & Fri at 09:00
- month - - - { - '{ every: "month", interval: 1, day: 1, hour: 0, minute: 0 }' - } - - 1st of each month at midnight
- year - - - { - '{ every: "year", interval: 1, month: 1, day: 1, hour: 0, minute: 0 }' - } - - Jan 1 each year at midnight
-
-
- - {/* ── Schedule calculator ─────────────────────────── */}
-

Schedule calculator

+

Retries and ownership

- Use ScheduleCalculator standalone to - preview or test schedule dates without running jobs: + Retries are opt-in and rerun the whole handler. Leases + fence stale state writes, but they cannot undo business + side effects. Execution with retries is at-least-once, + not exactly-once: use idempotent effects keyed by runId. + A timeout, crash or shutdown consumes an attempt. Throw + NonRetryableJobError for permanent failures.

+

+ Ordinary functions must cooperate with their AbortSignal + and retain their local concurrency slot until they + settle. Worker threads can be terminated. PostgreSQL + claims use database time and row locks; accepted work + and progress survive process restarts. +

+
+
+

Schema-driven periodic schedules

                          someCutoff) break;
-}`)
+                                __html: highlightTS(schedules)
                             }}
                         />
-
- - {/* ── Persistence ─────────────────────────────────── */} -
-

Custom persistence

- By default jobs are stored in memory. Implement{' '} - IJobRepository for durable storage: + Schedule is inferred from ScheduleSchema. All five + variants have individual schemas, exported directly and + through Schemas. Weekly schedules require weekdays; + monthly schedules require a day; yearly schedules + require both month and day. Minute schedules have no + local hour or minute. JSON dates are parsed at + validation boundaries.
+

+ Recurring occurrences are enqueued through the same + worker engine. Equivalent defaults, dates and weekday + order preserve cursors and the original start anchor; + updates affect future dispatch only. Pause/remove do not + cancel accepted runs. Use pauseSchedule(id), + pauseSchedule(id, false) or removeSchedule(id) to manage + triggers. +

+

+ Minutes use elapsed time. Days, weeks, months and years + use UTC or explicit IANA calendar time. DST gaps are + skipped; repeated wall times use the earlier instant + once. Calendar time defaults to 09:00; interval defaults + to 1. Monthly days are 1–28 or last. ScheduleCalculator + previews these same rules with one-based slot indexes. + Missed occurrences coalesce by default; skip drops + backlogs and replay enqueues bounded batches. Overlap + skip includes queued and retry-wait work. +

+
+
+

Limits and operations

+

+ Defaults: concurrency 1, polling 1 second, lease 30 + seconds, heartbeat 10 seconds, attempt timeout 5 + minutes, terminal retention 7 days. Payloads are limited + to 1 MiB, progress to 64 KiB per event and 10,000 events + per run. Define policies explicitly when changing these + limits. +

+

+ health returns queue counts, ready age and queued + definition versions. lastError and diagnostic callbacks + expose infrastructure failures. Cleanup removes terminal + runs and their history only; deduplication lasts for + retained runs, not forever. +

+

+ + Complete API guide + + {' · '} + + Runnable immediate and periodic examples + +

diff --git a/websites/docs/app/site.ts b/websites/docs/app/site.ts index 8a926bc6..0e357374 100644 --- a/websites/docs/app/site.ts +++ b/websites/docs/app/site.ts @@ -109,7 +109,7 @@ export const DOCS_ROUTES: RouteMetadata[] = [ path: '/scheduler', title: '@cleverbrush/scheduler', description: - 'Job scheduling documentation for workers, event handling, schedule calculators, and custom persistence.' + 'Typed durable jobs, PostgreSQL persistence, recurring triggers, worker leases, and replayable progress.' }, { path: '/log',