From 3e4f4a87b48dcf4214410e694feef37c835754ba Mon Sep 17 00:00:00 2001 From: flujo-app <300233937+flujo-app@users.noreply.github.com> Date: Tue, 3 Nov 2026 20:07:08 -0500 Subject: [PATCH 1/4] FLUJO as a worker program for lite and cloud fleets; retire waits for a busy replica; 0.7.0 A plan can set workerProgram: 'flujo'. Everything shared stays shared: the board, operator commands, stop receipts, traces, evidence and workflows talk to a driver through the contract Codex and Claude Code already use. - FlujoSessionDriver: one FLUJO conversation and workspace per agent, over FLUJO's HTTP API. - flujoProviderModel: a model provider becomes a stored FLUJO model record, so the key never enters the environment agent commands inherit. - startLiteWorker({ provider: 'flujo' }) attaches to a FLUJO or starts one. - packFactorySdk builds the SDK bundle from the installed package; plan.network gives a fleet its own private network. - runtime.mjs: what differs between programs is now one table. factory-selfhost retire now waits for a stopping app and retries refused deletes. Fixes #16 Fixes #20 --- .github/workflows/public-sdk.yml | 2 +- PUBLIC_SDK.md | 99 +++- bin/factory-cloud-mcp.mjs | 5 +- bin/factory-lite.mjs | 29 +- package-lock.json | 4 +- package.json | 2 +- scripts/verify-public-sdk.mjs | 6 +- selfhost/README.md | 2 +- src/cloud-lite/bootstrap.mjs | 18 + src/cloud-lite/bundle.mjs | 59 ++ src/cloud-lite/claude-driver.mjs | 2 +- src/cloud-lite/collect.mjs | 4 +- src/cloud-lite/evidence-worker.mjs | 2 +- src/cloud-lite/fleet.mjs | 4 +- src/cloud-lite/flujo-driver.mjs | 564 ++++++++++++++++++ src/cloud-lite/protocol.mjs | 30 +- src/cloud-lite/runtime.mjs | 111 +++- src/flujo-swarm/flujo-client.mjs | 13 +- src/flujo-swarm/template/factory-team.mjs | 5 +- src/lite-worker.mjs | 69 ++- src/model-provider.mjs | 53 +- src/public-sdk.mjs | 6 +- src/selfhost/modal.mjs | 32 +- test/cloud-lite-flujo.test.mjs | 677 ++++++++++++++++++++++ test/fixtures/fake-flujo.mjs | 197 +++++++ test/model-provider.test.mjs | 7 +- test/public-sdk.test.mjs | 2 +- test/selfhost-modal.test.mjs | 45 +- 28 files changed, 1950 insertions(+), 99 deletions(-) create mode 100644 src/cloud-lite/bundle.mjs create mode 100644 src/cloud-lite/flujo-driver.mjs create mode 100644 test/cloud-lite-flujo.test.mjs create mode 100644 test/fixtures/fake-flujo.mjs diff --git a/.github/workflows/public-sdk.yml b/.github/workflows/public-sdk.yml index e9d6740..dd1c96f 100644 --- a/.github/workflows/public-sdk.yml +++ b/.github/workflows/public-sdk.yml @@ -24,7 +24,7 @@ jobs: - run: npm ci - run: npm audit --omit=dev --audit-level=high - name: Focused tests, leaving nothing in the temp location - run: node scripts/test-temp-guard.mjs test/cloud-lite.test.mjs test/cloud-lite-claude.test.mjs test/cloud-lite-provider.test.mjs test/model-provider.test.mjs test/selfhost-modal.test.mjs test/public-sdk.test.mjs test/template-catalog.test.mjs test/factory-team-template.test.mjs test/flujo-workspace.test.mjs test/savia-case-plan.test.mjs test/savia-topology.test.mjs test/managed-fleet.test.mjs test/managed-fleet-real-module.test.mjs test/managed-fleet-provider-closure.test.mjs test/provider-cell-retirement.test.mjs test/presentation.test.mjs test/presentation-modal.test.mjs test/presentation-closure.test.mjs test/observatory-client.test.mjs test/workflows.test.mjs test/workflow-native-wake.test.mjs test/workflow-m0-lifecycle.test.mjs test/workflow-m1-portable-artifacts.test.mjs test/workflow-m2-ack-recovery.test.mjs test/workflow-m2-authority-recovery.test.mjs + run: node scripts/test-temp-guard.mjs test/cloud-lite.test.mjs test/cloud-lite-claude.test.mjs test/cloud-lite-flujo.test.mjs test/cloud-lite-provider.test.mjs test/model-provider.test.mjs test/selfhost-modal.test.mjs test/public-sdk.test.mjs test/template-catalog.test.mjs test/factory-team-template.test.mjs test/flujo-workspace.test.mjs test/savia-case-plan.test.mjs test/savia-topology.test.mjs test/managed-fleet.test.mjs test/managed-fleet-real-module.test.mjs test/managed-fleet-provider-closure.test.mjs test/provider-cell-retirement.test.mjs test/presentation.test.mjs test/presentation-modal.test.mjs test/presentation-closure.test.mjs test/observatory-client.test.mjs test/workflows.test.mjs test/workflow-native-wake.test.mjs test/workflow-m0-lifecycle.test.mjs test/workflow-m1-portable-artifacts.test.mjs test/workflow-m2-ack-recovery.test.mjs test/workflow-m2-authority-recovery.test.mjs - name: Self-hosted model settings (Python, offline) run: python3 -B -m unittest discover -s selfhost/modal -p "test_*.py" - name: Install and exercise the packed public package diff --git a/PUBLIC_SDK.md b/PUBLIC_SDK.md index 7f309e9..42810cc 100644 --- a/PUBLIC_SDK.md +++ b/PUBLIC_SDK.md @@ -1,17 +1,17 @@ # Public local Factory SDK, CLI, and MCP -Version 0.6.0 is distributed as a versioned package in -[GitHub Releases](https://github.com/flujo-app/FACTORY/releases/tag/v0.6.0): +Version 0.7.0 is distributed as a versioned package in +[GitHub Releases](https://github.com/flujo-app/FACTORY/releases/tag/v0.7.0): ```sh -npm install https://github.com/flujo-app/FACTORY/releases/download/v0.6.0/flujo-factory-0.6.0.tgz +npm install https://github.com/flujo-app/FACTORY/releases/download/v0.7.0/flujo-factory-0.7.0.tgz ``` FACTORY can serve an open-weight model on your own Modal GPUs and run workers on it. `ModalSelfHostedModel` and `factory-selfhost` deploy, verify, stop and retire a vLLM endpoint with Qwen3.8-27B (H100) or Nemotron 3.5 Super (B300) profiles; see [selfhost/README.md](selfhost/README.md). Its endpoint description is a model provider that -cloud and local lite workers (Codex or Claude Code) use directly. See +cloud and local lite workers (Codex, Claude Code or FLUJO) use directly. See [Self-hosted models and operator-driven jobs](#self-hosted-models-and-operator-driven-jobs) for the provider, read-only deploy keys, prepare hooks, operator-driven jobs and quieter traces added in 0.4.0. All of it is optional; existing plans behave as before, except that @@ -137,7 +137,7 @@ local overrides are not implicitly shared with a remote adapter. ### Lite workers `factory-lite` opens a regular interactive Codex or Claude CLI session in the -current directory. It does not provision a FLUJO worker or use the FLUJO +current directory, or runs a FLUJO worker (see below). A CLI session does not provision a FLUJO worker or use the FLUJO conversation harness. The CLI retains its built-in tools and unrestricted file and command access. The only MCP server supplied for that session is FACTORY's local board/swarm server, connected to the specified durable database. @@ -201,6 +201,23 @@ description the cloud plan uses (see command line, `factory-lite claude board.sqlite --model-provider endpoint.json` reads the key from the variable the file names in `envKey`. +A local FLUJO worker has no terminal session. It attaches to a running FLUJO, or starts +its own with a separate data folder, installs the generic team template with the board +as a tool server, and keeps its conversation across runs: + +```js +const worker = await factory.startLiteWorker({ provider: 'flujo', modelProvider, providerKey, + origin: 'http://127.0.0.1:4200', prompt: 'Work on the Factory board' }); +const turn = await worker.settled(); +await worker.send('Next job'); +await worker.stop(); +``` + +From the command line: `factory-lite flujo board.sqlite --model-provider endpoint.json +[--origin http://127.0.0.1:4200] "Work on the Factory board"`. Without `--origin`, the +`flujo` command from `npm install -g flujo-ai` must be on the path. A record from +`POST /api/model` can stand in for a provider as `flujoModel`. + ## FLUJO worker dispatch ### Cloud lite workers (development) @@ -320,6 +337,70 @@ await selfHostedFleet.deploy({ ...plan, workerProgram: 'claude', model: provider `thread/tokenUsage/updated`, and model retries become `model/retry`. Streaming deltas are skipped unless `traceDeltas` is on. +#### FLUJO as the worker program + +Set `workerProgram: 'flujo'` to run a FLUJO server on each machine. The board, operator +commands and stop receipts, durable traces, prepare hooks, evidence collection and +verified workflows are the same shared code as for Codex and Claude Code. + +```js +const fleet = new FactoryCloudLiteFleet({ directory: '/absolute/private/fleet-run', + flujoCloudPath: '/absolute/installed/flujo-cloud' }); +await fleet.deploy({ runId: 'my-flujo-run', app: 'my-flujo-app', goal: ownerGoal, + workerProgram: 'flujo', machines: 3, agentsPerMachine: 1, + modelProvider: model.endpoint(), + // Optional: each orchestrator leads a team of native FLUJO subflows. + flujo: { teamConcurrency: 4, agentTurns: 200, leadTurns: 600 }, +}, { image: immutableNodeImage, bundlePath: '/absolute/private/flujo-factory.tgz', providerKey: await model.key() }); +``` + +- **Model:** a plan needs a `modelProvider`; FLUJO workers have no subscription login. It + can offer `responses`, `chat` or `anthropic`, and FLUJO uses the first of those that the + provider has. `flujoProviderModel` is the exported mapping. FLUJO stores the key in its + own encrypted settings, so the FLUJO process and the agent commands it starts never have + it in their environment. `reasoningEffort` does not apply: FLUJO accepts it only for + models it knows by name. +- **Bundle:** `bundlePath` can be a release file, or what `packFactorySdk(file)` writes from + the installed package, which needs no `npm` or `tar` on the host. `network` in the plan + puts the fleet's app on its own private network. +- **Install:** the bootstrap installs one pinned `flujo-ai` release on the volume, checked + against its SHA-256, so a restarted Machine does not install it again. Allow a few minutes + on first start. A custom image can provide `flujo` on the path instead. Use a Node image of + at least 24.22.1 or 22.23.4, running as root; FLUJO's official image runs as a normal user. +- **Sessions:** each agent slot has its own FLUJO workspace with one conversation, so its + board identity, model record and tools stay apart. FLUJO runs on port 4210 on the + Machine's loopback, or `FACTORY_FLUJO_PORT`. Each workspace runs its own tool servers, so + plan on about 400 MB per agent plus 300 MB for FLUJO. FLUJO restarts by itself up to + three times in ten minutes; after that the worker stops, as with the other programs. +- **Team:** with `flujo.teamConcurrency` an orchestrator gets the `swarm_team` flow, with one + native gate for its subflows. The others get `swarm_agent`. A turn ends, and a stop receipt + is issued, only after every child conversation has ended too. Each conversation lists its + children under `/v1/chat/conversations?descendantsOf=`. +- **Tools:** `bash` and `filesystem`, plus `browser` if `flujo.servers` lists it and the image + has its browser; the bootstrap does not download one. The FLUJO control server stays off. + The board server takes its address and credential from a private file. A worker's prompt + names its working folder, because FLUJO's tools otherwise start in FLUJO's data folder. +- **Sandbox:** only `danger-full-access`. FLUJO's folder settings are a guide for its tools, + not a boundary, so the Machine is the sandbox, and a plan with a narrower `sandbox` is + refused. Agents get no GitHub token, as with other self-hosted model plans. An agent command + runs as the same user, so it could still read FLUJO's stored settings, including the model + key; use another program with a narrower sandbox when agents handle untrusted input. +- **Runs:** a run belongs to FLUJO, not to the connection that asked for it. A dropped event + stream is followed again from its last event. A submission whose answer was lost is read + back and never sent twice. While FLUJO is under pressure a turn waits for up to five + minutes. A question or form from a tool is declined, because nobody is there to answer it. +- **Messages and stops:** a message during a turn joins it at its next safe point; if the run + ended before taking it, the same turn goes on with it. A stop is requested again until FLUJO + has stopped, and wins over a queued message. If it never stops, no receipt is issued. +- **Restarts:** the conversation survives. A turn that was running when FLUJO died ends as + `interrupted`. It is not replayed; in autonomous mode the agent gets a new turn. If the + conversation is gone, the usual recovery for an unavailable session applies. +- **Traces and evidence:** messages, tool calls and results and subflow starts and ends become + `item/completed` records (`agentMessage`, `toolCall`, `toolResult`, `subflow`), with `depth` + and `lane` for children. Token use becomes `thread/tokenUsage/updated`, retries become + `model/retry`, and errors become `flujo/error`. Evidence holds the conversations from + `/data/flujo/workspaces`, not its keys, server settings or tool copies. + #### Self-hosted models and operator-driven jobs To run the model yourself, `ModalSelfHostedModel` (CLI `factory-selfhost`) deploys a @@ -357,12 +438,12 @@ await fleet.command('m0-a1', { id: 'job-1', action: 'message', message: 'First j maps to its own settings. `id`, `envKey` (the variable that carries the key) and `https` `baseUrls` (OpenAI-style, ending in `/v1`) without credentials are required. `apis` lists the wire APIs the server offers: `responses` (Codex), - `anthropic` (Claude Code, `/v1/messages`) and `chat`. vLLM serves all three; the - default is `['responses']`. `name`, `model`, `contextWindow`, + `anthropic` (Claude Code, `/v1/messages`) and `chat`; FLUJO can use any of them. vLLM + serves all three; the default is `['responses']`. `name`, `model`, `contextWindow`, `autoCompactTokenLimit`, `toolOutputTokenLimit`, `requestMaxRetries`, `streamMaxRetries`, `streamIdleTimeoutMs` and `requestTimeoutMs` are optional. - `codexProviderConfig`, `codexProviderOverrides` and `claudeProviderEnvironment` - are exported mappings; `providerPrograms(provider)` says which programs can use + `codexProviderConfig`, `codexProviderOverrides`, `claudeProviderEnvironment` and + `flujoProviderModel` are exported mappings; `providerPrograms(provider)` says which programs can use it. Each Codex gets a `config.toml` with the provider, the model's real context window and its compaction point; web search, multi-agent, apps, plugins, goals, memories and skill instructions are off, and agents' shell commands inherit only diff --git a/bin/factory-cloud-mcp.mjs b/bin/factory-cloud-mcp.mjs index 9e0b901..8447853 100644 --- a/bin/factory-cloud-mcp.mjs +++ b/bin/factory-cloud-mcp.mjs @@ -1,3 +1,6 @@ #!/usr/bin/env node +import { readFileSync } from 'node:fs'; import { startCloudBoardMcp } from '../src/cloud-lite/mcp.mjs'; -await startCloudBoardMcp(); +// `--config ` carries the address and credential for hosts that keep server settings in clear view. +const flag = process.argv.indexOf('--config'); +await startCloudBoardMcp(flag === -1 ? {} : JSON.parse(readFileSync(process.argv[flag + 1], 'utf8'))); diff --git a/bin/factory-lite.mjs b/bin/factory-lite.mjs index a93a8e9..52fbd17 100644 --- a/bin/factory-lite.mjs +++ b/bin/factory-lite.mjs @@ -2,9 +2,10 @@ import { readFileSync } from 'node:fs'; import { Factory } from '../src/public-sdk.mjs'; -const usage = 'Usage: factory-lite [--model-provider ] [prompt]\n' +const usage = 'Usage: factory-lite [--model-provider ] [--origin ] [prompt]\n' + ' --model-provider use a self-hosted model (e.g. from `factory-selfhost endpoint`); its key is read from the\n' - + ' environment variable the file names in envKey.\n'; + + ' environment variable the file names in envKey. FLUJO needs this.\n' + + ' --origin FLUJO only: attach to this running FLUJO instead of starting one.\n'; const args = process.argv.slice(2); const [provider, database] = args; if (provider === '--help' || provider === 'help' || !provider) { @@ -22,9 +23,31 @@ if (provider === '--help' || provider === 'help' || !provider) { providerKey = process.env[modelProvider.envKey]; if (!providerKey) throw new Error(`Set ${modelProvider.envKey} to the endpoint key.`); } - const child = new Factory(database).startLiteWorker({ provider, prompt: words.length ? words.join(' ') : undefined, modelProvider, providerKey }); + let origin; + const originFlag = words.indexOf('--origin'); + if (originFlag !== -1) { + origin = words[originFlag + 1]; + if (provider !== 'flujo' || !origin) throw new Error('--origin needs a URL and is for FLUJO workers.'); + words.splice(originFlag, 2); + } + const started = new Factory(database).startLiteWorker({ provider, prompt: words.length ? words.join(' ') : undefined, modelProvider, providerKey, ...(origin ? { origin } : {}) }); + if (provider === 'flujo') { + const worker = await started; + process.stdout.write(`FLUJO worker ready at ${worker.origin} in workspace ${worker.workspace}.\n`); + const finish = () => { void worker.stop(); }; + process.once('SIGINT', finish); process.once('SIGTERM', finish); + // With a prompt the worker ends with its turn; without one a started server stays up until stopped. + if (words.length) { + const turn = await worker.settled(); + process.stdout.write(`Turn ended: ${turn?.status}.\n`); + process.exitCode = turn?.status === 'completed' ? 0 : 1; + await worker.stop(); + } else if (origin) await worker.stop(); + } else { + const child = started; child.on('error', error => { process.stderr.write(`${error.message}\n`); process.exitCode = 1; }); child.on('exit', (code, signal) => { process.exitCode = code ?? (signal ? 1 : 0); }); + } } catch (error) { process.stderr.write(`${error.message}\n`); process.exitCode = 1; diff --git a/package-lock.json b/package-lock.json index 994859f..6eb85b8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "1.32.1" diff --git a/package.json b/package.json index 7a87709..1dc7466 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "description": "Durable FLUJO swarm engine and Factory SDK, CLI, and MCP server", "license": "MIT", "repository": { diff --git a/scripts/verify-public-sdk.mjs b/scripts/verify-public-sdk.mjs index 6d1090c..123d59e 100644 --- a/scripts/verify-public-sdk.mjs +++ b/scripts/verify-public-sdk.mjs @@ -96,7 +96,11 @@ assert.equal(providerPlan.modelProvider.autoCompactTokenLimit, 80000); assert.throws(() => sdk.validateCloudLitePlan({ ...providerPlan, modelProvider: { ...providerPlan.modelProvider, baseUrls: ['http://model.example/v1'] } }), TypeError); // One provider description maps to Codex and Claude Code. const endpoint = { ...providerPlan.modelProvider, model: 'served-model', apis: ['responses', 'anthropic'] }; -assert.deepEqual(sdk.providerPrograms(endpoint), ['codex', 'claude']); +assert.deepEqual(sdk.providerPrograms(endpoint), ['codex', 'claude', 'flujo']); +assert.equal(sdk.flujoProviderModel(endpoint, { key: 'package-smoke-key' }).adapter, 'openai-responses'); +assert.equal(typeof sdk.FlujoSessionDriver, 'function'); +assert.equal(typeof sdk.packFactorySdk, 'function'); +assert.match(sdk.cloudLiteBootstrap('a'.repeat(64), { workerProgram: 'flujo' }), /flujo-ai-\d+\.\d+\.\d+\.tgz/); assert.match(sdk.codexProviderConfig(endpoint), /wire_api = "responses"/); assert.equal(sdk.claudeProviderEnvironment(endpoint, { key: 'package-smoke-key' }).ANTHROPIC_BASE_URL, 'https://model.example'); // Self-hosted model serving ships its Python sources and profiles, but not its tests. diff --git a/selfhost/README.md b/selfhost/README.md index 5735da3..c9aade5 100644 --- a/selfhost/README.md +++ b/selfhost/README.md @@ -4,7 +4,7 @@ FACTORY can serve an open-weight model on your own Modal GPUs with vLLM, so work instead of a vendor API: source code and prompts stay in your account. The served endpoint speaks the OpenAI Responses API (Codex), the Anthropic Messages API (Claude Code) and Chat Completions, and plugs straight into `plan.modelProvider` and the local lite workers. Cloud lite workers can -run either program on it (`plan.workerProgram`). +run any of their programs on it (`plan.workerProgram`: Codex, Claude Code or FLUJO). This folder is separate from `modal/`, which holds the earlier pilot definitions. diff --git a/src/cloud-lite/bootstrap.mjs b/src/cloud-lite/bootstrap.mjs index a9c1bfe..0487e80 100644 --- a/src/cloud-lite/bootstrap.mjs +++ b/src/cloud-lite/bootstrap.mjs @@ -1,6 +1,8 @@ /** Bootstrap on the pinned upstream Node image when a custom registry is unavailable. */ const CLAUDE_CODE_VERSION = '2.1.311'; const CLAUDE_CODE_LINUX_X64_SHA256 = '86dc1fcc69498159eaf15f446613e847010f9036bfbbdffd4c49350be8489ada'; +const FLUJO_VERSION = '3.48.3'; +const FLUJO_SHA256 = '09e954ec5b15e30748c28576b2c0bbe3d4f5cdc34afbbbc139d92759474b6b98'; const programInstall = { codex: { folder: '/data/codex', packages: '', home: 'CODEX_HOME=/data/codex', @@ -21,6 +23,22 @@ echo '${CLAUDE_CODE_LINUX_X64_SHA256} /tmp/claude' | sha256sum -c - install -m 755 /tmp/claude /usr/local/bin/claude rm -f /tmp/claude`, }, + // Kept on the volume, so a restarted Machine does not install it again. An image that already has + // `flujo` on its path skips this. Install scripts stay off: they only download a browser workers leave off. + flujo: { + folder: '/data/flujo /data/factory-private /data/results', packages: ' ripgrep python3-venv python3-pip', home: 'FLUJO_DATA_DIR=/data/flujo', + install: `if ! command -v flujo >/dev/null 2>&1; then +if [ ! -f /data/flujo-program/.installed-${FLUJO_VERSION} ]; then +rm -rf /data/flujo-program +curl -fsSL https://registry.npmjs.org/flujo-ai/-/flujo-ai-${FLUJO_VERSION}.tgz -o /tmp/flujo.tgz +echo '${FLUJO_SHA256} /tmp/flujo.tgz' | sha256sum -c - +npm install --prefix /data/flujo-program /tmp/flujo.tgz --ignore-scripts --omit=dev --no-audit --no-fund +rm -f /tmp/flujo.tgz +touch /data/flujo-program/.installed-${FLUJO_VERSION} +fi +ln -sf /data/flujo-program/node_modules/.bin/flujo /usr/local/bin/flujo +fi`, + }, }; export function cloudLiteBootstrap(bundleSha256, { workerProgram = 'codex' } = {}) { diff --git a/src/cloud-lite/bundle.mjs b/src/cloud-lite/bundle.mjs new file mode 100644 index 0000000..bbb3b57 --- /dev/null +++ b/src/cloud-lite/bundle.mjs @@ -0,0 +1,59 @@ +import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { gzipSync } from 'node:zlib'; + +const PACKAGE_ROOT = fileURLToPath(new URL('../../', import.meta.url)); +const FOLDERS = ['src', 'bin', 'scripts', 'selfhost']; +const FILES = ['package.json', 'README.md', 'PUBLIC_SDK.md', 'WORKFLOWS.md', 'LICENSE']; + +function tarHeader(name, size, mode) { + const header = Buffer.alloc(512); + let path = name, prefix = ''; + if (Buffer.byteLength(path) > 100) { + const cut = path.indexOf('/', Buffer.byteLength(path) - 101); + if (cut === -1 || cut > 155) throw new Error(`Path is too long for the SDK bundle: ${name}`); + prefix = path.slice(0, cut); path = path.slice(cut + 1); + } + const put = (text, offset, length) => header.write(text, offset, length, 'utf8'); + const octal = (value, length) => value.toString(8).padStart(length - 1, '0') + '\0'; + put(path, 0, 100); put(octal(mode, 8), 100, 8); put(octal(0, 8), 108, 8); put(octal(0, 8), 116, 8); + put(octal(size, 12), 124, 12); put(octal(0, 12), 136, 12); put(' ', 148, 8); put('0', 156, 1); + put('ustar\0', 257, 6); put('00', 263, 2); put(prefix, 345, 155); + const checksum = header.reduce((sum, byte) => sum + byte, 0); + put(`${checksum.toString(8).padStart(6, '0')}\0 `, 148, 8); + return header; +} + +function listFiles(folder) { + const found = []; + for (const entry of readdirSync(folder, { withFileTypes: true })) { + if (['node_modules', '.git', '__pycache__'].includes(entry.name)) continue; + const path = join(folder, entry.name); + if (entry.isDirectory()) found.push(...listFiles(path)); + else if (entry.isFile()) found.push(path); + } + return found; +} + +/** + * Writes this installed package as the SDK tarball a fleet's bootstrap unpacks, so an app can deploy + * workers from its own dependency, with no copy of the release file and no `npm` or `tar` on the host. + * The same files give the same bytes, so a fleet can check it is resuming with the same bundle. + */ +export function packFactorySdk(destination, { root = PACKAGE_ROOT } = {}) { + if (typeof destination !== 'string' || !destination) throw new TypeError('Select where to write the SDK bundle.'); + const files = [...FILES.map(name => join(root, name)).filter(path => statSync(path, { throwIfNoEntry: false })?.isFile()), + ...FOLDERS.map(name => join(root, name)).filter(path => statSync(path, { throwIfNoEntry: false })?.isDirectory()).flatMap(listFiles)] + .map(path => ({ path, name: relative(root, path).split(/[\\/]/).join('/') })) + .sort((a, b) => (a.name < b.name ? -1 : 1)); + const parts = []; + for (const { path, name } of files) { + const bytes = readFileSync(path); + parts.push(tarHeader(`package/${name}`, bytes.length, name.startsWith('bin/') ? 0o755 : 0o644), bytes, Buffer.alloc((512 - bytes.length % 512) % 512)); + } + parts.push(Buffer.alloc(1024)); + mkdirSync(dirname(destination), { recursive: true, mode: 0o700 }); + writeFileSync(destination, gzipSync(Buffer.concat(parts), { level: 9 }), { mode: 0o600 }); + return { file: destination, files: files.length }; +} diff --git a/src/cloud-lite/claude-driver.mjs b/src/cloud-lite/claude-driver.mjs index 276a8bf..8394f40 100644 --- a/src/cloud-lite/claude-driver.mjs +++ b/src/cloud-lite/claude-driver.mjs @@ -157,7 +157,7 @@ export class ClaudeSessionDriver extends EventEmitter { const recovered = lastTurn?.status === 'inProgress' ? { ...lastTurn, status: 'interrupted' } : lastTurn; const thread = this.addThread(id, { ...params, lastTurn: recovered }); this.notify('thread/resumed', { threadId: id, thread: { id } }); - return { thread: this.describe(thread) }; + return { thread: this.describe(thread), ...(lastTurn?.status === 'inProgress' ? { recoveredTurn: recovered } : {}) }; } readThread({ threadId, includeTurns }) { diff --git a/src/cloud-lite/collect.mjs b/src/cloud-lite/collect.mjs index bb5a598..ada25ec 100644 --- a/src/cloud-lite/collect.mjs +++ b/src/cloud-lite/collect.mjs @@ -11,7 +11,9 @@ export async function collectCloudLiteSnapshot({ root = '/data', runId, destinat mkdirSync(destination, { mode: 0o700 }); const output = join(destination, 'snapshot'); mkdirSync(output, { recursive: true, mode: 0o700 }); const manifest = { runId, includeGitObjects, startedAt: new Date().toISOString(), files: [], omitted: [], changedDuringCopy: [], sqliteBackups: [] }; - const excluded = new Set(['node_modules', '.next', '.npm', '.cache', '__pycache__', '.pytest_cache', 'auth.json', 'lost+found']); + const excluded = new Set(['node_modules', '.next', '.npm', '.cache', '__pycache__', '.pytest_cache', 'auth.json', 'lost+found', + // FLUJO keeps its keys and server settings in these, and the board credential sits in the last one. + 'encryption_key.json', 'models.json', 'mcp_servers.json', 'factory-private']); const digest = async file => { const hash = createHash('sha256'); for await (const bytes of createReadStream(file)) hash.update(bytes); return hash.digest('hex'); }; const identicalFiles = new Map(); let hardLinkedFiles = 0; async function visit(source) { diff --git a/src/cloud-lite/evidence-worker.mjs b/src/cloud-lite/evidence-worker.mjs index 26fe2ae..b87bf1b 100644 --- a/src/cloud-lite/evidence-worker.mjs +++ b/src/cloud-lite/evidence-worker.mjs @@ -14,7 +14,7 @@ process.on('message',async message=>{ sessions=message.sessions; const directory='/data/factory-evidence';mkdirSync(directory,{recursive:true,mode:0o700}); const roots=[{name:'workspaces',path:'/data/worktrees',exclude:['node_modules','.git','.next','.cache','.npm','__pycache__']}, - {name:'sessions',path:message.sessionsPath??'/data/codex/sessions'}, {name:'results',path:'/data/results'}, + {name:'sessions',path:message.sessionsPath??'/data/codex/sessions',...(message.sessionsExclude?{exclude:message.sessionsExclude}:{})}, {name:'results',path:'/data/results'}, {name:'runtime',path:'/data/factory',exclude:['control.sqlite','control.sqlite-wal','control.sqlite-shm']}, {name:'board',path:resolve(directory,'board')}]; for(const root of roots)mkdirSync(root.path,{recursive:true,mode:0o700}); diff --git a/src/cloud-lite/fleet.mjs b/src/cloud-lite/fleet.mjs index 99ffc95..0464200 100644 --- a/src/cloud-lite/fleet.mjs +++ b/src/cloud-lite/fleet.mjs @@ -206,7 +206,7 @@ export class FactoryCloudLiteFleet { writeFileSync(join(this.directory, 'credentials.json'), JSON.stringify({ credentials, operatorToken: secret() }), { mode: 0o600, flag: 'wx' }); this.state = { schemaVersion: 1, plan, image: null, operations: {}, machines: [], createdAt: new Date().toISOString() }; this.save(); } - await this.stage('app.create', async () => { await this.fly(['apps', 'create', plan.app, '--org', plan.org ?? 'personal']); return { app: plan.app }; }); + await this.stage('app.create', async () => { await this.fly(['apps', 'create', plan.app, '--org', plan.org ?? 'personal', ...(plan.network ? ['--network', plan.network] : [])]); return { app: plan.app }; }); return this.snapshot(); } async deploy(input, { image, imageSource, channel, githubToken, bundlePath, evidenceTokens, deployKey, providerKey } = {}) { @@ -411,7 +411,7 @@ export class FactoryCloudLiteFleet { throw new TypeError('Select original journal-owned Machine identities.'); } const bootstrapPath = join(this.directory, `bootstrap-${digest}.sh`); - writeFileSync(bootstrapPath, cloudLiteBootstrap(digest), { mode: 0o600 }); + writeFileSync(bootstrapPath, cloudLiteBootstrap(digest, { workerProgram: this.state.plan.workerProgram }), { mode: 0o600 }); for (const machine of selected) { await this.stage(`${machine.id}.runtime.${digest}`, async () => { const inventory = JSON.parse(await this.fly(['machine', 'list', '--app', this.state.plan.app, '--json'])); diff --git a/src/cloud-lite/flujo-driver.mjs b/src/cloud-lite/flujo-driver.mjs new file mode 100644 index 0000000..59b71b7 --- /dev/null +++ b/src/cloud-lite/flujo-driver.mjs @@ -0,0 +1,564 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { EventEmitter } from 'node:events'; +import { createHash, randomUUID } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs'; +import http from 'node:http'; +import https from 'node:https'; +import { join } from 'node:path'; +import { FlujoClient } from '../flujo-swarm/flujo-client.mjs'; +import { buildFactoryTeamSpecs, FACTORY_FLOW_NAMES } from '../flujo-swarm/template/factory-team.mjs'; + +const invalid = message => Object.assign(new Error(message), { code: -32600 }); +const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); +/** A conversation still owns work in any of these states. Anything else is terminal. */ +const ACTIVE_STATUSES = new Set(['running', 'queued', 'pending', 'awaiting', 'awaiting_tool_approval', 'paused_debug']); +/** FLUJO refuses a run for a moment under pressure; the caller waits instead of failing the turn. */ +const RETRYABLE_SUBMISSION = new Set([408, 425, 429, 503]); +/** Each bundled server is one process per workspace, so a worker keeps only what it uses. */ +export const FLUJO_WORKER_SERVERS = Object.freeze(['bash', 'filesystem']); +/** The default port stays clear of the board, which uses 4200 on the coordinator Machine. */ +export const FLUJO_WORKER_PORT = 4210; + +/** The FLUJO environment. A model key lives in FLUJO's encrypted settings, never here: agent commands inherit all of this. */ +export function flujoEnvironment({ home, port = FLUJO_WORKER_PORT, path = process.env.PATH, extra = {} } = {}) { + return { PATH: path, HOME: '/data', FLUJO_DATA_DIR: home, FLUJO_PORT: String(port), FLUJO_EXPOSURE_MODE: 'localhost', NEXT_TELEMETRY_DISABLED: '1', GIT_TERMINAL_PROMPT: '0', ...extra }; +} + +/** One FLUJO process, restarted a few times if it dies. A worker that keeps crashing is reported, not hidden. */ +export class FlujoProgram extends EventEmitter { + constructor({ command = 'flujo', args = [], port = FLUJO_WORKER_PORT, environment = () => process.env, onOutput, startTimeoutMs = 300000, stopGraceMs = 10000, restartLimit = 3, restartWindowMs = 600000 } = {}) { + super(); + Object.assign(this, { command, baseArgs: args, port, environment, onOutput, startTimeoutMs, stopGraceMs, restartLimit, restartWindowMs }); + this.origin = `http://127.0.0.1:${port}`; + this.restarts = []; + } + + spawnProcess() { + // On Windows an npm command is a .cmd file, which only a shell can start. The port is a number, so nothing here needs quoting. + const child = spawn(this.command, [...this.baseArgs, '--no-open', '--port', String(this.port)], { env: this.environment(), stdio: ['ignore', 'pipe', 'pipe'], detached: process.platform !== 'win32', windowsHide: true, shell: process.platform === 'win32' }); + this.child = child; + for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => this.onOutput?.(chunk.toString())); + return child; + } + + async waitUntilReady(child) { + let failure; + const failed = new Promise((_, reject) => { + child.once('error', error => { failure = error; reject(error); }); + child.once('exit', code => { failure = new Error(`FLUJO exited before it was ready (${code}).`); reject(failure); }); + }); + failed.catch(() => {}); + const deadline = Date.now() + this.startTimeoutMs; + const client = new FlujoClient({ origin: this.origin }); + while (Date.now() < deadline) { + if (failure) throw failure; + const listed = await client.api('GET', '/api/workspaces', undefined, { workspace: null, timeoutMs: 5000 }).catch(() => null); + if (listed?.status === 200) return; + await Promise.race([sleep(500), failed]); + } + throw new Error('FLUJO did not become ready before the start deadline.'); + } + + async start() { + const child = this.spawnProcess(); + await this.waitUntilReady(child); + this.watch(child); + } + + watch(child) { + child.removeAllListeners('exit'); + child.once('exit', async (code, signal) => { + if (this.stopping || this.child !== child) { this.emit('exit', { code, signal }); return; } + for (;;) { + this.restarts = [...this.restarts.filter(at => Date.now() - at < this.restartWindowMs), Date.now()]; + this.emit('restarting', { code, signal, restarts: this.restarts.length }); + if (this.restarts.length > this.restartLimit) { this.emit('exit', { code, signal }); return; } + try { + const next = this.spawnProcess(); + await this.waitUntilReady(next); + this.watch(next); + this.emit('restarted', { restarts: this.restarts.length }); + return; + } catch { /* Counted above; the next attempt or the limit decides. */ } + } + }); + } + + signal(name) { + const child = this.child; + if (!child?.pid) return; + if (process.platform === 'win32') { spawnSync('taskkill', ['/pid', String(child.pid), '/T', ...(name === 'SIGKILL' ? ['/F'] : [])], { stdio: 'ignore', windowsHide: true }); return; } + // FLUJO leads its own process group, so the server and its tool processes stop with it. + try { process.kill(-child.pid, name); } catch { try { child.kill(name); } catch { /* Already gone. */ } } + } + + stop() { + this.stopping = true; + const child = this.child; + if (!child || child.exitCode !== null || child.signalCode !== null) return Promise.resolve(); + return new Promise(resolve => { + const timer = setTimeout(() => this.signal('SIGKILL'), this.stopGraceMs); + child.once('exit', () => { clearTimeout(timer); resolve(); }); + this.signal('SIGTERM'); + }); + } +} + +/** Send a run and return once FLUJO has taken it. The run does not depend on this connection. */ +function submitDetached(url, { headers, body }) { + return new Promise((resolve, reject) => { + const target = new URL(url), payload = Buffer.from(JSON.stringify(body)); + const request = (target.protocol === 'https:' ? https : http).request(target, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Content-Length': payload.length, ...headers } }, response => { + if (response.statusCode === 200) { resolve({ status: 200, text: '' }); response.destroy(); return; } + const chunks = []; + response.on('data', chunk => chunks.push(chunk)); + response.on('end', () => resolve({ status: response.statusCode, text: Buffer.concat(chunks).toString('utf8').slice(0, 2000) })); + response.on('error', reject); + }); + request.on('error', reject); + request.end(payload); + }); +} + +/** Follow one conversation's durable event stream until it closes. Resolves; a dropped stream is not an error here. */ +function followEvents(url, { headers, onEvent, onControl, signal }) { + return new Promise(resolve => { + const target = new URL(url); + let buffer = ''; + const done = () => resolve(); + const request = (target.protocol === 'https:' ? https : http).request(target, { headers: { Accept: 'text/event-stream', ...headers }, signal }, response => { + if (response.statusCode !== 200) { response.resume(); response.on('end', done); response.on('error', done); return; } + response.setEncoding('utf8'); + response.on('data', chunk => { + buffer += chunk; + for (let end = buffer.indexOf('\n\n'); end !== -1; end = buffer.indexOf('\n\n')) { + const frame = buffer.slice(0, end); buffer = buffer.slice(end + 2); + const lines = frame.split('\n'); + const data = lines.filter(line => line.startsWith('data:')).map(line => line.slice(5).trimStart()).join('\n'); + if (!data) continue; + let value; try { value = JSON.parse(data); } catch { continue; } + if (lines.some(line => line.startsWith('event:'))) onControl?.(value); else onEvent(value); + } + }); + response.on('end', done); response.on('error', done); response.on('close', done); + }); + request.on('error', done); + request.end(); + }); +} + +/** + * One FLUJO conversation per agent, behind the method and notification contract the cloud runtime + * already uses for a Codex app-server: thread/start|resume|read and turn/start|steer|interrupt, with + * turn and item notifications. Each agent gets its own workspace, so its board identity, model record + * and tools stay apart, and retiring it removes exactly those. A run belongs to FLUJO, not to this + * connection: a dropped stream is followed again, and an uncertain submission is read back, never sent twice. + */ +export class FlujoSessionDriver extends EventEmitter { + constructor({ origin, token, program, stateDirectory, privateDirectory = stateDirectory, modelRecord, team = {}, servers = FLUJO_WORKER_SERVERS, partialMessages = false, onStderr, clientFactory = settings => new FlujoClient(settings), version, + pollMs = 5000, interruptResendMs = 10000, submitTimeoutMs = 300000, toolsTimeoutMs = 120000, reconnectMs = 1000, drainMs = 3000 } = {}) { + super(); + if (!origin && !program) throw new TypeError('A FLUJO origin or program is required.'); + Object.assign(this, { program, token, stateDirectory, privateDirectory, modelRecord, team, servers, partialMessages, onStderr, clientFactory, knownVersion: version, pollMs, interruptResendMs, submitTimeoutMs, toolsTimeoutMs, reconnectMs, drainMs }); + this.origin = (program?.origin ?? origin).replace(/\/$/, ''); + this.threads = new Map(); + mkdirSync(stateDirectory, { recursive: true, mode: 0o700 }); + mkdirSync(privateDirectory, { recursive: true, mode: 0o700 }); + } + + client(workspace) { return this.clientFactory({ origin: this.origin, token: this.token, workspace }); } + headers() { return this.token ? { Authorization: `Bearer ${this.token}` } : {}; } + notify(method, params) { this.emit('notification', { method, params }); } + stateFile(id) { return join(this.stateDirectory, `${id}.flujo.json`); } + + async initialize() { + if (!this.program) return; + this.program.on('exit', result => this.emit('exit', result)); + this.program.on('restarting', result => this.onStderr?.(`FLUJO stopped unexpectedly (${result.code ?? result.signal}); restarting it.`)); + await this.program.start(); + // This instance is ours alone: nothing runs in its default workspace. + const home = this.client(null); + for (const server of await home.servers().catch(() => [])) { + if (!server.disabled) await home.api('PUT', `/api/mcp/servers/${encodeURIComponent(server.name)}`, { ...server, disabled: true }).catch(() => {}); + } + } + + async version() { + if (this.knownVersion) return this.knownVersion; + const read = await this.client(null).api('GET', '/api/update', undefined, { timeoutMs: 15000 }).catch(() => null); + return this.knownVersion = read?.body?.currentVersion ?? 'unknown'; + } + + async request(method, params = {}) { + if (method === 'thread/start') return this.startThread(params); + if (method === 'thread/resume') return this.resumeThread(params); + if (method === 'thread/read') return this.readThread(params); + if (method === 'turn/start') return this.startTurn(params); + if (method === 'turn/steer') return this.steerTurn(params); + if (method === 'turn/interrupt') return this.interruptTurn(params); + throw invalid(`Unsupported method ${method}.`); + } + + /** The board server reads its credential from a private file, so no secret enters FLUJO's server settings. */ + boardConfig(thread, server) { + const settings = { name: 'factory', transport: 'stdio', command: server.command, args: server.args ?? [], env: server.env ?? {}, cwd: thread.cwd, rootPath: thread.cwd, disabled: false, exposeAsMcpServer: false, enableMcpApps: false }; + if (!server.env?.FACTORY_BOARD_TOKEN) return settings; + const file = join(this.privateDirectory, `${thread.id}.board.json`); + writeFileSync(file, JSON.stringify({ url: server.env.FACTORY_BOARD_URL, token: server.env.FACTORY_BOARD_TOKEN }), { mode: 0o600 }); + const env = Object.fromEntries(Object.entries(server.env).filter(([name]) => !['FACTORY_BOARD_URL', 'FACTORY_BOARD_TOKEN'].includes(name))); + return { ...settings, args: [...settings.args, '--config', file], env }; + } + + async inventory(client, names) { + const tools = {}, deadline = Date.now() + this.toolsTimeoutMs; + for (const name of names) { + for (;;) { + const observed = await client.serverTools(name).catch(error => ({ tools: [], error: error.message })); + if (!observed.error && observed.tools.length) { tools[name] = observed.tools.map(tool => tool.name); break; } + // A worker never runs without its board or its tools: the same promise Codex gets from a required server. + if (Date.now() >= deadline) throw new Error(`The ${name} server did not connect in FLUJO.`); + await sleep(1000); + } + } + return tools; + } + + /** Idempotent, so a restarted Machine repairs its workspace before it resumes the conversation. */ + async install(thread, { server }) { + const client = this.client(thread.workspace); + await client.ensureWorkspace(); + await client.setWorkspaceRoots(thread.roots); + const wanted = new Set([...this.servers, ...(server ? ['factory'] : [])]); + for (const existing of await client.servers()) { + if (Boolean(existing.disabled) === wanted.has(existing.name)) { + await client.expect('PUT', `/api/mcp/servers/${encodeURIComponent(existing.name)}`, { ...existing, disabled: !wanted.has(existing.name) }); + } + } + const model = this.modelRecord(thread.model); + await client.upsertModel(model); + if (server) await client.upsertServer(this.boardConfig(thread, server)); + const availableServers = [...wanted]; + const availableTools = await this.inventory(client, availableServers); + const concurrency = this.team.concurrency; + const specs = buildFactoryTeamSpecs({ model: model.id, availableServers, availableTools, specialists: this.team.specialists, + // FLUJO's shell and file tools start in its own data folder unless told otherwise. + instructions: `Your working folder is ${thread.cwd}. Pass it as cwd on every shell command and use absolute paths under it for file tools.${thread.instructions ? `\n${thread.instructions}` : ''}`, + limits: { ...(this.team.agentTurns ? { agentTurns: this.team.agentTurns } : {}), ...(this.team.leadTurns ? { leadTurns: this.team.leadTurns } : {}), ...(concurrency ? { concurrency } : {}) }, + environment: { workingFolder: thread.cwd } }).filter(spec => concurrency || spec.name === FACTORY_FLOW_NAMES.agent); + const digest = createHash('sha256').update(JSON.stringify(specs)).digest('hex'); + // Saving a flow gives it a new identity, so an unchanged flow is left alone for the conversation that uses it. + const installed = new Set((await client.flows()).map(flow => flow.name)); + if (thread.flowDigest !== digest || specs.some(spec => !installed.has(spec.name))) { + for (const spec of specs) { + const flow = await client.saveFlowSpec(spec); + if (flow?.name !== spec.name) throw new Error('FLUJO did not confirm the installed flow.'); + } + thread.flowDigest = digest; + } + } + + save(thread) { + const file = this.stateFile(thread.id), next = `${file}.next`; + writeFileSync(next, JSON.stringify({ threadId: thread.id, workspace: thread.workspace, lastTurn: thread.lastTurn, nextSeq: thread.nextSeq, flowDigest: thread.flowDigest }), { mode: 0o600 }); + renameSync(next, file); + } + + addThread(id, { cwd, config, sandbox, model, developerInstructions, agentRole }, saved = {}) { + if ((sandbox ?? 'danger-full-access') !== 'danger-full-access') throw invalid('FLUJO workers run with full access on their Machine.'); + const thread = { id, workspace: saved.workspace ?? `factory-${id}`, cwd, model, instructions: developerInstructions, roots: [cwd, ...(config?.['sandbox_workspace_write.writable_roots'] ?? []), ...(this.team.roots ?? [])], + flow: agentRole === 'orchestrator' && this.team.concurrency ? FACTORY_FLOW_NAMES.team : FACTORY_FLOW_NAMES.agent, + lastTurn: saved.lastTurn ?? null, nextSeq: saved.nextSeq ?? 0, flowDigest: saved.flowDigest, turn: null, totals: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0 } }; + this.threads.set(id, thread); + return thread; + } + + thread(id) { + const thread = this.threads.get(id); + if (!thread) throw invalid(`Unknown thread ${id}.`); + return thread; + } + + async startThread(params) { + const thread = this.addThread(randomUUID(), params); + await this.install(thread, { server: params.config?.mcp_servers?.factory }); + this.save(thread); + this.notify('thread/started', { threadId: thread.id, thread: { id: thread.id } }); + return { thread: { id: thread.id, status: { type: 'idle' } } }; + } + + async resumeThread(params) { + const id = params.threadId; + const unavailable = () => invalid(`no rollout found for thread id ${id}`); + if (!existsSync(this.stateFile(id))) throw unavailable(); + const saved = JSON.parse(readFileSync(this.stateFile(id), 'utf8')); + const thread = this.addThread(id, params, saved); + await this.install(thread, { server: params.config?.mcp_servers?.factory }); + // A conversation without a turn has no history to lose; one with a turn must still have its history. + let status = 'idle', recoveredTurn; + if (thread.lastTurn) { + const read = await this.client(thread.workspace).conversation(id, { messageLimit: 1 }); + if (read.status === 404) throw unavailable(); + if (read.status !== 200) throw new Error(`FLUJO could not read the retained conversation (HTTP ${read.status}).`); + if (thread.lastTurn.status === 'inProgress') { + if (ACTIVE_STATUSES.has(read.body.status)) { + // Still running in FLUJO: observe it, never replay it. + status = 'active'; + this.adoptTurn(thread, thread.lastTurn.id); + } else recoveredTurn = thread.lastTurn = { ...thread.lastTurn, status: this.outcome(read.body).status === 'completed' ? 'completed' : 'interrupted' }; + } + } + this.save(thread); + this.notify('thread/resumed', { threadId: id, thread: { id } }); + return { thread: { id, status: { type: status } }, ...(recoveredTurn ? { recoveredTurn } : {}) }; + } + + /** A real readback: the thread is idle only when FLUJO agrees that the conversation and its children have ended. */ + async readThread({ threadId, includeTurns }) { + const thread = this.thread(threadId); + let active = Boolean(thread.turn); + if (!active && thread.lastTurn) { + const read = await this.client(thread.workspace).conversation(threadId, { messageLimit: 1 }); + // A turn FLUJO never took has no conversation, so nothing can be running. + if (read.status !== 200 && read.status !== 404) throw new Error(`FLUJO could not read the conversation (HTTP ${read.status}).`); + active = read.status === 200 && (ACTIVE_STATUSES.has(read.body.status) || await this.childrenActive(thread)); + } + return { thread: { id: thread.id, status: { type: active ? 'active' : 'idle' }, ...(includeTurns ? { turns: thread.lastTurn ? [thread.lastTurn] : [] } : {}) } }; + } + + /** Children keep working after a parent's run ends, and a stop must wait for them too. */ + async childrenActive(thread) { + const client = this.client(thread.workspace); + for (let cursor, page = 0; page < 100; page++) { + const read = await client.descendants(thread.id, cursor); + if (read.status !== 200 || !Array.isArray(read.body?.items)) throw new Error(`FLUJO could not list the conversation's children (HTTP ${read.status}).`); + if (read.body.items.some(item => ACTIVE_STATUSES.has(item.status))) return true; + if (!read.body.hasMore) return false; + cursor = read.body.nextCursor; + if (!cursor) return true; + } + return true; + } + + outcome(conversation) { + const classification = conversation.recovery?.classification; + if (classification === 'cancelled' || conversation.lastError?.code === 'cancelled') return { status: 'interrupted' }; + if (conversation.status === 'completed' || conversation.status === 'capped') return { status: 'completed' }; + const code = conversation.lastError?.httpStatus; + return { status: 'failed', ...(Number.isSafeInteger(code) ? { code } : {}) }; + } + + startTurn({ threadId, input }) { + const thread = this.thread(threadId); + if (thread.turn) throw invalid('This thread already has a running turn.'); + const id = randomUUID(); + const turn = this.beginTurn(thread, id); + thread.lastTurn = { id, status: 'inProgress' }; + this.save(thread); + this.notify('thread/status/changed', { threadId, status: { type: 'active' } }); + this.notify('turn/started', { threadId, turn: { id, status: 'inProgress' } }); + turn.submission = this.submit(thread, turn, textOf(input)).catch(error => { this.onStderr?.(`FLUJO did not take the turn: ${error.message}`); this.finish(thread, turn, { status: 'failed' }); }); + this.watch(thread, turn); + return { turn: { id, status: 'inProgress' } }; + } + + beginTurn(thread, id) { + const turn = { id, steers: new Map(), runSeen: false, finished: false, abort: new AbortController() }; + thread.turn = turn; + return turn; + } + + /** A run found still going after a restart. Its turn keeps its identity. */ + adoptTurn(thread, id) { + const turn = this.beginTurn(thread, id); + turn.runSeen = true; + this.watch(thread, turn); + } + + async submit(thread, turn, text) { + const url = `${this.origin}/v1/chat/completions?workspace=${encodeURIComponent(thread.workspace)}`; + const body = { model: `flow-${thread.flow}`, stream: true, metadata: { flujo: 'true', requireApproval: 'false', conversationId: thread.id }, messages: [{ role: 'user', content: text }] }; + const deadline = Date.now() + this.submitTimeoutMs; + // The run this turn is waiting for is whichever one is not already recorded. + const before = await this.client(thread.workspace).conversation(thread.id, { messageLimit: 1 }); + if (before.status !== 200 && before.status !== 404) throw new Error(`FLUJO could not read the conversation (HTTP ${before.status}).`); + turn.baselineRun = before.body?.recovery?.runId ?? null; + turn.runSeen = false; + for (let attempt = 0; ; attempt++) { + let response; + try { response = await submitDetached(url, { headers: { ...this.headers(), 'x-flujo-workspace': thread.workspace }, body }); } + catch (error) { + // The request may have arrived. Read it back rather than send the task a second time. + this.onStderr?.(`FLUJO submission outcome is being read back: ${error.message}`); + break; + } + if (response.status === 200) break; + if (turn.interrupting) { this.finish(thread, turn, { status: 'interrupted' }); return; } + if (!RETRYABLE_SUBMISSION.has(response.status) || Date.now() >= deadline || turn.finished) throw new Error(`HTTP ${response.status}: ${response.text.slice(0, 300)}`); + await sleep(Math.min(30000, 1000 * 2 ** attempt)); + } + turn.submittedAt = Date.now(); + } + + /** Follows the event stream and also reads the state on a timer, so a lost stream cannot leave a turn open. */ + watch(thread, turn) { + const settle = () => { void this.settle(thread, turn).catch(error => this.onStderr?.(`FLUJO state read is pending: ${error.message}`)); }; + turn.settle = settle; + turn.pollTimer = setInterval(settle, this.pollMs); + void (async () => { + while (!turn.finished && !this.closed) { + await followEvents(`${this.origin}/v1/chat/conversations/${encodeURIComponent(thread.id)}/events?workspace=${encodeURIComponent(thread.workspace)}&fromSeq=${thread.nextSeq}`, { + headers: { ...this.headers(), 'x-flujo-workspace': thread.workspace }, signal: turn.abort.signal, + onEvent: event => this.receive(thread, turn, event), + // FLUJO skipped ahead, for example past an event too large to send. + onControl: control => { if (Number.isSafeInteger(control?.nextSeq)) thread.nextSeq = Math.max(thread.nextSeq, control.nextSeq); }, + }); + if (turn.finished || this.closed) return; + settle(); + await sleep(this.reconnectMs); + } + })(); + } + + receive(thread, turn, event) { + if (Number.isSafeInteger(event.seq)) { + if (event.seq < thread.nextSeq) return; + thread.nextSeq = event.seq + 1; + } + const threadId = thread.id, turnId = turn.id; + const nesting = { ...(event.depth ? { depth: event.depth } : {}), ...(Number.isSafeInteger(event.laneIndex) ? { lane: event.laneIndex } : {}) }; + const item = body => this.notify('item/completed', { threadId, turnId, item: { ...body, ...nesting } }); + switch (event.type) { + case 'run:start': if (!event.depth) turn.runSeen = true; break; + case 'run:done': if (!event.depth) { turn.doneSeen = true; turn.settle?.(); } break; + case 'message': { + const { role, id, content, tool_calls: calls } = event.message ?? {}; + if (role === 'user') { turn.steers.delete(id); break; } + if (role !== 'assistant') break; + if (typeof content === 'string' && content) item({ type: 'agentMessage', text: content }); + for (const call of calls ?? []) item({ type: 'toolCall', id: call.id, tool: call.function?.name, arguments: call.function?.arguments }); + break; + } + case 'tool:result': item({ type: 'toolResult', id: event.toolCallId, tool: event.name, isError: event.isError === true, content: event.result }); break; + case 'usage': { + const last = { inputTokens: event.promptTokens ?? 0, cachedInputTokens: event.cacheReadTokens ?? 0, outputTokens: event.completionTokens ?? 0 }; + for (const name of Object.keys(last)) thread.totals[name] += last[name]; + this.notify('thread/tokenUsage/updated', { threadId, turnId, tokenUsage: { last, total: { ...thread.totals } } }); + break; + } + case 'model:delta': if (this.partialMessages) this.notify('item/stream/delta', { threadId, turnId, delta: event.delta, ...nesting }); break; + case 'recovery:retry': this.notify('model/retry', { threadId, turnId, attempt: event.attempt, retryAt: event.retryAt, error: event.failure?.category }); break; + case 'subflow:start': case 'subflow:done': item({ type: 'subflow', phase: event.type.slice(8), subflow: event.subflowId, conversationId: event.laneConversationId, status: event.status }); break; + case 'error': this.notify('flujo/error', { threadId, turnId, message: event.message, code: event.error?.code, ...nesting }); break; + // Nobody answers on an unattended worker, so a question or form never parks the run. + case 'run:awaiting_question': void this.respond(thread, { action: 'question-decline', questionId: event.questionId }); break; + case 'run:awaiting_elicitation': void this.respond(thread, { action: 'elicitation-cancel', elicitationId: event.elicitationId }); break; + default: break; + } + // Streaming tokens are most events and change nothing a restart needs; it just reads them again. + if (event.type !== 'model:delta' && event.type !== 'tool:progress') this.save(thread); + } + + async respond(thread, body) { + await this.client(thread.workspace).api('POST', `/v1/chat/conversations/${encodeURIComponent(thread.id)}/respond`, body).catch(error => this.onStderr?.(`FLUJO did not take the answer: ${error.message}`)); + } + + /** Ends the turn only after FLUJO itself says this run has ended, along with any children. */ + async settle(thread, turn) { + if (turn.finished || turn.settling) return; + turn.settling = true; + try { + await turn.submission; + if (turn.finished) return; + const read = await this.client(thread.workspace).conversation(thread.id, { messageLimit: turn.steers.size ? 200 : 1 }); + const conversation = read.status === 200 ? read.body : null; + if (read.status !== 200 && read.status !== 404) return; + const runId = conversation?.recovery?.runId; + if (runId && runId !== turn.baselineRun) turn.runSeen = true; + if (!turn.runSeen) { + // The previous run's end is not this turn's end. A run that never starts is not sent again; the turn ends unfinished. + if (Date.now() - (turn.submittedAt ?? Date.now()) >= this.submitTimeoutMs) this.finish(thread, turn, { status: 'failed' }); + return; + } + if (!conversation || ACTIVE_STATUSES.has(conversation.status)) return; + if (await this.childrenActive(thread)) return; + if (turn.finished) return; + // The state can be ahead of the stream. Wait for the run's last event so the trace is complete; a lost stream only delays this. + if (!turn.doneSeen) { + turn.terminalSince ??= Date.now(); + if (Date.now() - turn.terminalSince < this.drainMs) { setTimeout(() => turn.settle?.(), Math.min(this.drainMs, 200)); return; } + } + if (turn.steers.size && !turn.interrupting) { + const seen = new Set((conversation.messages ?? []).map(message => message.id)); + for (const id of [...turn.steers.keys()]) if (seen.has(id)) turn.steers.delete(id); + if (turn.steers.size) { + // A message that arrived as the run ended is still owed: this turn goes on with it. + const text = [...turn.steers.values()].join('\n\n'); + turn.steers.clear(); turn.doneSeen = false; turn.terminalSince = undefined; + turn.submission = this.submit(thread, turn, text).catch(error => { this.onStderr?.(`FLUJO did not take the queued message: ${error.message}`); this.finish(thread, turn, { status: 'failed' }); }); + return; + } + } + this.finish(thread, turn, this.outcome(conversation)); + } finally { turn.settling = false; } + } + + async steerTurn({ threadId, expectedTurnId, input }) { + const thread = this.thread(threadId), turn = thread.turn; + if (!turn || turn.interrupting) throw new Error('no active turn to steer'); + if (expectedTurnId && expectedTurnId !== turn.id) throw invalid('The expected turn is not the running turn.'); + await turn.submission; + const id = randomUUID(), text = textOf(input); + const response = await this.client(thread.workspace).api('POST', `/v1/chat/conversations/${encodeURIComponent(threadId)}/inject`, { content: text, id }); + if (response.status === 409) throw new Error('no active turn to steer'); + if (response.status !== 200 || response.body?.message_id !== id) throw new Error(`FLUJO did not confirm the queued message (HTTP ${response.status}).`); + if (thread.turn === turn) turn.steers.set(id, text); + return { turnId: turn.id }; + } + + async sendCancel(thread) { + const response = await this.client(thread.workspace).cancel(thread.id, 15000); + if (![200, 202].includes(response.status)) throw new Error(`FLUJO did not accept the stop (HTTP ${response.status}).`); + } + + async interruptTurn({ threadId, turnId }) { + const thread = this.thread(threadId), turn = thread.turn; + if (!turn || turn.id !== turnId) { + if (thread.lastTurn?.id === turnId) return {}; + throw invalid('The expected turn is not the running turn.'); + } + if (turn.interrupting) return {}; + turn.interrupting = true; + await turn.submission; + if (turn.finished) return {}; + try { await this.sendCancel(thread); } + catch (error) { turn.interrupting = false; throw error; } + // A cancel is a request; keep asking until FLUJO has actually stopped, children included. + turn.interruptTimer = setInterval(() => { void this.sendCancel(thread).catch(error => this.onStderr?.(`FLUJO stop is pending: ${error.message}`)); }, this.interruptResendMs); + turn.settle?.(); + return {}; + } + + finish(thread, turn, { status, code }) { + if (turn.finished) return; + turn.finished = true; + clearInterval(turn.pollTimer); clearInterval(turn.interruptTimer); + turn.abort.abort(); + thread.turn = null; thread.lastTurn = { id: turn.id, status }; + this.save(thread); + this.notify('turn/completed', { threadId: thread.id, turn: { id: turn.id, status, ...(status === 'completed' ? {} : { error: code === undefined ? {} : { code } }) } }); + this.notify('thread/status/changed', { threadId: thread.id, status: { type: 'idle' } }); + } + + close() { + this.closed = true; + for (const thread of this.threads.values()) { + if (!thread.turn) continue; + clearInterval(thread.turn.pollTimer); clearInterval(thread.turn.interruptTimer); + thread.turn.abort.abort(); + } + if (this.program) void this.program.stop(); + else this.emit('exit', { code: 0, signal: null }); + } +} + +const textOf = input => (input ?? []).filter(item => item.type === 'text').map(item => item.text).join('\n\n'); diff --git a/src/cloud-lite/protocol.mjs b/src/cloud-lite/protocol.mjs index b222fcc..08ee311 100644 --- a/src/cloud-lite/protocol.mjs +++ b/src/cloud-lite/protocol.mjs @@ -1,5 +1,5 @@ import { createHash, randomBytes } from 'node:crypto'; -import { validateModelProvider, providerApis, WORKER_PROGRAM_APIS } from '../model-provider.mjs'; +import { validateModelProvider, requireFleetApi, WORKER_PROGRAM_APIS } from '../model-provider.mjs'; export { validateModelProvider }; @@ -8,7 +8,7 @@ export const hash = value => createHash('sha256').update(typeof value === 'strin export function validateCloudLitePlan(input) { if (!input || typeof input !== 'object') throw new TypeError('A cloud lite-worker plan is required.'); const plan = { cpuKind: 'shared', cpus: 2, memoryMb: 4096, diskGb: 20, machines: 1, - agentsPerMachine: 10, region: 'iad', ...(input.workerProgram === 'claude' ? { model: 'opus', subagentModel: 'sonnet' } : { model: 'gpt-6.1-sol', subagentModel: 'gpt-6-luna' }), reasoningEffort: 'medium', ...input }; + agentsPerMachine: 10, region: 'iad', ...(input.workerProgram === 'claude' ? { model: 'opus', subagentModel: 'sonnet' } : input.workerProgram === 'flujo' ? {} : { model: 'gpt-6.1-sol', subagentModel: 'gpt-6-luna' }), reasoningEffort: 'medium', ...input }; if (plan.workerProgram !== undefined && !Object.hasOwn(WORKER_PROGRAM_APIS, plan.workerProgram)) throw new TypeError(`workerProgram must be one of ${Object.keys(WORKER_PROGRAM_APIS).join(', ')}.`); for (const name of ['runId', 'app']) if (!/^[a-z][a-z0-9-]{2,62}$/.test(plan[name] ?? '')) throw new TypeError(`Invalid ${name}.`); if (!['shared', 'performance'].includes(plan.cpuKind)) throw new TypeError('Invalid cpuKind.'); @@ -17,6 +17,7 @@ export function validateCloudLitePlan(input) { } if (plan.memoryMb % 256) throw new TypeError('memoryMb must be a multiple of 256.'); if (!/^[a-z]{3}$/.test(plan.region)) throw new TypeError('Invalid region.'); + if (plan.network !== undefined && !/^[a-z][a-z0-9-]{2,62}$/.test(plan.network)) throw new TypeError('Invalid network.'); if (plan.repository !== undefined && !/^[\w.-]+\/[\w.-]+$/.test(plan.repository)) throw new TypeError('Optional repository must be a GitHub owner/repository.'); if (typeof plan.goal !== 'string' || !plan.goal.trim()) throw new TypeError('A goal is required.'); if (plan.workflow !== undefined && plan.workflow !== 'verified-v1') throw new TypeError('Unsupported workflow profile.'); @@ -33,9 +34,13 @@ export function validateCloudLitePlan(input) { if (plan.modelProvider !== undefined) { validateModelProvider(plan.modelProvider); // Codex talks to custom providers through the Responses API, Claude Code through the Messages API. - const needed = WORKER_PROGRAM_APIS[plan.workerProgram ?? 'codex']; - if (!providerApis(plan.modelProvider).includes(needed)) throw new TypeError(`Cloud lite workers need a provider that offers the ${needed} API.`); - if (plan.workerProgram === 'claude' && !plan.modelProvider.model && !input.model) throw new TypeError('Select the served model name.'); + requireFleetApi(plan.modelProvider, plan.workerProgram ?? 'codex'); + if (['claude', 'flujo'].includes(plan.workerProgram) && !plan.modelProvider.model && !input.model) throw new TypeError('Select the served model name.'); + } + if (plan.flujo !== undefined && plan.workerProgram !== 'flujo') throw new TypeError('The flujo settings belong to FLUJO worker plans.'); + if (plan.workerProgram === 'flujo') { + validateFlujoSettings(plan); + if (plan.model && !plan.subagentModel) plan.subagentModel = plan.model; } if (plan.supervision !== undefined && !['autonomous', 'operator'].includes(plan.supervision)) throw new TypeError('supervision must be autonomous or operator.'); if (plan.sandbox !== undefined && !CODEX_SANDBOXES.includes(plan.sandbox)) throw new TypeError(`sandbox must be one of ${CODEX_SANDBOXES.join(', ')}.`); @@ -53,6 +58,19 @@ export function validateCloudLitePlan(input) { return plan; } export const CODEX_SANDBOXES = ['danger-full-access', 'workspace-write', 'read-only']; +/** The bundled tool servers a FLUJO worker may use. The FLUJO control server stays off: it can install and edit flows. */ +export const FLUJO_SERVERS = ['bash', 'filesystem', 'browser']; +function validateFlujoSettings(plan) { + // FLUJO has no subscription login to broker; its models are stored records with their own key. + if (!plan.modelProvider) throw new TypeError('FLUJO workers need a modelProvider.'); + if (plan.sandbox !== undefined && plan.sandbox !== 'danger-full-access') throw new TypeError('FLUJO workers run with full access on their Machine; choose danger-full-access.'); + const { teamConcurrency, agentTurns, leadTurns, servers } = plan.flujo ?? {}; + if (plan.flujo !== undefined && (!plan.flujo || typeof plan.flujo !== 'object' || Array.isArray(plan.flujo))) throw new TypeError('flujo must be an object.'); + if (teamConcurrency !== undefined && (!Number.isSafeInteger(teamConcurrency) || teamConcurrency < 1 || teamConcurrency > 9)) throw new TypeError('flujo.teamConcurrency must be from 1 to 9.'); + if (agentTurns !== undefined && (!Number.isSafeInteger(agentTurns) || agentTurns < 1 || agentTurns > 200)) throw new TypeError('flujo.agentTurns must be from 1 to 200.'); + if (leadTurns !== undefined && (!Number.isSafeInteger(leadTurns) || leadTurns < 1 || leadTurns > 600)) throw new TypeError('flujo.leadTurns must be from 1 to 600.'); + if (servers !== undefined && (!Array.isArray(servers) || !servers.length || new Set(servers).size !== servers.length || servers.some(name => !FLUJO_SERVERS.includes(name)))) throw new TypeError(`flujo.servers must list ${FLUJO_SERVERS.join(', ')}.`); +} export function agentInventory(plan) { return Array.from({ length: plan.machines }, (_, machine) => Array.from({ length: plan.agentsPerMachine }, (_, agent) => ({ id: `m${machine}-a${agent}`, machine: `m${machine}`, role: agent === 0 ? 'orchestrator' : 'subagent', @@ -75,7 +93,7 @@ export function machineConfig(plan, { image, volumeId, index, environment }) { image, env: environment, guest: { cpu_kind: plan.cpuKind, cpus: plan.cpus, memory_mb: plan.memoryMb }, mounts: [{ volume: volumeId, path: '/data' }], - metadata: { 'factory.run': plan.runId, 'factory.machine': `m${index}`, 'factory.runtime': plan.workerProgram === 'claude' ? 'claude-lite' : 'codex-lite' }, + metadata: { 'factory.run': plan.runId, 'factory.machine': `m${index}`, 'factory.runtime': `${plan.workerProgram ?? 'codex'}-lite` }, restart: { policy: 'no' }, // Same private topology as flujo-cloud: host connects using its owned proxy; // remote Codex sessions use the coordinator's exact Fly private Machine DNS. diff --git a/src/cloud-lite/runtime.mjs b/src/cloud-lite/runtime.mjs index 7c9e0f3..eeca7b9 100644 --- a/src/cloud-lite/runtime.mjs +++ b/src/cloud-lite/runtime.mjs @@ -1,4 +1,4 @@ -import { mkdirSync, readFileSync, writeFileSync, existsSync, copyFileSync, constants } from 'node:fs'; +import { mkdirSync, readFileSync, writeFileSync, existsSync, copyFileSync, constants, createWriteStream, renameSync } from 'node:fs'; import { open } from 'node:fs/promises'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -7,12 +7,13 @@ import { promisify } from 'node:util'; import { freemem, totalmem, loadavg } from 'node:os'; import { CodexRpc } from './rpc.mjs'; import { ClaudeSessionDriver, claudeCommandWrapper, claudeEnvironment } from './claude-driver.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_PORT, FLUJO_WORKER_SERVERS, flujoEnvironment } from './flujo-driver.mjs'; import { startCloudLiteCoordinator } from './server.mjs'; import { CloudLiteTrace } from './trace.mjs'; import { NativeOperatorStop } from './operator-stop.mjs'; import { NativeWorkflowWake } from './workflow-wake.mjs'; import { hash } from './protocol.mjs'; -import { codexProviderConfig } from '../model-provider.mjs'; +import { codexProviderConfig, flujoProviderModel } from '../model-provider.mjs'; const threadStatuses = new Set(['active','idle','archived','notLoaded','systemError','notFound']); const turnStatuses = new Set(['inProgress','completed','interrupted','failed']); @@ -40,13 +41,69 @@ export async function finalizeNativeStopCommand({ prove, publish, flush, acknowl markStopped(); return true; } +/** + * What differs between worker programs. Everything else (the board, operator commands, stop receipts, + * traces, evidence and workflows) is shared code that talks to a driver through one contract. + */ +const WORKER_PROGRAMS = { + codex: { + home: '/data/codex', sessions: '/data/codex/sessions', exitTrace: 'codex.exit', + prepare: ({ plan, index, home, providerMode }) => writeFileSync(join(home, 'config.toml'), providerMode ? providerConfigToml(plan, index) + : 'cli_auth_credentials_store="ephemeral"\nforced_login_method="chatgpt"\n', { mode: 0o600 }), + driver: ({ plan, home, githubToken, deployKey, providerKey, providerMode, repositoryDirectory, onStderr, refreshAuth }) => new CodexRpc({ + env: codexEnvironment({ plan, home, githubToken, deployKey, providerKey }), cwd: repositoryDirectory, onStderr, + onRequest: async (method, params) => { + if (method === 'account/chatgptAuthTokens/refresh' && !providerMode) return refreshAuth(); + throw new Error(`Unexpected Codex host request: ${method}`); + } }), + login: true, + version: ({ run }) => run('codex', ['--version']), + }, + claude: { + home: '/data/claude', sessions: '/data/claude/projects', exitTrace: 'codex.exit', resumeNeedsSettings: true, + prepare: ({ providerMode }) => writeFileSync(CLAUDE_COMMAND_WRAPPER, claudeCommandWrapper({ provider: providerMode }), { mode: 0o700 }), + driver: ({ plan, index, home, githubToken, deployKey, providerKey, onStderr, getLiveAuth }) => new ClaudeSessionDriver({ configDirectory: home, stateDirectory: '/data/factory', fixedModel: plan.modelProvider?.model, partialMessages: plan.traceDeltas === true, onStderr, + // Read for every turn, so a refreshed subscription token reaches the next process. + environment: () => claudeEnvironment({ plan, index, home, commandWrapper: CLAUDE_COMMAND_WRAPPER, githubToken, deployKey, providerKey, oauthToken: getLiveAuth()?.accessToken }) }), + version: ({ run }) => run('claude', ['--version']), + }, + flujo: { + home: '/data/flujo', sessions: '/data/flujo/workspaces', exitTrace: 'flujo.exit', resumeNeedsSettings: true, + // Conversations are the evidence. Keys, copied tool servers and browser data stay out of it. + sessionsExclude: ['encryption_key.json', 'models.json', 'mcp_servers.json', 'mcp-servers', 'browser-profile', 'bash-utils', 'screenshots', 'recordings', 'snapshots', '.runtime-locks', 'node_modules'], + sessionFields: agent => ({ agentRole: agent.role }), + prepare: () => { mkdirSync('/data/results', { recursive: true, mode: 0o700 }); }, + driver: ({ plan, index, home, providerKey, onStderr }) => { + const port = Number(process.env.FACTORY_FLUJO_PORT ?? FLUJO_WORKER_PORT); + const extra = process.env.NODE_EXTRA_CA_CERTS ? { NODE_EXTRA_CA_CERTS: process.env.NODE_EXTRA_CA_CERTS } : {}; + const { teamConcurrency: concurrency, agentTurns, leadTurns, specialists, servers = FLUJO_WORKER_SERVERS } = plan.flujo ?? {}; + return new FlujoSessionDriver({ program: new FlujoProgram({ port, environment: () => flujoEnvironment({ home, port, extra }), onOutput: rotatingLog('/data/factory/flujo.log') }), + stateDirectory: '/data/factory', privateDirectory: '/data/factory-private', servers, partialMessages: plan.traceDeltas === true, onStderr, + team: { concurrency, agentTurns, leadTurns, specialists, roots: ['/data/results'] }, + modelRecord: model => flujoProviderModel(plan.modelProvider, { index, key: providerKey, model }) }); + }, + version: ({ rpc }) => rpc.version(), + }, +}; +const CLAUDE_COMMAND_WRAPPER = '/data/factory/claude-command-env.sh'; + +/** FLUJO logs freely, so its log keeps only the newest two parts. */ +function rotatingLog(file, partBytes = 32 * 1024 * 1024) { + let stream = createWriteStream(file, { flags: 'a', mode: 0o600 }), bytes = 0; + return text => { + if (bytes >= partBytes) { stream.end(); renameSync(file, `${file}.1`); stream = createWriteStream(file, { flags: 'a', mode: 0o600 }); bytes = 0; } + bytes += Buffer.byteLength(text); stream.write(text); + }; +} + export async function runCloudLiteMachine(config) { const { plan, index, credentials, operatorToken, initialAuth = {}, githubToken, evidenceTokens, providerKey, deployKey } = config; // Optional: a self-hosted model provider instead of the subscription login, read-only // deploy-key repository access, a prepare hook for agent workspaces, operator-driven turns. const providerMode = Boolean(plan.modelProvider); - const claudeProgram = plan.workerProgram === 'claude'; - const evidenceSessionsPath = claudeProgram ? '/data/claude/projects' : '/data/codex/sessions'; + const programName = plan.workerProgram ?? 'codex'; + const program = WORKER_PROGRAMS[programName]; + const evidenceRoots = { sessionsPath: program.sessions, ...(program.sessionsExclude ? { sessionsExclude: program.sessionsExclude } : {}) }; const operatorSupervised = plan.supervision === 'operator'; const extraSecrets = [providerKey, deployKey].filter(value => typeof value === 'string' && value.length >= 8); mkdirSync('/data/factory', { recursive: true, mode: 0o700 }); @@ -60,7 +117,7 @@ export async function runCloudLiteMachine(config) { if(!retainedEvidence.receipt?.checkpoint.final){ const child=fork(fileURLToPath(new URL('./evidence-worker.mjs',import.meta.url)),[],{stdio:['ignore','ignore','ignore','ipc'],windowsHide:true}); child.on('error',()=>{}); - child.send({action:'start',runId:plan.runId,machineId:`m${index}`,url:plan.collection.url,token:evidenceTokens?.[`m${index}`],intervalMs:plan.collection.intervalMs??60000,sessions:sealedCheckpoint.sessions,sessionsPath:evidenceSessionsPath}); + child.send({action:'start',runId:plan.runId,machineId:`m${index}`,url:plan.collection.url,token:evidenceTokens?.[`m${index}`],intervalMs:plan.collection.intervalMs??60000,sessions:sealedCheckpoint.sessions,...evidenceRoots}); child.send({action:'finalize',sessions:sealedCheckpoint.sessions}); } return {sealed:true,stop:async()=>coordinator?.close()}; @@ -92,7 +149,7 @@ export async function runCloudLiteMachine(config) { return{agentId:agent.id,threadId:receipt.threadId,stopId:receipt.commandId}; }); const child=fork(fileURLToPath(new URL('./evidence-worker.mjs',import.meta.url)),[],{stdio:['ignore','ignore','ignore','ipc'],windowsHide:true});child.on('error',()=>{}); - child.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions,sessionsPath:evidenceSessionsPath}); + child.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions,...evidenceRoots}); child.send({action:'finalize',sessions});return{sealed:true,stop:async()=>coordinator?.close()}; } if(plan.shutdown && Date.now()>=Date.parse(plan.shutdown.deadlineUtc))throw new Error('Work deadline has passed; preserve existing evidence and reconcile shutdown without restarting agents.'); @@ -136,10 +193,9 @@ export async function runCloudLiteMachine(config) { prepared = await runPrepareHook({ plan, index, repositoryDirectory, env: gitEnv }); writeFileSync(preparedFile, JSON.stringify(prepared), { mode: 0o600 }); } - const home = claudeProgram ? '/data/claude' : '/data/codex'; mkdirSync(home, { recursive: true, mode: 0o700 }); + const home = program.home; mkdirSync(home, { recursive: true, mode: 0o700 }); // A fresh native harness has only the explicitly configured FACTORY MCP server. - if (!claudeProgram) writeFileSync(join(home, 'config.toml'), providerMode ? providerConfigToml(plan, index) - : 'cli_auth_credentials_store="ephemeral"\nforced_login_method="chatgpt"\n', { mode: 0o600 }); + program.prepare({ plan, index, home, providerMode }); const traces = new Map(); const workflowWakes = new Map(); for (const agent of bootstrap.agents) { @@ -155,20 +211,9 @@ export async function runCloudLiteMachine(config) { liveAuth = auth; return auth; }; - const commandWrapper = '/data/factory/claude-command-env.sh'; - if (claudeProgram) writeFileSync(commandWrapper, claudeCommandWrapper({ provider: providerMode }), { mode: 0o700 }); - const rpc = claudeProgram - ? new ClaudeSessionDriver({ configDirectory: home, stateDirectory: '/data/factory', fixedModel: plan.modelProvider?.model, partialMessages: plan.traceDeltas === true, - // Read for every turn, so a refreshed subscription token reaches the next process. - environment: () => claudeEnvironment({ plan, index, home, commandWrapper, githubToken, deployKey, providerKey, oauthToken: liveAuth?.accessToken }), - onStderr: text => rootTrace.record('claude.stderr', { text }) }) - : new CodexRpc({ env: codexEnvironment({ plan, home, githubToken, deployKey, providerKey }), cwd: repositoryDirectory, - onStderr: text => rootTrace.record('codex.stderr', { text }), - onRequest: async (method, params) => { - if (method === 'account/chatgptAuthTokens/refresh' && !providerMode) return refreshAuth(); - throw new Error(`Unexpected Codex host request: ${method}`); - } }); - if (claudeProgram && !providerMode) rpc.on('authFailed', () => { void refreshAuth().catch(error => rootTrace.record('auth.refresh_pending', { message: error.message })); }); + const rpc = program.driver({ plan, index, home, githubToken, deployKey, providerKey, providerMode, repositoryDirectory, refreshAuth, getLiveAuth: () => liveAuth, + onStderr: text => rootTrace.record(`${programName}.stderr`, { text }) }); + if (!providerMode) rpc.on('authFailed', () => { void refreshAuth().catch(error => rootTrace.record('auth.refresh_pending', { message: error.message })); }); rpc.on('notification', value => { const threadId = value.params?.threadId ?? value.params?.thread?.id; const agent = threadAgents.get(threadId) ?? rootId; @@ -201,7 +246,7 @@ export async function runCloudLiteMachine(config) { stopControllers.get(threadId)?.observeTerminal({ threadId, turnId, status }); } }); - rpc.on('exit', result => { rootTrace.record('codex.exit', result); void stop(); }); + rpc.on('exit', result => { rootTrace.record(program.exitTrace, result); void stop(); }); timer = setInterval(async () => { for (const [id, trace] of traces) { try { await trace.flush(); } catch (error) { rootTrace.record('trace.delivery_pending', { agent: id, message: error.message }); } @@ -209,8 +254,8 @@ export async function runCloudLiteMachine(config) { }, 1000); await rpc.initialize(); // A self-hosted provider authenticates with its key; no subscription login is involved. - if (!providerMode && !claudeProgram) await rpc.request('account/login/start', { type: 'chatgptAuthTokens', ...bootstrap.auth }); - rootTrace.record('machine.ready', { machine, node: process.version, ...(claudeProgram ? { claude: await run('claude', ['--version']) } : { codex: await run('codex', ['--version']) }), configuration: { cpuKind: plan.cpuKind, cpus: plan.cpus, memoryMb: plan.memoryMb, diskGb: plan.diskGb }, agents: bootstrap.agents.map(x => ({ id: x.id, role: x.role })) }); + if (!providerMode && program.login) await rpc.request('account/login/start', { type: 'chatgptAuthTokens', ...bootstrap.auth }); + rootTrace.record('machine.ready', { machine, node: process.version, [programName]: await program.version({ run, rpc }), configuration: { cpuKind: plan.cpuKind, cpus: plan.cpus, memoryMb: plan.memoryMb, diskGb: plan.diskGb }, agents: bootstrap.agents.map(x => ({ id: x.id, role: x.role })) }); const proxy = fileURLToPath(new URL('../../bin/factory-cloud-mcp.mjs', import.meta.url)); for (const agent of bootstrap.agents) { // With a prepare hook, each agent works in the folder the hook prepared for its slot. @@ -229,8 +274,8 @@ export async function runCloudLiteMachine(config) { const retained = JSON.parse(readFileSync(sessionFile)); try { result = await rpc.request('thread/resume', { threadId: retained.threadId, cwd: directory, config: configOverride, approvalPolicy: 'never', sandbox, - // Claude Code starts a new process per turn, so it needs these again after a restart. - ...(claudeProgram ? { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, developerInstructions: instructions } : {}) }); + // A driver without a long-lived native session needs these again after a restart. + ...(program.resumeNeedsSettings ? { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, developerInstructions: instructions } : {}), ...program.sessionFields?.(agent) }); } catch (error) { if (error.code !== -32600 || error.message !== `no rollout found for thread id ${retained.threadId}`) throw error; unavailableSession = retained.threadId; recovering = false; @@ -242,7 +287,7 @@ export async function runCloudLiteMachine(config) { } if (!result) { result = await rpc.request('thread/start', { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, cwd: directory, approvalPolicy: 'never', sandbox, config: configOverride, - developerInstructions: instructions, serviceName: `factory.${plan.runId}.${agent.id}`, historyMode: 'legacy' }); + developerInstructions: instructions, serviceName: `factory.${plan.runId}.${agent.id}`, historyMode: 'legacy', ...program.sessionFields?.(agent) }); const fd = await open(sessionFile, 'w', 0o600); try { await fd.writeFile(JSON.stringify({ threadId: result.thread.id, agentId: agent.id })); await fd.sync(); } finally { await fd.close(); } } @@ -259,6 +304,12 @@ export async function runCloudLiteMachine(config) { const threadEvent = recovering ? 'thread.resumed' : 'thread.started'; traces.get(agent.id).record('native.' + threadEvent, { threadId, ...(observedThreadStatus ? { threadStatus: nativeThreadStatus(observedThreadStatus) } : {}) }); + // A turn the old process never finished is closed on the board, or a later stop could not name its target. + if (result.recoveredTurn) { + turns.set(threadId, result.recoveredTurn.id); + terminalTurns.set(threadId, result.recoveredTurn); + traces.get(agent.id).record('native.turn.completed', { threadId, turn: result.recoveredTurn }); + } // Operator-supervised agents start idle; every turn comes from a durable operator message. if (!operatorSupervised && current?.thread.status?.type !== 'active') { const recoveryNote = unavailableSession ? `The prior prepared session ${unavailableSession} has no persisted rollout. Reconcile your existing FACTORY claims, inbox, checkout and GitHub state before continuing; do not replay previous delivery operations.\n` : ''; @@ -281,7 +332,7 @@ export async function runCloudLiteMachine(config) { if(message.type==='evidence.final')evidenceFinal=true; writeFileSync('/data/factory-evidence-status.json',JSON.stringify({at:new Date().toISOString(),...message}),{mode:0o600}); }); - evidenceProcess.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions:evidenceSessions(),sessionsPath:evidenceSessionsPath}); + evidenceProcess.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions:evidenceSessions(),...evidenceRoots}); }; launchCollector(); monitoring = setInterval(async () => { diff --git a/src/flujo-swarm/flujo-client.mjs b/src/flujo-swarm/flujo-client.mjs index d9f8404..f211833 100644 --- a/src/flujo-swarm/flujo-client.mjs +++ b/src/flujo-swarm/flujo-client.mjs @@ -90,6 +90,11 @@ export class FlujoClient { return true; } + /** The folders this workspace's file and shell tools may use. */ + async setWorkspaceRoots(roots, name = this.workspace) { + return this.expect('PATCH', '/api/workspaces', { name, newName: name, roots }, { workspace: null }); + } + /** Read-only observation; a missing workspace must stay missing throughout the window. */ async confirmWorkspaceAbsent(name, { verificationWindowMs = 15_000, pollIntervalMs = 500 } = {}) { if (!Number.isSafeInteger(verificationWindowMs) || verificationWindowMs < 1 || @@ -162,13 +167,13 @@ export class FlujoClient { return result.body.flow; } - async conversation(id) { - return this.api('GET', `/v1/chat/conversations/${encodeURIComponent(id)}`); + async conversation(id, { messageLimit } = {}) { + return this.api('GET', `/v1/chat/conversations/${encodeURIComponent(id)}${messageLimit ? `?messageLimit=${messageLimit}` : ''}`); } /** Read FLUJO's persisted conversation hierarchy without opening message bodies. */ - async descendants(id) { - return this.api('GET', `/v1/chat/conversations?paged=1&limit=200&descendantsOf=${encodeURIComponent(id)}`); + async descendants(id, cursor) { + return this.api('GET', `/v1/chat/conversations?paged=1&limit=200&descendantsOf=${encodeURIComponent(id)}${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`); } async inject(id, content, messageId = randomUUID()) { diff --git a/src/flujo-swarm/template/factory-team.mjs b/src/flujo-swarm/template/factory-team.mjs index 74b478d..b2ecaca 100644 --- a/src/flujo-swarm/template/factory-team.mjs +++ b/src/flujo-swarm/template/factory-team.mjs @@ -34,7 +34,7 @@ function environmentText(value) { /** FLUJO subflow team template. One lead plus up to nine local agents per workspace. */ export function buildFactoryTeamSpecs({ model, availableServers = [], availableTools = {}, - limits = {}, specialists, goalContext, environment } = {}) { + limits = {}, specialists, goalContext, environment, instructions } = {}) { if (typeof model !== 'string' || !model.trim() || !Array.isArray(availableServers) || new Set(availableServers).size !== availableServers.length || availableServers.some(name => typeof name !== 'string')) throw new TypeError('Model and connected server inventory are required'); @@ -56,7 +56,8 @@ export function buildFactoryTeamSpecs({ model, availableServers = [], availableT specialists.roles.map(role => `${role.id}: ${role.mission} ${role.instructions} Evidence: ${role.evidence}`).join('\n') : ''; const goal = context(goalContext, 'goalContext'); const surroundings = environmentText(environment); - const shared = `${goal ? `\nGoal context: ${goal}` : ''}${surroundings ? `\nEnvironment:\n${surroundings}` : ''}`; + if (instructions !== undefined && (typeof instructions !== 'string' || instructions.length > 64000)) throw new TypeError('instructions must be text under 64000 characters'); + const shared = `${goal ? `\nGoal context: ${goal}` : ''}${surroundings ? `\nEnvironment:\n${surroundings}` : ''}${instructions?.trim() ? `\n${instructions.trim()}` : ''}`; const principles = `Use the tools actually attached to this flow; never claim an action or result without its receipt. ` + `Reuse relevant installed upstream code and contracts before making a small adapter or product change. ` + `Deliver a bounded, useful product change where the task calls for one. Coordinate concisely on an available shared board and linked issue or PR. ` + diff --git a/src/lite-worker.mjs b/src/lite-worker.mjs index 8d4205f..532500f 100644 --- a/src/lite-worker.mjs +++ b/src/lite-worker.mjs @@ -2,8 +2,10 @@ import { resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { homedir } from 'node:os'; import { spawn } from 'node:child_process'; -import { mkdirSync } from 'node:fs'; -import { applyProviderEnvironment, claudeProviderEnvironment, codexProviderOverrides, validateModelProvider } from './model-provider.mjs'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { applyProviderEnvironment, claudeProviderEnvironment, codexProviderOverrides, flujoProviderModel, validateModelProvider } from './model-provider.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_PORT } from './cloud-lite/flujo-driver.mjs'; const mcp = fileURLToPath(new URL('../bin/factory-mcp.mjs', import.meta.url)); @@ -11,10 +13,11 @@ const mcp = fileURLToPath(new URL('../bin/factory-mcp.mjs', import.meta.url)); * Build a normal provider CLI command with its native tools and only FACTORY's MCP server. * With `modelProvider` (and its `providerKey`) the program talks to a self-hosted model * instead of its vendor login: Codex through the Responses API, Claude Code through the - * Anthropic Messages API. `providerIndex` picks one of several endpoints. + * Anthropic Messages API, FLUJO through a stored model record. `providerIndex` picks one of + * several endpoints. For FLUJO this is the command that starts its server. */ -export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = process.cwd(), modelProvider, providerKey, providerIndex = 0, env = process.env } = {}) { - if (!['codex', 'claude'].includes(provider)) throw new TypeError('provider must be codex or claude'); +export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = process.cwd(), modelProvider, providerKey, providerIndex = 0, env = process.env, port = FLUJO_WORKER_PORT, dataDirectory } = {}) { + if (!['codex', 'claude', 'flujo'].includes(provider)) throw new TypeError('provider must be codex, claude or flujo'); if (typeof database !== 'string' || !database.trim()) throw new TypeError('database path is required'); if (prompt !== undefined && (typeof prompt !== 'string' || !prompt.trim())) throw new TypeError('prompt must be nonempty'); const db = resolve(database); @@ -25,6 +28,11 @@ export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = p if (!Number.isSafeInteger(providerIndex) || providerIndex < 0) throw new TypeError('providerIndex must be a nonnegative integer.'); } const server = { command: process.execPath, args: [mcp, db, ...(agentId ? ['--actor',agentId] : [])] }; + if (provider === 'flujo') return { + command: 'flujo', cwd, server, args: ['--no-open', '--port', String(port)], + // Its own data folder, so a worker never touches the settings of a FLUJO someone uses by hand. + env: { ...env, FLUJO_DATA_DIR: dataDirectory ?? resolve(homedir(), '.factory', 'lite-flujo', 'data'), FLUJO_PORT: String(port) }, + }; if (provider === 'claude') return { command: 'claude', cwd, ...(modelProvider ? { env: applyProviderEnvironment(env, claudeProviderEnvironment(modelProvider, { index: providerIndex, key: providerKey })) } : {}), @@ -42,8 +50,57 @@ export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = p ...(prompt === undefined ? [] : [prompt])] }; } -/** Launch an interactive native CLI session and return its ChildProcess. */ +/** + * A FLUJO worker on this computer: attaches to `origin`, or starts its own FLUJO server, then installs + * one workspace with the generic team template and the board. The agent keeps its conversation across runs. + */ +export async function startFlujoLiteWorker({ origin, token, prompt, modelProvider, providerKey, providerIndex = 0, flujoModel, model, teamConcurrency, agentRole = 'orchestrator', servers, stateDirectory, developerInstructions, driverFactory = settings => new FlujoSessionDriver(settings), programFactory = settings => new FlujoProgram(settings), ...options } = {}) { + const spec = liteWorkerCommand({ ...options, provider: 'flujo', prompt, modelProvider, providerKey, providerIndex }); + if (!modelProvider && !flujoModel) throw new TypeError('A FLUJO worker needs a modelProvider or a flujoModel record.'); + const state = stateDirectory ?? resolve(homedir(), '.factory', 'lite-flujo', options.agentId ?? 'default'); + mkdirSync(state, { recursive: true, mode: 0o700 }); + const program = origin ? undefined : programFactory({ port: Number(spec.env.FLUJO_PORT), environment: () => spec.env }); + const driver = driverFactory({ origin, token, program, stateDirectory: state, servers, team: { ...(teamConcurrency ? { concurrency: teamConcurrency } : {}) }, + modelRecord: name => flujoModel ?? flujoProviderModel(modelProvider, { index: providerIndex, key: providerKey, model: model ?? name }) }); + await driver.initialize(); + const settings = { cwd: spec.cwd, model, sandbox: 'danger-full-access', agentRole, developerInstructions, config: { mcp_servers: { factory: spec.server } } }; + const sessionFile = join(state, 'session.json'); + let opened; + if (existsSync(sessionFile)) { + try { opened = await driver.request('thread/resume', { ...settings, threadId: JSON.parse(readFileSync(sessionFile, 'utf8')).threadId }); } + catch (error) { if (error.code !== -32600) throw error; } + } + opened ??= await driver.request('thread/start', settings); + const threadId = opened.thread.id; + writeFileSync(sessionFile, JSON.stringify({ threadId }), { mode: 0o600 }); + const conversation = text => [{ type: 'text', text }]; + const worker = { origin: driver.origin, workspace: driver.thread(threadId).workspace, threadId, driver, child: program?.child ?? null, + /** Starts a turn, or joins the one that is running. */ + send: async text => { + const running = driver.thread(threadId).turn; + if (!running) return driver.request('turn/start', { threadId, input: conversation(text) }); + return driver.request('turn/steer', { threadId, expectedTurnId: running.id, input: conversation(text) }); + }, + /** Resolves when the current turn has ended, with its final status. */ + settled: () => new Promise(accept => { + const last = driver.thread(threadId); + if (!last.turn) { accept(last.lastTurn); return; } + const listen = note => { if (note.method === 'turn/completed' && note.params.threadId === threadId) { driver.off('notification', listen); accept(note.params.turn); } }; + driver.on('notification', listen); + }), + stop: async () => { + const running = driver.thread(threadId).turn; + if (running) await driver.request('turn/interrupt', { threadId, turnId: running.id }).catch(() => {}); + driver.close(); + await program?.stop(); + } }; + if (prompt !== undefined) await worker.send(prompt); + return worker; +} + +/** Launch an interactive native CLI session and return its ChildProcess. FLUJO has no such session: it returns a promise of its worker handle. */ export function startLiteWorker({ stdio = 'inherit', spawnImpl = spawn, ...options } = {}) { + if (options.provider === 'flujo') return startFlujoLiteWorker(options); const spec = liteWorkerCommand(options); if (spec.env?.CODEX_HOME) mkdirSync(spec.env.CODEX_HOME, { recursive: true, mode: 0o700 }); return spawnImpl(spec.command, spec.args, { cwd: spec.cwd, env: spec.env, stdio }); diff --git a/src/model-provider.mjs b/src/model-provider.mjs index 6e03367..e8ce1d9 100644 --- a/src/model-provider.mjs +++ b/src/model-provider.mjs @@ -4,8 +4,11 @@ /** Wire APIs an endpoint can offer. */ export const MODEL_PROVIDER_APIS = ['responses', 'chat', 'anthropic']; -/** The wire API each worker program needs. */ -export const WORKER_PROGRAM_APIS = Object.freeze({ codex: 'responses', claude: 'anthropic' }); +/** + * The wire APIs each worker program can use, best first. Codex and Claude Code know one each; + * FLUJO has an adapter for all three and prefers the Responses API. + */ +export const WORKER_PROGRAM_APIS = Object.freeze({ codex: Object.freeze(['responses']), claude: Object.freeze(['anthropic']), flujo: Object.freeze(['responses', 'chat', 'anthropic']) }); // Variables the launchers set for the worker process itself; a provider key must not replace them. const RESERVED_ENVIRONMENT = new Set(['PATH', 'HOME', 'CODEX_HOME', 'GH_TOKEN', 'GIT_TERMINAL_PROMPT', 'ANTHROPIC_AUTH_TOKEN', 'ANTHROPIC_API_KEY', 'ANTHROPIC_BASE_URL', 'CLAUDE_CODE_OAUTH_TOKEN']); @@ -44,17 +47,27 @@ export function validateModelProvider(provider) { /** The wire APIs a provider offers (the default keeps 0.4.0 descriptions valid for Codex). */ export const providerApis = provider => provider.apis ?? ['responses']; +/** The API a worker program would use on this provider, or undefined when they share none. */ +export function programApi(provider, program) { + const apis = providerApis(provider); + return WORKER_PROGRAM_APIS[program]?.find(api => apis.includes(api)); +} + /** Which worker programs can use this provider. */ export function providerPrograms(provider) { - const apis = providerApis(provider); - return Object.entries(WORKER_PROGRAM_APIS).filter(([, api]) => apis.includes(api)).map(([program]) => program); + return Object.keys(WORKER_PROGRAM_APIS).filter(program => programApi(provider, program)); } +const apiNames = program => WORKER_PROGRAM_APIS[program].join(' or '); + function requireApi(provider, program) { validateModelProvider(provider); - if (!providerApis(provider).includes(WORKER_PROGRAM_APIS[program])) { - throw new TypeError(`${program} needs a provider that offers the ${WORKER_PROGRAM_APIS[program]} API.`); - } + if (!programApi(provider, program)) throw new TypeError(`${program} needs a provider that offers the ${apiNames(program)} API.`); +} + +/** A cloud plan's refusal, in the words its operator knows. */ +export function requireFleetApi(provider, program) { + if (!programApi(provider, program)) throw new TypeError(`Cloud lite workers need a provider that offers the ${apiNames(program)} API.`); } /** Spread workers over the endpoints so each keeps its prompt cache on one server. */ @@ -144,6 +157,32 @@ export function claudeProviderEnvironment(provider, { index = 0, key, model } = }; } +// FLUJO keeps a model as a stored record, so its settings differ by adapter rather than by variable. +const FLUJO_ADAPTERS = Object.freeze({ + responses: { provider: 'openai', adapter: 'openai-responses' }, + chat: { provider: 'openai', adapter: 'openai' }, + anthropic: { provider: 'anthropic', adapter: 'anthropic' }, +}); + +/** + * FLUJO model record for a self-hosted provider, for `POST /api/model`. FLUJO stores the key in its + * own encrypted settings, so it never enters the process environment that agent commands inherit. + */ +export function flujoProviderModel(provider, { index = 0, key, model } = {}) { + requireApi(provider, 'flujo'); + if (typeof key !== 'string' || key.length < 8) throw new TypeError('A model provider needs its API key.'); + const served = provider.model ?? model; + if (typeof served !== 'string' || !served) throw new TypeError('Select the served model name.'); + const api = programApi(provider, 'flujo'); + const base = providerBaseUrl(provider, index); + return { id: `factory-${provider.id}`, name: served, displayName: provider.name ?? provider.id, ...FLUJO_ADAPTERS[api], + // The Anthropic client adds /v1 itself. + baseUrl: api === 'anthropic' ? base.replace(/\/+$/, '').replace(/\/v1$/, '') : base, + ApiKey: key, supportsTools: true, + ...(provider.contextWindow ? { contextWindow: provider.contextWindow } : {}), + ...(provider.autoCompactTokenLimit ? { compactionThreshold: provider.autoCompactTokenLimit } : {}) }; +} + /** Apply a mapping's environment to an inherited one (null removes a variable). */ export function applyProviderEnvironment(base, overlay) { const env = { ...base }; diff --git a/src/public-sdk.mjs b/src/public-sdk.mjs index fd0c5e1..3f605d7 100644 --- a/src/public-sdk.mjs +++ b/src/public-sdk.mjs @@ -96,10 +96,12 @@ export { buildSaviaCasePlan } from './flujo-swarm/template/savia-case-plan.mjs'; export { observeSaviaCaseTopology } from './flujo-swarm/template/savia-topology.mjs'; export { createPresentationServer, startPresentationServer } from './presentation.mjs'; export { createObservatoryClient } from './observatory-client.mjs'; -export { liteWorkerCommand, startLiteWorker } from './lite-worker.mjs'; +export { liteWorkerCommand, startLiteWorker, startFlujoLiteWorker } from './lite-worker.mjs'; export { ModalSelfHostedModel, selfHostedProfiles, probeSelfHostedModel, modalResult, readBoundedJson, modalEndpointOrigin, runCommand, SELFHOST_MODAL_DIR, SELFHOSTED_KEY_ENV } from './selfhost/modal.mjs'; -export { MODEL_PROVIDER_APIS, WORKER_PROGRAM_APIS, validateModelProvider, providerPrograms, codexProviderConfig, codexProviderOverrides, claudeProviderEnvironment } from './model-provider.mjs'; +export { MODEL_PROVIDER_APIS, WORKER_PROGRAM_APIS, validateModelProvider, providerPrograms, codexProviderConfig, codexProviderOverrides, claudeProviderEnvironment, flujoProviderModel, programApi } from './model-provider.mjs'; +export { FlujoProgram, FlujoSessionDriver, flujoEnvironment } from './cloud-lite/flujo-driver.mjs'; export { FactoryCloudLiteFleet, CodexSubscriptionBroker, ClaudeSubscriptionBroker } from './cloud-lite/fleet.mjs'; export { validateCloudLitePlan, agentInventory, machineConfig as cloudLiteMachineConfig } from './cloud-lite/protocol.mjs'; export { cloudLiteBootstrap } from './cloud-lite/bootstrap.mjs'; +export { packFactorySdk } from './cloud-lite/bundle.mjs'; export * from './workflows/index.mjs'; diff --git a/src/selfhost/modal.mjs b/src/selfhost/modal.mjs index bdd369e..1a2ba0d 100644 --- a/src/selfhost/modal.mjs +++ b/src/selfhost/modal.mjs @@ -16,6 +16,11 @@ const MINUTE = 60_000; const notApplied = message => Object.assign(new Error(message), { outcome: 'not_applied' }); const unknown = message => Object.assign(new Error(message), { code: 'UNKNOWN', outcome: 'unknown', unknown: true }); +const pause = (ms, signal) => new Promise((resolve, reject) => { + if (signal?.aborted) { reject(uncertain()); return; } + const timer = setTimeout(resolve, ms); + signal?.addEventListener('abort', () => { clearTimeout(timer); reject(uncertain()); }, { once: true }); +}); const uncertain = () => unknown('The self-hosted model has an uncertain outcome. Its owned resources are kept for reconciliation; retire them before deploying again.'); const json = raw => { try { return JSON.parse(raw); } catch { return null; } }; const lastJsonLine = raw => json(String(raw ?? '').trim().split(/\r?\n/).reverse().find(line => line.trim().startsWith('{')) ?? ''); @@ -169,10 +174,10 @@ function fileCredentialStore(directory) { export class ModalSelfHostedModel { constructor({ directory, python = process.platform === 'win32' ? 'python' : 'python3', pythonArgs = [], env = process.env, commandRunner = runCommand, fetchImpl = fetch, credentialStore, keyName = 'selfhosted-model', namePrefix = 'factory', - clock = Date.now, deployTimeoutMs = 45 * MINUTE, prefetchTimeoutMs = 70 * MINUTE } = {}) { + clock = Date.now, deployTimeoutMs = 45 * MINUTE, prefetchTimeoutMs = 70 * MINUTE, stopWaitMs = 3 * MINUTE, stopPollMs = 3000, deleteAttempts = 5 } = {}) { if (typeof directory !== 'string' || !directory) throw new TypeError('Select a private state directory.'); if (!/^[a-z][a-z0-9]{1,15}$/.test(namePrefix)) throw new TypeError('namePrefix must be a short lowercase name.'); - Object.assign(this, { directory, python, pythonArgs: [...pythonArgs], commandRunner, fetch: fetchImpl, keyName, namePrefix, clock, deployTimeoutMs, prefetchTimeoutMs }); + Object.assign(this, { stopWaitMs, stopPollMs, deleteAttempts, directory, python, pythonArgs: [...pythonArgs], commandRunner, fetch: fetchImpl, keyName, namePrefix, clock, deployTimeoutMs, prefetchTimeoutMs }); this.environment = typeof env === 'function' ? env : () => env; this.store = credentialStore ?? fileCredentialStore(directory); this.path = join(directory, 'model.json'); @@ -349,11 +354,15 @@ export class ModalSelfHostedModel { async #stopApp(signal) { const stopped = await this.#cli(['app', 'stop', '--yes', this.record.appName], { signal }); - const states = await this.#appState(signal); - // Stopped (now or earlier), or never created after an early deployment failure, is a - // confirmed GPU-off state; anything else stays uncertain. - if (states && !states.every(state => /^stopped$/i.test(state))) throw uncertain(); - return { stopCommand: stopped?.code === 0 ? 'ok' : 'refused', listed: Boolean(states) }; + // A busy replica takes a while to stop. Stopped (now or earlier), or never created after an + // early deployment failure, is a confirmed GPU-off state; still stopping at the deadline stays uncertain. + const deadline = this.clock() + this.stopWaitMs; + for (;;) { + const states = await this.#appState(signal); + if (!states || states.every(state => /^stopped$/i.test(state))) return { stopCommand: stopped?.code === 0 ? 'ok' : 'refused', listed: Boolean(states) }; + if (!states.some(state => /stopping/i.test(state)) && stopped?.code !== 0 || this.clock() >= deadline) throw uncertain(); + await pause(this.stopPollMs, signal); + } } /** Stop the GPU containers now; key, secret and cached weights stay for a later redeploy. */ @@ -408,8 +417,13 @@ export class ModalSelfHostedModel { const app = record.appName; const owned = [['secret', `${app}-auth`], ['volume', `${app}-hf-cache`], ['volume', `${app}-vllm-cache`]]; for (const [kind, name] of owned) { - const deleted = await this.#cli([kind, 'delete', '--allow-missing', '--yes', name], { signal }); - if (deleted?.code !== 0) throw uncertain(); + // Modal can refuse a delete for a moment while the stopped app lets go of its volumes. + for (let attempt = 1; ; attempt++) { + const deleted = await this.#cli([kind, 'delete', '--allow-missing', '--yes', name], { signal }); + if (deleted?.code === 0) break; + if (attempt >= this.deleteAttempts) throw uncertain(); + await pause(this.stopPollMs, signal); + } } const [secrets, volumes] = await Promise.all([this.#cli(['secret', 'list', '--json'], { signal }), this.#cli(['volume', 'list', '--json'], { signal })]); const remaining = [json(secrets?.stdout), json(volumes?.stdout)]; diff --git a/test/cloud-lite-flujo.test.mjs b/test/cloud-lite-flujo.test.mjs new file mode 100644 index 0000000..7f5cfdc --- /dev/null +++ b/test/cloud-lite-flujo.test.mjs @@ -0,0 +1,677 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from 'node:fs'; +import { createServer } from 'node:net'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { validateCloudLitePlan, agentInventory, machineConfig, secret } from '../src/cloud-lite/protocol.mjs'; +import { cloudLiteBootstrap } from '../src/cloud-lite/bootstrap.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_SERVERS, flujoEnvironment } from '../src/cloud-lite/flujo-driver.mjs'; +import { packFactorySdk } from '../src/cloud-lite/bundle.mjs'; +import { FactoryCloudLiteFleet } from '../src/cloud-lite/fleet.mjs'; +import { NativeOperatorStop } from '../src/cloud-lite/operator-stop.mjs'; +import { NativeWorkflowWake } from '../src/cloud-lite/workflow-wake.mjs'; +import { CloudLiteTrace } from '../src/cloud-lite/trace.mjs'; +import { startCloudLiteCoordinator } from '../src/cloud-lite/server.mjs'; +import { nativeStopTurn } from '../src/cloud-lite/runtime.mjs'; +import { FactoryArtifactStore } from '../src/workflows/artifacts.mjs'; +import { captureWorkspace, readWorkspaceManifest } from '../src/workflows/workspace.mjs'; +import { flujoProviderModel, liteWorkerCommand, programApi, startLiteWorker } from '../src/public-sdk.mjs'; +import { FakeFlujo } from './fixtures/fake-flujo.mjs'; +import { tempDir } from './support/temp-dir.mjs'; + +const fresh = () => tempDir('factory-cloud-lite-flujo-'); +const fakeProgram = fileURLToPath(new URL('./fixtures/fake-flujo.mjs', import.meta.url)); +const key = 'provider-key-value-1234'; +const provider = (extra = {}) => ({ id: 'selfhosted', envKey: 'SELF_HOSTED_MODEL_KEY', model: 'served-model', name: 'Served', apis: ['responses', 'chat', 'anthropic'], + baseUrls: ['https://model-a.example.test/v1', 'https://model-b.example.test/v1'], contextWindow: 262144, autoCompactTokenLimit: 80000, ...extra }); +const plan = (extra = {}) => validateCloudLitePlan({ runId: 'flujo-test', app: 'factory-flujo-test', goal: 'Work on operator jobs', machines: 1, agentsPerMachine: 2, + workerProgram: 'flujo', modelProvider: provider(), ...extra }); +const text = value => [{ type: 'text', text: value }]; +const board = { command: process.execPath, args: ['board-proxy.mjs'], env: { FACTORY_BOARD_URL: 'http://board.invalid', FACTORY_BOARD_TOKEN: 'board-credential-value', FACTORY_WORKTREE: '/work/a0' } }; +const waitFor = async (condition, ms = 5000) => { const end = Date.now() + ms; for (;;) { const value = await condition(); if (value) return value; if (Date.now() > end) throw new Error('Condition was not reached.'); await new Promise(resolve => setTimeout(resolve, 10)); } }; + +async function harness({ script, root = fresh(), team, servers, fake: existing, ...options } = {}) { + const fake = existing ?? await new FakeFlujo({ script }).listen(); + const driver = new FlujoSessionDriver({ origin: fake.origin, stateDirectory: join(root, 'state'), privateDirectory: join(root, 'private'), team, servers, + modelRecord: name => flujoProviderModel(provider(), { key, model: name }), pollMs: 20, reconnectMs: 10, submitTimeoutMs: 400, interruptResendMs: 30, drainMs: 200, ...options }); + const notes = []; + driver.on('notification', note => notes.push(note)); + const finished = () => new Promise(resolve => { const check = note => { if (note.method === 'turn/completed') { driver.off('notification', check); resolve(note.params.turn); } }; driver.on('notification', check); }); + return { root, fake, driver, notes, finished, end: async () => { driver.close(); await fake.close(); } }; +} +const started = (h, extra = {}) => { mkdirSync(join(h.root, 'work'), { recursive: true }); return h.driver.request('thread/start', { model: 'served-model', cwd: join(h.root, 'work'), sandbox: 'danger-full-access', developerInstructions: 'Be useful.', ...extra }); }; +const items = (notes, type) => notes.filter(note => note.method === 'item/completed' && note.params.item.type === type).map(note => note.params.item); + +test('a plan can choose FLUJO, and refuses what FLUJO cannot do', () => { + assert.equal(plan().workerProgram, 'flujo'); + assert.equal(plan().modelProvider.model, 'served-model'); + assert.equal('model' in plan(), false, 'a FLUJO plan names no vendor model'); + assert.throws(() => plan({ modelProvider: undefined }), /need a modelProvider/); + assert.throws(() => plan({ modelProvider: provider({ model: undefined }) }), /served model/); + assert.equal(plan({ modelProvider: provider({ model: undefined }), model: 'named' }).subagentModel, 'named', 'every agent has a model'); + assert.throws(() => plan({ sandbox: 'read-only' }), /danger-full-access/); + assert.throws(() => plan({ sandbox: 'workspace-write' }), /danger-full-access/); + assert.equal(plan({ sandbox: 'danger-full-access' }).sandbox, 'danger-full-access'); + assert.throws(() => validateCloudLitePlan({ runId: 'codex-test', app: 'factory-codex-test', goal: 'x', flujo: {} }), /belong to FLUJO/); + assert.throws(() => plan({ flujo: { teamConcurrency: 10 } }), /teamConcurrency/); + assert.throws(() => plan({ flujo: { servers: ['flujo'] } }), /flujo\.servers/); + assert.throws(() => plan({ flujo: { agentTurns: 0 } }), /agentTurns/); + assert.equal(plan({ flujo: { teamConcurrency: 4, servers: ['bash'] } }).flujo.teamConcurrency, 4); + assert.equal(machineConfig(plan(), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'flujo-lite'); + assert.equal(machineConfig(plan({ workerProgram: 'claude', model: 'm', sandbox: undefined }), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'claude-lite'); + assert.equal(machineConfig(validateCloudLitePlan({ runId: 'codex-test', app: 'factory-codex-test', goal: 'x' }), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'codex-lite'); +}); + +test('the FLUJO mapping prefers Responses and stores the key in a model record', () => { + const responses = flujoProviderModel(provider(), { index: 1, key }); + assert.deepEqual([responses.id, responses.name, responses.provider, responses.adapter, responses.baseUrl, responses.ApiKey, responses.contextWindow, responses.compactionThreshold, responses.supportsTools], + ['factory-selfhosted', 'served-model', 'openai', 'openai-responses', 'https://model-b.example.test/v1', key, 262144, 80000, true]); + const chat = flujoProviderModel(provider({ apis: ['chat'] }), { key }); + assert.deepEqual([chat.adapter, chat.baseUrl], ['openai', 'https://model-a.example.test/v1']); + const anthropic = flujoProviderModel(provider({ apis: ['anthropic'] }), { key }); + assert.deepEqual([anthropic.provider, anthropic.adapter, anthropic.baseUrl], ['anthropic', 'anthropic', 'https://model-a.example.test']); + assert.equal(programApi(provider({ apis: ['chat', 'responses'] }), 'flujo'), 'responses', 'the order is the program\'s, not the provider\'s'); + assert.throws(() => flujoProviderModel(provider(), { key: 'short' }), /API key/); + assert.throws(() => flujoProviderModel(provider({ model: undefined }), { key }), /served model/); + assert.equal(flujoProviderModel(provider({ model: undefined }), { key, model: 'from-plan' }).name, 'from-plan'); +}); + +test('the FLUJO process environment carries no key, and the bootstrap pins the program by checksum', () => { + const env = flujoEnvironment({ home: '/data/flujo', path: '/bin' }); + assert.deepEqual(env, { PATH: '/bin', HOME: '/data', FLUJO_DATA_DIR: '/data/flujo', FLUJO_PORT: '4210', FLUJO_EXPOSURE_MODE: 'localhost', NEXT_TELEMETRY_DISABLED: '1', GIT_TERMINAL_PROMPT: '0' }); + const script = cloudLiteBootstrap('a'.repeat(64), { workerProgram: 'flujo' }); + assert.match(script, /flujo-ai-\d+\.\d+\.\d+\.tgz -o \/tmp\/flujo\.tgz/); + assert.match(script, /^echo '[a-f0-9]{64} {2}\/tmp\/flujo\.tgz' \| sha256sum -c -$/m); + assert.ok(script.indexOf('sha256sum -c -\nnpm install --prefix /data/flujo-program') > 0, 'the checksum is checked before install'); + assert.match(script, /--ignore-scripts/); + assert.match(script, /command -v flujo/, 'an image with FLUJO skips the install'); + assert.match(script, /FLUJO_DATA_DIR=\/data\/flujo/); + assert.doesNotMatch(script, /codex|claude/i); + assert.doesNotMatch(cloudLiteBootstrap('a'.repeat(64)), /flujo-ai/); +}); + +test('a thread gets its own workspace with only its tools, and no secret in FLUJO settings', async () => { + const h = await harness(); + try { + const { thread } = await started(h, { config: { mcp_servers: { factory: board }, 'sandbox_workspace_write.writable_roots': ['/extra'] } }); + assert.match(thread.id, /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/); + assert.equal(thread.status.type, 'idle'); + const space = h.fake.workspaces.get(`factory-${thread.id}`); + assert.deepEqual(space.roots, [join(h.root, 'work'), '/extra']); + assert.equal(space.models[0].ApiKey, key); + assert.equal(space.servers.find(server => server.name === 'flujo').disabled, true, 'the control server stays off'); + assert.deepEqual(space.servers.filter(server => !server.disabled).map(server => server.name), [...FLUJO_WORKER_SERVERS, 'factory']); + const server = space.servers.find(entry => entry.name === 'factory'); + assert.equal(JSON.stringify(server).includes('board-credential-value'), false); + assert.deepEqual(server.env, { FACTORY_WORKTREE: '/work/a0' }); + const config = server.args[server.args.indexOf('--config') + 1]; + assert.deepEqual(JSON.parse(readFileSync(config, 'utf8')), { url: 'http://board.invalid', token: 'board-credential-value' }); + if (process.platform !== 'win32') assert.equal(statSync(config).mode & 0o777, 0o600); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent']); + const agent = space.flows[0].spec.nodes.find(node => node.key === 'agent'); + assert.deepEqual(agent.servers.map(entry => entry.name), [...FLUJO_WORKER_SERVERS, 'factory']); + assert.match(space.flows[0].spec.nodes[0].prompt, /Your working folder is .*Pass it as cwd/s); + assert.match(space.flows[0].spec.nodes[0].prompt, /Be useful\./); + assert.equal(h.notes.some(note => note.method === 'thread/started' && note.params.threadId === thread.id), true); + } finally { await h.end(); } +}); + +test('an orchestrator leads a team when the plan asks for one', async () => { + const h = await harness({ team: { concurrency: 4, agentTurns: 6, leadTurns: 12 } }); + try { + const lead = (await started(h, { agentRole: 'orchestrator' })).thread; + const member = (await started(h, { agentRole: 'subagent' })).thread; + const space = h.fake.workspaces.get(`factory-${lead.id}`); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent', 'swarm_team']); + assert.equal(space.flows[1].spec.nodes.find(node => node.key === 'agents').concurrencyLimit, 4); + assert.equal(space.flows[1].spec.nodes.find(node => node.key === 'lead').maxTurns, 12); + await h.driver.request('turn/start', { threadId: lead.id, input: text('lead') }); + await h.driver.request('turn/start', { threadId: member.id, input: text('member') }); + await waitFor(() => h.fake.count('POST', '/v1/chat/completions') === 2); + const flows = h.fake.requests.filter(entry => entry.path === '/v1/chat/completions').map(entry => entry.body.model); + assert.deepEqual(flows, ['flow-swarm_team', 'flow-swarm_agent']); + } finally { await h.end(); } +}); + +test('the board must connect before a thread starts', async () => { + const h = await harness({ toolsTimeoutMs: 50 }); + h.fake.boardFails = true; + try { await assert.rejects(started(h, { config: { mcp_servers: { factory: board } } }), /factory server did not connect/); } + finally { await h.end(); } +}); + +test('a turn maps FLUJO events to the shared trace and ends only when FLUJO says so', async () => { + const h = await harness({ script: run => { run.say('working'); run.tool('bash_run'); run.usage(); run.emit({ type: 'recovery:retry', attempt: 2, retryAt: 5, failure: { category: 'rate_limit' } }); run.emit({ type: 'model:delta', delta: 'skipped' }); run.say('done'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + let readAtEnd; + h.driver.on('notification', note => { if (note.method === 'turn/completed') readAtEnd ??= h.driver.request('thread/read', { threadId: thread.id }); }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal(turn.status, 'inProgress'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + const final = await ended; + assert.deepEqual(final, { id: turn.id, status: 'completed' }); + assert.equal((await readAtEnd).thread.status.type, 'idle', 'idle is already readable when the turn ends'); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['working', 'done']); + assert.equal(items(h.notes, 'toolCall')[0].tool, 'bash_run'); + assert.equal(items(h.notes, 'toolResult')[0].isError, false); + const usage = h.notes.find(note => note.method === 'thread/tokenUsage/updated').params.tokenUsage; + assert.deepEqual(usage.last, { inputTokens: 40, cachedInputTokens: 10, outputTokens: 4 }); + assert.equal(h.notes.find(note => note.method === 'model/retry').params.error, 'rate_limit'); + assert.equal(h.notes.some(note => note.method === 'item/stream/delta'), false, 'streaming stays off unless asked for'); + const methods = h.notes.map(note => note.method); + assert.ok(methods.indexOf('turn/started') < methods.indexOf('turn/completed')); + assert.equal(methods.at(-1), 'thread/status/changed'); + // The next turn continues the same conversation and is not confused by the previous run's end. + const next = h.finished(); + const second = await h.driver.request('turn/start', { threadId: thread.id, input: text('again') }); + assert.equal((await next).id, second.turn.id); + assert.equal(h.fake.conversationOf(thread.id).messages.filter(message => message.role === 'user').length, 2); + } finally { await h.end(); } +}); + +test('a capped run counts as completed, and a failed run keeps its provider status', async () => { + let mode = 'capped'; + const h = await harness({ script: run => { if (mode === 'capped') run.capped(); else run.fail(429); } }); + try { + const { thread } = await started(h); + let ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('one') }); + assert.equal((await ended).status, 'completed'); + mode = 'fail'; ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('two') }); + const failed = await ended; + assert.equal(failed.status, 'failed'); + assert.deepEqual(failed.error, { code: 429 }); + } finally { await h.end(); } +}); + +test('a lost stream cannot leave a turn open: the state is read on a timer', async () => { + const h = await harness({ script: run => { run.say('quiet'); run.endSilently('completed'); }, drainMs: 60 }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'completed'); + } finally { await h.end(); } +}); + +test('a dropped stream is followed again from where it stopped, without repeating events or the task', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { run.say('before'); await gate; run.say('after'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => items(h.notes, 'agentMessage').length === 1); + for (const stream of h.fake.streams) stream.destroy(); + await waitFor(() => h.fake.count('GET', `/v1/chat/conversations/${thread.id}/events`) >= 2); + release(); + assert.equal((await ended).status, 'completed'); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['before', 'after']); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1); + } finally { await h.end(); } +}); + +test('an uncertain submission is read back, never sent twice', async () => { + const h = await harness(); + h.fake.dropAfterAccept = 1; + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'completed'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1); + } finally { await h.end(); } +}); + +test('a run that never starts ends the turn as unfinished, without sending it again', async () => { + const h = await harness({ script: () => {}, submitTimeoutMs: 60 }); + try { + const { thread } = await started(h); + const conversationStart = h.fake.complete.bind(h.fake); + // FLUJO answers, but no run appears: its conversation stays as it was. + h.fake.complete = (space, body, response) => { response.writeHead(200); response.write('data: {}\n\n'); return undefined; }; + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'failed'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + h.fake.complete = conversationStart; + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'it was sent once'); + } finally { await h.end(); } +}); + +test('under pressure a turn waits and retries; a refusal ends it', async () => { + const h = await harness({ submitTimeoutMs: 5000 }); + try { + const { thread } = await started(h); + h.fake.refuse = [503]; + let ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('busy') }); + assert.equal((await ended).status, 'completed'); + h.fake.refuse = [400]; + ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('bad') }); + assert.equal((await ended).status, 'failed'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + } finally { await h.end(); } +}); + +test('a message during a turn joins it; with nothing running the refusal has the exact text', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { await gate; for (const message of run.conversation.steered) { run.conversation.messages.push(message); run.emit({ type: 'message', message }); } run.say('done'); run.done(); } }); + try { + const { thread } = await started(h); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, input: text('nothing runs') }), { message: 'no active turn to steer' }); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: 'other', input: text('wrong') }), /expected turn/); + assert.deepEqual(await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('correction') }), { turnId: turn.id }); + release(); + assert.equal((await ended).status, 'completed'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'taken in by the run, so nothing is sent again'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('late') }), { message: 'no active turn to steer' }); + } finally { await h.end(); } +}); + +test('a message that arrives as the run ends is still delivered, in the same turn', async () => { + let release, runs = 0; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { runs++; if (runs === 1) { await gate; run.say('first'); run.done(); return; } run.say('second'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('owed') }); + release(); + assert.deepEqual(await ended, { id: turn.id, status: 'completed' }); + const posted = h.fake.requests.filter(entry => entry.path === '/v1/chat/completions'); + assert.deepEqual(posted.map(entry => entry.body.messages.at(-1).content), ['go', 'owed']); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['first', 'second']); + } finally { await h.end(); } +}); + +test('a stop interrupts the turn, keeps asking until FLUJO has stopped, and a refused stop can be retried', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + h.fake.cancelFails = 1; + await assert.rejects(h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), /did not accept the stop/); + await assert.rejects(h.driver.request('turn/interrupt', { threadId: thread.id, turnId: 'other' }), /expected turn/); + h.fake.holdCancel = true; + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), {}); + await waitFor(() => h.fake.cancels >= 2, 2000); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active', 'cancel requested is not stopped'); + h.fake.holdCancel = false; + const conversation = h.fake.conversationOf(thread.id), run = conversation.run; + run.ended = true; conversation.status = 'error'; conversation.lastError = { code: 'cancelled' }; run.emit({ type: 'run:done', status: 'error' }); + const final = await ended; + assert.equal(final.status, 'interrupted'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), {}, 'a finished turn is already stopped'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, input: text('late') }), { message: 'no active turn to steer' }); + } finally { await h.end(); } +}); + +test('a stop wins over a queued message, and a turn stopped before it ran is interrupted', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + let ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('queued') }); + await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }); + assert.equal((await ended).status, 'interrupted'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'the queued message does not start a new run'); + h.fake.refuse = [503]; + ended = h.finished(); + const second = await h.driver.request('turn/start', { threadId: thread.id, input: text('blocked') }); + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: second.turn.id }), {}); + assert.equal((await ended).status, 'interrupted'); + } finally { await h.end(); } +}); + +test('children keep a thread active until they have ended too', async () => { + let child; + const h = await harness({ script: run => { child = run.child('running'); run.say('lead is done'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => child && h.fake.conversationOf(thread.id).status === 'completed'); + await new Promise(resolve => setTimeout(resolve, 60)); + assert.equal(h.notes.some(note => note.method === 'turn/completed'), false, 'the lead has ended but a child still runs'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + child.status = 'completed'; + assert.equal((await ended).id, turn.id); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + // A child that appears later is caught by the readback, so a stop receipt cannot be wrong. + const late = h.fake.conversationOf(thread.id).run.child('running'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + late.status = 'error'; + } finally { await h.end(); } +}); + +test('an unattended worker answers no questions: questions and forms are declined so a run never parks', async () => { + const h = await harness({ script: run => { run.emit({ type: 'run:awaiting_question', questionId: 'q1', questions: [] }); run.emit({ type: 'run:awaiting_elicitation', elicitationId: 'e1', message: 'form', requestedSchema: {} }); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await ended; + await waitFor(() => h.fake.responses?.length === 2); + assert.deepEqual(h.fake.responses.map(body => body.action).sort(), ['elicitation-cancel', 'question-decline']); + } finally { await h.end(); } +}); + +test('resume keeps the conversation, and lost history gives the error the runtime recovers from', async () => { + const fake = await new FakeFlujo().listen(); + const root = fresh(); + const first = await harness({ fake, root }); + try { + const { thread } = await started(first); + const unavailable = id => ({ code: -32600, message: `no rollout found for thread id ${id}` }); + const missing = '00000000-0000-4000-8000-000000000000'; + await assert.rejects(first.driver.request('thread/resume', { threadId: missing, cwd: root }), unavailable(missing)); + const ended = first.finished(); + const { turn } = await first.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await ended; + first.driver.close(); + + const second = await harness({ fake, root }); + const flowsBefore = fake.workspaces.get(`factory-${thread.id}`).flows.map(flow => flow.id); + const resumed = await second.driver.request('thread/resume', { threadId: thread.id, model: 'served-model', cwd: join(root, 'work'), sandbox: 'danger-full-access', developerInstructions: 'Be useful.' }); + assert.equal(resumed.thread.status.type, 'idle'); + assert.equal(resumed.recoveredTurn, undefined); + assert.deepEqual(fake.workspaces.get(`factory-${thread.id}`).flows.map(flow => flow.id), flowsBefore, 'an unchanged flow keeps its identity'); + assert.deepEqual((await second.driver.request('thread/read', { threadId: thread.id, includeTurns: true })).thread.turns, [{ id: turn.id, status: 'completed' }]); + const next = second.finished(); + await second.driver.request('turn/start', { threadId: thread.id, input: text('more') }); + await next; + assert.equal(fake.conversationOf(thread.id).messages.filter(message => message.role === 'user').length, 2); + assert.deepEqual(items(second.notes, 'agentMessage').map(item => item.text), ['answer'], 'old events are not told again'); + second.driver.close(); + + fake.workspaces.get(`factory-${thread.id}`).conversations.delete(thread.id); + const third = await harness({ fake, root }); + await assert.rejects(third.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work') }), unavailable(thread.id)); + third.driver.close(); + } finally { await fake.close(); } +}); + +test('after a restart a dead turn is closed and a live one is observed, never replayed', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const fake = await new FakeFlujo({ script: async run => { await gate; run.say('late'); run.done(); } }).listen(); + const root = fresh(); + const first = await harness({ fake, root }); + try { + const { thread } = await started(first); + const { turn } = await first.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => fake.conversationOf(thread.id)?.status === 'running'); + first.driver.close(); + + // FLUJO kept running, so the new driver follows the same turn. + const second = await harness({ fake, root }); + const ended = second.finished(); + const resumed = await second.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(resumed.thread.status.type, 'active'); + release(); + assert.deepEqual(await ended, { id: turn.id, status: 'completed' }); + assert.equal(fake.count('POST', '/v1/chat/completions'), 1); + second.driver.close(); + + // FLUJO died with the Machine: the conversation is terminal, so the retained turn did not finish. + const state = join(root, 'state', `${thread.id}.flujo.json`); + writeFileSync(state, JSON.stringify({ ...JSON.parse(readFileSync(state, 'utf8')), lastTurn: { id: turn.id, status: 'inProgress' } })); + const conversation = fake.conversationOf(thread.id); + conversation.status = 'error'; conversation.recovery.classification = 'interrupted'; + const third = await harness({ fake, root }); + const closed = await third.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(closed.thread.status.type, 'idle'); + assert.deepEqual(closed.recoveredTurn, { id: turn.id, status: 'interrupted' }); + third.driver.close(); + } finally { await fake.close(); } +}); + +test('a durable workflow wake delivers a turn and reads it back', async () => { + const h = await harness(); + try { + const { thread } = await started(h); + const done = h.finished(); + const wake = new NativeWorkflowWake({ rpc: h.driver, threadId: thread.id, file: join(h.root, 'wake.json'), effort: 'medium' }); + const delivered = await wake.wake({ trigger: 'first', input: 'start', initial: true }); + assert.equal(delivered.status, 'delivered'); + const turn = await done; + assert.equal(delivered.turnId, turn.id); + assert.deepEqual(await wake.read(), { status: 'idle', turnId: turn.id }); + } finally { await h.end(); } +}); + +test('a real stop command reaches the board as a verified receipt', async () => { + const p = plan({ agentsPerMachine: 1 }); const agent = agentInventory(p)[0]; + const credentials = { [agent.id]: secret() }; const operatorToken = secret(); const root = fresh(); + const coordinator = await startCloudLiteCoordinator({ database: join(root, 'board.sqlite'), plan: p, credentials, operatorToken, auth: {}, host: '127.0.0.1', port: 0 }); + const url = `http://127.0.0.1:${coordinator.server.address().port}`; + const post = async (path, body, token = operatorToken) => { + const response = await fetch(url + path, { method: 'POST', headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); + assert.equal(response.status, 200); return response.json(); + }; + const h = await harness({ root, script: () => {} }); + try { + const { thread } = await started(h); + await post('/operator/session', { agent: agent.id, session: thread.id }); + const trace = new CloudLiteTrace({ file: join(root, 'trace.jsonl'), url, token: credentials[agent.id] }); + const turns = new Map(), terminal = new Map(); + h.driver.on('notification', ({ method, params }) => { + if (method === 'turn/started') { terminal.delete(thread.id); turns.set(thread.id, params.turn.id); trace.record('native.turn.started', { threadId: thread.id, turn: params.turn }); stop.observeTurnStarted({ threadId: thread.id, turnId: params.turn.id }); } + if (method === 'turn/completed') { terminal.set(thread.id, { id: params.turn.id, status: params.turn.status }); trace.record('native.turn.completed', { threadId: thread.id, turn: { id: params.turn.id, status: params.turn.status } }); stop.observeTerminal({ threadId: thread.id, turnId: params.turn.id, status: params.turn.status }); } + }); + const stop = new NativeOperatorStop({ rpc: h.driver, threadId: thread.id, getTurn: () => nativeStopTurn(turns, terminal, thread.id), trace: (kind, body) => trace.record(kind, body), flush: () => trace.flush(), timeoutMs: 5000 }); + trace.record('native.thread.started', { threadId: thread.id, threadStatus: 'idle' }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long job') }); + await trace.flush(); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await post('/operator/command', { id: 'stop-board', recipient: agent.id, action: 'stop', expectedThreadId: thread.id, expectedTurnId: turn.id }); + const { commands } = await post('/commands', {}, credentials[agent.id]); + const command = commands.find(value => value.id === 'stop-board'); + await stop.stop(command.id, { expectedThreadId: command.expectedThreadId, expectedTurnId: command.expectedTurnId }); + const ack = await post('/commands', { acknowledge: command.id }, credentials[agent.id]); + assert.equal(ack.acknowledged, true); + const native = coordinator.board.snapshot().agents.find(value => value.id === agent.id).nativeRuntime; + assert.equal(native.threadStatus, 'idle'); + assert.equal(native.turnStatus, 'interrupted'); + assert.equal(native.operatorStopReceipt.commandId, 'stop-board'); + assert.equal(native.operatorStopReceipt.threadId, thread.id); + assert.equal(native.operatorStopReceipt.turnId, turn.id); + } finally { await h.end(); await coordinator.close(); } +}); + +test('no stop receipt is issued while FLUJO still reports work', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + const turns = new Map(), terminal = new Map(); + let stop; + h.driver.on('notification', ({ method, params }) => { + if (method === 'turn/started') turns.set(thread.id, params.turn.id); + if (method === 'turn/completed') { terminal.set(thread.id, { id: params.turn.id, status: params.turn.status }); stop.observeTerminal({ threadId: thread.id, turnId: params.turn.id, status: params.turn.status }); } + }); + const receipts = []; + stop = new NativeOperatorStop({ rpc: h.driver, threadId: thread.id, getTurn: () => nativeStopTurn(turns, terminal, thread.id), trace: (kind, body) => receipts.push({ kind, body }), flush: async () => {}, timeoutMs: 60 }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + h.fake.holdCancel = true; + await assert.rejects(stop.stop('stop-held', { expectedThreadId: thread.id, expectedTurnId: turn.id }), /terminal notification/); + assert.equal(receipts.some(entry => entry.kind === 'native.operator.stop.completed'), false); + } finally { await h.end(); } +}); + +const freePort = () => new Promise(resolve => { const probe = createServer(); probe.listen(0, '127.0.0.1', () => { const { port } = probe.address(); probe.close(() => resolve(port)); }); }); +const programOptions = async (env = {}) => ({ command: process.execPath, args: [fakeProgram], port: await freePort(), environment: () => ({ ...process.env, ...env }), startTimeoutMs: 15000 }); + +test('the FLUJO process starts, restarts a few times if it dies, then reports the failure', async () => { + const program = new FlujoProgram({ ...await programOptions({ FAKE_FLUJO_CRASH_AFTER: '2000' }), restartLimit: 2, restartWindowMs: 60000 }); + const seen = []; + program.on('restarting', event => seen.push(`restarting ${event.restarts}`)); + program.on('restarted', event => seen.push(`restarted ${event.restarts}`)); + const exited = new Promise(resolve => program.on('exit', resolve)); + try { + await program.start(); + const result = await exited; + assert.equal(result.code, 7); + assert.deepEqual(seen, ['restarting 1', 'restarted 1', 'restarting 2', 'restarted 2', 'restarting 3']); + } finally { await program.stop(); } +}); + +test('a FLUJO that cannot start is reported, and a stop ends it', async () => { + const broken = new FlujoProgram(await programOptions({ FAKE_FLUJO_EXIT_AT_START: '3' })); + await assert.rejects(broken.start(), /exited before it was ready \(3\)/); + const program = new FlujoProgram(await programOptions()); + await program.start(); + const exited = new Promise(resolve => program.on('exit', resolve)); + await program.stop(); + await exited; + assert.ok(program.child.exitCode !== null || program.child.signalCode !== null); +}); + +test('a driver owns its FLUJO process: it starts it, reports its version and ends with it', async () => { + const root = fresh(); + const driver = new FlujoSessionDriver({ program: new FlujoProgram(await programOptions()), stateDirectory: join(root, 'state'), modelRecord: () => flujoProviderModel(provider(), { key }), pollMs: 20 }); + const exited = new Promise(resolve => driver.on('exit', resolve)); + await driver.initialize(); + assert.equal(await driver.version(), '9.9.9'); + mkdirSync(join(root, 'work')); + const { thread } = await driver.request('thread/start', { cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(thread.status.type, 'idle'); + driver.close(); + await exited; +}); + +test('evidence from a FLUJO worker holds its conversations and none of its keys', () => { + const root = fresh(), space = join(root, 'workspaces', 'factory-thread', 'db'); + mkdirSync(join(space, 'conversations'), { recursive: true }); mkdirSync(join(space, 'conversation-logs')); mkdirSync(join(root, 'workspaces', 'factory-thread', 'mcp-servers')); + writeFileSync(join(space, 'conversations', 'thread.json'), '{}'); writeFileSync(join(space, 'conversation-logs', 'thread.jsonl'), '{}\n'); + for (const name of ['encryption_key.json', 'models.json', 'mcp_servers.json']) writeFileSync(join(space, name), 'private'); + writeFileSync(join(root, 'workspaces', 'factory-thread', 'mcp-servers', 'server.js'), 'code'); + const store = new FactoryArtifactStore(join(fresh(), 'objects')); + const exclude = ['encryption_key.json', 'models.json', 'mcp_servers.json', 'mcp-servers', 'browser-profile', 'bash-utils', 'screenshots', 'recordings', 'snapshots', '.runtime-locks', 'node_modules']; + const captured = captureWorkspace(store, join(root, 'workspaces'), { exclude }); + const names = readWorkspaceManifest(store, captured.source.snapshot).files.map(file => file.path).sort(); + assert.deepEqual(names, ['factory-thread/db/conversation-logs/thread.jsonl', 'factory-thread/db/conversations/thread.json']); + // The list above is the one the worker uses. + const runtime = readFileSync(new URL('../src/cloud-lite/runtime.mjs', import.meta.url), 'utf8'); + assert.ok(runtime.includes(`sessionsExclude: ${JSON.stringify(exclude).replaceAll('"', "'").replaceAll(',', ', ')}`)); +}); + +test('a FLUJO fleet stages the key only as a secret, needs a provider and keeps its program on updates', async () => { + const root = fresh(); const calls = []; const machines = []; + const infrastructure = { + run: async (args, options = {}) => { calls.push({ args, input: options.input }); if (args[0] === 'volumes') return JSON.stringify({ id: 'vol_1' }); if (args[0] === 'machine' && args[1] === 'list') return JSON.stringify(machines.map(machine => ({ id: machine.id, name: machine.name, state: 'stopped', config: machine.config }))); return ''; }, + buildMachineConfig: () => ({ init: { cmd: ['upstream'] }, services: [] }), + createMachine: async ({ name, config }) => { const machine = { id: `machine-${machines.length}`, name, config }; machines.push(machine); return machine; }, + }; + const bundlePath = join(root, 'bundle.tgz'); writeFileSync(bundlePath, 'bundle'); + const image = 'ghcr.io/example/worker@sha256:' + 'a'.repeat(64); + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + await assert.rejects(fleet.deploy({ runId: 'flujo-test', app: 'factory-flujo-test', goal: 'x', workerProgram: 'flujo' }, { image, bundlePath }), /subscription auth broker \(or a model provider\)/); + await assert.rejects(fleet.deploy(plan(), { image, bundlePath }), /needs its API key/); + const result = await fleet.deploy(plan({ machines: 1 }), { image, bundlePath, providerKey: key }); + assert.equal(result.machines.length, 1); + const staged = calls.find(call => call.args[0] === 'secrets'); + const config = JSON.parse(Buffer.from(staged.input.trim().split('=').slice(1).join('='), 'base64').toString()); + assert.equal(config.providerKey, key); + assert.equal(config.plan.workerProgram, 'flujo'); + assert.equal(readFileSync(join(root, 'fleet', 'fleet.json'), 'utf8').includes(key), false); + assert.match(readFileSync(join(root, 'fleet', 'bootstrap.sh'), 'utf8'), /flujo-ai-/); + assert.equal(machines[0].config.metadata['factory.runtime'], 'flujo-lite'); + assert.equal(JSON.stringify(machines[0].config).includes(key), false); + // A runtime update must not switch the fleet to another program. + writeFileSync(bundlePath, 'bundle 2'); + try { await fleet.updateRuntime({ bundlePath }); } catch { /* The fake host has no update path; the script is what matters here. */ } + const updated = readFileSync(join(root, 'fleet', 'fleet.json'), 'utf8'); + assert.ok(updated); + const digest = (await import('node:crypto')).createHash('sha256').update('bundle 2').digest('hex'); + assert.match(readFileSync(join(root, 'fleet', `bootstrap-${digest}.sh`), 'utf8'), /flujo-ai-/); +}); + +test('an app can pack the SDK from its own dependency, and the same files give the same bytes', () => { + const root = fresh(), first = join(root, 'first.tgz'), second = join(root, 'nested', 'second.tgz'); + const packed = packFactorySdk(first); + assert.ok(packed.files > 50); + packFactorySdk(second); + assert.deepEqual(readFileSync(first), readFileSync(second)); + const unpacked = join(root, 'unpacked'); mkdirSync(unpacked); + // The bootstrap unpacks it the same way, and Git's tar on Windows reads a drive letter as a host. + const relative = spawnSync('tar', ['-xzf', 'first.tgz', '--strip-components=1', '-C', 'unpacked'], { cwd: root, encoding: 'utf8' }); + assert.equal(relative.status, 0, relative.stderr); + const sdk = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')); + assert.equal(JSON.parse(readFileSync(join(unpacked, 'package.json'), 'utf8')).version, sdk.version); + for (const file of ['bin/factory-cloud-runtime.mjs', 'bin/factory-cloud-mcp.mjs', 'src/cloud-lite/runtime.mjs', 'src/cloud-lite/flujo-driver.mjs']) assert.equal(existsSync(join(unpacked, file)), true, file); + assert.equal(existsSync(join(unpacked, 'node_modules')), false); + assert.equal(existsSync(join(unpacked, 'test')), false); + assert.ok(readdirSync(join(unpacked, 'src')).length > 10); + assert.deepEqual(readFileSync(join(unpacked, 'src', 'cloud-lite', 'bundle.mjs')), readFileSync(new URL('../src/cloud-lite/bundle.mjs', import.meta.url))); + if (process.platform !== 'win32') assert.equal(statSync(join(unpacked, 'bin', 'factory-cloud-runtime.mjs')).mode & 0o755, 0o755); + assert.throws(() => packFactorySdk(), /where to write/); +}); + +test('a fleet can have its own private network', async () => { + assert.throws(() => plan({ network: 'Bad Name' }), /Invalid network/); + const root = fresh(), calls = []; + const infrastructure = { run: async args => { calls.push(args); return ''; } }; + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + await fleet.prepare(plan({ org: 'my-org', network: 'goal-network' })); + assert.deepEqual(calls[0], ['apps', 'create', 'factory-flujo-test', '--org', 'my-org', '--network', 'goal-network']); + const other = new FactoryCloudLiteFleet({ directory: join(root, 'other'), infrastructure }); + await other.prepare(plan({ org: 'my-org' })); + assert.deepEqual(calls[1], ['apps', 'create', 'factory-flujo-test', '--org', 'my-org']); +}); + +test('a local FLUJO worker attaches, installs the template and the board, and keeps its conversation', async () => { + const fake = await new FakeFlujo().listen(); + const root = fresh(); + const options = { provider: 'flujo', database: join(root, 'board.sqlite'), cwd: root, origin: fake.origin, modelProvider: provider({ apis: ['chat'] }), providerKey: key, stateDirectory: join(root, 'state'), teamConcurrency: 2, agentId: 'builder' }; + try { + const spec = liteWorkerCommand({ ...options, port: 4321, dataDirectory: join(root, 'data'), env: { PATH: '/bin' } }); + assert.equal(spec.command, 'flujo'); + assert.deepEqual(spec.args, ['--no-open', '--port', '4321']); + assert.deepEqual(spec.env, { PATH: '/bin', FLUJO_DATA_DIR: join(root, 'data'), FLUJO_PORT: '4321' }); + assert.ok(spec.server.args.includes('--actor')); + assert.throws(() => liteWorkerCommand({ provider: 'other', database: 'x' }), /codex, claude or flujo/); + await assert.rejects(startLiteWorker({ provider: 'flujo', database: join(root, 'board.sqlite'), origin: fake.origin, stateDirectory: join(root, 'other') }), /modelProvider or a flujoModel/); + await assert.rejects(startLiteWorker({ ...options, providerKey: 'short' }), /API key/); + + const worker = await startLiteWorker({ ...options, prompt: 'Work on the board' }); + assert.equal(worker.origin, fake.origin); + assert.equal(worker.child, null, 'nothing is started when attaching'); + assert.equal((await worker.settled()).status, 'completed'); + const space = fake.workspaces.get(worker.workspace); + assert.equal(space.models[0].adapter, 'openai'); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent', 'swarm_team']); + const board = space.servers.find(server => server.name === 'factory'); + assert.equal(board.args.includes('--config'), false, 'a local board has no credential to hide'); + assert.ok(board.args.includes('builder')); + const threadId = worker.threadId; + await worker.stop(); + + const again = await startLiteWorker({ ...options }); + assert.equal(again.threadId, threadId, 'the agent keeps its conversation across runs'); + await again.send('More work'); + assert.equal((await again.settled()).status, 'completed'); + assert.equal(fake.conversationOf(threadId).messages.filter(message => message.role === 'user').length, 2); + await again.stop(); + assert.equal(existsSync(join(root, 'state', 'session.json')), true); + } finally { await fake.close(); } +}); diff --git a/test/fixtures/fake-flujo.mjs b/test/fixtures/fake-flujo.mjs new file mode 100644 index 0000000..01bec5e --- /dev/null +++ b/test/fixtures/fake-flujo.mjs @@ -0,0 +1,197 @@ +#!/usr/bin/env node +// A small stand-in for a FLUJO server: the HTTP routes the worker driver uses, with runs a test can script. +// It behaves as the real program did in the check against FLUJO 3.48.3: a run outlives the connection that +// asked for it, a cancel is a request that takes a moment, and terminal state comes from the conversation. +import http from 'node:http'; +import { randomUUID } from 'node:crypto'; +import { once } from 'node:events'; +import { pathToFileURL } from 'node:url'; + +const ACTIVE = new Set(['running', 'awaiting_tool_approval', 'paused_debug']); + +export class FakeFlujo { + constructor({ tools = { bash: ['run'], filesystem: ['read_file'] }, script } = {}) { + this.tools = tools; + // Called for each new run with (run, api); by default a run answers and ends. + this.script = script ?? (run => { run.say('answer'); run.done(); }); + this.workspaces = new Map(); + // Like FLUJO's event bus, a stream can attach before its conversation exists. + this.conversationStreams = new Map(); + this.requests = []; + this.streams = new Set(); + this.refuse = []; + this.dropAfterAccept = 0; + this.cancelFails = 0; + this.server = http.createServer((request, response) => { void this.route(request, response).catch(error => { response.writeHead(500); response.end(JSON.stringify({ error: error.message })); }); }); + } + + async listen(port = 0) { this.server.listen(port, '127.0.0.1'); await once(this.server, 'listening'); this.origin = `http://127.0.0.1:${this.server.address().port}`; return this; } + async close() { for (const stream of this.streams) stream.destroy(); this.server.closeAllConnections(); await new Promise(resolve => this.server.close(resolve)); } + count(method, path) { return this.requests.filter(entry => entry.method === method && entry.path.startsWith(path)).length; } + conversationOf(id) { for (const space of this.workspaces.values()) if (space.conversations.has(id)) return space.conversations.get(id); return null; } + + space(name) { + if (!this.workspaces.has(name)) this.workspaces.set(name, { name, roots: [], models: [], flows: [], conversations: new Map(), + servers: [...Object.keys(this.tools), 'flujo'].map(server => ({ name: server, disabled: false, transport: 'stdio' })) }); + return this.workspaces.get(name); + } + + json(response, status, body) { response.writeHead(status, { 'content-type': 'application/json' }); response.end(JSON.stringify(body)); } + + async route(request, response) { + const url = new URL(request.url, 'http://fake'); + const path = url.pathname, method = request.method; + const chunks = []; for await (const chunk of request) chunks.push(chunk); + const body = chunks.length ? JSON.parse(Buffer.concat(chunks).toString('utf8')) : undefined; + this.requests.push({ method, path, body, workspace: url.searchParams.get('workspace') }); + const workspace = url.searchParams.get('workspace') ?? 'default-workspace'; + if (path === '/api/workspaces') { + if (method === 'GET') return this.json(response, 200, { workspaces: [...this.workspaces.keys()].map(name => ({ name })) }); + if (method === 'POST') { this.space(body.name); return this.json(response, 201, { workspace: { name: body.name } }); } + if (method === 'PATCH') { this.space(body.name).roots = body.roots; return this.json(response, 200, { workspace: { name: body.name, roots: body.roots } }); } + if (method === 'DELETE') { this.workspaces.delete(body.name); return this.json(response, 200, { deleted: body.name }); } + } + const space = this.space(workspace); + if (path === '/api/init') return this.json(response, 200, { success: true }); + if (path === '/api/update') return this.json(response, 200, { currentVersion: '9.9.9' }); + if (path === '/api/model') { + if (method === 'GET') return this.json(response, 200, space.models); + space.models.push(body); return this.json(response, 201, { ...body, ApiKey: '********' }); + } + if (path.startsWith('/api/model/')) { space.models = space.models.map(model => model.id === body.id ? body : model); return this.json(response, 200, body); } + if (path === '/api/mcp/servers') { + if (method === 'GET') return this.json(response, 200, space.servers); + space.servers.push(body); return this.json(response, 201, body); + } + const toolRoute = /^\/api\/mcp\/servers\/([^/]+)\/tools$/.exec(path); + if (toolRoute) { + const name = decodeURIComponent(toolRoute[1]); + if (name === 'factory' && this.boardFails) return this.json(response, 200, { tools: [], error: 'not connected' }); + return this.json(response, 200, { tools: (name === 'factory' ? ['factory_status'] : this.tools[name] ?? []).map(tool => ({ name: tool })) }); + } + if (path.startsWith('/api/mcp/servers/')) { + const name = decodeURIComponent(path.split('/').at(-1)); + space.servers = space.servers.map(server => server.name === name ? { ...server, ...body } : server); + return this.json(response, 200, body); + } + if (path === '/api/flow') return this.json(response, 200, space.flows); + if (path === '/api/flow/compile') { + const flow = { id: randomUUID(), name: body.spec.name, spec: body.spec }; + space.flows.push(flow); return this.json(response, 201, { flow }); + } + if (path.startsWith('/api/flow/')) { space.flows = space.flows.filter(flow => flow.id !== path.split('/').at(-1)); return this.json(response, 200, { success: true }); } + if (path === '/v1/chat/completions') return this.complete(space, body, response); + if (path === '/v1/chat/conversations') { + const parent = url.searchParams.get('descendantsOf'); + const items = [...space.conversations.values()].filter(conversation => conversation.parentConversationId === parent).map(({ id, status, parentConversationId }) => ({ id, status, parentConversationId })); + return this.json(response, 200, { items, total: items.length, hasMore: false }); + } + const route = /^\/v1\/chat\/conversations\/([^/]+)(?:\/([a-z]+))?$/.exec(path); + if (route) return this.conversationRoute(space, route[1], route[2], { method, body, url, response }); + return this.json(response, 404, { error: `Unknown route ${method} ${path}` }); + } + + publish(conversation, event) { + const record = { conversationId: conversation.id, seq: conversation.nextSeq++, timestamp: Date.now(), ...event }; + conversation.events.push(record); + for (const stream of this.conversationStreams.get(conversation.id) ?? []) stream.write(`id: ${record.seq}\ndata: ${JSON.stringify(record)}\n\n`); + return record; + } + + makeRun(conversation) { + const emit = event => this.publish(conversation, event); + const end = (status, { error, classification } = {}) => { + if (run.ended) return; + run.ended = true; + conversation.status = status; + conversation.recovery.classification = classification ?? (status === 'completed' ? 'completed' : status === 'capped' ? 'capped' : 'permanent_failure'); + if (error) conversation.lastError = error; else delete conversation.lastError; + emit({ type: 'run:done', status: status === 'capped' ? 'capped' : status === 'completed' ? 'completed' : 'error' }); + }; + const run = { conversation, ended: false, emit, + say: (text, extra = {}) => { const message = { id: randomUUID(), role: 'assistant', content: text }; conversation.messages.push(message); emit({ type: 'message', message, ...extra }); }, + tool: (name, args = '{}') => { const id = randomUUID(); emit({ type: 'message', message: { id: randomUUID(), role: 'assistant', content: '', tool_calls: [{ id, type: 'function', function: { name, arguments: args } }] } }); emit({ type: 'tool:result', toolCallId: id, name, result: 'ok', isError: false }); }, + usage: () => emit({ type: 'usage', promptTokens: 40, completionTokens: 4, totalTokens: 44, cacheReadTokens: 10 }), + done: () => end('completed'), + capped: () => end('capped'), + fail: (httpStatus = 500) => end('error', { error: { message: 'failed', httpStatus } }), + // End without the final event, as a lost stream would see it. + endSilently: status => { run.ended = true; conversation.status = status; conversation.recovery.classification = status === 'completed' ? 'completed' : 'permanent_failure'; }, + child: status => { const id = randomUUID(); const child = { id, status, parentConversationId: conversation.id, messages: [], events: [], nextSeq: 0, recovery: {} }; conversation.space.conversations.set(id, child); return child; }, + }; + conversation.run = run; + return run; + } + + async complete(space, body, response) { + const refusal = this.refuse.shift(); + if (refusal) return this.json(response, refusal, { error: { message: 'busy' } }); + const id = body.metadata.conversationId; + const conversation = space.conversations.get(id) ?? { id, space, messages: [], events: [], nextSeq: 0, parentConversationId: null }; + space.conversations.set(id, conversation); + conversation.messages.push({ id: randomUUID(), role: 'user', content: body.messages.at(-1).content }); + conversation.flow = body.model; + conversation.status = 'running'; + conversation.recovery = { runId: randomUUID(), classification: 'running' }; + conversation.steered = []; + const run = this.makeRun(conversation); + run.emit({ type: 'run:start', flowId: body.model }); + // A dropped connection: the run was accepted, but the caller never hears so. + if (this.dropAfterAccept > 0) { this.dropAfterAccept--; response.destroy(); } + else { response.writeHead(200, { 'content-type': 'text/event-stream' }); response.write('data: {}\n\n'); } + setImmediate(() => { void this.script(run, this); }); + } + + conversationRoute(space, id, action, { method, body, url, response }) { + const conversation = space.conversations.get(id); + if (action === 'events') { + response.writeHead(200, { 'content-type': 'text/event-stream' }); + response.write('retry: 3000\n\n: connected\n\n'); + const from = Number(url.searchParams.get('fromSeq') ?? 0); + for (const record of conversation?.events ?? []) if (record.seq >= from) response.write(`id: ${record.seq}\ndata: ${JSON.stringify(record)}\n\n`); + if (!this.conversationStreams.has(id)) this.conversationStreams.set(id, new Set()); + const listeners = this.conversationStreams.get(id); + listeners.add(response); + response.on('close', () => listeners.delete(response)); + this.streams.add(response); + response.on('close', () => this.streams.delete(response)); + return; + } + if (!conversation) { + if (action === 'inject') return this.json(response, 409, { error: 'Conversation is not running', reason: 'not_running' }); + if (action === 'cancel') return this.json(response, 200, { success: true, message: 'Conversation not found, assumed cancelled.' }); + return this.json(response, 404, { error: 'Conversation not found' }); + } + if (action === 'inject') { + if (!ACTIVE.has(conversation.status)) return this.json(response, 409, { error: 'Conversation is not running', reason: 'not_running' }); + const message = { id: body.id, role: 'user', content: body.content }; + conversation.steered.push(message); + conversation.run?.onSteer?.(message); + return this.json(response, 200, { status: 'queued', conversation_id: id, message_id: body.id }); + } + if (action === 'cancel') { + if (this.cancelFails > 0) { this.cancelFails--; return this.json(response, 500, { error: 'cannot stop now' }); } + conversation.recovery.classification = 'cancelled'; + this.cancels = (this.cancels ?? 0) + 1; + const run = conversation.run; + if (run && !run.ended && !this.holdCancel) setTimeout(() => { if (run.ended) return; run.ended = true; conversation.status = 'error'; conversation.lastError = { code: 'cancelled' }; run.emit({ type: 'run:done', status: 'error' }); }, 5); + return this.json(response, 200, { success: true }); + } + if (action === 'respond') { this.responses = [...(this.responses ?? []), body]; return this.json(response, 200, { status: 'running' }); } + if (method === 'GET') { + const { id: identity, status, recovery, lastError, parentConversationId, messages } = conversation; + return this.json(response, 200, { id: identity, status, recovery, ...(lastError ? { lastError } : {}), parentConversationId, messages }); + } + return this.json(response, 404, { error: 'Unknown action' }); + } +} + +// Run as a program it stands in for the `flujo` command: `fake-flujo.mjs --no-open --port `. +if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) { + const port = Number(process.argv[process.argv.indexOf('--port') + 1]); + if (process.env.FAKE_FLUJO_EXIT_AT_START) process.exit(Number(process.env.FAKE_FLUJO_EXIT_AT_START)); + const fake = await new FakeFlujo().listen(port); + process.stdout.write(`listening ${fake.origin}\n`); + if (process.env.FAKE_FLUJO_CRASH_AFTER) setTimeout(() => process.exit(7), Number(process.env.FAKE_FLUJO_CRASH_AFTER)); + process.on('SIGTERM', () => process.exit(0)); +} diff --git a/test/model-provider.test.mjs b/test/model-provider.test.mjs index e5809fa..febc171 100644 --- a/test/model-provider.test.mjs +++ b/test/model-provider.test.mjs @@ -15,9 +15,10 @@ const endpoint = (extra = {}) => ({ id: 'selfhosted', name: 'Self-hosted model', contextWindow: 262144, autoCompactTokenLimit: 100000, ...extra }); test('one provider description states which worker programs can use it', () => { - assert.deepEqual(providerPrograms(endpoint()), ['codex', 'claude']); - assert.deepEqual(providerPrograms(endpoint({ apis: undefined })), ['codex'], '0.4.0 descriptions keep working for Codex'); - assert.deepEqual(providerPrograms(endpoint({ apis: ['anthropic'] })), ['claude']); + assert.deepEqual(providerPrograms(endpoint()), ['codex', 'claude', 'flujo']); + assert.deepEqual(providerPrograms(endpoint({ apis: undefined })), ['codex', 'flujo'], '0.4.0 descriptions keep working for Codex'); + assert.deepEqual(providerPrograms(endpoint({ apis: ['anthropic'] })), ['claude', 'flujo']); + assert.deepEqual(providerPrograms(endpoint({ apis: ['chat'] })), ['flujo']); for (const extra of [{ apis: [] }, { apis: ['grpc'] }, { envKey: 'ANTHROPIC_AUTH_TOKEN' }, { requestTimeoutMs: 10 }]) assert.throws(() => validateModelProvider(endpoint(extra)), TypeError, JSON.stringify(extra)); assert.throws(() => validateCloudLitePlan({ runId: 'claude-only', app: 'claude-only-app', goal: 'x', modelProvider: endpoint({ apis: ['anthropic'] }) }), /responses API/); }); diff --git a/test/public-sdk.test.mjs b/test/public-sdk.test.mjs index 27a9c9e..42249c9 100644 --- a/test/public-sdk.test.mjs +++ b/test/public-sdk.test.mjs @@ -46,7 +46,7 @@ test('lite workers launch native CLIs with only the Factory MCP configuration', assert.ok(claude.args.includes('--dangerously-skip-permissions')); assert.ok(claude.args.includes('--strict-mcp-config')); assert.deepEqual(Object.keys(JSON.parse(claude.args[claude.args.indexOf('--mcp-config') + 1]).mcpServers), ['factory']); - assert.throws(() => liteWorkerCommand({ provider: 'flujo', database: 'board.sqlite' }), /provider/); + assert.throws(() => liteWorkerCommand({ provider: 'other', database: 'board.sqlite' }), /provider/); }); test('public SDK starts a lite worker through the same launcher as the CLI', async t => { diff --git a/test/selfhost-modal.test.mjs b/test/selfhost-modal.test.mjs index f2447ea..ca7c485 100644 --- a/test/selfhost-modal.test.mjs +++ b/test/selfhost-modal.test.mjs @@ -11,8 +11,9 @@ const fresh = () => tempDir('factory-selfhost-'); const read = (path) => readFileSync(path, 'utf8'); // A fake Modal account: the deploy script, the Modal CLI and the served model. -function fixture({ deployError, describeError, holdDescribe, workspace = 'fixture-workspace', completion, redirect, catalogueBytes } = {}) { +function fixture({ deployError, describeError, holdDescribe, workspace = 'fixture-workspace', completion, redirect, catalogueBytes, stoppingReads = 0, refusedDeletes = 0, stopFor = Infinity, modelOptions = {} } = {}) { const calls = [], requests = [], apps = new Map(), secrets = new Set(), volumes = new Set(); + let reads = stoppingReads, refusals = refusedDeletes; let key, appName, account = workspace; const runner = async (command, args, options) => { calls.push({ command, args, input: options.input, env: options.env }); @@ -40,9 +41,15 @@ function fixture({ deployError, describeError, holdDescribe, workspace = 'fixtur } const cli = args.slice(args.indexOf('modal') + 1); if (cli[0] === 'profile') return { code: 0, stdout: JSON.stringify([{ name: 'fixture', workspace: account, active: true }]) }; - if (cli[0] === 'app' && cli[1] === 'stop') { const known = apps.has(cli.at(-1)); if (known) apps.set(cli.at(-1), 'stopped'); return { code: known ? 0 : 1, stdout: '' }; } - if (cli[0] === 'app' && cli[1] === 'list') return { code: 0, stdout: JSON.stringify([...apps].map(([description, state]) => ({ description, state }))) }; - if (cli[1] === 'delete') { (cli[0] === 'secret' ? secrets : volumes).delete(cli.at(-1)); return { code: 0, stdout: '' }; } + if (cli[0] === 'app' && cli[1] === 'stop') { const known = apps.has(cli.at(-1)); if (known) apps.set(cli.at(-1), reads > 0 ? 'stopping' : 'stopped'); return { code: known ? 0 : 1, stdout: '' }; } + if (cli[0] === 'app' && cli[1] === 'list') { + if (reads > 0 && --reads === 0 && stopFor === Infinity) for (const [name, state] of apps) if (state === 'stopping') apps.set(name, 'stopped'); + return { code: 0, stdout: JSON.stringify([...apps].map(([description, state]) => ({ description, state }))) }; + } + if (cli[1] === 'delete') { + if (refusals > 0) { refusals--; return { code: 1, stdout: '' }; } + (cli[0] === 'secret' ? secrets : volumes).delete(cli.at(-1)); return { code: 0, stdout: '' }; + } if (cli[1] === 'list') return { code: 0, stdout: JSON.stringify([...(cli[0] === 'secret' ? secrets : volumes)].map(name => ({ name }))) }; throw new Error(`Unexpected command ${cli.join(' ')}`); }; @@ -55,7 +62,7 @@ function fixture({ deployError, describeError, holdDescribe, workspace = 'fixtur const challenge = body.messages[0].content.match(/FACTORY-[A-F0-9]{8}/)?.[0]; return new Response(JSON.stringify(completion?.(body, challenge) ?? { model: body.model, choices: [{ message: { role: 'assistant', content: challenge }, finish_reason: 'stop' }] })); }; - const model = new ModalSelfHostedModel({ directory: fresh(), python: 'fixture-python', commandRunner: runner, fetchImpl, + const model = new ModalSelfHostedModel({ directory: fresh(), python: 'fixture-python', commandRunner: runner, fetchImpl, stopPollMs: 1, ...modelOptions, env: { PATH: 'fixture-path', FACTORY_SELFHOST_SETTINGS_B64: 'inherited-must-be-dropped' } }); return { model, calls, requests, apps, secrets, volumes, key: () => key, appName: () => appName, setWorkspace: value => { account = value; } }; } @@ -69,6 +76,34 @@ test('bundled profiles carry list-price estimates per running replica', () => { for (const profile of Object.values(profiles)) { assert.equal(profile.contextWindow, 262144); assert.equal(profile.autoCompactTokenLimit, 80000); } }); +test('one retire right after a busy replica finishes: it waits for the stop and retries refused deletes', async () => { + const f = fixture({ stoppingReads: 3, refusedDeletes: 2 }); + await f.model.deploy({ profile: 'qwen' }); + const retired = await f.model.retire(); + assert.equal(retired.phase, 'retired'); assert.equal(retired.cleanupVerified, true); + assert.equal(f.secrets.size, 0); assert.equal(f.volumes.size, 0); + const deletes = f.calls.filter(call => call.args.includes('delete') && call.args.includes('--allow-missing')); + assert.equal(deletes.length, 5, 'three resources, two of the tries refused'); +}); + +test('an app still stopping at the deadline, or a delete that keeps failing, stays uncertain and is never reported as success', async () => { + const stuck = fixture({ stoppingReads: 10000, stopFor: 0, modelOptions: { stopWaitMs: 20 } }); + await stuck.model.deploy({ profile: 'qwen' }); + await assert.rejects(stuck.model.retire(), /uncertain/); + assert.equal(stuck.model.status().phase, 'unknown'); + assert.equal(stuck.secrets.size, 1, 'nothing is deleted while the app may still run'); + const refused = fixture({ refusedDeletes: 100 }); + await refused.model.deploy({ profile: 'qwen' }); + await assert.rejects(refused.model.retire(), /uncertain/); + assert.equal(refused.model.status().phase, 'unknown'); + assert.equal(refused.calls.filter(call => call.args.includes('delete') && call.args.includes('--allow-missing')).length, 5, 'the tries are bounded'); + // The stop path waits too, and a later retire can still finish. + const stopping = fixture({ stoppingReads: 2 }); + await stopping.model.deploy({ profile: 'qwen' }); + assert.equal((await stopping.model.stop()).phase, 'stopped'); + assert.equal((await stopping.model.retire()).phase, 'retired'); +}); + test('deploy, verify, describe, stop, redeploy and retire touch only the owned resources', async () => { const f = fixture(); const ready = await f.model.deploy({ profile: 'qwen', overrides: { replicas: 2 } }); From 4178ba79d24a868a0c6438708f26ad0f34dd104f Mon Sep 17 00:00:00 2001 From: flujo-app <300233937+flujo-app@users.noreply.github.com> Date: Tue, 3 Nov 2026 22:37:16 -0500 Subject: [PATCH 2/4] Quote FLUJO's command on Windows --- src/cloud-lite/flujo-driver.mjs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/cloud-lite/flujo-driver.mjs b/src/cloud-lite/flujo-driver.mjs index 59b71b7..3abbdd4 100644 --- a/src/cloud-lite/flujo-driver.mjs +++ b/src/cloud-lite/flujo-driver.mjs @@ -34,8 +34,10 @@ export class FlujoProgram extends EventEmitter { } spawnProcess() { - // On Windows an npm command is a .cmd file, which only a shell can start. The port is a number, so nothing here needs quoting. - const child = spawn(this.command, [...this.baseArgs, '--no-open', '--port', String(this.port)], { env: this.environment(), stdio: ['ignore', 'pipe', 'pipe'], detached: process.platform !== 'win32', windowsHide: true, shell: process.platform === 'win32' }); + // On Windows an npm command is a .cmd file, which only a shell can start. + const windows = process.platform === 'win32'; + const word = value => windows && /\s/.test(value) ? `"${value}"` : value; + const child = spawn(word(this.command), [...this.baseArgs, '--no-open', '--port', String(this.port)].map(word), { env: this.environment(), stdio: ['ignore', 'pipe', 'pipe'], detached: !windows, windowsHide: true, shell: windows }); this.child = child; for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => this.onOutput?.(chunk.toString())); return child; From b769ad06beddc404617fe52c0d173e3c14315d09 Mon Sep 17 00:00:00 2001 From: flujo-app <300233937+flujo-app@users.noreply.github.com> Date: Tue, 3 Nov 2026 22:49:40 -0500 Subject: [PATCH 3/4] A self-hosted FLUJO model is not OpenAI's; a fleet that stopped part way can be retired --- PUBLIC_SDK.md | 1 + src/cloud-lite/fleet.mjs | 10 +++++++--- src/model-provider.mjs | 11 +++++------ test/cloud-lite-flujo.test.mjs | 28 +++++++++++++++++++++++++++- 4 files changed, 40 insertions(+), 10 deletions(-) diff --git a/PUBLIC_SDK.md b/PUBLIC_SDK.md index 42810cc..6cfd301 100644 --- a/PUBLIC_SDK.md +++ b/PUBLIC_SDK.md @@ -367,6 +367,7 @@ await fleet.deploy({ runId: 'my-flujo-run', app: 'my-flujo-app', goal: ownerGoal against its SHA-256, so a restarted Machine does not install it again. Allow a few minutes on first start. A custom image can provide `flujo` on the path instead. Use a Node image of at least 24.22.1 or 22.23.4, running as root; FLUJO's official image runs as a normal user. + Fly needs the full reference, such as `docker.io/library/node@sha256:...`. - **Sessions:** each agent slot has its own FLUJO workspace with one conversation, so its board identity, model record and tools stay apart. FLUJO runs on port 4210 on the Machine's loopback, or `FACTORY_FLUJO_PORT`. Each workspace runs its own tool servers, so diff --git a/src/cloud-lite/fleet.mjs b/src/cloud-lite/fleet.mjs index 0464200..635d9be 100644 --- a/src/cloud-lite/fleet.mjs +++ b/src/cloud-lite/fleet.mjs @@ -657,9 +657,13 @@ export class FactoryCloudLiteFleet { if (this.state.operations['app.retire']?.state !== 'succeeded') { const inventory = JSON.parse(await this.fly(['machine', 'list', '--app', this.state.plan.app, '--json'])); const volumes = JSON.parse(await this.fly(['volumes', 'list', '--app', this.state.plan.app, '--json'])); - if (inventory.length !== this.state.machines.length || inventory.some(value => !this.state.machines.some(machine => - machine.id === value.id && machine.name === value.name && value.config?.metadata?.['factory.run'] === this.state.plan.runId && value.config?.mounts?.some(mount => mount.volume === machine.volumeId))) || - volumes.length !== this.state.machines.length || volumes.some(value => !this.state.machines.some(machine => machine.volumeId === value.id))) { + // A deployment can stop part way, so what this journal created is the inventory, not the plan's full size. + // A Machine whose creation has an unknown outcome still proves itself by its run label. + const journaledVolumes = new Set(Object.entries(this.state.operations).filter(([key, operation]) => /^m\d+\.volume$/.test(key) && operation.state === 'succeeded').map(([, operation]) => operation.result?.id)); + const uncertainMachines = new Set(Object.entries(this.state.operations).filter(([key, operation]) => /^m\d+\.machine-create$/.test(key) && operation.state !== 'succeeded').map(([key]) => `factory-${this.state.plan.runId}-${key.split('.')[0]}`)); + const labelled = value => value.config?.metadata?.['factory.run'] === this.state.plan.runId; + if (inventory.some(value => !labelled(value) || !(this.state.machines.some(machine => machine.id === value.id && machine.name === value.name && value.config?.mounts?.some(mount => mount.volume === machine.volumeId)) || uncertainMachines.has(value.name))) + || volumes.some(value => !journaledVolumes.has(value.id))) { throw new Error('Retirement requires the exact original Machine and volume inventory; retain resources whose ownership differs.'); } } diff --git a/src/model-provider.mjs b/src/model-provider.mjs index e8ce1d9..3e2f83c 100644 --- a/src/model-provider.mjs +++ b/src/model-provider.mjs @@ -158,11 +158,10 @@ export function claudeProviderEnvironment(provider, { index = 0, key, model } = } // FLUJO keeps a model as a stored record, so its settings differ by adapter rather than by variable. -const FLUJO_ADAPTERS = Object.freeze({ - responses: { provider: 'openai', adapter: 'openai-responses' }, - chat: { provider: 'openai', adapter: 'openai' }, - anthropic: { provider: 'anthropic', adapter: 'anthropic' }, -}); +const FLUJO_ADAPTERS = Object.freeze({ responses: 'openai-responses', chat: 'openai', anthropic: 'anthropic' }); +// FLUJO moves the system message behind the conversation for OpenAI's own prompt cache. Open models' +// chat templates refuse that, so only OpenAI's endpoint may carry the openai name. +const flujoProviderName = (api, base) => api === 'anthropic' ? 'anthropic' : new URL(base).hostname === 'api.openai.com' ? 'openai' : 'litellm'; /** * FLUJO model record for a self-hosted provider, for `POST /api/model`. FLUJO stores the key in its @@ -175,7 +174,7 @@ export function flujoProviderModel(provider, { index = 0, key, model } = {}) { if (typeof served !== 'string' || !served) throw new TypeError('Select the served model name.'); const api = programApi(provider, 'flujo'); const base = providerBaseUrl(provider, index); - return { id: `factory-${provider.id}`, name: served, displayName: provider.name ?? provider.id, ...FLUJO_ADAPTERS[api], + return { id: `factory-${provider.id}`, name: served, displayName: provider.name ?? provider.id, provider: flujoProviderName(api, base), adapter: FLUJO_ADAPTERS[api], // The Anthropic client adds /v1 itself. baseUrl: api === 'anthropic' ? base.replace(/\/+$/, '').replace(/\/v1$/, '') : base, ApiKey: key, supportsTools: true, diff --git a/test/cloud-lite-flujo.test.mjs b/test/cloud-lite-flujo.test.mjs index 7f5cfdc..9ea1558 100644 --- a/test/cloud-lite-flujo.test.mjs +++ b/test/cloud-lite-flujo.test.mjs @@ -67,7 +67,9 @@ test('a plan can choose FLUJO, and refuses what FLUJO cannot do', () => { test('the FLUJO mapping prefers Responses and stores the key in a model record', () => { const responses = flujoProviderModel(provider(), { index: 1, key }); assert.deepEqual([responses.id, responses.name, responses.provider, responses.adapter, responses.baseUrl, responses.ApiKey, responses.contextWindow, responses.compactionThreshold, responses.supportsTools], - ['factory-selfhosted', 'served-model', 'openai', 'openai-responses', 'https://model-b.example.test/v1', key, 262144, 80000, true]); + ['factory-selfhosted', 'served-model', 'litellm', 'openai-responses', 'https://model-b.example.test/v1', key, 262144, 80000, true]); + // FLUJO's OpenAI cache order puts the system message last, which open models refuse. + assert.equal(flujoProviderModel(provider({ baseUrls: ['https://api.openai.com/v1'] }), { key }).provider, 'openai'); const chat = flujoProviderModel(provider({ apis: ['chat'] }), { key }); assert.deepEqual([chat.adapter, chat.baseUrl], ['openai', 'https://model-a.example.test/v1']); const anthropic = flujoProviderModel(provider({ apis: ['anthropic'] }), { key }); @@ -639,6 +641,30 @@ test('a fleet can have its own private network', async () => { assert.deepEqual(calls[1], ['apps', 'create', 'factory-flujo-test', '--org', 'my-org']); }); +test('a fleet that stopped part way can still be retired, and foreign resources still block it', async () => { + const root = fresh(); const calls = []; + let machines = [], volumes = [{ id: 'vol_1' }]; + const infrastructure = { + run: async args => { calls.push(args); if (args[0] === 'volumes' && args[1] === 'create') return JSON.stringify({ id: 'vol_1' }); if (args[0] === 'volumes') return JSON.stringify(volumes); if (args[0] === 'machine') return JSON.stringify(machines); return ''; }, + buildMachineConfig: () => ({ init: { cmd: ['upstream'] }, services: [] }), + createMachine: async () => { throw new Error('refused'); }, + }; + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + const image = 'ghcr.io/example/worker@sha256:' + 'a'.repeat(64); + await assert.rejects(fleet.deploy(plan({ machines: 1 }), { image, providerKey: key }), /refused/); + assert.equal(fleet.snapshot().machines.length, 0); + machines = [{ id: 'other', name: 'someone-else', config: { metadata: { 'factory.run': 'another-run' } } }]; + await assert.rejects(fleet.retire(), /exact original/); + machines = []; volumes = [{ id: 'vol_foreign' }]; + await assert.rejects(fleet.retire(), /exact original/); + // A Machine whose creation looked refused may exist anyway; its name and label prove it is ours. + volumes = [{ id: 'vol_1' }]; + machines = [{ id: 'late', name: 'factory-flujo-test-m0', config: { metadata: { 'factory.run': 'flujo-test' } } }]; + await fleet.retire(); + assert.equal(fleet.snapshot().status, 'retired'); + assert.deepEqual(calls.at(-1), ['apps', 'destroy', 'factory-flujo-test', '--yes']); +}); + test('a local FLUJO worker attaches, installs the template and the board, and keeps its conversation', async () => { const fake = await new FakeFlujo().listen(); const root = fresh(); From de6b2b40fe34ee1d93b396554a1919dd158c52de Mon Sep 17 00:00:00 2001 From: flujo-app <300233937+flujo-app@users.noreply.github.com> Date: Tue, 3 Nov 2026 22:55:59 -0500 Subject: [PATCH 4/4] Say how long a FLUJO worker takes to start --- PUBLIC_SDK.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/PUBLIC_SDK.md b/PUBLIC_SDK.md index 6cfd301..135734a 100644 --- a/PUBLIC_SDK.md +++ b/PUBLIC_SDK.md @@ -364,8 +364,8 @@ await fleet.deploy({ runId: 'my-flujo-run', app: 'my-flujo-app', goal: ownerGoal the installed package, which needs no `npm` or `tar` on the host. `network` in the plan puts the fleet's app on its own private network. - **Install:** the bootstrap installs one pinned `flujo-ai` release on the volume, checked - against its SHA-256, so a restarted Machine does not install it again. Allow a few minutes - on first start. A custom image can provide `flujo` on the path instead. Use a Node image of + against its SHA-256, so a restarted Machine does not install it again. Allow about four + minutes on first start. A custom image can provide `flujo` on the path instead. Use a Node image of at least 24.22.1 or 22.23.4, running as root; FLUJO's official image runs as a normal user. Fly needs the full reference, such as `docker.io/library/node@sha256:...`. - **Sessions:** each agent slot has its own FLUJO workspace with one conversation, so its @@ -374,7 +374,7 @@ await fleet.deploy({ runId: 'my-flujo-run', app: 'my-flujo-app', goal: ownerGoal plan on about 400 MB per agent plus 300 MB for FLUJO. FLUJO restarts by itself up to three times in ten minutes; after that the worker stops, as with the other programs. - **Team:** with `flujo.teamConcurrency` an orchestrator gets the `swarm_team` flow, with one - native gate for its subflows. The others get `swarm_agent`. A turn ends, and a stop receipt + native gate for its subflows, and `flujo.specialists` can give it a role catalog. The others get `swarm_agent`. A turn ends, and a stop receipt is issued, only after every child conversation has ended too. Each conversation lists its children under `/v1/chat/conversations?descendantsOf=`. - **Tools:** `bash` and `filesystem`, plus `browser` if `flujo.servers` lists it and the image