diff --git a/.github/workflows/public-sdk.yml b/.github/workflows/public-sdk.yml index e9d6740..dd1c96f 100644 --- a/.github/workflows/public-sdk.yml +++ b/.github/workflows/public-sdk.yml @@ -24,7 +24,7 @@ jobs: - run: npm ci - run: npm audit --omit=dev --audit-level=high - name: Focused tests, leaving nothing in the temp location - run: node scripts/test-temp-guard.mjs test/cloud-lite.test.mjs test/cloud-lite-claude.test.mjs test/cloud-lite-provider.test.mjs test/model-provider.test.mjs test/selfhost-modal.test.mjs test/public-sdk.test.mjs test/template-catalog.test.mjs test/factory-team-template.test.mjs test/flujo-workspace.test.mjs test/savia-case-plan.test.mjs test/savia-topology.test.mjs test/managed-fleet.test.mjs test/managed-fleet-real-module.test.mjs test/managed-fleet-provider-closure.test.mjs test/provider-cell-retirement.test.mjs test/presentation.test.mjs test/presentation-modal.test.mjs test/presentation-closure.test.mjs test/observatory-client.test.mjs test/workflows.test.mjs test/workflow-native-wake.test.mjs test/workflow-m0-lifecycle.test.mjs test/workflow-m1-portable-artifacts.test.mjs test/workflow-m2-ack-recovery.test.mjs test/workflow-m2-authority-recovery.test.mjs + run: node scripts/test-temp-guard.mjs test/cloud-lite.test.mjs test/cloud-lite-claude.test.mjs test/cloud-lite-flujo.test.mjs test/cloud-lite-provider.test.mjs test/model-provider.test.mjs test/selfhost-modal.test.mjs test/public-sdk.test.mjs test/template-catalog.test.mjs test/factory-team-template.test.mjs test/flujo-workspace.test.mjs test/savia-case-plan.test.mjs test/savia-topology.test.mjs test/managed-fleet.test.mjs test/managed-fleet-real-module.test.mjs test/managed-fleet-provider-closure.test.mjs test/provider-cell-retirement.test.mjs test/presentation.test.mjs test/presentation-modal.test.mjs test/presentation-closure.test.mjs test/observatory-client.test.mjs test/workflows.test.mjs test/workflow-native-wake.test.mjs test/workflow-m0-lifecycle.test.mjs test/workflow-m1-portable-artifacts.test.mjs test/workflow-m2-ack-recovery.test.mjs test/workflow-m2-authority-recovery.test.mjs - name: Self-hosted model settings (Python, offline) run: python3 -B -m unittest discover -s selfhost/modal -p "test_*.py" - name: Install and exercise the packed public package diff --git a/PUBLIC_SDK.md b/PUBLIC_SDK.md index 7f309e9..135734a 100644 --- a/PUBLIC_SDK.md +++ b/PUBLIC_SDK.md @@ -1,17 +1,17 @@ # Public local Factory SDK, CLI, and MCP -Version 0.6.0 is distributed as a versioned package in -[GitHub Releases](https://github.com/flujo-app/FACTORY/releases/tag/v0.6.0): +Version 0.7.0 is distributed as a versioned package in +[GitHub Releases](https://github.com/flujo-app/FACTORY/releases/tag/v0.7.0): ```sh -npm install https://github.com/flujo-app/FACTORY/releases/download/v0.6.0/flujo-factory-0.6.0.tgz +npm install https://github.com/flujo-app/FACTORY/releases/download/v0.7.0/flujo-factory-0.7.0.tgz ``` FACTORY can serve an open-weight model on your own Modal GPUs and run workers on it. `ModalSelfHostedModel` and `factory-selfhost` deploy, verify, stop and retire a vLLM endpoint with Qwen3.8-27B (H100) or Nemotron 3.5 Super (B300) profiles; see [selfhost/README.md](selfhost/README.md). Its endpoint description is a model provider that -cloud and local lite workers (Codex or Claude Code) use directly. See +cloud and local lite workers (Codex, Claude Code or FLUJO) use directly. See [Self-hosted models and operator-driven jobs](#self-hosted-models-and-operator-driven-jobs) for the provider, read-only deploy keys, prepare hooks, operator-driven jobs and quieter traces added in 0.4.0. All of it is optional; existing plans behave as before, except that @@ -137,7 +137,7 @@ local overrides are not implicitly shared with a remote adapter. ### Lite workers `factory-lite` opens a regular interactive Codex or Claude CLI session in the -current directory. It does not provision a FLUJO worker or use the FLUJO +current directory, or runs a FLUJO worker (see below). A CLI session does not provision a FLUJO worker or use the FLUJO conversation harness. The CLI retains its built-in tools and unrestricted file and command access. The only MCP server supplied for that session is FACTORY's local board/swarm server, connected to the specified durable database. @@ -201,6 +201,23 @@ description the cloud plan uses (see command line, `factory-lite claude board.sqlite --model-provider endpoint.json` reads the key from the variable the file names in `envKey`. +A local FLUJO worker has no terminal session. It attaches to a running FLUJO, or starts +its own with a separate data folder, installs the generic team template with the board +as a tool server, and keeps its conversation across runs: + +```js +const worker = await factory.startLiteWorker({ provider: 'flujo', modelProvider, providerKey, + origin: 'http://127.0.0.1:4200', prompt: 'Work on the Factory board' }); +const turn = await worker.settled(); +await worker.send('Next job'); +await worker.stop(); +``` + +From the command line: `factory-lite flujo board.sqlite --model-provider endpoint.json +[--origin http://127.0.0.1:4200] "Work on the Factory board"`. Without `--origin`, the +`flujo` command from `npm install -g flujo-ai` must be on the path. A record from +`POST /api/model` can stand in for a provider as `flujoModel`. + ## FLUJO worker dispatch ### Cloud lite workers (development) @@ -320,6 +337,71 @@ await selfHostedFleet.deploy({ ...plan, workerProgram: 'claude', model: provider `thread/tokenUsage/updated`, and model retries become `model/retry`. Streaming deltas are skipped unless `traceDeltas` is on. +#### FLUJO as the worker program + +Set `workerProgram: 'flujo'` to run a FLUJO server on each machine. The board, operator +commands and stop receipts, durable traces, prepare hooks, evidence collection and +verified workflows are the same shared code as for Codex and Claude Code. + +```js +const fleet = new FactoryCloudLiteFleet({ directory: '/absolute/private/fleet-run', + flujoCloudPath: '/absolute/installed/flujo-cloud' }); +await fleet.deploy({ runId: 'my-flujo-run', app: 'my-flujo-app', goal: ownerGoal, + workerProgram: 'flujo', machines: 3, agentsPerMachine: 1, + modelProvider: model.endpoint(), + // Optional: each orchestrator leads a team of native FLUJO subflows. + flujo: { teamConcurrency: 4, agentTurns: 200, leadTurns: 600 }, +}, { image: immutableNodeImage, bundlePath: '/absolute/private/flujo-factory.tgz', providerKey: await model.key() }); +``` + +- **Model:** a plan needs a `modelProvider`; FLUJO workers have no subscription login. It + can offer `responses`, `chat` or `anthropic`, and FLUJO uses the first of those that the + provider has. `flujoProviderModel` is the exported mapping. FLUJO stores the key in its + own encrypted settings, so the FLUJO process and the agent commands it starts never have + it in their environment. `reasoningEffort` does not apply: FLUJO accepts it only for + models it knows by name. +- **Bundle:** `bundlePath` can be a release file, or what `packFactorySdk(file)` writes from + the installed package, which needs no `npm` or `tar` on the host. `network` in the plan + puts the fleet's app on its own private network. +- **Install:** the bootstrap installs one pinned `flujo-ai` release on the volume, checked + against its SHA-256, so a restarted Machine does not install it again. Allow about four + minutes on first start. A custom image can provide `flujo` on the path instead. Use a Node image of + at least 24.22.1 or 22.23.4, running as root; FLUJO's official image runs as a normal user. + Fly needs the full reference, such as `docker.io/library/node@sha256:...`. +- **Sessions:** each agent slot has its own FLUJO workspace with one conversation, so its + board identity, model record and tools stay apart. FLUJO runs on port 4210 on the + Machine's loopback, or `FACTORY_FLUJO_PORT`. Each workspace runs its own tool servers, so + plan on about 400 MB per agent plus 300 MB for FLUJO. FLUJO restarts by itself up to + three times in ten minutes; after that the worker stops, as with the other programs. +- **Team:** with `flujo.teamConcurrency` an orchestrator gets the `swarm_team` flow, with one + native gate for its subflows, and `flujo.specialists` can give it a role catalog. The others get `swarm_agent`. A turn ends, and a stop receipt + is issued, only after every child conversation has ended too. Each conversation lists its + children under `/v1/chat/conversations?descendantsOf=`. +- **Tools:** `bash` and `filesystem`, plus `browser` if `flujo.servers` lists it and the image + has its browser; the bootstrap does not download one. The FLUJO control server stays off. + The board server takes its address and credential from a private file. A worker's prompt + names its working folder, because FLUJO's tools otherwise start in FLUJO's data folder. +- **Sandbox:** only `danger-full-access`. FLUJO's folder settings are a guide for its tools, + not a boundary, so the Machine is the sandbox, and a plan with a narrower `sandbox` is + refused. Agents get no GitHub token, as with other self-hosted model plans. An agent command + runs as the same user, so it could still read FLUJO's stored settings, including the model + key; use another program with a narrower sandbox when agents handle untrusted input. +- **Runs:** a run belongs to FLUJO, not to the connection that asked for it. A dropped event + stream is followed again from its last event. A submission whose answer was lost is read + back and never sent twice. While FLUJO is under pressure a turn waits for up to five + minutes. A question or form from a tool is declined, because nobody is there to answer it. +- **Messages and stops:** a message during a turn joins it at its next safe point; if the run + ended before taking it, the same turn goes on with it. A stop is requested again until FLUJO + has stopped, and wins over a queued message. If it never stops, no receipt is issued. +- **Restarts:** the conversation survives. A turn that was running when FLUJO died ends as + `interrupted`. It is not replayed; in autonomous mode the agent gets a new turn. If the + conversation is gone, the usual recovery for an unavailable session applies. +- **Traces and evidence:** messages, tool calls and results and subflow starts and ends become + `item/completed` records (`agentMessage`, `toolCall`, `toolResult`, `subflow`), with `depth` + and `lane` for children. Token use becomes `thread/tokenUsage/updated`, retries become + `model/retry`, and errors become `flujo/error`. Evidence holds the conversations from + `/data/flujo/workspaces`, not its keys, server settings or tool copies. + #### Self-hosted models and operator-driven jobs To run the model yourself, `ModalSelfHostedModel` (CLI `factory-selfhost`) deploys a @@ -357,12 +439,12 @@ await fleet.command('m0-a1', { id: 'job-1', action: 'message', message: 'First j maps to its own settings. `id`, `envKey` (the variable that carries the key) and `https` `baseUrls` (OpenAI-style, ending in `/v1`) without credentials are required. `apis` lists the wire APIs the server offers: `responses` (Codex), - `anthropic` (Claude Code, `/v1/messages`) and `chat`. vLLM serves all three; the - default is `['responses']`. `name`, `model`, `contextWindow`, + `anthropic` (Claude Code, `/v1/messages`) and `chat`; FLUJO can use any of them. vLLM + serves all three; the default is `['responses']`. `name`, `model`, `contextWindow`, `autoCompactTokenLimit`, `toolOutputTokenLimit`, `requestMaxRetries`, `streamMaxRetries`, `streamIdleTimeoutMs` and `requestTimeoutMs` are optional. - `codexProviderConfig`, `codexProviderOverrides` and `claudeProviderEnvironment` - are exported mappings; `providerPrograms(provider)` says which programs can use + `codexProviderConfig`, `codexProviderOverrides`, `claudeProviderEnvironment` and + `flujoProviderModel` are exported mappings; `providerPrograms(provider)` says which programs can use it. Each Codex gets a `config.toml` with the provider, the model's real context window and its compaction point; web search, multi-agent, apps, plugins, goals, memories and skill instructions are off, and agents' shell commands inherit only diff --git a/bin/factory-cloud-mcp.mjs b/bin/factory-cloud-mcp.mjs index 9e0b901..8447853 100644 --- a/bin/factory-cloud-mcp.mjs +++ b/bin/factory-cloud-mcp.mjs @@ -1,3 +1,6 @@ #!/usr/bin/env node +import { readFileSync } from 'node:fs'; import { startCloudBoardMcp } from '../src/cloud-lite/mcp.mjs'; -await startCloudBoardMcp(); +// `--config ` carries the address and credential for hosts that keep server settings in clear view. +const flag = process.argv.indexOf('--config'); +await startCloudBoardMcp(flag === -1 ? {} : JSON.parse(readFileSync(process.argv[flag + 1], 'utf8'))); diff --git a/bin/factory-lite.mjs b/bin/factory-lite.mjs index a93a8e9..52fbd17 100644 --- a/bin/factory-lite.mjs +++ b/bin/factory-lite.mjs @@ -2,9 +2,10 @@ import { readFileSync } from 'node:fs'; import { Factory } from '../src/public-sdk.mjs'; -const usage = 'Usage: factory-lite [--model-provider ] [prompt]\n' +const usage = 'Usage: factory-lite [--model-provider ] [--origin ] [prompt]\n' + ' --model-provider use a self-hosted model (e.g. from `factory-selfhost endpoint`); its key is read from the\n' - + ' environment variable the file names in envKey.\n'; + + ' environment variable the file names in envKey. FLUJO needs this.\n' + + ' --origin FLUJO only: attach to this running FLUJO instead of starting one.\n'; const args = process.argv.slice(2); const [provider, database] = args; if (provider === '--help' || provider === 'help' || !provider) { @@ -22,9 +23,31 @@ if (provider === '--help' || provider === 'help' || !provider) { providerKey = process.env[modelProvider.envKey]; if (!providerKey) throw new Error(`Set ${modelProvider.envKey} to the endpoint key.`); } - const child = new Factory(database).startLiteWorker({ provider, prompt: words.length ? words.join(' ') : undefined, modelProvider, providerKey }); + let origin; + const originFlag = words.indexOf('--origin'); + if (originFlag !== -1) { + origin = words[originFlag + 1]; + if (provider !== 'flujo' || !origin) throw new Error('--origin needs a URL and is for FLUJO workers.'); + words.splice(originFlag, 2); + } + const started = new Factory(database).startLiteWorker({ provider, prompt: words.length ? words.join(' ') : undefined, modelProvider, providerKey, ...(origin ? { origin } : {}) }); + if (provider === 'flujo') { + const worker = await started; + process.stdout.write(`FLUJO worker ready at ${worker.origin} in workspace ${worker.workspace}.\n`); + const finish = () => { void worker.stop(); }; + process.once('SIGINT', finish); process.once('SIGTERM', finish); + // With a prompt the worker ends with its turn; without one a started server stays up until stopped. + if (words.length) { + const turn = await worker.settled(); + process.stdout.write(`Turn ended: ${turn?.status}.\n`); + process.exitCode = turn?.status === 'completed' ? 0 : 1; + await worker.stop(); + } else if (origin) await worker.stop(); + } else { + const child = started; child.on('error', error => { process.stderr.write(`${error.message}\n`); process.exitCode = 1; }); child.on('exit', (code, signal) => { process.exitCode = code ?? (signal ? 1 : 0); }); + } } catch (error) { process.stderr.write(`${error.message}\n`); process.exitCode = 1; diff --git a/package-lock.json b/package-lock.json index 994859f..6eb85b8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "1.32.1" diff --git a/package.json b/package.json index 7a87709..1dc7466 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "flujo-factory", - "version": "0.6.0", + "version": "0.7.0", "description": "Durable FLUJO swarm engine and Factory SDK, CLI, and MCP server", "license": "MIT", "repository": { diff --git a/scripts/verify-public-sdk.mjs b/scripts/verify-public-sdk.mjs index 6d1090c..123d59e 100644 --- a/scripts/verify-public-sdk.mjs +++ b/scripts/verify-public-sdk.mjs @@ -96,7 +96,11 @@ assert.equal(providerPlan.modelProvider.autoCompactTokenLimit, 80000); assert.throws(() => sdk.validateCloudLitePlan({ ...providerPlan, modelProvider: { ...providerPlan.modelProvider, baseUrls: ['http://model.example/v1'] } }), TypeError); // One provider description maps to Codex and Claude Code. const endpoint = { ...providerPlan.modelProvider, model: 'served-model', apis: ['responses', 'anthropic'] }; -assert.deepEqual(sdk.providerPrograms(endpoint), ['codex', 'claude']); +assert.deepEqual(sdk.providerPrograms(endpoint), ['codex', 'claude', 'flujo']); +assert.equal(sdk.flujoProviderModel(endpoint, { key: 'package-smoke-key' }).adapter, 'openai-responses'); +assert.equal(typeof sdk.FlujoSessionDriver, 'function'); +assert.equal(typeof sdk.packFactorySdk, 'function'); +assert.match(sdk.cloudLiteBootstrap('a'.repeat(64), { workerProgram: 'flujo' }), /flujo-ai-\d+\.\d+\.\d+\.tgz/); assert.match(sdk.codexProviderConfig(endpoint), /wire_api = "responses"/); assert.equal(sdk.claudeProviderEnvironment(endpoint, { key: 'package-smoke-key' }).ANTHROPIC_BASE_URL, 'https://model.example'); // Self-hosted model serving ships its Python sources and profiles, but not its tests. diff --git a/selfhost/README.md b/selfhost/README.md index 5735da3..c9aade5 100644 --- a/selfhost/README.md +++ b/selfhost/README.md @@ -4,7 +4,7 @@ FACTORY can serve an open-weight model on your own Modal GPUs with vLLM, so work instead of a vendor API: source code and prompts stay in your account. The served endpoint speaks the OpenAI Responses API (Codex), the Anthropic Messages API (Claude Code) and Chat Completions, and plugs straight into `plan.modelProvider` and the local lite workers. Cloud lite workers can -run either program on it (`plan.workerProgram`). +run any of their programs on it (`plan.workerProgram`: Codex, Claude Code or FLUJO). This folder is separate from `modal/`, which holds the earlier pilot definitions. diff --git a/src/cloud-lite/bootstrap.mjs b/src/cloud-lite/bootstrap.mjs index a9c1bfe..0487e80 100644 --- a/src/cloud-lite/bootstrap.mjs +++ b/src/cloud-lite/bootstrap.mjs @@ -1,6 +1,8 @@ /** Bootstrap on the pinned upstream Node image when a custom registry is unavailable. */ const CLAUDE_CODE_VERSION = '2.1.311'; const CLAUDE_CODE_LINUX_X64_SHA256 = '86dc1fcc69498159eaf15f446613e847010f9036bfbbdffd4c49350be8489ada'; +const FLUJO_VERSION = '3.48.3'; +const FLUJO_SHA256 = '09e954ec5b15e30748c28576b2c0bbe3d4f5cdc34afbbbc139d92759474b6b98'; const programInstall = { codex: { folder: '/data/codex', packages: '', home: 'CODEX_HOME=/data/codex', @@ -21,6 +23,22 @@ echo '${CLAUDE_CODE_LINUX_X64_SHA256} /tmp/claude' | sha256sum -c - install -m 755 /tmp/claude /usr/local/bin/claude rm -f /tmp/claude`, }, + // Kept on the volume, so a restarted Machine does not install it again. An image that already has + // `flujo` on its path skips this. Install scripts stay off: they only download a browser workers leave off. + flujo: { + folder: '/data/flujo /data/factory-private /data/results', packages: ' ripgrep python3-venv python3-pip', home: 'FLUJO_DATA_DIR=/data/flujo', + install: `if ! command -v flujo >/dev/null 2>&1; then +if [ ! -f /data/flujo-program/.installed-${FLUJO_VERSION} ]; then +rm -rf /data/flujo-program +curl -fsSL https://registry.npmjs.org/flujo-ai/-/flujo-ai-${FLUJO_VERSION}.tgz -o /tmp/flujo.tgz +echo '${FLUJO_SHA256} /tmp/flujo.tgz' | sha256sum -c - +npm install --prefix /data/flujo-program /tmp/flujo.tgz --ignore-scripts --omit=dev --no-audit --no-fund +rm -f /tmp/flujo.tgz +touch /data/flujo-program/.installed-${FLUJO_VERSION} +fi +ln -sf /data/flujo-program/node_modules/.bin/flujo /usr/local/bin/flujo +fi`, + }, }; export function cloudLiteBootstrap(bundleSha256, { workerProgram = 'codex' } = {}) { diff --git a/src/cloud-lite/bundle.mjs b/src/cloud-lite/bundle.mjs new file mode 100644 index 0000000..bbb3b57 --- /dev/null +++ b/src/cloud-lite/bundle.mjs @@ -0,0 +1,59 @@ +import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { gzipSync } from 'node:zlib'; + +const PACKAGE_ROOT = fileURLToPath(new URL('../../', import.meta.url)); +const FOLDERS = ['src', 'bin', 'scripts', 'selfhost']; +const FILES = ['package.json', 'README.md', 'PUBLIC_SDK.md', 'WORKFLOWS.md', 'LICENSE']; + +function tarHeader(name, size, mode) { + const header = Buffer.alloc(512); + let path = name, prefix = ''; + if (Buffer.byteLength(path) > 100) { + const cut = path.indexOf('/', Buffer.byteLength(path) - 101); + if (cut === -1 || cut > 155) throw new Error(`Path is too long for the SDK bundle: ${name}`); + prefix = path.slice(0, cut); path = path.slice(cut + 1); + } + const put = (text, offset, length) => header.write(text, offset, length, 'utf8'); + const octal = (value, length) => value.toString(8).padStart(length - 1, '0') + '\0'; + put(path, 0, 100); put(octal(mode, 8), 100, 8); put(octal(0, 8), 108, 8); put(octal(0, 8), 116, 8); + put(octal(size, 12), 124, 12); put(octal(0, 12), 136, 12); put(' ', 148, 8); put('0', 156, 1); + put('ustar\0', 257, 6); put('00', 263, 2); put(prefix, 345, 155); + const checksum = header.reduce((sum, byte) => sum + byte, 0); + put(`${checksum.toString(8).padStart(6, '0')}\0 `, 148, 8); + return header; +} + +function listFiles(folder) { + const found = []; + for (const entry of readdirSync(folder, { withFileTypes: true })) { + if (['node_modules', '.git', '__pycache__'].includes(entry.name)) continue; + const path = join(folder, entry.name); + if (entry.isDirectory()) found.push(...listFiles(path)); + else if (entry.isFile()) found.push(path); + } + return found; +} + +/** + * Writes this installed package as the SDK tarball a fleet's bootstrap unpacks, so an app can deploy + * workers from its own dependency, with no copy of the release file and no `npm` or `tar` on the host. + * The same files give the same bytes, so a fleet can check it is resuming with the same bundle. + */ +export function packFactorySdk(destination, { root = PACKAGE_ROOT } = {}) { + if (typeof destination !== 'string' || !destination) throw new TypeError('Select where to write the SDK bundle.'); + const files = [...FILES.map(name => join(root, name)).filter(path => statSync(path, { throwIfNoEntry: false })?.isFile()), + ...FOLDERS.map(name => join(root, name)).filter(path => statSync(path, { throwIfNoEntry: false })?.isDirectory()).flatMap(listFiles)] + .map(path => ({ path, name: relative(root, path).split(/[\\/]/).join('/') })) + .sort((a, b) => (a.name < b.name ? -1 : 1)); + const parts = []; + for (const { path, name } of files) { + const bytes = readFileSync(path); + parts.push(tarHeader(`package/${name}`, bytes.length, name.startsWith('bin/') ? 0o755 : 0o644), bytes, Buffer.alloc((512 - bytes.length % 512) % 512)); + } + parts.push(Buffer.alloc(1024)); + mkdirSync(dirname(destination), { recursive: true, mode: 0o700 }); + writeFileSync(destination, gzipSync(Buffer.concat(parts), { level: 9 }), { mode: 0o600 }); + return { file: destination, files: files.length }; +} diff --git a/src/cloud-lite/claude-driver.mjs b/src/cloud-lite/claude-driver.mjs index 276a8bf..8394f40 100644 --- a/src/cloud-lite/claude-driver.mjs +++ b/src/cloud-lite/claude-driver.mjs @@ -157,7 +157,7 @@ export class ClaudeSessionDriver extends EventEmitter { const recovered = lastTurn?.status === 'inProgress' ? { ...lastTurn, status: 'interrupted' } : lastTurn; const thread = this.addThread(id, { ...params, lastTurn: recovered }); this.notify('thread/resumed', { threadId: id, thread: { id } }); - return { thread: this.describe(thread) }; + return { thread: this.describe(thread), ...(lastTurn?.status === 'inProgress' ? { recoveredTurn: recovered } : {}) }; } readThread({ threadId, includeTurns }) { diff --git a/src/cloud-lite/collect.mjs b/src/cloud-lite/collect.mjs index bb5a598..ada25ec 100644 --- a/src/cloud-lite/collect.mjs +++ b/src/cloud-lite/collect.mjs @@ -11,7 +11,9 @@ export async function collectCloudLiteSnapshot({ root = '/data', runId, destinat mkdirSync(destination, { mode: 0o700 }); const output = join(destination, 'snapshot'); mkdirSync(output, { recursive: true, mode: 0o700 }); const manifest = { runId, includeGitObjects, startedAt: new Date().toISOString(), files: [], omitted: [], changedDuringCopy: [], sqliteBackups: [] }; - const excluded = new Set(['node_modules', '.next', '.npm', '.cache', '__pycache__', '.pytest_cache', 'auth.json', 'lost+found']); + const excluded = new Set(['node_modules', '.next', '.npm', '.cache', '__pycache__', '.pytest_cache', 'auth.json', 'lost+found', + // FLUJO keeps its keys and server settings in these, and the board credential sits in the last one. + 'encryption_key.json', 'models.json', 'mcp_servers.json', 'factory-private']); const digest = async file => { const hash = createHash('sha256'); for await (const bytes of createReadStream(file)) hash.update(bytes); return hash.digest('hex'); }; const identicalFiles = new Map(); let hardLinkedFiles = 0; async function visit(source) { diff --git a/src/cloud-lite/evidence-worker.mjs b/src/cloud-lite/evidence-worker.mjs index 26fe2ae..b87bf1b 100644 --- a/src/cloud-lite/evidence-worker.mjs +++ b/src/cloud-lite/evidence-worker.mjs @@ -14,7 +14,7 @@ process.on('message',async message=>{ sessions=message.sessions; const directory='/data/factory-evidence';mkdirSync(directory,{recursive:true,mode:0o700}); const roots=[{name:'workspaces',path:'/data/worktrees',exclude:['node_modules','.git','.next','.cache','.npm','__pycache__']}, - {name:'sessions',path:message.sessionsPath??'/data/codex/sessions'}, {name:'results',path:'/data/results'}, + {name:'sessions',path:message.sessionsPath??'/data/codex/sessions',...(message.sessionsExclude?{exclude:message.sessionsExclude}:{})}, {name:'results',path:'/data/results'}, {name:'runtime',path:'/data/factory',exclude:['control.sqlite','control.sqlite-wal','control.sqlite-shm']}, {name:'board',path:resolve(directory,'board')}]; for(const root of roots)mkdirSync(root.path,{recursive:true,mode:0o700}); diff --git a/src/cloud-lite/fleet.mjs b/src/cloud-lite/fleet.mjs index 99ffc95..635d9be 100644 --- a/src/cloud-lite/fleet.mjs +++ b/src/cloud-lite/fleet.mjs @@ -206,7 +206,7 @@ export class FactoryCloudLiteFleet { writeFileSync(join(this.directory, 'credentials.json'), JSON.stringify({ credentials, operatorToken: secret() }), { mode: 0o600, flag: 'wx' }); this.state = { schemaVersion: 1, plan, image: null, operations: {}, machines: [], createdAt: new Date().toISOString() }; this.save(); } - await this.stage('app.create', async () => { await this.fly(['apps', 'create', plan.app, '--org', plan.org ?? 'personal']); return { app: plan.app }; }); + await this.stage('app.create', async () => { await this.fly(['apps', 'create', plan.app, '--org', plan.org ?? 'personal', ...(plan.network ? ['--network', plan.network] : [])]); return { app: plan.app }; }); return this.snapshot(); } async deploy(input, { image, imageSource, channel, githubToken, bundlePath, evidenceTokens, deployKey, providerKey } = {}) { @@ -411,7 +411,7 @@ export class FactoryCloudLiteFleet { throw new TypeError('Select original journal-owned Machine identities.'); } const bootstrapPath = join(this.directory, `bootstrap-${digest}.sh`); - writeFileSync(bootstrapPath, cloudLiteBootstrap(digest), { mode: 0o600 }); + writeFileSync(bootstrapPath, cloudLiteBootstrap(digest, { workerProgram: this.state.plan.workerProgram }), { mode: 0o600 }); for (const machine of selected) { await this.stage(`${machine.id}.runtime.${digest}`, async () => { const inventory = JSON.parse(await this.fly(['machine', 'list', '--app', this.state.plan.app, '--json'])); @@ -657,9 +657,13 @@ export class FactoryCloudLiteFleet { if (this.state.operations['app.retire']?.state !== 'succeeded') { const inventory = JSON.parse(await this.fly(['machine', 'list', '--app', this.state.plan.app, '--json'])); const volumes = JSON.parse(await this.fly(['volumes', 'list', '--app', this.state.plan.app, '--json'])); - if (inventory.length !== this.state.machines.length || inventory.some(value => !this.state.machines.some(machine => - machine.id === value.id && machine.name === value.name && value.config?.metadata?.['factory.run'] === this.state.plan.runId && value.config?.mounts?.some(mount => mount.volume === machine.volumeId))) || - volumes.length !== this.state.machines.length || volumes.some(value => !this.state.machines.some(machine => machine.volumeId === value.id))) { + // A deployment can stop part way, so what this journal created is the inventory, not the plan's full size. + // A Machine whose creation has an unknown outcome still proves itself by its run label. + const journaledVolumes = new Set(Object.entries(this.state.operations).filter(([key, operation]) => /^m\d+\.volume$/.test(key) && operation.state === 'succeeded').map(([, operation]) => operation.result?.id)); + const uncertainMachines = new Set(Object.entries(this.state.operations).filter(([key, operation]) => /^m\d+\.machine-create$/.test(key) && operation.state !== 'succeeded').map(([key]) => `factory-${this.state.plan.runId}-${key.split('.')[0]}`)); + const labelled = value => value.config?.metadata?.['factory.run'] === this.state.plan.runId; + if (inventory.some(value => !labelled(value) || !(this.state.machines.some(machine => machine.id === value.id && machine.name === value.name && value.config?.mounts?.some(mount => mount.volume === machine.volumeId)) || uncertainMachines.has(value.name))) + || volumes.some(value => !journaledVolumes.has(value.id))) { throw new Error('Retirement requires the exact original Machine and volume inventory; retain resources whose ownership differs.'); } } diff --git a/src/cloud-lite/flujo-driver.mjs b/src/cloud-lite/flujo-driver.mjs new file mode 100644 index 0000000..3abbdd4 --- /dev/null +++ b/src/cloud-lite/flujo-driver.mjs @@ -0,0 +1,566 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { EventEmitter } from 'node:events'; +import { createHash, randomUUID } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs'; +import http from 'node:http'; +import https from 'node:https'; +import { join } from 'node:path'; +import { FlujoClient } from '../flujo-swarm/flujo-client.mjs'; +import { buildFactoryTeamSpecs, FACTORY_FLOW_NAMES } from '../flujo-swarm/template/factory-team.mjs'; + +const invalid = message => Object.assign(new Error(message), { code: -32600 }); +const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); +/** A conversation still owns work in any of these states. Anything else is terminal. */ +const ACTIVE_STATUSES = new Set(['running', 'queued', 'pending', 'awaiting', 'awaiting_tool_approval', 'paused_debug']); +/** FLUJO refuses a run for a moment under pressure; the caller waits instead of failing the turn. */ +const RETRYABLE_SUBMISSION = new Set([408, 425, 429, 503]); +/** Each bundled server is one process per workspace, so a worker keeps only what it uses. */ +export const FLUJO_WORKER_SERVERS = Object.freeze(['bash', 'filesystem']); +/** The default port stays clear of the board, which uses 4200 on the coordinator Machine. */ +export const FLUJO_WORKER_PORT = 4210; + +/** The FLUJO environment. A model key lives in FLUJO's encrypted settings, never here: agent commands inherit all of this. */ +export function flujoEnvironment({ home, port = FLUJO_WORKER_PORT, path = process.env.PATH, extra = {} } = {}) { + return { PATH: path, HOME: '/data', FLUJO_DATA_DIR: home, FLUJO_PORT: String(port), FLUJO_EXPOSURE_MODE: 'localhost', NEXT_TELEMETRY_DISABLED: '1', GIT_TERMINAL_PROMPT: '0', ...extra }; +} + +/** One FLUJO process, restarted a few times if it dies. A worker that keeps crashing is reported, not hidden. */ +export class FlujoProgram extends EventEmitter { + constructor({ command = 'flujo', args = [], port = FLUJO_WORKER_PORT, environment = () => process.env, onOutput, startTimeoutMs = 300000, stopGraceMs = 10000, restartLimit = 3, restartWindowMs = 600000 } = {}) { + super(); + Object.assign(this, { command, baseArgs: args, port, environment, onOutput, startTimeoutMs, stopGraceMs, restartLimit, restartWindowMs }); + this.origin = `http://127.0.0.1:${port}`; + this.restarts = []; + } + + spawnProcess() { + // On Windows an npm command is a .cmd file, which only a shell can start. + const windows = process.platform === 'win32'; + const word = value => windows && /\s/.test(value) ? `"${value}"` : value; + const child = spawn(word(this.command), [...this.baseArgs, '--no-open', '--port', String(this.port)].map(word), { env: this.environment(), stdio: ['ignore', 'pipe', 'pipe'], detached: !windows, windowsHide: true, shell: windows }); + this.child = child; + for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => this.onOutput?.(chunk.toString())); + return child; + } + + async waitUntilReady(child) { + let failure; + const failed = new Promise((_, reject) => { + child.once('error', error => { failure = error; reject(error); }); + child.once('exit', code => { failure = new Error(`FLUJO exited before it was ready (${code}).`); reject(failure); }); + }); + failed.catch(() => {}); + const deadline = Date.now() + this.startTimeoutMs; + const client = new FlujoClient({ origin: this.origin }); + while (Date.now() < deadline) { + if (failure) throw failure; + const listed = await client.api('GET', '/api/workspaces', undefined, { workspace: null, timeoutMs: 5000 }).catch(() => null); + if (listed?.status === 200) return; + await Promise.race([sleep(500), failed]); + } + throw new Error('FLUJO did not become ready before the start deadline.'); + } + + async start() { + const child = this.spawnProcess(); + await this.waitUntilReady(child); + this.watch(child); + } + + watch(child) { + child.removeAllListeners('exit'); + child.once('exit', async (code, signal) => { + if (this.stopping || this.child !== child) { this.emit('exit', { code, signal }); return; } + for (;;) { + this.restarts = [...this.restarts.filter(at => Date.now() - at < this.restartWindowMs), Date.now()]; + this.emit('restarting', { code, signal, restarts: this.restarts.length }); + if (this.restarts.length > this.restartLimit) { this.emit('exit', { code, signal }); return; } + try { + const next = this.spawnProcess(); + await this.waitUntilReady(next); + this.watch(next); + this.emit('restarted', { restarts: this.restarts.length }); + return; + } catch { /* Counted above; the next attempt or the limit decides. */ } + } + }); + } + + signal(name) { + const child = this.child; + if (!child?.pid) return; + if (process.platform === 'win32') { spawnSync('taskkill', ['/pid', String(child.pid), '/T', ...(name === 'SIGKILL' ? ['/F'] : [])], { stdio: 'ignore', windowsHide: true }); return; } + // FLUJO leads its own process group, so the server and its tool processes stop with it. + try { process.kill(-child.pid, name); } catch { try { child.kill(name); } catch { /* Already gone. */ } } + } + + stop() { + this.stopping = true; + const child = this.child; + if (!child || child.exitCode !== null || child.signalCode !== null) return Promise.resolve(); + return new Promise(resolve => { + const timer = setTimeout(() => this.signal('SIGKILL'), this.stopGraceMs); + child.once('exit', () => { clearTimeout(timer); resolve(); }); + this.signal('SIGTERM'); + }); + } +} + +/** Send a run and return once FLUJO has taken it. The run does not depend on this connection. */ +function submitDetached(url, { headers, body }) { + return new Promise((resolve, reject) => { + const target = new URL(url), payload = Buffer.from(JSON.stringify(body)); + const request = (target.protocol === 'https:' ? https : http).request(target, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Content-Length': payload.length, ...headers } }, response => { + if (response.statusCode === 200) { resolve({ status: 200, text: '' }); response.destroy(); return; } + const chunks = []; + response.on('data', chunk => chunks.push(chunk)); + response.on('end', () => resolve({ status: response.statusCode, text: Buffer.concat(chunks).toString('utf8').slice(0, 2000) })); + response.on('error', reject); + }); + request.on('error', reject); + request.end(payload); + }); +} + +/** Follow one conversation's durable event stream until it closes. Resolves; a dropped stream is not an error here. */ +function followEvents(url, { headers, onEvent, onControl, signal }) { + return new Promise(resolve => { + const target = new URL(url); + let buffer = ''; + const done = () => resolve(); + const request = (target.protocol === 'https:' ? https : http).request(target, { headers: { Accept: 'text/event-stream', ...headers }, signal }, response => { + if (response.statusCode !== 200) { response.resume(); response.on('end', done); response.on('error', done); return; } + response.setEncoding('utf8'); + response.on('data', chunk => { + buffer += chunk; + for (let end = buffer.indexOf('\n\n'); end !== -1; end = buffer.indexOf('\n\n')) { + const frame = buffer.slice(0, end); buffer = buffer.slice(end + 2); + const lines = frame.split('\n'); + const data = lines.filter(line => line.startsWith('data:')).map(line => line.slice(5).trimStart()).join('\n'); + if (!data) continue; + let value; try { value = JSON.parse(data); } catch { continue; } + if (lines.some(line => line.startsWith('event:'))) onControl?.(value); else onEvent(value); + } + }); + response.on('end', done); response.on('error', done); response.on('close', done); + }); + request.on('error', done); + request.end(); + }); +} + +/** + * One FLUJO conversation per agent, behind the method and notification contract the cloud runtime + * already uses for a Codex app-server: thread/start|resume|read and turn/start|steer|interrupt, with + * turn and item notifications. Each agent gets its own workspace, so its board identity, model record + * and tools stay apart, and retiring it removes exactly those. A run belongs to FLUJO, not to this + * connection: a dropped stream is followed again, and an uncertain submission is read back, never sent twice. + */ +export class FlujoSessionDriver extends EventEmitter { + constructor({ origin, token, program, stateDirectory, privateDirectory = stateDirectory, modelRecord, team = {}, servers = FLUJO_WORKER_SERVERS, partialMessages = false, onStderr, clientFactory = settings => new FlujoClient(settings), version, + pollMs = 5000, interruptResendMs = 10000, submitTimeoutMs = 300000, toolsTimeoutMs = 120000, reconnectMs = 1000, drainMs = 3000 } = {}) { + super(); + if (!origin && !program) throw new TypeError('A FLUJO origin or program is required.'); + Object.assign(this, { program, token, stateDirectory, privateDirectory, modelRecord, team, servers, partialMessages, onStderr, clientFactory, knownVersion: version, pollMs, interruptResendMs, submitTimeoutMs, toolsTimeoutMs, reconnectMs, drainMs }); + this.origin = (program?.origin ?? origin).replace(/\/$/, ''); + this.threads = new Map(); + mkdirSync(stateDirectory, { recursive: true, mode: 0o700 }); + mkdirSync(privateDirectory, { recursive: true, mode: 0o700 }); + } + + client(workspace) { return this.clientFactory({ origin: this.origin, token: this.token, workspace }); } + headers() { return this.token ? { Authorization: `Bearer ${this.token}` } : {}; } + notify(method, params) { this.emit('notification', { method, params }); } + stateFile(id) { return join(this.stateDirectory, `${id}.flujo.json`); } + + async initialize() { + if (!this.program) return; + this.program.on('exit', result => this.emit('exit', result)); + this.program.on('restarting', result => this.onStderr?.(`FLUJO stopped unexpectedly (${result.code ?? result.signal}); restarting it.`)); + await this.program.start(); + // This instance is ours alone: nothing runs in its default workspace. + const home = this.client(null); + for (const server of await home.servers().catch(() => [])) { + if (!server.disabled) await home.api('PUT', `/api/mcp/servers/${encodeURIComponent(server.name)}`, { ...server, disabled: true }).catch(() => {}); + } + } + + async version() { + if (this.knownVersion) return this.knownVersion; + const read = await this.client(null).api('GET', '/api/update', undefined, { timeoutMs: 15000 }).catch(() => null); + return this.knownVersion = read?.body?.currentVersion ?? 'unknown'; + } + + async request(method, params = {}) { + if (method === 'thread/start') return this.startThread(params); + if (method === 'thread/resume') return this.resumeThread(params); + if (method === 'thread/read') return this.readThread(params); + if (method === 'turn/start') return this.startTurn(params); + if (method === 'turn/steer') return this.steerTurn(params); + if (method === 'turn/interrupt') return this.interruptTurn(params); + throw invalid(`Unsupported method ${method}.`); + } + + /** The board server reads its credential from a private file, so no secret enters FLUJO's server settings. */ + boardConfig(thread, server) { + const settings = { name: 'factory', transport: 'stdio', command: server.command, args: server.args ?? [], env: server.env ?? {}, cwd: thread.cwd, rootPath: thread.cwd, disabled: false, exposeAsMcpServer: false, enableMcpApps: false }; + if (!server.env?.FACTORY_BOARD_TOKEN) return settings; + const file = join(this.privateDirectory, `${thread.id}.board.json`); + writeFileSync(file, JSON.stringify({ url: server.env.FACTORY_BOARD_URL, token: server.env.FACTORY_BOARD_TOKEN }), { mode: 0o600 }); + const env = Object.fromEntries(Object.entries(server.env).filter(([name]) => !['FACTORY_BOARD_URL', 'FACTORY_BOARD_TOKEN'].includes(name))); + return { ...settings, args: [...settings.args, '--config', file], env }; + } + + async inventory(client, names) { + const tools = {}, deadline = Date.now() + this.toolsTimeoutMs; + for (const name of names) { + for (;;) { + const observed = await client.serverTools(name).catch(error => ({ tools: [], error: error.message })); + if (!observed.error && observed.tools.length) { tools[name] = observed.tools.map(tool => tool.name); break; } + // A worker never runs without its board or its tools: the same promise Codex gets from a required server. + if (Date.now() >= deadline) throw new Error(`The ${name} server did not connect in FLUJO.`); + await sleep(1000); + } + } + return tools; + } + + /** Idempotent, so a restarted Machine repairs its workspace before it resumes the conversation. */ + async install(thread, { server }) { + const client = this.client(thread.workspace); + await client.ensureWorkspace(); + await client.setWorkspaceRoots(thread.roots); + const wanted = new Set([...this.servers, ...(server ? ['factory'] : [])]); + for (const existing of await client.servers()) { + if (Boolean(existing.disabled) === wanted.has(existing.name)) { + await client.expect('PUT', `/api/mcp/servers/${encodeURIComponent(existing.name)}`, { ...existing, disabled: !wanted.has(existing.name) }); + } + } + const model = this.modelRecord(thread.model); + await client.upsertModel(model); + if (server) await client.upsertServer(this.boardConfig(thread, server)); + const availableServers = [...wanted]; + const availableTools = await this.inventory(client, availableServers); + const concurrency = this.team.concurrency; + const specs = buildFactoryTeamSpecs({ model: model.id, availableServers, availableTools, specialists: this.team.specialists, + // FLUJO's shell and file tools start in its own data folder unless told otherwise. + instructions: `Your working folder is ${thread.cwd}. Pass it as cwd on every shell command and use absolute paths under it for file tools.${thread.instructions ? `\n${thread.instructions}` : ''}`, + limits: { ...(this.team.agentTurns ? { agentTurns: this.team.agentTurns } : {}), ...(this.team.leadTurns ? { leadTurns: this.team.leadTurns } : {}), ...(concurrency ? { concurrency } : {}) }, + environment: { workingFolder: thread.cwd } }).filter(spec => concurrency || spec.name === FACTORY_FLOW_NAMES.agent); + const digest = createHash('sha256').update(JSON.stringify(specs)).digest('hex'); + // Saving a flow gives it a new identity, so an unchanged flow is left alone for the conversation that uses it. + const installed = new Set((await client.flows()).map(flow => flow.name)); + if (thread.flowDigest !== digest || specs.some(spec => !installed.has(spec.name))) { + for (const spec of specs) { + const flow = await client.saveFlowSpec(spec); + if (flow?.name !== spec.name) throw new Error('FLUJO did not confirm the installed flow.'); + } + thread.flowDigest = digest; + } + } + + save(thread) { + const file = this.stateFile(thread.id), next = `${file}.next`; + writeFileSync(next, JSON.stringify({ threadId: thread.id, workspace: thread.workspace, lastTurn: thread.lastTurn, nextSeq: thread.nextSeq, flowDigest: thread.flowDigest }), { mode: 0o600 }); + renameSync(next, file); + } + + addThread(id, { cwd, config, sandbox, model, developerInstructions, agentRole }, saved = {}) { + if ((sandbox ?? 'danger-full-access') !== 'danger-full-access') throw invalid('FLUJO workers run with full access on their Machine.'); + const thread = { id, workspace: saved.workspace ?? `factory-${id}`, cwd, model, instructions: developerInstructions, roots: [cwd, ...(config?.['sandbox_workspace_write.writable_roots'] ?? []), ...(this.team.roots ?? [])], + flow: agentRole === 'orchestrator' && this.team.concurrency ? FACTORY_FLOW_NAMES.team : FACTORY_FLOW_NAMES.agent, + lastTurn: saved.lastTurn ?? null, nextSeq: saved.nextSeq ?? 0, flowDigest: saved.flowDigest, turn: null, totals: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0 } }; + this.threads.set(id, thread); + return thread; + } + + thread(id) { + const thread = this.threads.get(id); + if (!thread) throw invalid(`Unknown thread ${id}.`); + return thread; + } + + async startThread(params) { + const thread = this.addThread(randomUUID(), params); + await this.install(thread, { server: params.config?.mcp_servers?.factory }); + this.save(thread); + this.notify('thread/started', { threadId: thread.id, thread: { id: thread.id } }); + return { thread: { id: thread.id, status: { type: 'idle' } } }; + } + + async resumeThread(params) { + const id = params.threadId; + const unavailable = () => invalid(`no rollout found for thread id ${id}`); + if (!existsSync(this.stateFile(id))) throw unavailable(); + const saved = JSON.parse(readFileSync(this.stateFile(id), 'utf8')); + const thread = this.addThread(id, params, saved); + await this.install(thread, { server: params.config?.mcp_servers?.factory }); + // A conversation without a turn has no history to lose; one with a turn must still have its history. + let status = 'idle', recoveredTurn; + if (thread.lastTurn) { + const read = await this.client(thread.workspace).conversation(id, { messageLimit: 1 }); + if (read.status === 404) throw unavailable(); + if (read.status !== 200) throw new Error(`FLUJO could not read the retained conversation (HTTP ${read.status}).`); + if (thread.lastTurn.status === 'inProgress') { + if (ACTIVE_STATUSES.has(read.body.status)) { + // Still running in FLUJO: observe it, never replay it. + status = 'active'; + this.adoptTurn(thread, thread.lastTurn.id); + } else recoveredTurn = thread.lastTurn = { ...thread.lastTurn, status: this.outcome(read.body).status === 'completed' ? 'completed' : 'interrupted' }; + } + } + this.save(thread); + this.notify('thread/resumed', { threadId: id, thread: { id } }); + return { thread: { id, status: { type: status } }, ...(recoveredTurn ? { recoveredTurn } : {}) }; + } + + /** A real readback: the thread is idle only when FLUJO agrees that the conversation and its children have ended. */ + async readThread({ threadId, includeTurns }) { + const thread = this.thread(threadId); + let active = Boolean(thread.turn); + if (!active && thread.lastTurn) { + const read = await this.client(thread.workspace).conversation(threadId, { messageLimit: 1 }); + // A turn FLUJO never took has no conversation, so nothing can be running. + if (read.status !== 200 && read.status !== 404) throw new Error(`FLUJO could not read the conversation (HTTP ${read.status}).`); + active = read.status === 200 && (ACTIVE_STATUSES.has(read.body.status) || await this.childrenActive(thread)); + } + return { thread: { id: thread.id, status: { type: active ? 'active' : 'idle' }, ...(includeTurns ? { turns: thread.lastTurn ? [thread.lastTurn] : [] } : {}) } }; + } + + /** Children keep working after a parent's run ends, and a stop must wait for them too. */ + async childrenActive(thread) { + const client = this.client(thread.workspace); + for (let cursor, page = 0; page < 100; page++) { + const read = await client.descendants(thread.id, cursor); + if (read.status !== 200 || !Array.isArray(read.body?.items)) throw new Error(`FLUJO could not list the conversation's children (HTTP ${read.status}).`); + if (read.body.items.some(item => ACTIVE_STATUSES.has(item.status))) return true; + if (!read.body.hasMore) return false; + cursor = read.body.nextCursor; + if (!cursor) return true; + } + return true; + } + + outcome(conversation) { + const classification = conversation.recovery?.classification; + if (classification === 'cancelled' || conversation.lastError?.code === 'cancelled') return { status: 'interrupted' }; + if (conversation.status === 'completed' || conversation.status === 'capped') return { status: 'completed' }; + const code = conversation.lastError?.httpStatus; + return { status: 'failed', ...(Number.isSafeInteger(code) ? { code } : {}) }; + } + + startTurn({ threadId, input }) { + const thread = this.thread(threadId); + if (thread.turn) throw invalid('This thread already has a running turn.'); + const id = randomUUID(); + const turn = this.beginTurn(thread, id); + thread.lastTurn = { id, status: 'inProgress' }; + this.save(thread); + this.notify('thread/status/changed', { threadId, status: { type: 'active' } }); + this.notify('turn/started', { threadId, turn: { id, status: 'inProgress' } }); + turn.submission = this.submit(thread, turn, textOf(input)).catch(error => { this.onStderr?.(`FLUJO did not take the turn: ${error.message}`); this.finish(thread, turn, { status: 'failed' }); }); + this.watch(thread, turn); + return { turn: { id, status: 'inProgress' } }; + } + + beginTurn(thread, id) { + const turn = { id, steers: new Map(), runSeen: false, finished: false, abort: new AbortController() }; + thread.turn = turn; + return turn; + } + + /** A run found still going after a restart. Its turn keeps its identity. */ + adoptTurn(thread, id) { + const turn = this.beginTurn(thread, id); + turn.runSeen = true; + this.watch(thread, turn); + } + + async submit(thread, turn, text) { + const url = `${this.origin}/v1/chat/completions?workspace=${encodeURIComponent(thread.workspace)}`; + const body = { model: `flow-${thread.flow}`, stream: true, metadata: { flujo: 'true', requireApproval: 'false', conversationId: thread.id }, messages: [{ role: 'user', content: text }] }; + const deadline = Date.now() + this.submitTimeoutMs; + // The run this turn is waiting for is whichever one is not already recorded. + const before = await this.client(thread.workspace).conversation(thread.id, { messageLimit: 1 }); + if (before.status !== 200 && before.status !== 404) throw new Error(`FLUJO could not read the conversation (HTTP ${before.status}).`); + turn.baselineRun = before.body?.recovery?.runId ?? null; + turn.runSeen = false; + for (let attempt = 0; ; attempt++) { + let response; + try { response = await submitDetached(url, { headers: { ...this.headers(), 'x-flujo-workspace': thread.workspace }, body }); } + catch (error) { + // The request may have arrived. Read it back rather than send the task a second time. + this.onStderr?.(`FLUJO submission outcome is being read back: ${error.message}`); + break; + } + if (response.status === 200) break; + if (turn.interrupting) { this.finish(thread, turn, { status: 'interrupted' }); return; } + if (!RETRYABLE_SUBMISSION.has(response.status) || Date.now() >= deadline || turn.finished) throw new Error(`HTTP ${response.status}: ${response.text.slice(0, 300)}`); + await sleep(Math.min(30000, 1000 * 2 ** attempt)); + } + turn.submittedAt = Date.now(); + } + + /** Follows the event stream and also reads the state on a timer, so a lost stream cannot leave a turn open. */ + watch(thread, turn) { + const settle = () => { void this.settle(thread, turn).catch(error => this.onStderr?.(`FLUJO state read is pending: ${error.message}`)); }; + turn.settle = settle; + turn.pollTimer = setInterval(settle, this.pollMs); + void (async () => { + while (!turn.finished && !this.closed) { + await followEvents(`${this.origin}/v1/chat/conversations/${encodeURIComponent(thread.id)}/events?workspace=${encodeURIComponent(thread.workspace)}&fromSeq=${thread.nextSeq}`, { + headers: { ...this.headers(), 'x-flujo-workspace': thread.workspace }, signal: turn.abort.signal, + onEvent: event => this.receive(thread, turn, event), + // FLUJO skipped ahead, for example past an event too large to send. + onControl: control => { if (Number.isSafeInteger(control?.nextSeq)) thread.nextSeq = Math.max(thread.nextSeq, control.nextSeq); }, + }); + if (turn.finished || this.closed) return; + settle(); + await sleep(this.reconnectMs); + } + })(); + } + + receive(thread, turn, event) { + if (Number.isSafeInteger(event.seq)) { + if (event.seq < thread.nextSeq) return; + thread.nextSeq = event.seq + 1; + } + const threadId = thread.id, turnId = turn.id; + const nesting = { ...(event.depth ? { depth: event.depth } : {}), ...(Number.isSafeInteger(event.laneIndex) ? { lane: event.laneIndex } : {}) }; + const item = body => this.notify('item/completed', { threadId, turnId, item: { ...body, ...nesting } }); + switch (event.type) { + case 'run:start': if (!event.depth) turn.runSeen = true; break; + case 'run:done': if (!event.depth) { turn.doneSeen = true; turn.settle?.(); } break; + case 'message': { + const { role, id, content, tool_calls: calls } = event.message ?? {}; + if (role === 'user') { turn.steers.delete(id); break; } + if (role !== 'assistant') break; + if (typeof content === 'string' && content) item({ type: 'agentMessage', text: content }); + for (const call of calls ?? []) item({ type: 'toolCall', id: call.id, tool: call.function?.name, arguments: call.function?.arguments }); + break; + } + case 'tool:result': item({ type: 'toolResult', id: event.toolCallId, tool: event.name, isError: event.isError === true, content: event.result }); break; + case 'usage': { + const last = { inputTokens: event.promptTokens ?? 0, cachedInputTokens: event.cacheReadTokens ?? 0, outputTokens: event.completionTokens ?? 0 }; + for (const name of Object.keys(last)) thread.totals[name] += last[name]; + this.notify('thread/tokenUsage/updated', { threadId, turnId, tokenUsage: { last, total: { ...thread.totals } } }); + break; + } + case 'model:delta': if (this.partialMessages) this.notify('item/stream/delta', { threadId, turnId, delta: event.delta, ...nesting }); break; + case 'recovery:retry': this.notify('model/retry', { threadId, turnId, attempt: event.attempt, retryAt: event.retryAt, error: event.failure?.category }); break; + case 'subflow:start': case 'subflow:done': item({ type: 'subflow', phase: event.type.slice(8), subflow: event.subflowId, conversationId: event.laneConversationId, status: event.status }); break; + case 'error': this.notify('flujo/error', { threadId, turnId, message: event.message, code: event.error?.code, ...nesting }); break; + // Nobody answers on an unattended worker, so a question or form never parks the run. + case 'run:awaiting_question': void this.respond(thread, { action: 'question-decline', questionId: event.questionId }); break; + case 'run:awaiting_elicitation': void this.respond(thread, { action: 'elicitation-cancel', elicitationId: event.elicitationId }); break; + default: break; + } + // Streaming tokens are most events and change nothing a restart needs; it just reads them again. + if (event.type !== 'model:delta' && event.type !== 'tool:progress') this.save(thread); + } + + async respond(thread, body) { + await this.client(thread.workspace).api('POST', `/v1/chat/conversations/${encodeURIComponent(thread.id)}/respond`, body).catch(error => this.onStderr?.(`FLUJO did not take the answer: ${error.message}`)); + } + + /** Ends the turn only after FLUJO itself says this run has ended, along with any children. */ + async settle(thread, turn) { + if (turn.finished || turn.settling) return; + turn.settling = true; + try { + await turn.submission; + if (turn.finished) return; + const read = await this.client(thread.workspace).conversation(thread.id, { messageLimit: turn.steers.size ? 200 : 1 }); + const conversation = read.status === 200 ? read.body : null; + if (read.status !== 200 && read.status !== 404) return; + const runId = conversation?.recovery?.runId; + if (runId && runId !== turn.baselineRun) turn.runSeen = true; + if (!turn.runSeen) { + // The previous run's end is not this turn's end. A run that never starts is not sent again; the turn ends unfinished. + if (Date.now() - (turn.submittedAt ?? Date.now()) >= this.submitTimeoutMs) this.finish(thread, turn, { status: 'failed' }); + return; + } + if (!conversation || ACTIVE_STATUSES.has(conversation.status)) return; + if (await this.childrenActive(thread)) return; + if (turn.finished) return; + // The state can be ahead of the stream. Wait for the run's last event so the trace is complete; a lost stream only delays this. + if (!turn.doneSeen) { + turn.terminalSince ??= Date.now(); + if (Date.now() - turn.terminalSince < this.drainMs) { setTimeout(() => turn.settle?.(), Math.min(this.drainMs, 200)); return; } + } + if (turn.steers.size && !turn.interrupting) { + const seen = new Set((conversation.messages ?? []).map(message => message.id)); + for (const id of [...turn.steers.keys()]) if (seen.has(id)) turn.steers.delete(id); + if (turn.steers.size) { + // A message that arrived as the run ended is still owed: this turn goes on with it. + const text = [...turn.steers.values()].join('\n\n'); + turn.steers.clear(); turn.doneSeen = false; turn.terminalSince = undefined; + turn.submission = this.submit(thread, turn, text).catch(error => { this.onStderr?.(`FLUJO did not take the queued message: ${error.message}`); this.finish(thread, turn, { status: 'failed' }); }); + return; + } + } + this.finish(thread, turn, this.outcome(conversation)); + } finally { turn.settling = false; } + } + + async steerTurn({ threadId, expectedTurnId, input }) { + const thread = this.thread(threadId), turn = thread.turn; + if (!turn || turn.interrupting) throw new Error('no active turn to steer'); + if (expectedTurnId && expectedTurnId !== turn.id) throw invalid('The expected turn is not the running turn.'); + await turn.submission; + const id = randomUUID(), text = textOf(input); + const response = await this.client(thread.workspace).api('POST', `/v1/chat/conversations/${encodeURIComponent(threadId)}/inject`, { content: text, id }); + if (response.status === 409) throw new Error('no active turn to steer'); + if (response.status !== 200 || response.body?.message_id !== id) throw new Error(`FLUJO did not confirm the queued message (HTTP ${response.status}).`); + if (thread.turn === turn) turn.steers.set(id, text); + return { turnId: turn.id }; + } + + async sendCancel(thread) { + const response = await this.client(thread.workspace).cancel(thread.id, 15000); + if (![200, 202].includes(response.status)) throw new Error(`FLUJO did not accept the stop (HTTP ${response.status}).`); + } + + async interruptTurn({ threadId, turnId }) { + const thread = this.thread(threadId), turn = thread.turn; + if (!turn || turn.id !== turnId) { + if (thread.lastTurn?.id === turnId) return {}; + throw invalid('The expected turn is not the running turn.'); + } + if (turn.interrupting) return {}; + turn.interrupting = true; + await turn.submission; + if (turn.finished) return {}; + try { await this.sendCancel(thread); } + catch (error) { turn.interrupting = false; throw error; } + // A cancel is a request; keep asking until FLUJO has actually stopped, children included. + turn.interruptTimer = setInterval(() => { void this.sendCancel(thread).catch(error => this.onStderr?.(`FLUJO stop is pending: ${error.message}`)); }, this.interruptResendMs); + turn.settle?.(); + return {}; + } + + finish(thread, turn, { status, code }) { + if (turn.finished) return; + turn.finished = true; + clearInterval(turn.pollTimer); clearInterval(turn.interruptTimer); + turn.abort.abort(); + thread.turn = null; thread.lastTurn = { id: turn.id, status }; + this.save(thread); + this.notify('turn/completed', { threadId: thread.id, turn: { id: turn.id, status, ...(status === 'completed' ? {} : { error: code === undefined ? {} : { code } }) } }); + this.notify('thread/status/changed', { threadId: thread.id, status: { type: 'idle' } }); + } + + close() { + this.closed = true; + for (const thread of this.threads.values()) { + if (!thread.turn) continue; + clearInterval(thread.turn.pollTimer); clearInterval(thread.turn.interruptTimer); + thread.turn.abort.abort(); + } + if (this.program) void this.program.stop(); + else this.emit('exit', { code: 0, signal: null }); + } +} + +const textOf = input => (input ?? []).filter(item => item.type === 'text').map(item => item.text).join('\n\n'); diff --git a/src/cloud-lite/protocol.mjs b/src/cloud-lite/protocol.mjs index b222fcc..08ee311 100644 --- a/src/cloud-lite/protocol.mjs +++ b/src/cloud-lite/protocol.mjs @@ -1,5 +1,5 @@ import { createHash, randomBytes } from 'node:crypto'; -import { validateModelProvider, providerApis, WORKER_PROGRAM_APIS } from '../model-provider.mjs'; +import { validateModelProvider, requireFleetApi, WORKER_PROGRAM_APIS } from '../model-provider.mjs'; export { validateModelProvider }; @@ -8,7 +8,7 @@ export const hash = value => createHash('sha256').update(typeof value === 'strin export function validateCloudLitePlan(input) { if (!input || typeof input !== 'object') throw new TypeError('A cloud lite-worker plan is required.'); const plan = { cpuKind: 'shared', cpus: 2, memoryMb: 4096, diskGb: 20, machines: 1, - agentsPerMachine: 10, region: 'iad', ...(input.workerProgram === 'claude' ? { model: 'opus', subagentModel: 'sonnet' } : { model: 'gpt-6.1-sol', subagentModel: 'gpt-6-luna' }), reasoningEffort: 'medium', ...input }; + agentsPerMachine: 10, region: 'iad', ...(input.workerProgram === 'claude' ? { model: 'opus', subagentModel: 'sonnet' } : input.workerProgram === 'flujo' ? {} : { model: 'gpt-6.1-sol', subagentModel: 'gpt-6-luna' }), reasoningEffort: 'medium', ...input }; if (plan.workerProgram !== undefined && !Object.hasOwn(WORKER_PROGRAM_APIS, plan.workerProgram)) throw new TypeError(`workerProgram must be one of ${Object.keys(WORKER_PROGRAM_APIS).join(', ')}.`); for (const name of ['runId', 'app']) if (!/^[a-z][a-z0-9-]{2,62}$/.test(plan[name] ?? '')) throw new TypeError(`Invalid ${name}.`); if (!['shared', 'performance'].includes(plan.cpuKind)) throw new TypeError('Invalid cpuKind.'); @@ -17,6 +17,7 @@ export function validateCloudLitePlan(input) { } if (plan.memoryMb % 256) throw new TypeError('memoryMb must be a multiple of 256.'); if (!/^[a-z]{3}$/.test(plan.region)) throw new TypeError('Invalid region.'); + if (plan.network !== undefined && !/^[a-z][a-z0-9-]{2,62}$/.test(plan.network)) throw new TypeError('Invalid network.'); if (plan.repository !== undefined && !/^[\w.-]+\/[\w.-]+$/.test(plan.repository)) throw new TypeError('Optional repository must be a GitHub owner/repository.'); if (typeof plan.goal !== 'string' || !plan.goal.trim()) throw new TypeError('A goal is required.'); if (plan.workflow !== undefined && plan.workflow !== 'verified-v1') throw new TypeError('Unsupported workflow profile.'); @@ -33,9 +34,13 @@ export function validateCloudLitePlan(input) { if (plan.modelProvider !== undefined) { validateModelProvider(plan.modelProvider); // Codex talks to custom providers through the Responses API, Claude Code through the Messages API. - const needed = WORKER_PROGRAM_APIS[plan.workerProgram ?? 'codex']; - if (!providerApis(plan.modelProvider).includes(needed)) throw new TypeError(`Cloud lite workers need a provider that offers the ${needed} API.`); - if (plan.workerProgram === 'claude' && !plan.modelProvider.model && !input.model) throw new TypeError('Select the served model name.'); + requireFleetApi(plan.modelProvider, plan.workerProgram ?? 'codex'); + if (['claude', 'flujo'].includes(plan.workerProgram) && !plan.modelProvider.model && !input.model) throw new TypeError('Select the served model name.'); + } + if (plan.flujo !== undefined && plan.workerProgram !== 'flujo') throw new TypeError('The flujo settings belong to FLUJO worker plans.'); + if (plan.workerProgram === 'flujo') { + validateFlujoSettings(plan); + if (plan.model && !plan.subagentModel) plan.subagentModel = plan.model; } if (plan.supervision !== undefined && !['autonomous', 'operator'].includes(plan.supervision)) throw new TypeError('supervision must be autonomous or operator.'); if (plan.sandbox !== undefined && !CODEX_SANDBOXES.includes(plan.sandbox)) throw new TypeError(`sandbox must be one of ${CODEX_SANDBOXES.join(', ')}.`); @@ -53,6 +58,19 @@ export function validateCloudLitePlan(input) { return plan; } export const CODEX_SANDBOXES = ['danger-full-access', 'workspace-write', 'read-only']; +/** The bundled tool servers a FLUJO worker may use. The FLUJO control server stays off: it can install and edit flows. */ +export const FLUJO_SERVERS = ['bash', 'filesystem', 'browser']; +function validateFlujoSettings(plan) { + // FLUJO has no subscription login to broker; its models are stored records with their own key. + if (!plan.modelProvider) throw new TypeError('FLUJO workers need a modelProvider.'); + if (plan.sandbox !== undefined && plan.sandbox !== 'danger-full-access') throw new TypeError('FLUJO workers run with full access on their Machine; choose danger-full-access.'); + const { teamConcurrency, agentTurns, leadTurns, servers } = plan.flujo ?? {}; + if (plan.flujo !== undefined && (!plan.flujo || typeof plan.flujo !== 'object' || Array.isArray(plan.flujo))) throw new TypeError('flujo must be an object.'); + if (teamConcurrency !== undefined && (!Number.isSafeInteger(teamConcurrency) || teamConcurrency < 1 || teamConcurrency > 9)) throw new TypeError('flujo.teamConcurrency must be from 1 to 9.'); + if (agentTurns !== undefined && (!Number.isSafeInteger(agentTurns) || agentTurns < 1 || agentTurns > 200)) throw new TypeError('flujo.agentTurns must be from 1 to 200.'); + if (leadTurns !== undefined && (!Number.isSafeInteger(leadTurns) || leadTurns < 1 || leadTurns > 600)) throw new TypeError('flujo.leadTurns must be from 1 to 600.'); + if (servers !== undefined && (!Array.isArray(servers) || !servers.length || new Set(servers).size !== servers.length || servers.some(name => !FLUJO_SERVERS.includes(name)))) throw new TypeError(`flujo.servers must list ${FLUJO_SERVERS.join(', ')}.`); +} export function agentInventory(plan) { return Array.from({ length: plan.machines }, (_, machine) => Array.from({ length: plan.agentsPerMachine }, (_, agent) => ({ id: `m${machine}-a${agent}`, machine: `m${machine}`, role: agent === 0 ? 'orchestrator' : 'subagent', @@ -75,7 +93,7 @@ export function machineConfig(plan, { image, volumeId, index, environment }) { image, env: environment, guest: { cpu_kind: plan.cpuKind, cpus: plan.cpus, memory_mb: plan.memoryMb }, mounts: [{ volume: volumeId, path: '/data' }], - metadata: { 'factory.run': plan.runId, 'factory.machine': `m${index}`, 'factory.runtime': plan.workerProgram === 'claude' ? 'claude-lite' : 'codex-lite' }, + metadata: { 'factory.run': plan.runId, 'factory.machine': `m${index}`, 'factory.runtime': `${plan.workerProgram ?? 'codex'}-lite` }, restart: { policy: 'no' }, // Same private topology as flujo-cloud: host connects using its owned proxy; // remote Codex sessions use the coordinator's exact Fly private Machine DNS. diff --git a/src/cloud-lite/runtime.mjs b/src/cloud-lite/runtime.mjs index 7c9e0f3..eeca7b9 100644 --- a/src/cloud-lite/runtime.mjs +++ b/src/cloud-lite/runtime.mjs @@ -1,4 +1,4 @@ -import { mkdirSync, readFileSync, writeFileSync, existsSync, copyFileSync, constants } from 'node:fs'; +import { mkdirSync, readFileSync, writeFileSync, existsSync, copyFileSync, constants, createWriteStream, renameSync } from 'node:fs'; import { open } from 'node:fs/promises'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -7,12 +7,13 @@ import { promisify } from 'node:util'; import { freemem, totalmem, loadavg } from 'node:os'; import { CodexRpc } from './rpc.mjs'; import { ClaudeSessionDriver, claudeCommandWrapper, claudeEnvironment } from './claude-driver.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_PORT, FLUJO_WORKER_SERVERS, flujoEnvironment } from './flujo-driver.mjs'; import { startCloudLiteCoordinator } from './server.mjs'; import { CloudLiteTrace } from './trace.mjs'; import { NativeOperatorStop } from './operator-stop.mjs'; import { NativeWorkflowWake } from './workflow-wake.mjs'; import { hash } from './protocol.mjs'; -import { codexProviderConfig } from '../model-provider.mjs'; +import { codexProviderConfig, flujoProviderModel } from '../model-provider.mjs'; const threadStatuses = new Set(['active','idle','archived','notLoaded','systemError','notFound']); const turnStatuses = new Set(['inProgress','completed','interrupted','failed']); @@ -40,13 +41,69 @@ export async function finalizeNativeStopCommand({ prove, publish, flush, acknowl markStopped(); return true; } +/** + * What differs between worker programs. Everything else (the board, operator commands, stop receipts, + * traces, evidence and workflows) is shared code that talks to a driver through one contract. + */ +const WORKER_PROGRAMS = { + codex: { + home: '/data/codex', sessions: '/data/codex/sessions', exitTrace: 'codex.exit', + prepare: ({ plan, index, home, providerMode }) => writeFileSync(join(home, 'config.toml'), providerMode ? providerConfigToml(plan, index) + : 'cli_auth_credentials_store="ephemeral"\nforced_login_method="chatgpt"\n', { mode: 0o600 }), + driver: ({ plan, home, githubToken, deployKey, providerKey, providerMode, repositoryDirectory, onStderr, refreshAuth }) => new CodexRpc({ + env: codexEnvironment({ plan, home, githubToken, deployKey, providerKey }), cwd: repositoryDirectory, onStderr, + onRequest: async (method, params) => { + if (method === 'account/chatgptAuthTokens/refresh' && !providerMode) return refreshAuth(); + throw new Error(`Unexpected Codex host request: ${method}`); + } }), + login: true, + version: ({ run }) => run('codex', ['--version']), + }, + claude: { + home: '/data/claude', sessions: '/data/claude/projects', exitTrace: 'codex.exit', resumeNeedsSettings: true, + prepare: ({ providerMode }) => writeFileSync(CLAUDE_COMMAND_WRAPPER, claudeCommandWrapper({ provider: providerMode }), { mode: 0o700 }), + driver: ({ plan, index, home, githubToken, deployKey, providerKey, onStderr, getLiveAuth }) => new ClaudeSessionDriver({ configDirectory: home, stateDirectory: '/data/factory', fixedModel: plan.modelProvider?.model, partialMessages: plan.traceDeltas === true, onStderr, + // Read for every turn, so a refreshed subscription token reaches the next process. + environment: () => claudeEnvironment({ plan, index, home, commandWrapper: CLAUDE_COMMAND_WRAPPER, githubToken, deployKey, providerKey, oauthToken: getLiveAuth()?.accessToken }) }), + version: ({ run }) => run('claude', ['--version']), + }, + flujo: { + home: '/data/flujo', sessions: '/data/flujo/workspaces', exitTrace: 'flujo.exit', resumeNeedsSettings: true, + // Conversations are the evidence. Keys, copied tool servers and browser data stay out of it. + sessionsExclude: ['encryption_key.json', 'models.json', 'mcp_servers.json', 'mcp-servers', 'browser-profile', 'bash-utils', 'screenshots', 'recordings', 'snapshots', '.runtime-locks', 'node_modules'], + sessionFields: agent => ({ agentRole: agent.role }), + prepare: () => { mkdirSync('/data/results', { recursive: true, mode: 0o700 }); }, + driver: ({ plan, index, home, providerKey, onStderr }) => { + const port = Number(process.env.FACTORY_FLUJO_PORT ?? FLUJO_WORKER_PORT); + const extra = process.env.NODE_EXTRA_CA_CERTS ? { NODE_EXTRA_CA_CERTS: process.env.NODE_EXTRA_CA_CERTS } : {}; + const { teamConcurrency: concurrency, agentTurns, leadTurns, specialists, servers = FLUJO_WORKER_SERVERS } = plan.flujo ?? {}; + return new FlujoSessionDriver({ program: new FlujoProgram({ port, environment: () => flujoEnvironment({ home, port, extra }), onOutput: rotatingLog('/data/factory/flujo.log') }), + stateDirectory: '/data/factory', privateDirectory: '/data/factory-private', servers, partialMessages: plan.traceDeltas === true, onStderr, + team: { concurrency, agentTurns, leadTurns, specialists, roots: ['/data/results'] }, + modelRecord: model => flujoProviderModel(plan.modelProvider, { index, key: providerKey, model }) }); + }, + version: ({ rpc }) => rpc.version(), + }, +}; +const CLAUDE_COMMAND_WRAPPER = '/data/factory/claude-command-env.sh'; + +/** FLUJO logs freely, so its log keeps only the newest two parts. */ +function rotatingLog(file, partBytes = 32 * 1024 * 1024) { + let stream = createWriteStream(file, { flags: 'a', mode: 0o600 }), bytes = 0; + return text => { + if (bytes >= partBytes) { stream.end(); renameSync(file, `${file}.1`); stream = createWriteStream(file, { flags: 'a', mode: 0o600 }); bytes = 0; } + bytes += Buffer.byteLength(text); stream.write(text); + }; +} + export async function runCloudLiteMachine(config) { const { plan, index, credentials, operatorToken, initialAuth = {}, githubToken, evidenceTokens, providerKey, deployKey } = config; // Optional: a self-hosted model provider instead of the subscription login, read-only // deploy-key repository access, a prepare hook for agent workspaces, operator-driven turns. const providerMode = Boolean(plan.modelProvider); - const claudeProgram = plan.workerProgram === 'claude'; - const evidenceSessionsPath = claudeProgram ? '/data/claude/projects' : '/data/codex/sessions'; + const programName = plan.workerProgram ?? 'codex'; + const program = WORKER_PROGRAMS[programName]; + const evidenceRoots = { sessionsPath: program.sessions, ...(program.sessionsExclude ? { sessionsExclude: program.sessionsExclude } : {}) }; const operatorSupervised = plan.supervision === 'operator'; const extraSecrets = [providerKey, deployKey].filter(value => typeof value === 'string' && value.length >= 8); mkdirSync('/data/factory', { recursive: true, mode: 0o700 }); @@ -60,7 +117,7 @@ export async function runCloudLiteMachine(config) { if(!retainedEvidence.receipt?.checkpoint.final){ const child=fork(fileURLToPath(new URL('./evidence-worker.mjs',import.meta.url)),[],{stdio:['ignore','ignore','ignore','ipc'],windowsHide:true}); child.on('error',()=>{}); - child.send({action:'start',runId:plan.runId,machineId:`m${index}`,url:plan.collection.url,token:evidenceTokens?.[`m${index}`],intervalMs:plan.collection.intervalMs??60000,sessions:sealedCheckpoint.sessions,sessionsPath:evidenceSessionsPath}); + child.send({action:'start',runId:plan.runId,machineId:`m${index}`,url:plan.collection.url,token:evidenceTokens?.[`m${index}`],intervalMs:plan.collection.intervalMs??60000,sessions:sealedCheckpoint.sessions,...evidenceRoots}); child.send({action:'finalize',sessions:sealedCheckpoint.sessions}); } return {sealed:true,stop:async()=>coordinator?.close()}; @@ -92,7 +149,7 @@ export async function runCloudLiteMachine(config) { return{agentId:agent.id,threadId:receipt.threadId,stopId:receipt.commandId}; }); const child=fork(fileURLToPath(new URL('./evidence-worker.mjs',import.meta.url)),[],{stdio:['ignore','ignore','ignore','ipc'],windowsHide:true});child.on('error',()=>{}); - child.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions,sessionsPath:evidenceSessionsPath}); + child.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions,...evidenceRoots}); child.send({action:'finalize',sessions});return{sealed:true,stop:async()=>coordinator?.close()}; } if(plan.shutdown && Date.now()>=Date.parse(plan.shutdown.deadlineUtc))throw new Error('Work deadline has passed; preserve existing evidence and reconcile shutdown without restarting agents.'); @@ -136,10 +193,9 @@ export async function runCloudLiteMachine(config) { prepared = await runPrepareHook({ plan, index, repositoryDirectory, env: gitEnv }); writeFileSync(preparedFile, JSON.stringify(prepared), { mode: 0o600 }); } - const home = claudeProgram ? '/data/claude' : '/data/codex'; mkdirSync(home, { recursive: true, mode: 0o700 }); + const home = program.home; mkdirSync(home, { recursive: true, mode: 0o700 }); // A fresh native harness has only the explicitly configured FACTORY MCP server. - if (!claudeProgram) writeFileSync(join(home, 'config.toml'), providerMode ? providerConfigToml(plan, index) - : 'cli_auth_credentials_store="ephemeral"\nforced_login_method="chatgpt"\n', { mode: 0o600 }); + program.prepare({ plan, index, home, providerMode }); const traces = new Map(); const workflowWakes = new Map(); for (const agent of bootstrap.agents) { @@ -155,20 +211,9 @@ export async function runCloudLiteMachine(config) { liveAuth = auth; return auth; }; - const commandWrapper = '/data/factory/claude-command-env.sh'; - if (claudeProgram) writeFileSync(commandWrapper, claudeCommandWrapper({ provider: providerMode }), { mode: 0o700 }); - const rpc = claudeProgram - ? new ClaudeSessionDriver({ configDirectory: home, stateDirectory: '/data/factory', fixedModel: plan.modelProvider?.model, partialMessages: plan.traceDeltas === true, - // Read for every turn, so a refreshed subscription token reaches the next process. - environment: () => claudeEnvironment({ plan, index, home, commandWrapper, githubToken, deployKey, providerKey, oauthToken: liveAuth?.accessToken }), - onStderr: text => rootTrace.record('claude.stderr', { text }) }) - : new CodexRpc({ env: codexEnvironment({ plan, home, githubToken, deployKey, providerKey }), cwd: repositoryDirectory, - onStderr: text => rootTrace.record('codex.stderr', { text }), - onRequest: async (method, params) => { - if (method === 'account/chatgptAuthTokens/refresh' && !providerMode) return refreshAuth(); - throw new Error(`Unexpected Codex host request: ${method}`); - } }); - if (claudeProgram && !providerMode) rpc.on('authFailed', () => { void refreshAuth().catch(error => rootTrace.record('auth.refresh_pending', { message: error.message })); }); + const rpc = program.driver({ plan, index, home, githubToken, deployKey, providerKey, providerMode, repositoryDirectory, refreshAuth, getLiveAuth: () => liveAuth, + onStderr: text => rootTrace.record(`${programName}.stderr`, { text }) }); + if (!providerMode) rpc.on('authFailed', () => { void refreshAuth().catch(error => rootTrace.record('auth.refresh_pending', { message: error.message })); }); rpc.on('notification', value => { const threadId = value.params?.threadId ?? value.params?.thread?.id; const agent = threadAgents.get(threadId) ?? rootId; @@ -201,7 +246,7 @@ export async function runCloudLiteMachine(config) { stopControllers.get(threadId)?.observeTerminal({ threadId, turnId, status }); } }); - rpc.on('exit', result => { rootTrace.record('codex.exit', result); void stop(); }); + rpc.on('exit', result => { rootTrace.record(program.exitTrace, result); void stop(); }); timer = setInterval(async () => { for (const [id, trace] of traces) { try { await trace.flush(); } catch (error) { rootTrace.record('trace.delivery_pending', { agent: id, message: error.message }); } @@ -209,8 +254,8 @@ export async function runCloudLiteMachine(config) { }, 1000); await rpc.initialize(); // A self-hosted provider authenticates with its key; no subscription login is involved. - if (!providerMode && !claudeProgram) await rpc.request('account/login/start', { type: 'chatgptAuthTokens', ...bootstrap.auth }); - rootTrace.record('machine.ready', { machine, node: process.version, ...(claudeProgram ? { claude: await run('claude', ['--version']) } : { codex: await run('codex', ['--version']) }), configuration: { cpuKind: plan.cpuKind, cpus: plan.cpus, memoryMb: plan.memoryMb, diskGb: plan.diskGb }, agents: bootstrap.agents.map(x => ({ id: x.id, role: x.role })) }); + if (!providerMode && program.login) await rpc.request('account/login/start', { type: 'chatgptAuthTokens', ...bootstrap.auth }); + rootTrace.record('machine.ready', { machine, node: process.version, [programName]: await program.version({ run, rpc }), configuration: { cpuKind: plan.cpuKind, cpus: plan.cpus, memoryMb: plan.memoryMb, diskGb: plan.diskGb }, agents: bootstrap.agents.map(x => ({ id: x.id, role: x.role })) }); const proxy = fileURLToPath(new URL('../../bin/factory-cloud-mcp.mjs', import.meta.url)); for (const agent of bootstrap.agents) { // With a prepare hook, each agent works in the folder the hook prepared for its slot. @@ -229,8 +274,8 @@ export async function runCloudLiteMachine(config) { const retained = JSON.parse(readFileSync(sessionFile)); try { result = await rpc.request('thread/resume', { threadId: retained.threadId, cwd: directory, config: configOverride, approvalPolicy: 'never', sandbox, - // Claude Code starts a new process per turn, so it needs these again after a restart. - ...(claudeProgram ? { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, developerInstructions: instructions } : {}) }); + // A driver without a long-lived native session needs these again after a restart. + ...(program.resumeNeedsSettings ? { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, developerInstructions: instructions } : {}), ...program.sessionFields?.(agent) }); } catch (error) { if (error.code !== -32600 || error.message !== `no rollout found for thread id ${retained.threadId}`) throw error; unavailableSession = retained.threadId; recovering = false; @@ -242,7 +287,7 @@ export async function runCloudLiteMachine(config) { } if (!result) { result = await rpc.request('thread/start', { model: agent.role === 'orchestrator' ? plan.model : plan.subagentModel, cwd: directory, approvalPolicy: 'never', sandbox, config: configOverride, - developerInstructions: instructions, serviceName: `factory.${plan.runId}.${agent.id}`, historyMode: 'legacy' }); + developerInstructions: instructions, serviceName: `factory.${plan.runId}.${agent.id}`, historyMode: 'legacy', ...program.sessionFields?.(agent) }); const fd = await open(sessionFile, 'w', 0o600); try { await fd.writeFile(JSON.stringify({ threadId: result.thread.id, agentId: agent.id })); await fd.sync(); } finally { await fd.close(); } } @@ -259,6 +304,12 @@ export async function runCloudLiteMachine(config) { const threadEvent = recovering ? 'thread.resumed' : 'thread.started'; traces.get(agent.id).record('native.' + threadEvent, { threadId, ...(observedThreadStatus ? { threadStatus: nativeThreadStatus(observedThreadStatus) } : {}) }); + // A turn the old process never finished is closed on the board, or a later stop could not name its target. + if (result.recoveredTurn) { + turns.set(threadId, result.recoveredTurn.id); + terminalTurns.set(threadId, result.recoveredTurn); + traces.get(agent.id).record('native.turn.completed', { threadId, turn: result.recoveredTurn }); + } // Operator-supervised agents start idle; every turn comes from a durable operator message. if (!operatorSupervised && current?.thread.status?.type !== 'active') { const recoveryNote = unavailableSession ? `The prior prepared session ${unavailableSession} has no persisted rollout. Reconcile your existing FACTORY claims, inbox, checkout and GitHub state before continuing; do not replay previous delivery operations.\n` : ''; @@ -281,7 +332,7 @@ export async function runCloudLiteMachine(config) { if(message.type==='evidence.final')evidenceFinal=true; writeFileSync('/data/factory-evidence-status.json',JSON.stringify({at:new Date().toISOString(),...message}),{mode:0o600}); }); - evidenceProcess.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions:evidenceSessions(),sessionsPath:evidenceSessionsPath}); + evidenceProcess.send({action:'start',runId:plan.runId,machineId:machine,url:plan.collection.url,token:evidenceTokens?.[machine],intervalMs:plan.collection.intervalMs??60000,sessions:evidenceSessions(),...evidenceRoots}); }; launchCollector(); monitoring = setInterval(async () => { diff --git a/src/flujo-swarm/flujo-client.mjs b/src/flujo-swarm/flujo-client.mjs index d9f8404..f211833 100644 --- a/src/flujo-swarm/flujo-client.mjs +++ b/src/flujo-swarm/flujo-client.mjs @@ -90,6 +90,11 @@ export class FlujoClient { return true; } + /** The folders this workspace's file and shell tools may use. */ + async setWorkspaceRoots(roots, name = this.workspace) { + return this.expect('PATCH', '/api/workspaces', { name, newName: name, roots }, { workspace: null }); + } + /** Read-only observation; a missing workspace must stay missing throughout the window. */ async confirmWorkspaceAbsent(name, { verificationWindowMs = 15_000, pollIntervalMs = 500 } = {}) { if (!Number.isSafeInteger(verificationWindowMs) || verificationWindowMs < 1 || @@ -162,13 +167,13 @@ export class FlujoClient { return result.body.flow; } - async conversation(id) { - return this.api('GET', `/v1/chat/conversations/${encodeURIComponent(id)}`); + async conversation(id, { messageLimit } = {}) { + return this.api('GET', `/v1/chat/conversations/${encodeURIComponent(id)}${messageLimit ? `?messageLimit=${messageLimit}` : ''}`); } /** Read FLUJO's persisted conversation hierarchy without opening message bodies. */ - async descendants(id) { - return this.api('GET', `/v1/chat/conversations?paged=1&limit=200&descendantsOf=${encodeURIComponent(id)}`); + async descendants(id, cursor) { + return this.api('GET', `/v1/chat/conversations?paged=1&limit=200&descendantsOf=${encodeURIComponent(id)}${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`); } async inject(id, content, messageId = randomUUID()) { diff --git a/src/flujo-swarm/template/factory-team.mjs b/src/flujo-swarm/template/factory-team.mjs index 74b478d..b2ecaca 100644 --- a/src/flujo-swarm/template/factory-team.mjs +++ b/src/flujo-swarm/template/factory-team.mjs @@ -34,7 +34,7 @@ function environmentText(value) { /** FLUJO subflow team template. One lead plus up to nine local agents per workspace. */ export function buildFactoryTeamSpecs({ model, availableServers = [], availableTools = {}, - limits = {}, specialists, goalContext, environment } = {}) { + limits = {}, specialists, goalContext, environment, instructions } = {}) { if (typeof model !== 'string' || !model.trim() || !Array.isArray(availableServers) || new Set(availableServers).size !== availableServers.length || availableServers.some(name => typeof name !== 'string')) throw new TypeError('Model and connected server inventory are required'); @@ -56,7 +56,8 @@ export function buildFactoryTeamSpecs({ model, availableServers = [], availableT specialists.roles.map(role => `${role.id}: ${role.mission} ${role.instructions} Evidence: ${role.evidence}`).join('\n') : ''; const goal = context(goalContext, 'goalContext'); const surroundings = environmentText(environment); - const shared = `${goal ? `\nGoal context: ${goal}` : ''}${surroundings ? `\nEnvironment:\n${surroundings}` : ''}`; + if (instructions !== undefined && (typeof instructions !== 'string' || instructions.length > 64000)) throw new TypeError('instructions must be text under 64000 characters'); + const shared = `${goal ? `\nGoal context: ${goal}` : ''}${surroundings ? `\nEnvironment:\n${surroundings}` : ''}${instructions?.trim() ? `\n${instructions.trim()}` : ''}`; const principles = `Use the tools actually attached to this flow; never claim an action or result without its receipt. ` + `Reuse relevant installed upstream code and contracts before making a small adapter or product change. ` + `Deliver a bounded, useful product change where the task calls for one. Coordinate concisely on an available shared board and linked issue or PR. ` + diff --git a/src/lite-worker.mjs b/src/lite-worker.mjs index 8d4205f..532500f 100644 --- a/src/lite-worker.mjs +++ b/src/lite-worker.mjs @@ -2,8 +2,10 @@ import { resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { homedir } from 'node:os'; import { spawn } from 'node:child_process'; -import { mkdirSync } from 'node:fs'; -import { applyProviderEnvironment, claudeProviderEnvironment, codexProviderOverrides, validateModelProvider } from './model-provider.mjs'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { applyProviderEnvironment, claudeProviderEnvironment, codexProviderOverrides, flujoProviderModel, validateModelProvider } from './model-provider.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_PORT } from './cloud-lite/flujo-driver.mjs'; const mcp = fileURLToPath(new URL('../bin/factory-mcp.mjs', import.meta.url)); @@ -11,10 +13,11 @@ const mcp = fileURLToPath(new URL('../bin/factory-mcp.mjs', import.meta.url)); * Build a normal provider CLI command with its native tools and only FACTORY's MCP server. * With `modelProvider` (and its `providerKey`) the program talks to a self-hosted model * instead of its vendor login: Codex through the Responses API, Claude Code through the - * Anthropic Messages API. `providerIndex` picks one of several endpoints. + * Anthropic Messages API, FLUJO through a stored model record. `providerIndex` picks one of + * several endpoints. For FLUJO this is the command that starts its server. */ -export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = process.cwd(), modelProvider, providerKey, providerIndex = 0, env = process.env } = {}) { - if (!['codex', 'claude'].includes(provider)) throw new TypeError('provider must be codex or claude'); +export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = process.cwd(), modelProvider, providerKey, providerIndex = 0, env = process.env, port = FLUJO_WORKER_PORT, dataDirectory } = {}) { + if (!['codex', 'claude', 'flujo'].includes(provider)) throw new TypeError('provider must be codex, claude or flujo'); if (typeof database !== 'string' || !database.trim()) throw new TypeError('database path is required'); if (prompt !== undefined && (typeof prompt !== 'string' || !prompt.trim())) throw new TypeError('prompt must be nonempty'); const db = resolve(database); @@ -25,6 +28,11 @@ export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = p if (!Number.isSafeInteger(providerIndex) || providerIndex < 0) throw new TypeError('providerIndex must be a nonnegative integer.'); } const server = { command: process.execPath, args: [mcp, db, ...(agentId ? ['--actor',agentId] : [])] }; + if (provider === 'flujo') return { + command: 'flujo', cwd, server, args: ['--no-open', '--port', String(port)], + // Its own data folder, so a worker never touches the settings of a FLUJO someone uses by hand. + env: { ...env, FLUJO_DATA_DIR: dataDirectory ?? resolve(homedir(), '.factory', 'lite-flujo', 'data'), FLUJO_PORT: String(port) }, + }; if (provider === 'claude') return { command: 'claude', cwd, ...(modelProvider ? { env: applyProviderEnvironment(env, claudeProviderEnvironment(modelProvider, { index: providerIndex, key: providerKey })) } : {}), @@ -42,8 +50,57 @@ export function liteWorkerCommand({ provider, database, prompt, agentId, cwd = p ...(prompt === undefined ? [] : [prompt])] }; } -/** Launch an interactive native CLI session and return its ChildProcess. */ +/** + * A FLUJO worker on this computer: attaches to `origin`, or starts its own FLUJO server, then installs + * one workspace with the generic team template and the board. The agent keeps its conversation across runs. + */ +export async function startFlujoLiteWorker({ origin, token, prompt, modelProvider, providerKey, providerIndex = 0, flujoModel, model, teamConcurrency, agentRole = 'orchestrator', servers, stateDirectory, developerInstructions, driverFactory = settings => new FlujoSessionDriver(settings), programFactory = settings => new FlujoProgram(settings), ...options } = {}) { + const spec = liteWorkerCommand({ ...options, provider: 'flujo', prompt, modelProvider, providerKey, providerIndex }); + if (!modelProvider && !flujoModel) throw new TypeError('A FLUJO worker needs a modelProvider or a flujoModel record.'); + const state = stateDirectory ?? resolve(homedir(), '.factory', 'lite-flujo', options.agentId ?? 'default'); + mkdirSync(state, { recursive: true, mode: 0o700 }); + const program = origin ? undefined : programFactory({ port: Number(spec.env.FLUJO_PORT), environment: () => spec.env }); + const driver = driverFactory({ origin, token, program, stateDirectory: state, servers, team: { ...(teamConcurrency ? { concurrency: teamConcurrency } : {}) }, + modelRecord: name => flujoModel ?? flujoProviderModel(modelProvider, { index: providerIndex, key: providerKey, model: model ?? name }) }); + await driver.initialize(); + const settings = { cwd: spec.cwd, model, sandbox: 'danger-full-access', agentRole, developerInstructions, config: { mcp_servers: { factory: spec.server } } }; + const sessionFile = join(state, 'session.json'); + let opened; + if (existsSync(sessionFile)) { + try { opened = await driver.request('thread/resume', { ...settings, threadId: JSON.parse(readFileSync(sessionFile, 'utf8')).threadId }); } + catch (error) { if (error.code !== -32600) throw error; } + } + opened ??= await driver.request('thread/start', settings); + const threadId = opened.thread.id; + writeFileSync(sessionFile, JSON.stringify({ threadId }), { mode: 0o600 }); + const conversation = text => [{ type: 'text', text }]; + const worker = { origin: driver.origin, workspace: driver.thread(threadId).workspace, threadId, driver, child: program?.child ?? null, + /** Starts a turn, or joins the one that is running. */ + send: async text => { + const running = driver.thread(threadId).turn; + if (!running) return driver.request('turn/start', { threadId, input: conversation(text) }); + return driver.request('turn/steer', { threadId, expectedTurnId: running.id, input: conversation(text) }); + }, + /** Resolves when the current turn has ended, with its final status. */ + settled: () => new Promise(accept => { + const last = driver.thread(threadId); + if (!last.turn) { accept(last.lastTurn); return; } + const listen = note => { if (note.method === 'turn/completed' && note.params.threadId === threadId) { driver.off('notification', listen); accept(note.params.turn); } }; + driver.on('notification', listen); + }), + stop: async () => { + const running = driver.thread(threadId).turn; + if (running) await driver.request('turn/interrupt', { threadId, turnId: running.id }).catch(() => {}); + driver.close(); + await program?.stop(); + } }; + if (prompt !== undefined) await worker.send(prompt); + return worker; +} + +/** Launch an interactive native CLI session and return its ChildProcess. FLUJO has no such session: it returns a promise of its worker handle. */ export function startLiteWorker({ stdio = 'inherit', spawnImpl = spawn, ...options } = {}) { + if (options.provider === 'flujo') return startFlujoLiteWorker(options); const spec = liteWorkerCommand(options); if (spec.env?.CODEX_HOME) mkdirSync(spec.env.CODEX_HOME, { recursive: true, mode: 0o700 }); return spawnImpl(spec.command, spec.args, { cwd: spec.cwd, env: spec.env, stdio }); diff --git a/src/model-provider.mjs b/src/model-provider.mjs index 6e03367..3e2f83c 100644 --- a/src/model-provider.mjs +++ b/src/model-provider.mjs @@ -4,8 +4,11 @@ /** Wire APIs an endpoint can offer. */ export const MODEL_PROVIDER_APIS = ['responses', 'chat', 'anthropic']; -/** The wire API each worker program needs. */ -export const WORKER_PROGRAM_APIS = Object.freeze({ codex: 'responses', claude: 'anthropic' }); +/** + * The wire APIs each worker program can use, best first. Codex and Claude Code know one each; + * FLUJO has an adapter for all three and prefers the Responses API. + */ +export const WORKER_PROGRAM_APIS = Object.freeze({ codex: Object.freeze(['responses']), claude: Object.freeze(['anthropic']), flujo: Object.freeze(['responses', 'chat', 'anthropic']) }); // Variables the launchers set for the worker process itself; a provider key must not replace them. const RESERVED_ENVIRONMENT = new Set(['PATH', 'HOME', 'CODEX_HOME', 'GH_TOKEN', 'GIT_TERMINAL_PROMPT', 'ANTHROPIC_AUTH_TOKEN', 'ANTHROPIC_API_KEY', 'ANTHROPIC_BASE_URL', 'CLAUDE_CODE_OAUTH_TOKEN']); @@ -44,17 +47,27 @@ export function validateModelProvider(provider) { /** The wire APIs a provider offers (the default keeps 0.4.0 descriptions valid for Codex). */ export const providerApis = provider => provider.apis ?? ['responses']; +/** The API a worker program would use on this provider, or undefined when they share none. */ +export function programApi(provider, program) { + const apis = providerApis(provider); + return WORKER_PROGRAM_APIS[program]?.find(api => apis.includes(api)); +} + /** Which worker programs can use this provider. */ export function providerPrograms(provider) { - const apis = providerApis(provider); - return Object.entries(WORKER_PROGRAM_APIS).filter(([, api]) => apis.includes(api)).map(([program]) => program); + return Object.keys(WORKER_PROGRAM_APIS).filter(program => programApi(provider, program)); } +const apiNames = program => WORKER_PROGRAM_APIS[program].join(' or '); + function requireApi(provider, program) { validateModelProvider(provider); - if (!providerApis(provider).includes(WORKER_PROGRAM_APIS[program])) { - throw new TypeError(`${program} needs a provider that offers the ${WORKER_PROGRAM_APIS[program]} API.`); - } + if (!programApi(provider, program)) throw new TypeError(`${program} needs a provider that offers the ${apiNames(program)} API.`); +} + +/** A cloud plan's refusal, in the words its operator knows. */ +export function requireFleetApi(provider, program) { + if (!programApi(provider, program)) throw new TypeError(`Cloud lite workers need a provider that offers the ${apiNames(program)} API.`); } /** Spread workers over the endpoints so each keeps its prompt cache on one server. */ @@ -144,6 +157,31 @@ export function claudeProviderEnvironment(provider, { index = 0, key, model } = }; } +// FLUJO keeps a model as a stored record, so its settings differ by adapter rather than by variable. +const FLUJO_ADAPTERS = Object.freeze({ responses: 'openai-responses', chat: 'openai', anthropic: 'anthropic' }); +// FLUJO moves the system message behind the conversation for OpenAI's own prompt cache. Open models' +// chat templates refuse that, so only OpenAI's endpoint may carry the openai name. +const flujoProviderName = (api, base) => api === 'anthropic' ? 'anthropic' : new URL(base).hostname === 'api.openai.com' ? 'openai' : 'litellm'; + +/** + * FLUJO model record for a self-hosted provider, for `POST /api/model`. FLUJO stores the key in its + * own encrypted settings, so it never enters the process environment that agent commands inherit. + */ +export function flujoProviderModel(provider, { index = 0, key, model } = {}) { + requireApi(provider, 'flujo'); + if (typeof key !== 'string' || key.length < 8) throw new TypeError('A model provider needs its API key.'); + const served = provider.model ?? model; + if (typeof served !== 'string' || !served) throw new TypeError('Select the served model name.'); + const api = programApi(provider, 'flujo'); + const base = providerBaseUrl(provider, index); + return { id: `factory-${provider.id}`, name: served, displayName: provider.name ?? provider.id, provider: flujoProviderName(api, base), adapter: FLUJO_ADAPTERS[api], + // The Anthropic client adds /v1 itself. + baseUrl: api === 'anthropic' ? base.replace(/\/+$/, '').replace(/\/v1$/, '') : base, + ApiKey: key, supportsTools: true, + ...(provider.contextWindow ? { contextWindow: provider.contextWindow } : {}), + ...(provider.autoCompactTokenLimit ? { compactionThreshold: provider.autoCompactTokenLimit } : {}) }; +} + /** Apply a mapping's environment to an inherited one (null removes a variable). */ export function applyProviderEnvironment(base, overlay) { const env = { ...base }; diff --git a/src/public-sdk.mjs b/src/public-sdk.mjs index fd0c5e1..3f605d7 100644 --- a/src/public-sdk.mjs +++ b/src/public-sdk.mjs @@ -96,10 +96,12 @@ export { buildSaviaCasePlan } from './flujo-swarm/template/savia-case-plan.mjs'; export { observeSaviaCaseTopology } from './flujo-swarm/template/savia-topology.mjs'; export { createPresentationServer, startPresentationServer } from './presentation.mjs'; export { createObservatoryClient } from './observatory-client.mjs'; -export { liteWorkerCommand, startLiteWorker } from './lite-worker.mjs'; +export { liteWorkerCommand, startLiteWorker, startFlujoLiteWorker } from './lite-worker.mjs'; export { ModalSelfHostedModel, selfHostedProfiles, probeSelfHostedModel, modalResult, readBoundedJson, modalEndpointOrigin, runCommand, SELFHOST_MODAL_DIR, SELFHOSTED_KEY_ENV } from './selfhost/modal.mjs'; -export { MODEL_PROVIDER_APIS, WORKER_PROGRAM_APIS, validateModelProvider, providerPrograms, codexProviderConfig, codexProviderOverrides, claudeProviderEnvironment } from './model-provider.mjs'; +export { MODEL_PROVIDER_APIS, WORKER_PROGRAM_APIS, validateModelProvider, providerPrograms, codexProviderConfig, codexProviderOverrides, claudeProviderEnvironment, flujoProviderModel, programApi } from './model-provider.mjs'; +export { FlujoProgram, FlujoSessionDriver, flujoEnvironment } from './cloud-lite/flujo-driver.mjs'; export { FactoryCloudLiteFleet, CodexSubscriptionBroker, ClaudeSubscriptionBroker } from './cloud-lite/fleet.mjs'; export { validateCloudLitePlan, agentInventory, machineConfig as cloudLiteMachineConfig } from './cloud-lite/protocol.mjs'; export { cloudLiteBootstrap } from './cloud-lite/bootstrap.mjs'; +export { packFactorySdk } from './cloud-lite/bundle.mjs'; export * from './workflows/index.mjs'; diff --git a/src/selfhost/modal.mjs b/src/selfhost/modal.mjs index bdd369e..1a2ba0d 100644 --- a/src/selfhost/modal.mjs +++ b/src/selfhost/modal.mjs @@ -16,6 +16,11 @@ const MINUTE = 60_000; const notApplied = message => Object.assign(new Error(message), { outcome: 'not_applied' }); const unknown = message => Object.assign(new Error(message), { code: 'UNKNOWN', outcome: 'unknown', unknown: true }); +const pause = (ms, signal) => new Promise((resolve, reject) => { + if (signal?.aborted) { reject(uncertain()); return; } + const timer = setTimeout(resolve, ms); + signal?.addEventListener('abort', () => { clearTimeout(timer); reject(uncertain()); }, { once: true }); +}); const uncertain = () => unknown('The self-hosted model has an uncertain outcome. Its owned resources are kept for reconciliation; retire them before deploying again.'); const json = raw => { try { return JSON.parse(raw); } catch { return null; } }; const lastJsonLine = raw => json(String(raw ?? '').trim().split(/\r?\n/).reverse().find(line => line.trim().startsWith('{')) ?? ''); @@ -169,10 +174,10 @@ function fileCredentialStore(directory) { export class ModalSelfHostedModel { constructor({ directory, python = process.platform === 'win32' ? 'python' : 'python3', pythonArgs = [], env = process.env, commandRunner = runCommand, fetchImpl = fetch, credentialStore, keyName = 'selfhosted-model', namePrefix = 'factory', - clock = Date.now, deployTimeoutMs = 45 * MINUTE, prefetchTimeoutMs = 70 * MINUTE } = {}) { + clock = Date.now, deployTimeoutMs = 45 * MINUTE, prefetchTimeoutMs = 70 * MINUTE, stopWaitMs = 3 * MINUTE, stopPollMs = 3000, deleteAttempts = 5 } = {}) { if (typeof directory !== 'string' || !directory) throw new TypeError('Select a private state directory.'); if (!/^[a-z][a-z0-9]{1,15}$/.test(namePrefix)) throw new TypeError('namePrefix must be a short lowercase name.'); - Object.assign(this, { directory, python, pythonArgs: [...pythonArgs], commandRunner, fetch: fetchImpl, keyName, namePrefix, clock, deployTimeoutMs, prefetchTimeoutMs }); + Object.assign(this, { stopWaitMs, stopPollMs, deleteAttempts, directory, python, pythonArgs: [...pythonArgs], commandRunner, fetch: fetchImpl, keyName, namePrefix, clock, deployTimeoutMs, prefetchTimeoutMs }); this.environment = typeof env === 'function' ? env : () => env; this.store = credentialStore ?? fileCredentialStore(directory); this.path = join(directory, 'model.json'); @@ -349,11 +354,15 @@ export class ModalSelfHostedModel { async #stopApp(signal) { const stopped = await this.#cli(['app', 'stop', '--yes', this.record.appName], { signal }); - const states = await this.#appState(signal); - // Stopped (now or earlier), or never created after an early deployment failure, is a - // confirmed GPU-off state; anything else stays uncertain. - if (states && !states.every(state => /^stopped$/i.test(state))) throw uncertain(); - return { stopCommand: stopped?.code === 0 ? 'ok' : 'refused', listed: Boolean(states) }; + // A busy replica takes a while to stop. Stopped (now or earlier), or never created after an + // early deployment failure, is a confirmed GPU-off state; still stopping at the deadline stays uncertain. + const deadline = this.clock() + this.stopWaitMs; + for (;;) { + const states = await this.#appState(signal); + if (!states || states.every(state => /^stopped$/i.test(state))) return { stopCommand: stopped?.code === 0 ? 'ok' : 'refused', listed: Boolean(states) }; + if (!states.some(state => /stopping/i.test(state)) && stopped?.code !== 0 || this.clock() >= deadline) throw uncertain(); + await pause(this.stopPollMs, signal); + } } /** Stop the GPU containers now; key, secret and cached weights stay for a later redeploy. */ @@ -408,8 +417,13 @@ export class ModalSelfHostedModel { const app = record.appName; const owned = [['secret', `${app}-auth`], ['volume', `${app}-hf-cache`], ['volume', `${app}-vllm-cache`]]; for (const [kind, name] of owned) { - const deleted = await this.#cli([kind, 'delete', '--allow-missing', '--yes', name], { signal }); - if (deleted?.code !== 0) throw uncertain(); + // Modal can refuse a delete for a moment while the stopped app lets go of its volumes. + for (let attempt = 1; ; attempt++) { + const deleted = await this.#cli([kind, 'delete', '--allow-missing', '--yes', name], { signal }); + if (deleted?.code === 0) break; + if (attempt >= this.deleteAttempts) throw uncertain(); + await pause(this.stopPollMs, signal); + } } const [secrets, volumes] = await Promise.all([this.#cli(['secret', 'list', '--json'], { signal }), this.#cli(['volume', 'list', '--json'], { signal })]); const remaining = [json(secrets?.stdout), json(volumes?.stdout)]; diff --git a/test/cloud-lite-flujo.test.mjs b/test/cloud-lite-flujo.test.mjs new file mode 100644 index 0000000..9ea1558 --- /dev/null +++ b/test/cloud-lite-flujo.test.mjs @@ -0,0 +1,703 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from 'node:fs'; +import { createServer } from 'node:net'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { validateCloudLitePlan, agentInventory, machineConfig, secret } from '../src/cloud-lite/protocol.mjs'; +import { cloudLiteBootstrap } from '../src/cloud-lite/bootstrap.mjs'; +import { FlujoProgram, FlujoSessionDriver, FLUJO_WORKER_SERVERS, flujoEnvironment } from '../src/cloud-lite/flujo-driver.mjs'; +import { packFactorySdk } from '../src/cloud-lite/bundle.mjs'; +import { FactoryCloudLiteFleet } from '../src/cloud-lite/fleet.mjs'; +import { NativeOperatorStop } from '../src/cloud-lite/operator-stop.mjs'; +import { NativeWorkflowWake } from '../src/cloud-lite/workflow-wake.mjs'; +import { CloudLiteTrace } from '../src/cloud-lite/trace.mjs'; +import { startCloudLiteCoordinator } from '../src/cloud-lite/server.mjs'; +import { nativeStopTurn } from '../src/cloud-lite/runtime.mjs'; +import { FactoryArtifactStore } from '../src/workflows/artifacts.mjs'; +import { captureWorkspace, readWorkspaceManifest } from '../src/workflows/workspace.mjs'; +import { flujoProviderModel, liteWorkerCommand, programApi, startLiteWorker } from '../src/public-sdk.mjs'; +import { FakeFlujo } from './fixtures/fake-flujo.mjs'; +import { tempDir } from './support/temp-dir.mjs'; + +const fresh = () => tempDir('factory-cloud-lite-flujo-'); +const fakeProgram = fileURLToPath(new URL('./fixtures/fake-flujo.mjs', import.meta.url)); +const key = 'provider-key-value-1234'; +const provider = (extra = {}) => ({ id: 'selfhosted', envKey: 'SELF_HOSTED_MODEL_KEY', model: 'served-model', name: 'Served', apis: ['responses', 'chat', 'anthropic'], + baseUrls: ['https://model-a.example.test/v1', 'https://model-b.example.test/v1'], contextWindow: 262144, autoCompactTokenLimit: 80000, ...extra }); +const plan = (extra = {}) => validateCloudLitePlan({ runId: 'flujo-test', app: 'factory-flujo-test', goal: 'Work on operator jobs', machines: 1, agentsPerMachine: 2, + workerProgram: 'flujo', modelProvider: provider(), ...extra }); +const text = value => [{ type: 'text', text: value }]; +const board = { command: process.execPath, args: ['board-proxy.mjs'], env: { FACTORY_BOARD_URL: 'http://board.invalid', FACTORY_BOARD_TOKEN: 'board-credential-value', FACTORY_WORKTREE: '/work/a0' } }; +const waitFor = async (condition, ms = 5000) => { const end = Date.now() + ms; for (;;) { const value = await condition(); if (value) return value; if (Date.now() > end) throw new Error('Condition was not reached.'); await new Promise(resolve => setTimeout(resolve, 10)); } }; + +async function harness({ script, root = fresh(), team, servers, fake: existing, ...options } = {}) { + const fake = existing ?? await new FakeFlujo({ script }).listen(); + const driver = new FlujoSessionDriver({ origin: fake.origin, stateDirectory: join(root, 'state'), privateDirectory: join(root, 'private'), team, servers, + modelRecord: name => flujoProviderModel(provider(), { key, model: name }), pollMs: 20, reconnectMs: 10, submitTimeoutMs: 400, interruptResendMs: 30, drainMs: 200, ...options }); + const notes = []; + driver.on('notification', note => notes.push(note)); + const finished = () => new Promise(resolve => { const check = note => { if (note.method === 'turn/completed') { driver.off('notification', check); resolve(note.params.turn); } }; driver.on('notification', check); }); + return { root, fake, driver, notes, finished, end: async () => { driver.close(); await fake.close(); } }; +} +const started = (h, extra = {}) => { mkdirSync(join(h.root, 'work'), { recursive: true }); return h.driver.request('thread/start', { model: 'served-model', cwd: join(h.root, 'work'), sandbox: 'danger-full-access', developerInstructions: 'Be useful.', ...extra }); }; +const items = (notes, type) => notes.filter(note => note.method === 'item/completed' && note.params.item.type === type).map(note => note.params.item); + +test('a plan can choose FLUJO, and refuses what FLUJO cannot do', () => { + assert.equal(plan().workerProgram, 'flujo'); + assert.equal(plan().modelProvider.model, 'served-model'); + assert.equal('model' in plan(), false, 'a FLUJO plan names no vendor model'); + assert.throws(() => plan({ modelProvider: undefined }), /need a modelProvider/); + assert.throws(() => plan({ modelProvider: provider({ model: undefined }) }), /served model/); + assert.equal(plan({ modelProvider: provider({ model: undefined }), model: 'named' }).subagentModel, 'named', 'every agent has a model'); + assert.throws(() => plan({ sandbox: 'read-only' }), /danger-full-access/); + assert.throws(() => plan({ sandbox: 'workspace-write' }), /danger-full-access/); + assert.equal(plan({ sandbox: 'danger-full-access' }).sandbox, 'danger-full-access'); + assert.throws(() => validateCloudLitePlan({ runId: 'codex-test', app: 'factory-codex-test', goal: 'x', flujo: {} }), /belong to FLUJO/); + assert.throws(() => plan({ flujo: { teamConcurrency: 10 } }), /teamConcurrency/); + assert.throws(() => plan({ flujo: { servers: ['flujo'] } }), /flujo\.servers/); + assert.throws(() => plan({ flujo: { agentTurns: 0 } }), /agentTurns/); + assert.equal(plan({ flujo: { teamConcurrency: 4, servers: ['bash'] } }).flujo.teamConcurrency, 4); + assert.equal(machineConfig(plan(), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'flujo-lite'); + assert.equal(machineConfig(plan({ workerProgram: 'claude', model: 'm', sandbox: undefined }), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'claude-lite'); + assert.equal(machineConfig(validateCloudLitePlan({ runId: 'codex-test', app: 'factory-codex-test', goal: 'x' }), { image: 'i', volumeId: 'v', index: 0, environment: {} }).metadata['factory.runtime'], 'codex-lite'); +}); + +test('the FLUJO mapping prefers Responses and stores the key in a model record', () => { + const responses = flujoProviderModel(provider(), { index: 1, key }); + assert.deepEqual([responses.id, responses.name, responses.provider, responses.adapter, responses.baseUrl, responses.ApiKey, responses.contextWindow, responses.compactionThreshold, responses.supportsTools], + ['factory-selfhosted', 'served-model', 'litellm', 'openai-responses', 'https://model-b.example.test/v1', key, 262144, 80000, true]); + // FLUJO's OpenAI cache order puts the system message last, which open models refuse. + assert.equal(flujoProviderModel(provider({ baseUrls: ['https://api.openai.com/v1'] }), { key }).provider, 'openai'); + const chat = flujoProviderModel(provider({ apis: ['chat'] }), { key }); + assert.deepEqual([chat.adapter, chat.baseUrl], ['openai', 'https://model-a.example.test/v1']); + const anthropic = flujoProviderModel(provider({ apis: ['anthropic'] }), { key }); + assert.deepEqual([anthropic.provider, anthropic.adapter, anthropic.baseUrl], ['anthropic', 'anthropic', 'https://model-a.example.test']); + assert.equal(programApi(provider({ apis: ['chat', 'responses'] }), 'flujo'), 'responses', 'the order is the program\'s, not the provider\'s'); + assert.throws(() => flujoProviderModel(provider(), { key: 'short' }), /API key/); + assert.throws(() => flujoProviderModel(provider({ model: undefined }), { key }), /served model/); + assert.equal(flujoProviderModel(provider({ model: undefined }), { key, model: 'from-plan' }).name, 'from-plan'); +}); + +test('the FLUJO process environment carries no key, and the bootstrap pins the program by checksum', () => { + const env = flujoEnvironment({ home: '/data/flujo', path: '/bin' }); + assert.deepEqual(env, { PATH: '/bin', HOME: '/data', FLUJO_DATA_DIR: '/data/flujo', FLUJO_PORT: '4210', FLUJO_EXPOSURE_MODE: 'localhost', NEXT_TELEMETRY_DISABLED: '1', GIT_TERMINAL_PROMPT: '0' }); + const script = cloudLiteBootstrap('a'.repeat(64), { workerProgram: 'flujo' }); + assert.match(script, /flujo-ai-\d+\.\d+\.\d+\.tgz -o \/tmp\/flujo\.tgz/); + assert.match(script, /^echo '[a-f0-9]{64} {2}\/tmp\/flujo\.tgz' \| sha256sum -c -$/m); + assert.ok(script.indexOf('sha256sum -c -\nnpm install --prefix /data/flujo-program') > 0, 'the checksum is checked before install'); + assert.match(script, /--ignore-scripts/); + assert.match(script, /command -v flujo/, 'an image with FLUJO skips the install'); + assert.match(script, /FLUJO_DATA_DIR=\/data\/flujo/); + assert.doesNotMatch(script, /codex|claude/i); + assert.doesNotMatch(cloudLiteBootstrap('a'.repeat(64)), /flujo-ai/); +}); + +test('a thread gets its own workspace with only its tools, and no secret in FLUJO settings', async () => { + const h = await harness(); + try { + const { thread } = await started(h, { config: { mcp_servers: { factory: board }, 'sandbox_workspace_write.writable_roots': ['/extra'] } }); + assert.match(thread.id, /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/); + assert.equal(thread.status.type, 'idle'); + const space = h.fake.workspaces.get(`factory-${thread.id}`); + assert.deepEqual(space.roots, [join(h.root, 'work'), '/extra']); + assert.equal(space.models[0].ApiKey, key); + assert.equal(space.servers.find(server => server.name === 'flujo').disabled, true, 'the control server stays off'); + assert.deepEqual(space.servers.filter(server => !server.disabled).map(server => server.name), [...FLUJO_WORKER_SERVERS, 'factory']); + const server = space.servers.find(entry => entry.name === 'factory'); + assert.equal(JSON.stringify(server).includes('board-credential-value'), false); + assert.deepEqual(server.env, { FACTORY_WORKTREE: '/work/a0' }); + const config = server.args[server.args.indexOf('--config') + 1]; + assert.deepEqual(JSON.parse(readFileSync(config, 'utf8')), { url: 'http://board.invalid', token: 'board-credential-value' }); + if (process.platform !== 'win32') assert.equal(statSync(config).mode & 0o777, 0o600); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent']); + const agent = space.flows[0].spec.nodes.find(node => node.key === 'agent'); + assert.deepEqual(agent.servers.map(entry => entry.name), [...FLUJO_WORKER_SERVERS, 'factory']); + assert.match(space.flows[0].spec.nodes[0].prompt, /Your working folder is .*Pass it as cwd/s); + assert.match(space.flows[0].spec.nodes[0].prompt, /Be useful\./); + assert.equal(h.notes.some(note => note.method === 'thread/started' && note.params.threadId === thread.id), true); + } finally { await h.end(); } +}); + +test('an orchestrator leads a team when the plan asks for one', async () => { + const h = await harness({ team: { concurrency: 4, agentTurns: 6, leadTurns: 12 } }); + try { + const lead = (await started(h, { agentRole: 'orchestrator' })).thread; + const member = (await started(h, { agentRole: 'subagent' })).thread; + const space = h.fake.workspaces.get(`factory-${lead.id}`); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent', 'swarm_team']); + assert.equal(space.flows[1].spec.nodes.find(node => node.key === 'agents').concurrencyLimit, 4); + assert.equal(space.flows[1].spec.nodes.find(node => node.key === 'lead').maxTurns, 12); + await h.driver.request('turn/start', { threadId: lead.id, input: text('lead') }); + await h.driver.request('turn/start', { threadId: member.id, input: text('member') }); + await waitFor(() => h.fake.count('POST', '/v1/chat/completions') === 2); + const flows = h.fake.requests.filter(entry => entry.path === '/v1/chat/completions').map(entry => entry.body.model); + assert.deepEqual(flows, ['flow-swarm_team', 'flow-swarm_agent']); + } finally { await h.end(); } +}); + +test('the board must connect before a thread starts', async () => { + const h = await harness({ toolsTimeoutMs: 50 }); + h.fake.boardFails = true; + try { await assert.rejects(started(h, { config: { mcp_servers: { factory: board } } }), /factory server did not connect/); } + finally { await h.end(); } +}); + +test('a turn maps FLUJO events to the shared trace and ends only when FLUJO says so', async () => { + const h = await harness({ script: run => { run.say('working'); run.tool('bash_run'); run.usage(); run.emit({ type: 'recovery:retry', attempt: 2, retryAt: 5, failure: { category: 'rate_limit' } }); run.emit({ type: 'model:delta', delta: 'skipped' }); run.say('done'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + let readAtEnd; + h.driver.on('notification', note => { if (note.method === 'turn/completed') readAtEnd ??= h.driver.request('thread/read', { threadId: thread.id }); }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal(turn.status, 'inProgress'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + const final = await ended; + assert.deepEqual(final, { id: turn.id, status: 'completed' }); + assert.equal((await readAtEnd).thread.status.type, 'idle', 'idle is already readable when the turn ends'); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['working', 'done']); + assert.equal(items(h.notes, 'toolCall')[0].tool, 'bash_run'); + assert.equal(items(h.notes, 'toolResult')[0].isError, false); + const usage = h.notes.find(note => note.method === 'thread/tokenUsage/updated').params.tokenUsage; + assert.deepEqual(usage.last, { inputTokens: 40, cachedInputTokens: 10, outputTokens: 4 }); + assert.equal(h.notes.find(note => note.method === 'model/retry').params.error, 'rate_limit'); + assert.equal(h.notes.some(note => note.method === 'item/stream/delta'), false, 'streaming stays off unless asked for'); + const methods = h.notes.map(note => note.method); + assert.ok(methods.indexOf('turn/started') < methods.indexOf('turn/completed')); + assert.equal(methods.at(-1), 'thread/status/changed'); + // The next turn continues the same conversation and is not confused by the previous run's end. + const next = h.finished(); + const second = await h.driver.request('turn/start', { threadId: thread.id, input: text('again') }); + assert.equal((await next).id, second.turn.id); + assert.equal(h.fake.conversationOf(thread.id).messages.filter(message => message.role === 'user').length, 2); + } finally { await h.end(); } +}); + +test('a capped run counts as completed, and a failed run keeps its provider status', async () => { + let mode = 'capped'; + const h = await harness({ script: run => { if (mode === 'capped') run.capped(); else run.fail(429); } }); + try { + const { thread } = await started(h); + let ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('one') }); + assert.equal((await ended).status, 'completed'); + mode = 'fail'; ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('two') }); + const failed = await ended; + assert.equal(failed.status, 'failed'); + assert.deepEqual(failed.error, { code: 429 }); + } finally { await h.end(); } +}); + +test('a lost stream cannot leave a turn open: the state is read on a timer', async () => { + const h = await harness({ script: run => { run.say('quiet'); run.endSilently('completed'); }, drainMs: 60 }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'completed'); + } finally { await h.end(); } +}); + +test('a dropped stream is followed again from where it stopped, without repeating events or the task', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { run.say('before'); await gate; run.say('after'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => items(h.notes, 'agentMessage').length === 1); + for (const stream of h.fake.streams) stream.destroy(); + await waitFor(() => h.fake.count('GET', `/v1/chat/conversations/${thread.id}/events`) >= 2); + release(); + assert.equal((await ended).status, 'completed'); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['before', 'after']); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1); + } finally { await h.end(); } +}); + +test('an uncertain submission is read back, never sent twice', async () => { + const h = await harness(); + h.fake.dropAfterAccept = 1; + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'completed'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1); + } finally { await h.end(); } +}); + +test('a run that never starts ends the turn as unfinished, without sending it again', async () => { + const h = await harness({ script: () => {}, submitTimeoutMs: 60 }); + try { + const { thread } = await started(h); + const conversationStart = h.fake.complete.bind(h.fake); + // FLUJO answers, but no run appears: its conversation stays as it was. + h.fake.complete = (space, body, response) => { response.writeHead(200); response.write('data: {}\n\n'); return undefined; }; + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + assert.equal((await ended).status, 'failed'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + h.fake.complete = conversationStart; + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'it was sent once'); + } finally { await h.end(); } +}); + +test('under pressure a turn waits and retries; a refusal ends it', async () => { + const h = await harness({ submitTimeoutMs: 5000 }); + try { + const { thread } = await started(h); + h.fake.refuse = [503]; + let ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('busy') }); + assert.equal((await ended).status, 'completed'); + h.fake.refuse = [400]; + ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('bad') }); + assert.equal((await ended).status, 'failed'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + } finally { await h.end(); } +}); + +test('a message during a turn joins it; with nothing running the refusal has the exact text', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { await gate; for (const message of run.conversation.steered) { run.conversation.messages.push(message); run.emit({ type: 'message', message }); } run.say('done'); run.done(); } }); + try { + const { thread } = await started(h); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, input: text('nothing runs') }), { message: 'no active turn to steer' }); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: 'other', input: text('wrong') }), /expected turn/); + assert.deepEqual(await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('correction') }), { turnId: turn.id }); + release(); + assert.equal((await ended).status, 'completed'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'taken in by the run, so nothing is sent again'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('late') }), { message: 'no active turn to steer' }); + } finally { await h.end(); } +}); + +test('a message that arrives as the run ends is still delivered, in the same turn', async () => { + let release, runs = 0; + const gate = new Promise(resolve => { release = resolve; }); + const h = await harness({ script: async run => { runs++; if (runs === 1) { await gate; run.say('first'); run.done(); return; } run.say('second'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('owed') }); + release(); + assert.deepEqual(await ended, { id: turn.id, status: 'completed' }); + const posted = h.fake.requests.filter(entry => entry.path === '/v1/chat/completions'); + assert.deepEqual(posted.map(entry => entry.body.messages.at(-1).content), ['go', 'owed']); + assert.deepEqual(items(h.notes, 'agentMessage').map(item => item.text), ['first', 'second']); + } finally { await h.end(); } +}); + +test('a stop interrupts the turn, keeps asking until FLUJO has stopped, and a refused stop can be retried', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + h.fake.cancelFails = 1; + await assert.rejects(h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), /did not accept the stop/); + await assert.rejects(h.driver.request('turn/interrupt', { threadId: thread.id, turnId: 'other' }), /expected turn/); + h.fake.holdCancel = true; + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), {}); + await waitFor(() => h.fake.cancels >= 2, 2000); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active', 'cancel requested is not stopped'); + h.fake.holdCancel = false; + const conversation = h.fake.conversationOf(thread.id), run = conversation.run; + run.ended = true; conversation.status = 'error'; conversation.lastError = { code: 'cancelled' }; run.emit({ type: 'run:done', status: 'error' }); + const final = await ended; + assert.equal(final.status, 'interrupted'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }), {}, 'a finished turn is already stopped'); + await assert.rejects(h.driver.request('turn/steer', { threadId: thread.id, input: text('late') }), { message: 'no active turn to steer' }); + } finally { await h.end(); } +}); + +test('a stop wins over a queued message, and a turn stopped before it ran is interrupted', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + let ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await h.driver.request('turn/steer', { threadId: thread.id, expectedTurnId: turn.id, input: text('queued') }); + await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: turn.id }); + assert.equal((await ended).status, 'interrupted'); + assert.equal(h.fake.count('POST', '/v1/chat/completions'), 1, 'the queued message does not start a new run'); + h.fake.refuse = [503]; + ended = h.finished(); + const second = await h.driver.request('turn/start', { threadId: thread.id, input: text('blocked') }); + assert.deepEqual(await h.driver.request('turn/interrupt', { threadId: thread.id, turnId: second.turn.id }), {}); + assert.equal((await ended).status, 'interrupted'); + } finally { await h.end(); } +}); + +test('children keep a thread active until they have ended too', async () => { + let child; + const h = await harness({ script: run => { child = run.child('running'); run.say('lead is done'); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => child && h.fake.conversationOf(thread.id).status === 'completed'); + await new Promise(resolve => setTimeout(resolve, 60)); + assert.equal(h.notes.some(note => note.method === 'turn/completed'), false, 'the lead has ended but a child still runs'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + child.status = 'completed'; + assert.equal((await ended).id, turn.id); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'idle'); + // A child that appears later is caught by the readback, so a stop receipt cannot be wrong. + const late = h.fake.conversationOf(thread.id).run.child('running'); + assert.equal((await h.driver.request('thread/read', { threadId: thread.id })).thread.status.type, 'active'); + late.status = 'error'; + } finally { await h.end(); } +}); + +test('an unattended worker answers no questions: questions and forms are declined so a run never parks', async () => { + const h = await harness({ script: run => { run.emit({ type: 'run:awaiting_question', questionId: 'q1', questions: [] }); run.emit({ type: 'run:awaiting_elicitation', elicitationId: 'e1', message: 'form', requestedSchema: {} }); run.done(); } }); + try { + const { thread } = await started(h); + const ended = h.finished(); + await h.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await ended; + await waitFor(() => h.fake.responses?.length === 2); + assert.deepEqual(h.fake.responses.map(body => body.action).sort(), ['elicitation-cancel', 'question-decline']); + } finally { await h.end(); } +}); + +test('resume keeps the conversation, and lost history gives the error the runtime recovers from', async () => { + const fake = await new FakeFlujo().listen(); + const root = fresh(); + const first = await harness({ fake, root }); + try { + const { thread } = await started(first); + const unavailable = id => ({ code: -32600, message: `no rollout found for thread id ${id}` }); + const missing = '00000000-0000-4000-8000-000000000000'; + await assert.rejects(first.driver.request('thread/resume', { threadId: missing, cwd: root }), unavailable(missing)); + const ended = first.finished(); + const { turn } = await first.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await ended; + first.driver.close(); + + const second = await harness({ fake, root }); + const flowsBefore = fake.workspaces.get(`factory-${thread.id}`).flows.map(flow => flow.id); + const resumed = await second.driver.request('thread/resume', { threadId: thread.id, model: 'served-model', cwd: join(root, 'work'), sandbox: 'danger-full-access', developerInstructions: 'Be useful.' }); + assert.equal(resumed.thread.status.type, 'idle'); + assert.equal(resumed.recoveredTurn, undefined); + assert.deepEqual(fake.workspaces.get(`factory-${thread.id}`).flows.map(flow => flow.id), flowsBefore, 'an unchanged flow keeps its identity'); + assert.deepEqual((await second.driver.request('thread/read', { threadId: thread.id, includeTurns: true })).thread.turns, [{ id: turn.id, status: 'completed' }]); + const next = second.finished(); + await second.driver.request('turn/start', { threadId: thread.id, input: text('more') }); + await next; + assert.equal(fake.conversationOf(thread.id).messages.filter(message => message.role === 'user').length, 2); + assert.deepEqual(items(second.notes, 'agentMessage').map(item => item.text), ['answer'], 'old events are not told again'); + second.driver.close(); + + fake.workspaces.get(`factory-${thread.id}`).conversations.delete(thread.id); + const third = await harness({ fake, root }); + await assert.rejects(third.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work') }), unavailable(thread.id)); + third.driver.close(); + } finally { await fake.close(); } +}); + +test('after a restart a dead turn is closed and a live one is observed, never replayed', async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const fake = await new FakeFlujo({ script: async run => { await gate; run.say('late'); run.done(); } }).listen(); + const root = fresh(); + const first = await harness({ fake, root }); + try { + const { thread } = await started(first); + const { turn } = await first.driver.request('turn/start', { threadId: thread.id, input: text('go') }); + await waitFor(() => fake.conversationOf(thread.id)?.status === 'running'); + first.driver.close(); + + // FLUJO kept running, so the new driver follows the same turn. + const second = await harness({ fake, root }); + const ended = second.finished(); + const resumed = await second.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(resumed.thread.status.type, 'active'); + release(); + assert.deepEqual(await ended, { id: turn.id, status: 'completed' }); + assert.equal(fake.count('POST', '/v1/chat/completions'), 1); + second.driver.close(); + + // FLUJO died with the Machine: the conversation is terminal, so the retained turn did not finish. + const state = join(root, 'state', `${thread.id}.flujo.json`); + writeFileSync(state, JSON.stringify({ ...JSON.parse(readFileSync(state, 'utf8')), lastTurn: { id: turn.id, status: 'inProgress' } })); + const conversation = fake.conversationOf(thread.id); + conversation.status = 'error'; conversation.recovery.classification = 'interrupted'; + const third = await harness({ fake, root }); + const closed = await third.driver.request('thread/resume', { threadId: thread.id, cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(closed.thread.status.type, 'idle'); + assert.deepEqual(closed.recoveredTurn, { id: turn.id, status: 'interrupted' }); + third.driver.close(); + } finally { await fake.close(); } +}); + +test('a durable workflow wake delivers a turn and reads it back', async () => { + const h = await harness(); + try { + const { thread } = await started(h); + const done = h.finished(); + const wake = new NativeWorkflowWake({ rpc: h.driver, threadId: thread.id, file: join(h.root, 'wake.json'), effort: 'medium' }); + const delivered = await wake.wake({ trigger: 'first', input: 'start', initial: true }); + assert.equal(delivered.status, 'delivered'); + const turn = await done; + assert.equal(delivered.turnId, turn.id); + assert.deepEqual(await wake.read(), { status: 'idle', turnId: turn.id }); + } finally { await h.end(); } +}); + +test('a real stop command reaches the board as a verified receipt', async () => { + const p = plan({ agentsPerMachine: 1 }); const agent = agentInventory(p)[0]; + const credentials = { [agent.id]: secret() }; const operatorToken = secret(); const root = fresh(); + const coordinator = await startCloudLiteCoordinator({ database: join(root, 'board.sqlite'), plan: p, credentials, operatorToken, auth: {}, host: '127.0.0.1', port: 0 }); + const url = `http://127.0.0.1:${coordinator.server.address().port}`; + const post = async (path, body, token = operatorToken) => { + const response = await fetch(url + path, { method: 'POST', headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); + assert.equal(response.status, 200); return response.json(); + }; + const h = await harness({ root, script: () => {} }); + try { + const { thread } = await started(h); + await post('/operator/session', { agent: agent.id, session: thread.id }); + const trace = new CloudLiteTrace({ file: join(root, 'trace.jsonl'), url, token: credentials[agent.id] }); + const turns = new Map(), terminal = new Map(); + h.driver.on('notification', ({ method, params }) => { + if (method === 'turn/started') { terminal.delete(thread.id); turns.set(thread.id, params.turn.id); trace.record('native.turn.started', { threadId: thread.id, turn: params.turn }); stop.observeTurnStarted({ threadId: thread.id, turnId: params.turn.id }); } + if (method === 'turn/completed') { terminal.set(thread.id, { id: params.turn.id, status: params.turn.status }); trace.record('native.turn.completed', { threadId: thread.id, turn: { id: params.turn.id, status: params.turn.status } }); stop.observeTerminal({ threadId: thread.id, turnId: params.turn.id, status: params.turn.status }); } + }); + const stop = new NativeOperatorStop({ rpc: h.driver, threadId: thread.id, getTurn: () => nativeStopTurn(turns, terminal, thread.id), trace: (kind, body) => trace.record(kind, body), flush: () => trace.flush(), timeoutMs: 5000 }); + trace.record('native.thread.started', { threadId: thread.id, threadStatus: 'idle' }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long job') }); + await trace.flush(); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + await post('/operator/command', { id: 'stop-board', recipient: agent.id, action: 'stop', expectedThreadId: thread.id, expectedTurnId: turn.id }); + const { commands } = await post('/commands', {}, credentials[agent.id]); + const command = commands.find(value => value.id === 'stop-board'); + await stop.stop(command.id, { expectedThreadId: command.expectedThreadId, expectedTurnId: command.expectedTurnId }); + const ack = await post('/commands', { acknowledge: command.id }, credentials[agent.id]); + assert.equal(ack.acknowledged, true); + const native = coordinator.board.snapshot().agents.find(value => value.id === agent.id).nativeRuntime; + assert.equal(native.threadStatus, 'idle'); + assert.equal(native.turnStatus, 'interrupted'); + assert.equal(native.operatorStopReceipt.commandId, 'stop-board'); + assert.equal(native.operatorStopReceipt.threadId, thread.id); + assert.equal(native.operatorStopReceipt.turnId, turn.id); + } finally { await h.end(); await coordinator.close(); } +}); + +test('no stop receipt is issued while FLUJO still reports work', async () => { + const h = await harness({ script: () => {} }); + try { + const { thread } = await started(h); + const turns = new Map(), terminal = new Map(); + let stop; + h.driver.on('notification', ({ method, params }) => { + if (method === 'turn/started') turns.set(thread.id, params.turn.id); + if (method === 'turn/completed') { terminal.set(thread.id, { id: params.turn.id, status: params.turn.status }); stop.observeTerminal({ threadId: thread.id, turnId: params.turn.id, status: params.turn.status }); } + }); + const receipts = []; + stop = new NativeOperatorStop({ rpc: h.driver, threadId: thread.id, getTurn: () => nativeStopTurn(turns, terminal, thread.id), trace: (kind, body) => receipts.push({ kind, body }), flush: async () => {}, timeoutMs: 60 }); + const { turn } = await h.driver.request('turn/start', { threadId: thread.id, input: text('long') }); + await waitFor(() => h.fake.conversationOf(thread.id)?.status === 'running'); + h.fake.holdCancel = true; + await assert.rejects(stop.stop('stop-held', { expectedThreadId: thread.id, expectedTurnId: turn.id }), /terminal notification/); + assert.equal(receipts.some(entry => entry.kind === 'native.operator.stop.completed'), false); + } finally { await h.end(); } +}); + +const freePort = () => new Promise(resolve => { const probe = createServer(); probe.listen(0, '127.0.0.1', () => { const { port } = probe.address(); probe.close(() => resolve(port)); }); }); +const programOptions = async (env = {}) => ({ command: process.execPath, args: [fakeProgram], port: await freePort(), environment: () => ({ ...process.env, ...env }), startTimeoutMs: 15000 }); + +test('the FLUJO process starts, restarts a few times if it dies, then reports the failure', async () => { + const program = new FlujoProgram({ ...await programOptions({ FAKE_FLUJO_CRASH_AFTER: '2000' }), restartLimit: 2, restartWindowMs: 60000 }); + const seen = []; + program.on('restarting', event => seen.push(`restarting ${event.restarts}`)); + program.on('restarted', event => seen.push(`restarted ${event.restarts}`)); + const exited = new Promise(resolve => program.on('exit', resolve)); + try { + await program.start(); + const result = await exited; + assert.equal(result.code, 7); + assert.deepEqual(seen, ['restarting 1', 'restarted 1', 'restarting 2', 'restarted 2', 'restarting 3']); + } finally { await program.stop(); } +}); + +test('a FLUJO that cannot start is reported, and a stop ends it', async () => { + const broken = new FlujoProgram(await programOptions({ FAKE_FLUJO_EXIT_AT_START: '3' })); + await assert.rejects(broken.start(), /exited before it was ready \(3\)/); + const program = new FlujoProgram(await programOptions()); + await program.start(); + const exited = new Promise(resolve => program.on('exit', resolve)); + await program.stop(); + await exited; + assert.ok(program.child.exitCode !== null || program.child.signalCode !== null); +}); + +test('a driver owns its FLUJO process: it starts it, reports its version and ends with it', async () => { + const root = fresh(); + const driver = new FlujoSessionDriver({ program: new FlujoProgram(await programOptions()), stateDirectory: join(root, 'state'), modelRecord: () => flujoProviderModel(provider(), { key }), pollMs: 20 }); + const exited = new Promise(resolve => driver.on('exit', resolve)); + await driver.initialize(); + assert.equal(await driver.version(), '9.9.9'); + mkdirSync(join(root, 'work')); + const { thread } = await driver.request('thread/start', { cwd: join(root, 'work'), sandbox: 'danger-full-access' }); + assert.equal(thread.status.type, 'idle'); + driver.close(); + await exited; +}); + +test('evidence from a FLUJO worker holds its conversations and none of its keys', () => { + const root = fresh(), space = join(root, 'workspaces', 'factory-thread', 'db'); + mkdirSync(join(space, 'conversations'), { recursive: true }); mkdirSync(join(space, 'conversation-logs')); mkdirSync(join(root, 'workspaces', 'factory-thread', 'mcp-servers')); + writeFileSync(join(space, 'conversations', 'thread.json'), '{}'); writeFileSync(join(space, 'conversation-logs', 'thread.jsonl'), '{}\n'); + for (const name of ['encryption_key.json', 'models.json', 'mcp_servers.json']) writeFileSync(join(space, name), 'private'); + writeFileSync(join(root, 'workspaces', 'factory-thread', 'mcp-servers', 'server.js'), 'code'); + const store = new FactoryArtifactStore(join(fresh(), 'objects')); + const exclude = ['encryption_key.json', 'models.json', 'mcp_servers.json', 'mcp-servers', 'browser-profile', 'bash-utils', 'screenshots', 'recordings', 'snapshots', '.runtime-locks', 'node_modules']; + const captured = captureWorkspace(store, join(root, 'workspaces'), { exclude }); + const names = readWorkspaceManifest(store, captured.source.snapshot).files.map(file => file.path).sort(); + assert.deepEqual(names, ['factory-thread/db/conversation-logs/thread.jsonl', 'factory-thread/db/conversations/thread.json']); + // The list above is the one the worker uses. + const runtime = readFileSync(new URL('../src/cloud-lite/runtime.mjs', import.meta.url), 'utf8'); + assert.ok(runtime.includes(`sessionsExclude: ${JSON.stringify(exclude).replaceAll('"', "'").replaceAll(',', ', ')}`)); +}); + +test('a FLUJO fleet stages the key only as a secret, needs a provider and keeps its program on updates', async () => { + const root = fresh(); const calls = []; const machines = []; + const infrastructure = { + run: async (args, options = {}) => { calls.push({ args, input: options.input }); if (args[0] === 'volumes') return JSON.stringify({ id: 'vol_1' }); if (args[0] === 'machine' && args[1] === 'list') return JSON.stringify(machines.map(machine => ({ id: machine.id, name: machine.name, state: 'stopped', config: machine.config }))); return ''; }, + buildMachineConfig: () => ({ init: { cmd: ['upstream'] }, services: [] }), + createMachine: async ({ name, config }) => { const machine = { id: `machine-${machines.length}`, name, config }; machines.push(machine); return machine; }, + }; + const bundlePath = join(root, 'bundle.tgz'); writeFileSync(bundlePath, 'bundle'); + const image = 'ghcr.io/example/worker@sha256:' + 'a'.repeat(64); + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + await assert.rejects(fleet.deploy({ runId: 'flujo-test', app: 'factory-flujo-test', goal: 'x', workerProgram: 'flujo' }, { image, bundlePath }), /subscription auth broker \(or a model provider\)/); + await assert.rejects(fleet.deploy(plan(), { image, bundlePath }), /needs its API key/); + const result = await fleet.deploy(plan({ machines: 1 }), { image, bundlePath, providerKey: key }); + assert.equal(result.machines.length, 1); + const staged = calls.find(call => call.args[0] === 'secrets'); + const config = JSON.parse(Buffer.from(staged.input.trim().split('=').slice(1).join('='), 'base64').toString()); + assert.equal(config.providerKey, key); + assert.equal(config.plan.workerProgram, 'flujo'); + assert.equal(readFileSync(join(root, 'fleet', 'fleet.json'), 'utf8').includes(key), false); + assert.match(readFileSync(join(root, 'fleet', 'bootstrap.sh'), 'utf8'), /flujo-ai-/); + assert.equal(machines[0].config.metadata['factory.runtime'], 'flujo-lite'); + assert.equal(JSON.stringify(machines[0].config).includes(key), false); + // A runtime update must not switch the fleet to another program. + writeFileSync(bundlePath, 'bundle 2'); + try { await fleet.updateRuntime({ bundlePath }); } catch { /* The fake host has no update path; the script is what matters here. */ } + const updated = readFileSync(join(root, 'fleet', 'fleet.json'), 'utf8'); + assert.ok(updated); + const digest = (await import('node:crypto')).createHash('sha256').update('bundle 2').digest('hex'); + assert.match(readFileSync(join(root, 'fleet', `bootstrap-${digest}.sh`), 'utf8'), /flujo-ai-/); +}); + +test('an app can pack the SDK from its own dependency, and the same files give the same bytes', () => { + const root = fresh(), first = join(root, 'first.tgz'), second = join(root, 'nested', 'second.tgz'); + const packed = packFactorySdk(first); + assert.ok(packed.files > 50); + packFactorySdk(second); + assert.deepEqual(readFileSync(first), readFileSync(second)); + const unpacked = join(root, 'unpacked'); mkdirSync(unpacked); + // The bootstrap unpacks it the same way, and Git's tar on Windows reads a drive letter as a host. + const relative = spawnSync('tar', ['-xzf', 'first.tgz', '--strip-components=1', '-C', 'unpacked'], { cwd: root, encoding: 'utf8' }); + assert.equal(relative.status, 0, relative.stderr); + const sdk = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')); + assert.equal(JSON.parse(readFileSync(join(unpacked, 'package.json'), 'utf8')).version, sdk.version); + for (const file of ['bin/factory-cloud-runtime.mjs', 'bin/factory-cloud-mcp.mjs', 'src/cloud-lite/runtime.mjs', 'src/cloud-lite/flujo-driver.mjs']) assert.equal(existsSync(join(unpacked, file)), true, file); + assert.equal(existsSync(join(unpacked, 'node_modules')), false); + assert.equal(existsSync(join(unpacked, 'test')), false); + assert.ok(readdirSync(join(unpacked, 'src')).length > 10); + assert.deepEqual(readFileSync(join(unpacked, 'src', 'cloud-lite', 'bundle.mjs')), readFileSync(new URL('../src/cloud-lite/bundle.mjs', import.meta.url))); + if (process.platform !== 'win32') assert.equal(statSync(join(unpacked, 'bin', 'factory-cloud-runtime.mjs')).mode & 0o755, 0o755); + assert.throws(() => packFactorySdk(), /where to write/); +}); + +test('a fleet can have its own private network', async () => { + assert.throws(() => plan({ network: 'Bad Name' }), /Invalid network/); + const root = fresh(), calls = []; + const infrastructure = { run: async args => { calls.push(args); return ''; } }; + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + await fleet.prepare(plan({ org: 'my-org', network: 'goal-network' })); + assert.deepEqual(calls[0], ['apps', 'create', 'factory-flujo-test', '--org', 'my-org', '--network', 'goal-network']); + const other = new FactoryCloudLiteFleet({ directory: join(root, 'other'), infrastructure }); + await other.prepare(plan({ org: 'my-org' })); + assert.deepEqual(calls[1], ['apps', 'create', 'factory-flujo-test', '--org', 'my-org']); +}); + +test('a fleet that stopped part way can still be retired, and foreign resources still block it', async () => { + const root = fresh(); const calls = []; + let machines = [], volumes = [{ id: 'vol_1' }]; + const infrastructure = { + run: async args => { calls.push(args); if (args[0] === 'volumes' && args[1] === 'create') return JSON.stringify({ id: 'vol_1' }); if (args[0] === 'volumes') return JSON.stringify(volumes); if (args[0] === 'machine') return JSON.stringify(machines); return ''; }, + buildMachineConfig: () => ({ init: { cmd: ['upstream'] }, services: [] }), + createMachine: async () => { throw new Error('refused'); }, + }; + const fleet = new FactoryCloudLiteFleet({ directory: join(root, 'fleet'), infrastructure }); + const image = 'ghcr.io/example/worker@sha256:' + 'a'.repeat(64); + await assert.rejects(fleet.deploy(plan({ machines: 1 }), { image, providerKey: key }), /refused/); + assert.equal(fleet.snapshot().machines.length, 0); + machines = [{ id: 'other', name: 'someone-else', config: { metadata: { 'factory.run': 'another-run' } } }]; + await assert.rejects(fleet.retire(), /exact original/); + machines = []; volumes = [{ id: 'vol_foreign' }]; + await assert.rejects(fleet.retire(), /exact original/); + // A Machine whose creation looked refused may exist anyway; its name and label prove it is ours. + volumes = [{ id: 'vol_1' }]; + machines = [{ id: 'late', name: 'factory-flujo-test-m0', config: { metadata: { 'factory.run': 'flujo-test' } } }]; + await fleet.retire(); + assert.equal(fleet.snapshot().status, 'retired'); + assert.deepEqual(calls.at(-1), ['apps', 'destroy', 'factory-flujo-test', '--yes']); +}); + +test('a local FLUJO worker attaches, installs the template and the board, and keeps its conversation', async () => { + const fake = await new FakeFlujo().listen(); + const root = fresh(); + const options = { provider: 'flujo', database: join(root, 'board.sqlite'), cwd: root, origin: fake.origin, modelProvider: provider({ apis: ['chat'] }), providerKey: key, stateDirectory: join(root, 'state'), teamConcurrency: 2, agentId: 'builder' }; + try { + const spec = liteWorkerCommand({ ...options, port: 4321, dataDirectory: join(root, 'data'), env: { PATH: '/bin' } }); + assert.equal(spec.command, 'flujo'); + assert.deepEqual(spec.args, ['--no-open', '--port', '4321']); + assert.deepEqual(spec.env, { PATH: '/bin', FLUJO_DATA_DIR: join(root, 'data'), FLUJO_PORT: '4321' }); + assert.ok(spec.server.args.includes('--actor')); + assert.throws(() => liteWorkerCommand({ provider: 'other', database: 'x' }), /codex, claude or flujo/); + await assert.rejects(startLiteWorker({ provider: 'flujo', database: join(root, 'board.sqlite'), origin: fake.origin, stateDirectory: join(root, 'other') }), /modelProvider or a flujoModel/); + await assert.rejects(startLiteWorker({ ...options, providerKey: 'short' }), /API key/); + + const worker = await startLiteWorker({ ...options, prompt: 'Work on the board' }); + assert.equal(worker.origin, fake.origin); + assert.equal(worker.child, null, 'nothing is started when attaching'); + assert.equal((await worker.settled()).status, 'completed'); + const space = fake.workspaces.get(worker.workspace); + assert.equal(space.models[0].adapter, 'openai'); + assert.deepEqual(space.flows.map(flow => flow.name), ['swarm_agent', 'swarm_team']); + const board = space.servers.find(server => server.name === 'factory'); + assert.equal(board.args.includes('--config'), false, 'a local board has no credential to hide'); + assert.ok(board.args.includes('builder')); + const threadId = worker.threadId; + await worker.stop(); + + const again = await startLiteWorker({ ...options }); + assert.equal(again.threadId, threadId, 'the agent keeps its conversation across runs'); + await again.send('More work'); + assert.equal((await again.settled()).status, 'completed'); + assert.equal(fake.conversationOf(threadId).messages.filter(message => message.role === 'user').length, 2); + await again.stop(); + assert.equal(existsSync(join(root, 'state', 'session.json')), true); + } finally { await fake.close(); } +}); diff --git a/test/fixtures/fake-flujo.mjs b/test/fixtures/fake-flujo.mjs new file mode 100644 index 0000000..01bec5e --- /dev/null +++ b/test/fixtures/fake-flujo.mjs @@ -0,0 +1,197 @@ +#!/usr/bin/env node +// A small stand-in for a FLUJO server: the HTTP routes the worker driver uses, with runs a test can script. +// It behaves as the real program did in the check against FLUJO 3.48.3: a run outlives the connection that +// asked for it, a cancel is a request that takes a moment, and terminal state comes from the conversation. +import http from 'node:http'; +import { randomUUID } from 'node:crypto'; +import { once } from 'node:events'; +import { pathToFileURL } from 'node:url'; + +const ACTIVE = new Set(['running', 'awaiting_tool_approval', 'paused_debug']); + +export class FakeFlujo { + constructor({ tools = { bash: ['run'], filesystem: ['read_file'] }, script } = {}) { + this.tools = tools; + // Called for each new run with (run, api); by default a run answers and ends. + this.script = script ?? (run => { run.say('answer'); run.done(); }); + this.workspaces = new Map(); + // Like FLUJO's event bus, a stream can attach before its conversation exists. + this.conversationStreams = new Map(); + this.requests = []; + this.streams = new Set(); + this.refuse = []; + this.dropAfterAccept = 0; + this.cancelFails = 0; + this.server = http.createServer((request, response) => { void this.route(request, response).catch(error => { response.writeHead(500); response.end(JSON.stringify({ error: error.message })); }); }); + } + + async listen(port = 0) { this.server.listen(port, '127.0.0.1'); await once(this.server, 'listening'); this.origin = `http://127.0.0.1:${this.server.address().port}`; return this; } + async close() { for (const stream of this.streams) stream.destroy(); this.server.closeAllConnections(); await new Promise(resolve => this.server.close(resolve)); } + count(method, path) { return this.requests.filter(entry => entry.method === method && entry.path.startsWith(path)).length; } + conversationOf(id) { for (const space of this.workspaces.values()) if (space.conversations.has(id)) return space.conversations.get(id); return null; } + + space(name) { + if (!this.workspaces.has(name)) this.workspaces.set(name, { name, roots: [], models: [], flows: [], conversations: new Map(), + servers: [...Object.keys(this.tools), 'flujo'].map(server => ({ name: server, disabled: false, transport: 'stdio' })) }); + return this.workspaces.get(name); + } + + json(response, status, body) { response.writeHead(status, { 'content-type': 'application/json' }); response.end(JSON.stringify(body)); } + + async route(request, response) { + const url = new URL(request.url, 'http://fake'); + const path = url.pathname, method = request.method; + const chunks = []; for await (const chunk of request) chunks.push(chunk); + const body = chunks.length ? JSON.parse(Buffer.concat(chunks).toString('utf8')) : undefined; + this.requests.push({ method, path, body, workspace: url.searchParams.get('workspace') }); + const workspace = url.searchParams.get('workspace') ?? 'default-workspace'; + if (path === '/api/workspaces') { + if (method === 'GET') return this.json(response, 200, { workspaces: [...this.workspaces.keys()].map(name => ({ name })) }); + if (method === 'POST') { this.space(body.name); return this.json(response, 201, { workspace: { name: body.name } }); } + if (method === 'PATCH') { this.space(body.name).roots = body.roots; return this.json(response, 200, { workspace: { name: body.name, roots: body.roots } }); } + if (method === 'DELETE') { this.workspaces.delete(body.name); return this.json(response, 200, { deleted: body.name }); } + } + const space = this.space(workspace); + if (path === '/api/init') return this.json(response, 200, { success: true }); + if (path === '/api/update') return this.json(response, 200, { currentVersion: '9.9.9' }); + if (path === '/api/model') { + if (method === 'GET') return this.json(response, 200, space.models); + space.models.push(body); return this.json(response, 201, { ...body, ApiKey: '********' }); + } + if (path.startsWith('/api/model/')) { space.models = space.models.map(model => model.id === body.id ? body : model); return this.json(response, 200, body); } + if (path === '/api/mcp/servers') { + if (method === 'GET') return this.json(response, 200, space.servers); + space.servers.push(body); return this.json(response, 201, body); + } + const toolRoute = /^\/api\/mcp\/servers\/([^/]+)\/tools$/.exec(path); + if (toolRoute) { + const name = decodeURIComponent(toolRoute[1]); + if (name === 'factory' && this.boardFails) return this.json(response, 200, { tools: [], error: 'not connected' }); + return this.json(response, 200, { tools: (name === 'factory' ? ['factory_status'] : this.tools[name] ?? []).map(tool => ({ name: tool })) }); + } + if (path.startsWith('/api/mcp/servers/')) { + const name = decodeURIComponent(path.split('/').at(-1)); + space.servers = space.servers.map(server => server.name === name ? { ...server, ...body } : server); + return this.json(response, 200, body); + } + if (path === '/api/flow') return this.json(response, 200, space.flows); + if (path === '/api/flow/compile') { + const flow = { id: randomUUID(), name: body.spec.name, spec: body.spec }; + space.flows.push(flow); return this.json(response, 201, { flow }); + } + if (path.startsWith('/api/flow/')) { space.flows = space.flows.filter(flow => flow.id !== path.split('/').at(-1)); return this.json(response, 200, { success: true }); } + if (path === '/v1/chat/completions') return this.complete(space, body, response); + if (path === '/v1/chat/conversations') { + const parent = url.searchParams.get('descendantsOf'); + const items = [...space.conversations.values()].filter(conversation => conversation.parentConversationId === parent).map(({ id, status, parentConversationId }) => ({ id, status, parentConversationId })); + return this.json(response, 200, { items, total: items.length, hasMore: false }); + } + const route = /^\/v1\/chat\/conversations\/([^/]+)(?:\/([a-z]+))?$/.exec(path); + if (route) return this.conversationRoute(space, route[1], route[2], { method, body, url, response }); + return this.json(response, 404, { error: `Unknown route ${method} ${path}` }); + } + + publish(conversation, event) { + const record = { conversationId: conversation.id, seq: conversation.nextSeq++, timestamp: Date.now(), ...event }; + conversation.events.push(record); + for (const stream of this.conversationStreams.get(conversation.id) ?? []) stream.write(`id: ${record.seq}\ndata: ${JSON.stringify(record)}\n\n`); + return record; + } + + makeRun(conversation) { + const emit = event => this.publish(conversation, event); + const end = (status, { error, classification } = {}) => { + if (run.ended) return; + run.ended = true; + conversation.status = status; + conversation.recovery.classification = classification ?? (status === 'completed' ? 'completed' : status === 'capped' ? 'capped' : 'permanent_failure'); + if (error) conversation.lastError = error; else delete conversation.lastError; + emit({ type: 'run:done', status: status === 'capped' ? 'capped' : status === 'completed' ? 'completed' : 'error' }); + }; + const run = { conversation, ended: false, emit, + say: (text, extra = {}) => { const message = { id: randomUUID(), role: 'assistant', content: text }; conversation.messages.push(message); emit({ type: 'message', message, ...extra }); }, + tool: (name, args = '{}') => { const id = randomUUID(); emit({ type: 'message', message: { id: randomUUID(), role: 'assistant', content: '', tool_calls: [{ id, type: 'function', function: { name, arguments: args } }] } }); emit({ type: 'tool:result', toolCallId: id, name, result: 'ok', isError: false }); }, + usage: () => emit({ type: 'usage', promptTokens: 40, completionTokens: 4, totalTokens: 44, cacheReadTokens: 10 }), + done: () => end('completed'), + capped: () => end('capped'), + fail: (httpStatus = 500) => end('error', { error: { message: 'failed', httpStatus } }), + // End without the final event, as a lost stream would see it. + endSilently: status => { run.ended = true; conversation.status = status; conversation.recovery.classification = status === 'completed' ? 'completed' : 'permanent_failure'; }, + child: status => { const id = randomUUID(); const child = { id, status, parentConversationId: conversation.id, messages: [], events: [], nextSeq: 0, recovery: {} }; conversation.space.conversations.set(id, child); return child; }, + }; + conversation.run = run; + return run; + } + + async complete(space, body, response) { + const refusal = this.refuse.shift(); + if (refusal) return this.json(response, refusal, { error: { message: 'busy' } }); + const id = body.metadata.conversationId; + const conversation = space.conversations.get(id) ?? { id, space, messages: [], events: [], nextSeq: 0, parentConversationId: null }; + space.conversations.set(id, conversation); + conversation.messages.push({ id: randomUUID(), role: 'user', content: body.messages.at(-1).content }); + conversation.flow = body.model; + conversation.status = 'running'; + conversation.recovery = { runId: randomUUID(), classification: 'running' }; + conversation.steered = []; + const run = this.makeRun(conversation); + run.emit({ type: 'run:start', flowId: body.model }); + // A dropped connection: the run was accepted, but the caller never hears so. + if (this.dropAfterAccept > 0) { this.dropAfterAccept--; response.destroy(); } + else { response.writeHead(200, { 'content-type': 'text/event-stream' }); response.write('data: {}\n\n'); } + setImmediate(() => { void this.script(run, this); }); + } + + conversationRoute(space, id, action, { method, body, url, response }) { + const conversation = space.conversations.get(id); + if (action === 'events') { + response.writeHead(200, { 'content-type': 'text/event-stream' }); + response.write('retry: 3000\n\n: connected\n\n'); + const from = Number(url.searchParams.get('fromSeq') ?? 0); + for (const record of conversation?.events ?? []) if (record.seq >= from) response.write(`id: ${record.seq}\ndata: ${JSON.stringify(record)}\n\n`); + if (!this.conversationStreams.has(id)) this.conversationStreams.set(id, new Set()); + const listeners = this.conversationStreams.get(id); + listeners.add(response); + response.on('close', () => listeners.delete(response)); + this.streams.add(response); + response.on('close', () => this.streams.delete(response)); + return; + } + if (!conversation) { + if (action === 'inject') return this.json(response, 409, { error: 'Conversation is not running', reason: 'not_running' }); + if (action === 'cancel') return this.json(response, 200, { success: true, message: 'Conversation not found, assumed cancelled.' }); + return this.json(response, 404, { error: 'Conversation not found' }); + } + if (action === 'inject') { + if (!ACTIVE.has(conversation.status)) return this.json(response, 409, { error: 'Conversation is not running', reason: 'not_running' }); + const message = { id: body.id, role: 'user', content: body.content }; + conversation.steered.push(message); + conversation.run?.onSteer?.(message); + return this.json(response, 200, { status: 'queued', conversation_id: id, message_id: body.id }); + } + if (action === 'cancel') { + if (this.cancelFails > 0) { this.cancelFails--; return this.json(response, 500, { error: 'cannot stop now' }); } + conversation.recovery.classification = 'cancelled'; + this.cancels = (this.cancels ?? 0) + 1; + const run = conversation.run; + if (run && !run.ended && !this.holdCancel) setTimeout(() => { if (run.ended) return; run.ended = true; conversation.status = 'error'; conversation.lastError = { code: 'cancelled' }; run.emit({ type: 'run:done', status: 'error' }); }, 5); + return this.json(response, 200, { success: true }); + } + if (action === 'respond') { this.responses = [...(this.responses ?? []), body]; return this.json(response, 200, { status: 'running' }); } + if (method === 'GET') { + const { id: identity, status, recovery, lastError, parentConversationId, messages } = conversation; + return this.json(response, 200, { id: identity, status, recovery, ...(lastError ? { lastError } : {}), parentConversationId, messages }); + } + return this.json(response, 404, { error: 'Unknown action' }); + } +} + +// Run as a program it stands in for the `flujo` command: `fake-flujo.mjs --no-open --port `. +if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) { + const port = Number(process.argv[process.argv.indexOf('--port') + 1]); + if (process.env.FAKE_FLUJO_EXIT_AT_START) process.exit(Number(process.env.FAKE_FLUJO_EXIT_AT_START)); + const fake = await new FakeFlujo().listen(port); + process.stdout.write(`listening ${fake.origin}\n`); + if (process.env.FAKE_FLUJO_CRASH_AFTER) setTimeout(() => process.exit(7), Number(process.env.FAKE_FLUJO_CRASH_AFTER)); + process.on('SIGTERM', () => process.exit(0)); +} diff --git a/test/model-provider.test.mjs b/test/model-provider.test.mjs index e5809fa..febc171 100644 --- a/test/model-provider.test.mjs +++ b/test/model-provider.test.mjs @@ -15,9 +15,10 @@ const endpoint = (extra = {}) => ({ id: 'selfhosted', name: 'Self-hosted model', contextWindow: 262144, autoCompactTokenLimit: 100000, ...extra }); test('one provider description states which worker programs can use it', () => { - assert.deepEqual(providerPrograms(endpoint()), ['codex', 'claude']); - assert.deepEqual(providerPrograms(endpoint({ apis: undefined })), ['codex'], '0.4.0 descriptions keep working for Codex'); - assert.deepEqual(providerPrograms(endpoint({ apis: ['anthropic'] })), ['claude']); + assert.deepEqual(providerPrograms(endpoint()), ['codex', 'claude', 'flujo']); + assert.deepEqual(providerPrograms(endpoint({ apis: undefined })), ['codex', 'flujo'], '0.4.0 descriptions keep working for Codex'); + assert.deepEqual(providerPrograms(endpoint({ apis: ['anthropic'] })), ['claude', 'flujo']); + assert.deepEqual(providerPrograms(endpoint({ apis: ['chat'] })), ['flujo']); for (const extra of [{ apis: [] }, { apis: ['grpc'] }, { envKey: 'ANTHROPIC_AUTH_TOKEN' }, { requestTimeoutMs: 10 }]) assert.throws(() => validateModelProvider(endpoint(extra)), TypeError, JSON.stringify(extra)); assert.throws(() => validateCloudLitePlan({ runId: 'claude-only', app: 'claude-only-app', goal: 'x', modelProvider: endpoint({ apis: ['anthropic'] }) }), /responses API/); }); diff --git a/test/public-sdk.test.mjs b/test/public-sdk.test.mjs index 27a9c9e..42249c9 100644 --- a/test/public-sdk.test.mjs +++ b/test/public-sdk.test.mjs @@ -46,7 +46,7 @@ test('lite workers launch native CLIs with only the Factory MCP configuration', assert.ok(claude.args.includes('--dangerously-skip-permissions')); assert.ok(claude.args.includes('--strict-mcp-config')); assert.deepEqual(Object.keys(JSON.parse(claude.args[claude.args.indexOf('--mcp-config') + 1]).mcpServers), ['factory']); - assert.throws(() => liteWorkerCommand({ provider: 'flujo', database: 'board.sqlite' }), /provider/); + assert.throws(() => liteWorkerCommand({ provider: 'other', database: 'board.sqlite' }), /provider/); }); test('public SDK starts a lite worker through the same launcher as the CLI', async t => { diff --git a/test/selfhost-modal.test.mjs b/test/selfhost-modal.test.mjs index f2447ea..ca7c485 100644 --- a/test/selfhost-modal.test.mjs +++ b/test/selfhost-modal.test.mjs @@ -11,8 +11,9 @@ const fresh = () => tempDir('factory-selfhost-'); const read = (path) => readFileSync(path, 'utf8'); // A fake Modal account: the deploy script, the Modal CLI and the served model. -function fixture({ deployError, describeError, holdDescribe, workspace = 'fixture-workspace', completion, redirect, catalogueBytes } = {}) { +function fixture({ deployError, describeError, holdDescribe, workspace = 'fixture-workspace', completion, redirect, catalogueBytes, stoppingReads = 0, refusedDeletes = 0, stopFor = Infinity, modelOptions = {} } = {}) { const calls = [], requests = [], apps = new Map(), secrets = new Set(), volumes = new Set(); + let reads = stoppingReads, refusals = refusedDeletes; let key, appName, account = workspace; const runner = async (command, args, options) => { calls.push({ command, args, input: options.input, env: options.env }); @@ -40,9 +41,15 @@ function fixture({ deployError, describeError, holdDescribe, workspace = 'fixtur } const cli = args.slice(args.indexOf('modal') + 1); if (cli[0] === 'profile') return { code: 0, stdout: JSON.stringify([{ name: 'fixture', workspace: account, active: true }]) }; - if (cli[0] === 'app' && cli[1] === 'stop') { const known = apps.has(cli.at(-1)); if (known) apps.set(cli.at(-1), 'stopped'); return { code: known ? 0 : 1, stdout: '' }; } - if (cli[0] === 'app' && cli[1] === 'list') return { code: 0, stdout: JSON.stringify([...apps].map(([description, state]) => ({ description, state }))) }; - if (cli[1] === 'delete') { (cli[0] === 'secret' ? secrets : volumes).delete(cli.at(-1)); return { code: 0, stdout: '' }; } + if (cli[0] === 'app' && cli[1] === 'stop') { const known = apps.has(cli.at(-1)); if (known) apps.set(cli.at(-1), reads > 0 ? 'stopping' : 'stopped'); return { code: known ? 0 : 1, stdout: '' }; } + if (cli[0] === 'app' && cli[1] === 'list') { + if (reads > 0 && --reads === 0 && stopFor === Infinity) for (const [name, state] of apps) if (state === 'stopping') apps.set(name, 'stopped'); + return { code: 0, stdout: JSON.stringify([...apps].map(([description, state]) => ({ description, state }))) }; + } + if (cli[1] === 'delete') { + if (refusals > 0) { refusals--; return { code: 1, stdout: '' }; } + (cli[0] === 'secret' ? secrets : volumes).delete(cli.at(-1)); return { code: 0, stdout: '' }; + } if (cli[1] === 'list') return { code: 0, stdout: JSON.stringify([...(cli[0] === 'secret' ? secrets : volumes)].map(name => ({ name }))) }; throw new Error(`Unexpected command ${cli.join(' ')}`); }; @@ -55,7 +62,7 @@ function fixture({ deployError, describeError, holdDescribe, workspace = 'fixtur const challenge = body.messages[0].content.match(/FACTORY-[A-F0-9]{8}/)?.[0]; return new Response(JSON.stringify(completion?.(body, challenge) ?? { model: body.model, choices: [{ message: { role: 'assistant', content: challenge }, finish_reason: 'stop' }] })); }; - const model = new ModalSelfHostedModel({ directory: fresh(), python: 'fixture-python', commandRunner: runner, fetchImpl, + const model = new ModalSelfHostedModel({ directory: fresh(), python: 'fixture-python', commandRunner: runner, fetchImpl, stopPollMs: 1, ...modelOptions, env: { PATH: 'fixture-path', FACTORY_SELFHOST_SETTINGS_B64: 'inherited-must-be-dropped' } }); return { model, calls, requests, apps, secrets, volumes, key: () => key, appName: () => appName, setWorkspace: value => { account = value; } }; } @@ -69,6 +76,34 @@ test('bundled profiles carry list-price estimates per running replica', () => { for (const profile of Object.values(profiles)) { assert.equal(profile.contextWindow, 262144); assert.equal(profile.autoCompactTokenLimit, 80000); } }); +test('one retire right after a busy replica finishes: it waits for the stop and retries refused deletes', async () => { + const f = fixture({ stoppingReads: 3, refusedDeletes: 2 }); + await f.model.deploy({ profile: 'qwen' }); + const retired = await f.model.retire(); + assert.equal(retired.phase, 'retired'); assert.equal(retired.cleanupVerified, true); + assert.equal(f.secrets.size, 0); assert.equal(f.volumes.size, 0); + const deletes = f.calls.filter(call => call.args.includes('delete') && call.args.includes('--allow-missing')); + assert.equal(deletes.length, 5, 'three resources, two of the tries refused'); +}); + +test('an app still stopping at the deadline, or a delete that keeps failing, stays uncertain and is never reported as success', async () => { + const stuck = fixture({ stoppingReads: 10000, stopFor: 0, modelOptions: { stopWaitMs: 20 } }); + await stuck.model.deploy({ profile: 'qwen' }); + await assert.rejects(stuck.model.retire(), /uncertain/); + assert.equal(stuck.model.status().phase, 'unknown'); + assert.equal(stuck.secrets.size, 1, 'nothing is deleted while the app may still run'); + const refused = fixture({ refusedDeletes: 100 }); + await refused.model.deploy({ profile: 'qwen' }); + await assert.rejects(refused.model.retire(), /uncertain/); + assert.equal(refused.model.status().phase, 'unknown'); + assert.equal(refused.calls.filter(call => call.args.includes('delete') && call.args.includes('--allow-missing')).length, 5, 'the tries are bounded'); + // The stop path waits too, and a later retire can still finish. + const stopping = fixture({ stoppingReads: 2 }); + await stopping.model.deploy({ profile: 'qwen' }); + assert.equal((await stopping.model.stop()).phase, 'stopped'); + assert.equal((await stopping.model.retire()).phase, 'retired'); +}); + test('deploy, verify, describe, stop, redeploy and retire touch only the owned resources', async () => { const f = fixture(); const ready = await f.model.deploy({ profile: 'qwen', overrides: { replicas: 2 } });