diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f17b0d1 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,156 @@ +# Contributing to clicked + +## Pull requests must target `dev` + +**Open every pull request against `dev`, not `main`.** + +If you open a PR against `main` and you are not the repository maintainer (or a collaborator +with admin/maintain permission), the `guard-main-branch.yml` workflow automatically: + +1. Posts a comment on the PR explaining that PRs must target `dev`. +2. Closes the PR (`state: closed`). + +**This is not a rejection of your work.** It is an automated branch-policy check, not a review +outcome — nobody looked at your code and decided against it. To continue: open a new PR with +the same branch against `dev` (or retarget the base branch on the existing PR and reopen it, +per the bot's comment — GitHub's base-branch editing UI is available on an open or reopened +PR; once the workflow has already closed it, opening a fresh PR against `dev` is the reliable +path). + +`dev` is the integration branch contributors merge into; `main` is reserved for the +maintainer. This is why the `close-linked-issues.yml` workflow (below) exists at all — GitHub's +built-in "Closes #N" auto-close only fires on merges to the *default* branch, and this repo's +default branch is `main`, not `dev`. + +## Forking and branching + +```bash +# 1. Fork the repo on GitHub, then clone your fork +git clone https://github.com//clicked.git +cd clicked + +# 2. Add the upstream repo so you can pull in new dev commits later +git remote add upstream https://github.com/codebestia/clicked.git + +# 3. Make sure you're branching from an up-to-date dev +git fetch upstream +git checkout -b your-feature-branch upstream/dev + +# 4. ...make your changes, keeping the branch focused on one concern... + +# 5. Push to your fork +git push origin your-feature-branch + +# 6. Open a PR: base = codebestia/clicked:dev, compare = /clicked:your-feature-branch +``` + +Keep unrelated changes out of the branch — a docs fix and a feature change belong in separate +PRs so each can be reviewed and merged independently. + +## Commit conventions + +This repository follows [Conventional Commits](https://www.conventionalcommits.org/), with an +optional scope naming the area touched. Examples consistent with the existing commit history: + +``` +feat: add protection workflow +feat(backend): lock in Signal-protocol server invariants with guards and tests +feat(web): session reset and re-handshake on peer identity key change +fix(messages): apply shared payload validation to file messages +fix(conversations): serialize list preview to prevent plaintext leaks +docs: add testing strategy, migration workflow, and security policy +docs(backend): add background GC jobs reference +docs(web): add frontend accessibility guide +docs(contracts): add contract testing guide +``` + +- **Format:** `(): `. +- **Common types:** `feat`, `fix`, `docs` (also seen in history: `style`, `refactor`, `chore`, + `test`). +- **Scopes in use:** `backend`, `web`, `contracts`, `ai_agent`, `messages`, `conversations`, + `security`, among others — pick the one matching the app or area your change touches, or omit + the scope for changes that span multiple areas. +- Write the description in the imperative, present tense (`add`, not `added`/`adds`), matching + the pattern above. + +## Linking issues + +If your PR closes an issue, reference it with a closing keyword directly in the PR title or +body: `Closes #123`, `Fixes #123`, or `Resolves #123` (case-insensitive; `-s`/`-es`/`-d`/`-ed` +suffixes are all recognized). + +When your PR merges into `dev`, the `close-linked-issues.yml` workflow scans the PR title and +body for that pattern and closes each matched issue, with a comment +(`Closed by #, merged into dev.`) — this replaces GitHub's native auto-close, which only +triggers on merges to the default branch (`main`). + +**Limitation to know about:** the workflow's matching regex requires the closing keyword +immediately before each issue number. `Closes #123, #124` only closes `#123` — `#124` has no +keyword directly in front of it and will be left open. If your PR closes multiple issues, repeat +the keyword for each: `Closes #123, Closes #124`. + +## PR template + +Every PR uses the template at +[`.github/pull_request_template.md`](.github/pull_request_template.md), which asks for: + +- A **description** of the change. +- The **type of change** (bug fix / new feature / documentation update / other). +- A **checklist**: contributing guidelines read, changes tested locally, code follows the + project's coding standards. + +Fill in the description and check off the boxes that genuinely apply — an unchecked box is a +signal to the reviewer, not just a formality. + +## CI requirements + +CI is split by area, using GitHub Actions `paths` filters — a workflow only runs when a PR +touches the paths it's scoped to. Two checks run on every PR regardless of what changed. + +### Repository-wide (always run) + +- **PR Check** (`pr.yml`) — installs dependencies and runs `pnpm run lint` at the repo root. +- **Security CI** (`security-ci.yml`) — runs on every PR (and every push to `main`): + - `regression`: the ciphertext-only guard and secret-field scan + (`apps/backend/src/__tests__/security.regression.test.ts`). + - `dependency-audit`: `pnpm audit` scoped to crypto-relevant backend dependencies (`ioredis`, + `jsonwebtoken`, `web-push`, `@stellar/stellar-sdk`, `drizzle-orm`, `socket.io`). + +### Backend (`apps/backend/**`) + +`backend-ci.yml` spins up Postgres, Redis, and MinIO containers, runs migrations, then: +`pnpm format:check`, `pnpm lint`, `pnpm test`. (The suite's actual tests don't dial into those +containers — see [`apps/backend/docs/testing.md`](apps/backend/docs/testing.md) — the +containers exist so CI can validate migrations against a real Postgres.) + +### Web (`apps/web/**`) + +`frontend-ci.yml`: `pnpm --filter web lint`, `pnpm --filter web test`, +`pnpm --filter web build`. + +### AI agent (`apps/ai_agent/**`) + +`ai-agent-ci.yml`, three jobs: `ruff check` + `ruff format --check` (lint), `mypy main.py` +(typecheck), and `pytest --cov=main` (test + coverage, uploaded to Codecov). + +### Contracts (`contracts/**`) + +`contracts-ci.yml`, matrixed over the `token_transfer`, `group_treasury`, and `proposals` +packages: `cargo test -p `, then `cargo build -p --target +wasm32-unknown-unknown --release` with a 100 KB per-WASM size gate reported as a PR comment. +A separate `clippy` job runs `cargo clippy --workspace --target wasm32-unknown-unknown -- -D +warnings` (with `dead_code` and `too-many-arguments` allowed), and an `audit` job runs +`cargo audit`. This workflow also runs on a weekly schedule independent of any PR. + +### Running checks locally + +Match the workflow's own commands where practical: `pnpm lint` / `pnpm format:check` / +`pnpm test` for the app you touched, or `cargo test -p ` / +`cargo clippy --workspace --target wasm32-unknown-unknown` for contracts. Running the same +commands the workflow runs, in the affected app's directory, is the most reliable way to know +CI will pass before you push. + +--- + +See also the top-level [documentation index](docs/README.md) for architecture and per-app +references. diff --git a/IMPLEMENTATION_DOCS.md b/IMPLEMENTATION_DOCS.md new file mode 100644 index 0000000..e483a81 --- /dev/null +++ b/IMPLEMENTATION_DOCS.md @@ -0,0 +1,31 @@ +# Implementation docs + +Index of the docs that describe how this tree actually behaves. Concept +and contract pages live next to the code they cover; this file is only +the map. + +## Decisions + +- [docs/adr/README.md](./docs/adr/README.md) — Architecture Decision + Records (devices table, message ordering, ciphertext-only storage, + per-device envelopes, sealed-box → Signal, MLS for groups). + +## Backend concepts + +- [apps/backend/docs/concepts-configuration.md](./apps/backend/docs/concepts-configuration.md) — + boot-validated vs lazy env, `loadEnv()`, object-store singleton. +- [apps/backend/docs/concepts-backpressure.md](./apps/backend/docs/concepts-backpressure.md) — + socket buffer tiers; only disconnect is enforced today. +- [apps/backend/docs/concepts-stellar-listener.md](./apps/backend/docs/concepts-stellar-listener.md) — + Soroban event watch, cursors, idempotency, RPC failure. +- [apps/backend/docs/concepts-gateway-architecture.md](./apps/backend/docs/concepts-gateway-architecture.md) +- [apps/backend/docs/concepts-delivery-fanout.md](./apps/backend/docs/concepts-delivery-fanout.md) +- [apps/backend/docs/concepts-storage-push-jobs.md](./apps/backend/docs/concepts-storage-push-jobs.md) + +## Operations and security + +- [docs/runbook.md](./docs/runbook.md) +- [docs/observability.md](./docs/observability.md) +- [docs/threat-model.md](./docs/threat-model.md) +- [docs/security/rate-limits.md](./docs/security/rate-limits.md) +- [`.env.example`](./.env.example) — environment-variable reference diff --git a/README.md b/README.md index f5e35b0..c0db05e 100644 --- a/README.md +++ b/README.md @@ -152,8 +152,19 @@ pnpm test ## 📚 Documentation +**➡️ [Documentation index](docs/README.md) — every document in this repository, grouped by +reader: new contributor, backend, frontend, contract developer, operator, security reviewer.** +Start there; it links to everything below and is kept exhaustive. + +New to the project? Read the [system architecture overview](docs/architecture-overview.md) +first — one diagram of all four apps and the external services, plus two end-to-end traces. + +Frequently opened: + - [Devices & Prekeys API](apps/backend/docs/api-devices.md) — every `/devices` and `/user-devices` route: request/response shapes, ownership checks, prekey upload contract, and revocation side effects - [E2EE Onboarding Sequence](apps/backend/docs/e2ee-onboarding.md) — device registration and prekey upload flow for first-contact DM setup +- [Contract events reference](contracts/docs/contracts-events.md) — every on-chain event, its payload, and which ones the backend listener consumes +- [WASM size and resource budget](contracts/docs/concepts-resource-budget.md) — the 100 KB per-contract CI gate, current sizes, and headroom --- @@ -161,6 +172,9 @@ pnpm test We welcome contributions from developers, designers, and researchers. +**➡️ See [CONTRIBUTING.md](CONTRIBUTING.md)** for the branch policy (PRs target `dev`, not +`main`), commit conventions, issue linking, and CI requirements before opening a PR. + --- ## 📌 How to Contribute diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..2d4e6b7 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,223 @@ +# Security Policy + +Clicked is an end-to-end encrypted messaging product that also moves funds on-chain. A +vulnerability here can expose private conversations or drain a treasury, and in the on-chain +case the damage may be irreversible. We take reports seriously and we would much rather hear +about a problem from you than from an incident. + +--- + +## Reporting a vulnerability + +**Report privately, through GitHub's private vulnerability reporting:** + +> **** + +That form is private between you and the maintainers. It creates a draft security advisory, +gives us a place to discuss the issue and share a fix with you before it is public, and lets +us credit you when the advisory is published. + +If the form is unavailable to you for any reason, contact the maintainer +([@codebestia](https://github.com/codebestia)) directly on GitHub and ask for a private +channel. Do not include vulnerability details in that first message. + +### Do not report vulnerabilities publicly + +**Do not open a public issue, a public pull request, or a public discussion for a security +vulnerability, and do not post details on social media, in a chat channel, or in a commit +message before a fix has shipped.** + +A public issue is a working exploit advertisement. The report is visible to everyone the +moment it is filed, while a fix takes days to write, review, and deploy — and in the case of +a non-upgradeable contract it may not be deployable at all. A pull request is worse: the diff +itself explains the bug, and anyone watching the repository is notified. + +This applies even when the issue seems minor or you are not sure it is exploitable. Let us +make that call in private. If you are unsure whether something counts as a security issue, +report it privately anyway — a misfiled private report costs us nothing. + +### What to include + +The more of this you can supply, the faster we can triage: + +- Which component is affected (backend, web client, contracts, AI agent) and the commit, + branch, or deployed contract ID you tested against. +- A description of the vulnerability and the security property it breaks — for example + "the server can recover message plaintext", "device A can read device B's envelopes", + "a non-admin can withdraw from a treasury". +- Reproduction steps, ideally a minimal proof of concept: a failing test, a `curl` sequence, + or a Soroban invocation. +- Your assessment of the impact and of any preconditions (does it need an authenticated + session, a compromised device, a specific group state?). +- Whether you have disclosed this to anyone else, and any disclosure deadline you intend to + hold us to. + +### What to expect + +| Stage | Target | +| ------------------------------------------------------------------- | -------------------------- | +| Acknowledgement that we received the report | **3 business days** | +| Initial triage: severity, in scope or not, whether we can reproduce | **10 business days** | +| Status updates while a fix is in progress | At least every **14 days** | +| Fix or documented mitigation for a confirmed high-severity issue | **90 days** from triage | + +If you do not hear from us within the acknowledgement window, please ping the maintainer on +GitHub referencing the advisory — assume the notification was missed rather than ignored. + +We aim to publish an advisory once a fix is available, crediting you by the name or handle +you ask for (or keeping you anonymous, if you prefer). Please give us a chance to ship the +fix before disclosing publicly; if you have a fixed disclosure deadline, say so in your first +message so we can plan against it. + +There is no paid bug bounty for this project today. + +--- + +## Scope + +### In scope + +**Backend gateway and API** (`apps/backend`) + +- Authentication and session handling: wallet-signature challenge/verify, JWT issuance, + claim validation, session or device binding bypass. +- Authorization: reading or writing a conversation, message, device, file, or treasury record + you are not a member of or do not own. +- Anything that lets the server, or an attacker with server access, obtain message plaintext + or private key material — the ciphertext-only invariants are load-bearing for every claim + this product makes. +- Injection into the database, the object store, or the Socket.IO event path; envelope + validation or idempotency bypass on the `dispatch` event. +- Rate-limit bypass that enables credential stuffing, prekey exhaustion, or resource + exhaustion of the gateway. +- Presigned upload/download URL flaws: forgery, scope escalation, or access to another + conversation's blobs. + +**Web client cryptography** (`apps/web`) + +- Flaws in the E2EE implementation: X3DH, the double ratchet, MLS group operations, session + or epoch handling, safety-number computation. +- Key generation, storage, or lifetime bugs — weak randomness, a private key leaving the + device, keys surviving a revocation, keys readable from another origin. +- File encryption: key reuse, nonce reuse, unauthenticated ciphertext, a file key reaching + the wrong recipient. +- Identity-trust bugs: accepting an unverified identity key change, or failing to surface one. +- Client-side XSS or a content-injection path that reaches decrypted message content or + IndexedDB. + +**Smart contracts** (`contracts/`: `token_transfer`, `group_treasury`, `proposals`) + +- Any path that moves funds without correct authorization, or that lets a caller bypass + `require_auth`. +- Treasury multisig or voting logic that can be subverted: double-voting, quorum manipulation, + replaying or re-executing a proposal, executing a proposal that did not pass. +- Arithmetic overflow/underflow, accounting errors, and storage-key collisions. +- Abuse of the admin-gated `upgrade` entrypoint on `token_transfer`. + +**AI agent** (`apps/ai_agent`) — in scope where a flaw exposes user data or is reachable from +untrusted input; prompt-quality complaints are not security issues. + +**Repository and supply chain** — leaked credentials committed to the repository, a +compromised or typosquatted dependency, or a CI workflow that can be made to execute +attacker-controlled code. + +### Out of scope + +- **Denial of service through raw volume.** Load-testing a deployment, traffic floods, or + resource exhaustion that requires attacker bandwidth rather than an application flaw. +- **Automated scanner output with no demonstrated impact** — a CVE in a transitive dependency + on a code path this project does not reach, or a "missing header" report with no exploit. +- **Missing hardening that is not itself a vulnerability**: absent security headers, cookie + flags on endpoints that set no cookies, verbose non-sensitive error messages, TLS + configuration of a deployment we do not control. +- **Social engineering, phishing, or physical attacks** against maintainers or users. +- **Vulnerabilities requiring a fully compromised device or a malicious OS/browser + extension.** The threat model assumes the endpoint is trusted; an attacker with the user's + device already has their keys. +- **Third-party services** — the Stellar network itself, a wallet extension such as Freighter, + browser push providers, the LLM or vector-store vendors. Report those to their owners. +- **Anything already documented as an accepted residual risk** — see below. +- Findings in a fork, a stale branch, or code that was never merged to `dev` or `main`. + +### Testing rules + +Test against your own local deployment and your own accounts. Do not test against another +person's account or data, do not exfiltrate data beyond the minimum needed to demonstrate the +issue, and do not degrade a shared environment. For contracts, use **testnet** — see +[Contract-specific reporting](#contract-specific-reporting) below. + +--- + +## Known and accepted residual risk + +Before reporting, please read the **[threat model](docs/threat-model.md)**. It states exactly +what the server can and cannot see, and its "Residual metadata risk" section lists risks that +are inherent to operating a centralized delivery service and are **known and accepted**, +including: + +- the **social graph** — conversation membership reveals who talks to whom, +- **traffic analysis** — message timing and ciphertext size leak coarse signals, +- **presence and activity patterns**, +- **device fingerprinting** via device name, platform, and prekey consumption rate. + +A report that these are observable is not a vulnerability report; it is a restatement of the +documented design. A report that some _content_, _key material_, or _session state_ is +observable when the threat model says it is not — that is exactly what we want to hear about, +and it is a high-severity finding. + +Related reading, all of which describes intended behaviour rather than bugs: + +- [Threat model](docs/threat-model.md) — trust boundaries and what the server sees. +- [Backend security hardening](apps/backend/docs/security-hardening.md) — measures already in + place and the threats each closes. +- [Rate limits](docs/security/rate-limits.md) — every bucket and its threshold. +- [Audit logging](docs/security/audit-logging.md) — what is logged and what is deliberately + excluded. +- [TLS and pinning](docs/security/tls-and-pinning.md) — transport security expectations. +- [E2EE architecture](apps/web/docs/concepts-e2ee-architecture.md) — the client-side key model. + +--- + +## Contract-specific reporting + +**Treat an on-chain finding as more urgent than an equivalent server-side one, and be more +careful with it.** A backend bug is fixed by a deploy; an on-chain bug frequently is not. + +- **Most of this system's contracts cannot be patched in place.** Only `token_transfer` has an + `upgrade` entrypoint (admin-gated `update_current_contract_wasm`). `group_treasury` and + `proposals` expose **no upgrade function at all** — their deployed WASM is immutable for the + life of the contract instance. Fixing a bug in either one means deploying a _new_ contract, + migrating state, and repointing every consumer at the new contract ID. See + [Contract upgrades and versioning](contracts/docs/concepts-upgrades.md). +- **Funds already in a vulnerable contract may not be recoverable.** Because a redeploy does + not move existing balances or proposal state, a live exploit can be unfixable after the + fact. Time between report and mitigation matters much more than usual, and mitigation may + have to start with pausing usage and moving funds rather than with a code change. +- **Never test a contract vulnerability against mainnet or against a deployed instance holding + real funds.** Reproduce it in the Soroban test environment (`cargo test` against + `Env::default()`) or on testnet with your own deployment. A proof of concept executed + against a live treasury is an exploit, not research, regardless of intent. +- **The strongest proof of concept for a contract finding is a failing Rust test.** Add a + `#[test]` against the contract in question and send us the test — it removes all ambiguity + about preconditions and cannot be mistaken for an attack. See the + [contract testing guide](contracts/docs/testing.md). +- **Include the contract ID and network** you tested against, and say explicitly whether the + issue affects an already-deployed instance or only the current source. +- **If the finding involves the `upgrade` entrypoint or the admin key**, flag that in the first + line of your report. Admin compromise on `token_transfer` is the highest-severity class of + finding in this repository, because it converts an upgradeable contract into an + attacker-controlled one. + +--- + +## Supported versions + +This project is pre-1.0 and under active development. Security fixes are applied to the +`dev` branch and flow to `main`; there are no maintained release branches and no backports to +older tags. Run a current checkout of `main`. + +| Version | Supported | +| --------------- | --------- | +| `main` (latest) | Yes | +| `dev` (latest) | Yes | +| Anything older | No | diff --git a/apps/ai_agent/docs/concepts-prompts-and-models.md b/apps/ai_agent/docs/concepts-prompts-and-models.md new file mode 100644 index 0000000..00397f4 --- /dev/null +++ b/apps/ai_agent/docs/concepts-prompts-and-models.md @@ -0,0 +1,311 @@ +# Prompts and model configuration + +The system prompts behind each LLM-backed endpoint, the model used per endpoint, how raw model output is parsed back into typed responses, and what happens when the model returns something malformed. + +Everything here reflects [apps/ai_agent/main.py](../main.py) as implemented. Prompts are reproduced **verbatim** — if you change one in code, change it here in the same commit, because a drifted prompt doc is worse than no prompt doc. + +## Models at a glance + +| Endpoint | Model | Call type | JSON mode | Timeout | +| ---------------------- | ------------------------ | --------------- | --------- | ------- | +| `POST /chat` | `gpt-4o-mini` | chat completion | no | 30s | +| `POST /transfers/analyse` | `gpt-4o-mini` | chat completion | **yes** | 10s | +| `POST /proposals/summarise` | `gpt-4o-mini` | chat completion | **yes** | 10s | +| `POST /index/message` | `text-embedding-3-small` | embedding | n/a | none | +| `GET /search` | `text-embedding-3-small` | embedding | n/a | none | + +Model identifiers are **hardcoded string literals at each call site**. There is no configuration object, no environment variable, and no central constant. Changing a model means editing `main.py`. + +`GET /health` involves no model at all. + +## The prompts + +### `POST /chat` — system prompt + +This is the only true *system* prompt in the service. It is a module-level constant, applied to every `/chat` request: + +```python +_SYSTEM_PROMPT = ( + "You are an AI assistant for Clicked, a decentralised messaging and payment " + "platform built on the Stellar blockchain. Clicked lets users send token " + "payments inside chat conversations, manage group treasuries, and participate " + "in DAO-style governance. Help users with questions about transactions, wallet " + "management, group finances, and platform features." +) +``` + +Assembled as: + +```python +response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[ + {"role": "system", "content": _SYSTEM_PROMPT}, + {"role": "user", "content": request.message}, + ], + timeout=30, +) +``` + +**Behaviour it is designed to produce.** The prompt does one job: domain grounding. It tells the model what Clicked is (decentralised messaging + payments), what chain it runs on (Stellar), what users can do (in-chat payments, group treasuries, DAO governance), and which question areas are in scope (transactions, wallet management, group finances, platform features). Without it the model would answer Stellar questions generically and would have no idea what "group treasury" means in this product. + +Three properties of this prompt matter operationally: + +- **It does not constrain output format.** `/chat` returns free-form prose; there is no JSON mode and no schema. The reply is passed through unmodified. +- **It does not constrain output length.** There is no `max_tokens`, so response length — and cost — is bounded only by the model. +- **It carries no conversation history.** Each request sends exactly two messages: system and the current user turn. `ChatRequest.conversation_id` is accepted and validated by Pydantic but **is never sent to the model and never used to load prior turns**. `/chat` is stateless and single-turn despite the field's name implying otherwise. + +`test_system_prompt_contains_context` in [tests/test_chat.py](../tests/test_chat.py) pins the presence of the "Clicked", "Stellar", and messaging/payment grounding, so gutting the prompt fails CI. + +### `POST /transfers/analyse` — user prompt + +No system message. The whole instruction is a single user-role message, built per request: + +```python + prompt = ( + "Analyse this Stellar transfer for fraud risk.\n" + f"Amount: {request.amount} XLM\n" + f"Sender: {request.sender}\n" + f"Recipient: {request.recipient}\n" + f"Memo: {request.memo}\n\n" + "Reply with JSON only using keys: flagged (bool), reason (string under 100 chars or null), " + "confidence (float 0-1). Flag if suspicious patterns are detected." + ) +``` + +Sent as: + +```python +response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": prompt}], + response_format={"type": "json_object"}, + timeout=10, +) +``` + +**Behaviour it is designed to produce.** A three-key JSON verdict — `flagged`, `reason`, `confidence` — matching the `TransferAnalyseResponse` shape field for field. The `reason` length hint keeps the string renderable in a UI, and the `confidence` range makes the score comparable across calls. `response_format={"type": "json_object"}` puts the model in JSON mode so output is guaranteed to be syntactically valid JSON — though *not* guaranteed to contain the keys asked for, which is the whole reason the parsing layer below exists. + +**This prompt is only reached for transfers at or below the threshold.** Amounts strictly greater than `_HIGH_VALUE_THRESHOLD` (`10_000.0`) return a rule-based verdict without any model call: + +```python +if request.amount > _HIGH_VALUE_THRESHOLD: + return TransferAnalyseResponse( + flagged=True, + reason=f"Amount {request.amount} XLM exceeds {_HIGH_VALUE_THRESHOLD} XLM threshold", + confidence=0.99, + ) +``` + +The high-value flag is deliberately not delegated to the model — it is a security property that must not depend on model behaviour. See [concepts-transfer-risk-analysis.md](concepts-transfer-risk-analysis.md) for the rationale. + +### `POST /proposals/summarise` — user prompt + +Again a single user-role message, no system message: + +```python + prompt = ( + "Summarise this Clicked governance proposal for a frontend reader and " + "rate its risk level.\n" + f"Title: {request.title}\n" + f"Description: {request.description}\n" + f"Amount: {request.amount} XLM\n\n" + "Reply with JSON only using keys: summary (a plain-English summary of " + 'exactly 2 sentences), risk (one of "low", "medium", "high"). ' + 'Use "high" for large amounts, unclear intent, or obvious red flags; ' + '"low" for small, well-scoped, low-impact proposals; otherwise "medium".' + ) +``` + +Sent as: + +```python +response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": prompt}], + response_format={"type": "json_object"}, + timeout=10, +) +``` + +**Behaviour it is designed to produce.** Two outputs in one call. The `summary` is constrained to *exactly two sentences* and to plain English, because it renders in a fixed-size UI slot and is read by voters who will not read the full proposal. The `risk` field is constrained to a closed three-value set, and — unlike the transfer prompt — the prompt supplies an explicit rubric for choosing between them: `high` for large amounts, unclear intent, or obvious red flags; `low` for small, well-scoped, low-impact proposals; `medium` otherwise. `medium` is the stated default, which matters because it is also the code's fallback (see below), so a malformed response degrades toward the same value the rubric already treats as neutral. + +### Embedding calls — `/index/message` and `/search` + +No prompt. Both endpoints embed raw text directly with `text-embedding-3-small`: + +```python +# /index/message +res = openai_client.embeddings.create(input=request.content, model="text-embedding-3-small") +vector = res.data[0].embedding + +# /search +res = openai_client.embeddings.create(input=q, model="text-embedding-3-small") +vector = res.data[0].embedding +``` + +**Both sides must use the same model.** Vectors from different embedding models are not comparable, so a query embedded with one model against a corpus indexed with another returns semantically meaningless results — with no error. This is the most dangerous coupling in the service; see [Changing a model safely](#changing-a-model-safely). + +## Output parsing and validation + +Each endpoint hands model output back through a different amount of validation. Ordered from least to most defensive: + +### `/chat` — no parsing + +```python +return ChatResponse(reply=response.choices[0].message.content) +``` + +The content string is passed straight into `ChatResponse`. There is no JSON parsing and no content validation. + +One sharp edge: the OpenAI API can return `None` for `message.content` (for example on a content-filter stop). `ChatResponse.reply` is typed `str`, so a `None` raises a Pydantic `ValidationError` inside the handler, which surfaces to the caller as an unhandled `500`. There is no explicit guard for this case. + +### `/transfers/analyse` — parse with per-field defaults + +This is the worked example the parsing strategy is best understood through: + +```python +result = json.loads(response.choices[0].message.content) +return TransferAnalyseResponse( + flagged=bool(result.get("flagged", False)), + reason=result.get("reason"), + confidence=float(result.get("confidence", 0.0)), +) +``` + +Every field is read with `.get()` and a default, so a **partial** JSON object still produces a valid typed response: + +| Model returns | Parsed result | Rationale | +| ------------- | ------------- | --------- | +| `{"flagged": true, "reason": "Suspicious memo", "confidence": 0.9}` | `flagged=True, reason="Suspicious memo", confidence=0.9` | Complete response, passed through. | +| `{"flagged": false, "reason": null}` — **`confidence` missing** | `flagged=False, reason=None, **confidence=0.0**` | `.get("confidence", 0.0)` supplies `0.0`. | +| `{"reason": null, "confidence": 0.5}` — **`flagged` missing** | **`flagged=False`**, `reason=None, confidence=0.5` | `.get("flagged", False)` supplies `False`. | +| `{}` | `flagged=False, reason=None, confidence=0.0` | Both defaults apply. | + +**The two defaults fail in opposite directions, and the asymmetry is intentional.** + +A missing `flagged` defaults to `False` — *do not flag*. A missing `confidence` defaults to `0.0` — *no confidence at all*. So a model that omits `flagged` produces a "not flagged" verdict, and a model that omits `confidence` produces a verdict the caller can see is worthless. A consumer that reads `confidence` before acting on `flagged` will correctly distrust both malformed cases. A consumer that reads `flagged` alone will silently treat a malformed response as a clean transfer. + +**This is the operationally important consequence: on this endpoint, model failure looks like "transfer is fine".** Callers making a security decision must gate on `confidence`, not on `flagged` alone. Both default paths are pinned by `test_llm_path_missing_confidence_defaults_to_zero` and `test_llm_path_missing_flagged_defaults_to_false` in [tests/test_transfers.py](../tests/test_transfers.py). + +Note also that `confidence` is **not range-checked**. The prompt asks for `0-1`, but a model returning `7.5` produces `confidence=7.5`; `float()` only enforces the type. Similarly `reason` has no length check despite the prompt's "under 100 chars". + +If the model returns syntactically invalid JSON, `json.loads` raises and the request fails as an unhandled `500`. JSON mode makes this unlikely but not impossible (a response truncated by the token limit is still invalid JSON). There is no `try/except` around the parse. + +### `/proposals/summarise` — parse, reject, then clamp + +The most defensive of the three, and the only one that deliberately returns a `502`: + +```python +result = json.loads(response.choices[0].message.content) + +summary = (result.get("summary") or "").strip() +if not summary: + raise HTTPException(status_code=502, detail="LLM did not return a summary") + +risk = str(result.get("risk", "")).strip().lower() +if risk not in ("low", "medium", "high"): + # Defensive fallback: never return an invalid risk level to the caller. + risk = "medium" + +# Pydantic re-validates via response_model before the response is sent. +return ProposalSummariseResponse( + summary=summary, risk=cast(Literal["low", "medium", "high"], risk) +) +``` + +The two fields are handled by opposite strategies, because their failure modes differ in kind: + +- **`summary` is mandatory — a missing one is an error.** Absent, `null`, empty, or whitespace-only all collapse to `""` via `(... or "").strip()` and raise `502 "LLM did not return a summary"`. There is no sensible fabricated summary, so the endpoint fails loudly rather than returning an empty string that would render as a blank card. +- **`risk` is clamped — an invalid one degrades.** The value is normalised (`str()`, `.strip()`, `.lower()`, so `"HIGH"` and `" high "` both survive) and checked against the closed set. Anything else — a missing key, `"critical"`, `"unknown"`, `null` — becomes `"medium"`. Returning an out-of-set risk would break `RiskLevel` typing for every downstream consumer, so the code guarantees a valid value. + +`"medium"` is a reasonable clamp target precisely because it is also the prompt's stated default for the unremarkable case. But note the ambiguity it creates: **a `risk` of `"medium"` may mean the model judged the proposal middling, or that it returned garbage.** The two are indistinguishable to the caller, and no signal is logged. Do not treat `"medium"` as a confident assessment. + +The `cast()` is a static-typing assertion only — it performs no runtime check. The real runtime guarantee comes from `response_model=ProposalSummariseResponse` on the route decorator, which re-validates the outgoing payload against the `RiskLevel` literal before it is serialised. + +### Parsing strategy summary + +| Endpoint | Malformed field | Behaviour | Caller sees | +| -------- | --------------- | --------- | ----------- | +| `/chat` | content is `None` | Pydantic `ValidationError`, unhandled | `500` | +| `/transfers/analyse` | invalid JSON | `json.loads` raises, unhandled | `500` | +| `/transfers/analyse` | `flagged` missing | defaults to `False` | `200`, unflagged | +| `/transfers/analyse` | `confidence` missing | defaults to `0.0` | `200`, zero confidence | +| `/transfers/analyse` | `confidence` out of range | passed through unchecked | `200`, e.g. `7.5` | +| `/proposals/summarise` | invalid JSON | `json.loads` raises, unhandled | `500` | +| `/proposals/summarise` | `summary` missing/empty | explicit rejection | `502` | +| `/proposals/summarise` | `risk` missing/invalid | clamped to `"medium"` | `200` | + +The typed response shapes themselves are documented in [contracts-pydantic-models.md](contracts-pydantic-models.md). + +## Changing a model safely + +Model IDs are string literals at each call site, so a change is a code edit and a deploy. The safe procedure differs sharply between the two model families. + +### Changing a chat model + +Applies to `/chat`, `/transfers/analyse`, and `/proposals/summarise`. + +1. **Confirm JSON mode support.** `/transfers/analyse` and `/proposals/summarise` pass `response_format={"type": "json_object"}`. A model that does not support it will error on every request. `/chat` does not use JSON mode and is unconstrained here. +2. **Change one endpoint at a time.** The three call sites are independent — there is no shared constant — so a model change is naturally scoped to one endpoint. Keep it that way; do not sweep all three in one edit. +3. **Re-test the parse layer, not just the happy path.** A new model may phrase `risk` differently (triggering the `"medium"` clamp more often) or omit `confidence` more frequently (silently producing unflagged verdicts). The defaults will hide this — no error, no log, just quietly degraded output. Diff real outputs before and after. +4. **Re-check the timeouts.** `/transfers/analyse` and `/proposals/summarise` allow only 10s. A slower or reasoning-style model can exceed that and turn a working endpoint into a timeout. +5. **Re-check cost.** There is no `max_tokens` on any call, so a more expensive model multiplies cost directly with no ceiling. See [operations.md](operations.md#cost-controls-on-the-llm-path). +6. **Update `test_correct_model_used`** in [tests/test_chat.py](../tests/test_chat.py), which asserts the `/chat` model literal and will fail on any change. + +Changing a chat model is reversible: revert the literal and redeploy. + +### Changing the embedding model + +**This is not a routine change and is not reversible by redeploy alone.** + +`text-embedding-3-small` appears at two call sites — the write path in `/index/message` and the read path in `/search`. Vectors produced by different embedding models occupy different spaces and are not comparable. Changing the model on only one side produces a search that returns confident nonsense with **no error and no warning**. Changing it on both sides leaves every previously indexed message in the old space, which is the same failure for all existing data. + +A correct migration requires: + +1. Change the literal at **both** call sites in the same commit. +2. Re-embed and re-index the entire existing `Message` corpus with the new model. +3. Cut over reads only once the re-index is complete — ideally via a new collection and an atomic switch, since `/search` against a half-migrated corpus silently mixes both spaces. + +Also confirm the new model's dimensionality is compatible with the `Message` collection configuration before starting — see [contracts-weaviate-schema.md](contracts-weaviate-schema.md). + +## Privacy boundary + +**What leaves the system.** Every LLM-backed endpoint sends its payload to OpenAI, the sole external model provider. Specifically: + +| Endpoint | Sent to OpenAI | +| -------- | -------------- | +| `POST /chat` | The `_SYSTEM_PROMPT` and the full `message` string, verbatim | +| `POST /transfers/analyse` | `amount`, `sender`, `recipient`, `memo` — interpolated into the prompt (LLM path only) | +| `POST /proposals/summarise` | `title`, `description`, `amount` — interpolated into the prompt | +| `POST /index/message` | The full `content` string of the message being indexed | +| `GET /search` | The full `q` query string | + +Two of these deserve emphasis. **`/transfers/analyse` transmits both Stellar addresses and the transfer memo** — the memo is application-controlled and may carry a chat message reference or user-supplied text. **`/index/message` transmits the complete plaintext body of every indexed message**, which makes it the single largest flow of user content to a third party in the service. + +**What never leaves the system.** + +- **`OPENAI_API_KEY`** is used to authenticate to OpenAI and is never placed in a prompt. +- **`ChatRequest.conversation_id`** is accepted and validated but never sent to the model — `/chat` builds its message list from `request.message` alone. +- **`IndexMessageRequest.messageId`, `conversationId`, and `senderId`** are never sent to OpenAI. Only `content` is embedded. The identifiers are written to Weaviate as properties alongside the vector, so **message content reaches OpenAI while message metadata stays local**. +- **The `conversationId` filter on `/search`** is applied inside Weaviate via `Filter.by_property("conversationId").equal(conversationId)`, not by the model. Conversation scoping is enforced locally. +- **The stored vectors and all Weaviate contents.** Weaviate is a local dependency (`connect_to_local()`); the corpus is never shipped to the model provider. Only the text being embedded transits, one call at a time. +- **Everything on the `/transfers/analyse` rule-based path.** Transfers above `10_000.0` XLM are resolved entirely in-process with no network call, so their addresses and memos never reach OpenAI at all. +- **`GET /health`** involves no external call whatsoever. + +**Boundary properties to be aware of.** + +- **The service performs no redaction, masking, or PII stripping.** Whatever the caller supplies is what the provider receives. +- **Message content sent to OpenAI is plaintext at that point.** Whatever end-to-end encryption applies in the wider platform (see [apps/web/docs/concepts-e2ee-architecture.md](../../web/docs/concepts-e2ee-architecture.md)) does not extend to `/index/message`: the endpoint receives and forwards cleartext. Indexing a message is a deliberate decision to expose its content to the model provider. +- **Retention is governed by the OpenAI account's data policy**, not by this service. The service keeps no record of what it sent — no prompt logging, no request archive. +- **There is no authentication on any endpoint.** Any party able to reach the service can cause data to be sent to OpenAI under your account. Network isolation is the controlling mitigation — see [operations.md](operations.md#known-operational-risks). + +## Related documents + +- [Operations guide](operations.md) — running the service, health checks, scaling, cost controls +- [Testing guide](testing.md) — testing prompt and parsing behaviour without a real model +- [Pydantic models](contracts-pydantic-models.md) — the typed request/response shapes +- [Transfer risk analysis](concepts-transfer-risk-analysis.md) — why the high-value rule bypasses the model +- [RAG search architecture](concepts-rag-search-architecture.md) — how embeddings are indexed and retrieved +- [Weaviate schema](contracts-weaviate-schema.md) — the `Message` collection and vector configuration +- [Chat API](api-chat.md) · [Transfers analyse API](api-transfers-analyse.md) · [Proposals summarise API](api-proposals-summarise.md) · [Index and search API](api-index-search.md) diff --git a/apps/ai_agent/docs/configuration.md b/apps/ai_agent/docs/configuration.md new file mode 100644 index 0000000..e83c80f --- /dev/null +++ b/apps/ai_agent/docs/configuration.md @@ -0,0 +1,289 @@ +# AI agent configuration & environment reference + +Every setting the `ai_agent` service reads, what happens when one is missing, and +which settings are not configurable at all. + +The headline is short, and it is deliberately stated up front because it is the +opposite of what most services look like: **the service reads exactly one +environment variable, `OPENAI_API_KEY`.** Everything else — the model, the +Weaviate connection, the bind host and port — is hardcoded in +[`apps/ai_agent/main.py`](../main.py). Changing any of them is a code change, not +a deployment setting. + +--- + +## 1. Environment variables + +| Variable | Required | Default | Read at | Effect when unset | +| ---------------- | -------- | ------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `OPENAI_API_KEY` | Yes\* | _none_ | Per request, inside `_openai_client()` — not at import and not at boot | The process still starts and `/health` still returns 200; every OpenAI-backed endpoint returns 500. See [§3](#3-startup-and-missing-api-key-behaviour). | + +\* Required for the service to do anything useful. It is _not_ required for the +process to start, and that distinction is the single most important thing in this +document. + +It is the same key the backend uses — it is declared once in the repo-root +[`.env.example`](../../../.env.example) under `# AI Service`: + +```bash +# AI Service +OPENAI_API_KEY= +``` + +There is no `.env` loading in `main.py` — no `python-dotenv`, no +`pydantic-settings`. The variable must be present in the process environment. In +local development that is whatever your shell or process manager exports; in +Docker or Kubernetes it is the container environment. + +### Variables the service does **not** read + +Worth stating explicitly, because their absence is easy to mistake for a bug: + +| Variable | Status | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `OPENAI_MODEL` (or any model-selection variable) | Not read. Models are hardcoded — see [§2](#2-hardcoded-settings). | +| `WEAVIATE_URL` / `WEAVIATE_HOST` / `WEAVIATE_PORT` | Not read. The connection is `weaviate.connect_to_local()` — see [§4](#4-weaviate-connection). | +| `WEAVIATE_API_KEY` | Not read. The client connects unauthenticated. | +| `HOST` / `PORT` | Not read. See [§2](#2-hardcoded-settings). | +| `OPENAI_BASE_URL` | Not read. `OpenAI(api_key=...)` is constructed with no base URL override, so the SDK's own environment handling is the only way to redirect it. | + +Setting any of these has no effect on this service. + +--- + +## 2. Hardcoded settings + +| Setting | Value | Where | +| -------------------- | ------------------------------------------------------ | ---------------------------------------------------- | +| Chat model | `gpt-4o-mini` | `chat`, `analyse_transfer`, `summarise_proposal` | +| Embedding model | `text-embedding-3-small` | `index_message`, `search_messages` | +| Bind host | `0.0.0.0` | `uvicorn.run(...)` under `if __name__ == "__main__"` | +| Bind port | `8000` | same | +| Weaviate collection | `Message` | `index_message`, `search_messages` | +| Search result limit | `5` | `search_messages` | +| High-value threshold | `10_000.0` XLM | `_HIGH_VALUE_THRESHOLD`, used by `analyse_transfer` | +| Request timeouts | 30 s for `/chat`, 10 s for the two JSON-mode endpoints | passed per call as `timeout=` | +| System prompt | `_SYSTEM_PROMPT` | module level | + +Two consequences: + +- **Model changes require a code change and a deploy.** There is no way to move + to a different model, or to run two environments on different models, through + configuration. +- **The host and port literals only apply to `python main.py`.** Running the + module directly executes the `__main__` block and binds `0.0.0.0:8000`. Under + any ASGI server invoked against the app object — which is how the repo README + runs it (`uv run fastapi dev main.py`) and how a container normally runs it — + the `__main__` block never executes and the host/port come from that command: + + ```bash + # __main__ block runs: binds 0.0.0.0:8000 + python main.py + + # __main__ block does NOT run: fastapi/uvicorn decides the bind address + uv run fastapi dev main.py # defaults to 127.0.0.1:8000 + uv run uvicorn main:app --host 0.0.0.0 --port 9000 + ``` + + So the port is effectively set by the launch command, and the `8000` in + `main.py` is a default only for the direct-execution path. A deployment that + needs a different port sets it on the server command line. + +--- + +## 3. Startup and missing-API-key behaviour + +`OPENAI_API_KEY` is read **per request**, inside the `_openai_client()` helper: + +```python +def _openai_client(): + api_key = os.environ.get("OPENAI_API_KEY") + if not api_key: + raise HTTPException(status_code=500, detail="OPENAI_API_KEY is not configured") + if OpenAI is None: + raise HTTPException(status_code=500, detail="openai package is not installed") + + return OpenAI(api_key=api_key) +``` + +Nothing validates it at import time or at startup. The service therefore **starts +successfully with no API key at all** and fails only when an OpenAI-backed +endpoint is called. + +### The exact behaviour, endpoint by endpoint + +| Endpoint | With no `OPENAI_API_KEY` | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `GET /health` | **200** `{"status": "ok"}` | +| `POST /chat` | **500** `{"detail": "OPENAI_API_KEY is not configured"}` | +| `POST /transfers/analyse` | **500** — unless `amount > 10000`, which is answered by the rule-based branch **before** any OpenAI client is constructed, and still returns **200** | +| `POST /proposals/summarise` | **500** | +| `POST /index/message` | **500** if Weaviate is reachable (the embedding call needs the key); **503** if Weaviate is unreachable, because the connection is attempted first | +| `GET /search` | Same as above — **503** on a Weaviate failure, **500** on the missing key when Weaviate is up, and **200** with `{"results": []}` when the `Message` collection does not exist, because that path returns before any embedding is requested | + +### This is deliberate, and it is load-balancer-visible + +`/health` is a liveness probe: it answers whether the process is up and serving, +not whether every downstream dependency is configured. Because it does not touch +`_openai_client()`, a misconfigured deployment presents as: + +- **healthy** to any load balancer, orchestrator probe, or uptime monitor + pointed at `/health`, and +- **completely broken** to every real caller, with a 500 on each request. + +That combination is the failure mode to watch for. An instance rolled out with a +missing or misspelled key will pass its health check, be added to the pool, and +serve nothing but 500s. Kubernetes will not restart it; a load balancer will not +drain it. + +The test suite pins this behaviour rather than leaving it incidental — +`apps/ai_agent/tests/test_health.py::test_health_works_without_api_key` asserts +the 200, and `apps/ai_agent/tests/test_chat.py::test_missing_api_key_returns_500` +asserts the 500. Changing either is a deliberate contract change. + +**Operational recommendation.** Do not rely on `/health` to catch a +misconfiguration. Either: + +- assert the key is present at deploy time (a startup check in your orchestration, + or a required secret rather than an optional env var), or +- monitor the 5xx rate on `/chat` separately from the liveness probe, or +- add a readiness endpoint that checks configuration — deliberately distinct from + `/health`, which should keep its current semantics. + +### Related failure: the `openai` package missing + +`main.py` imports `openai` inside a `try/except ImportError` and sets +`OpenAI = None` when it is absent, so an environment without the dependency also +starts cleanly and fails per-request with **500** +`{"detail": "openai package is not installed"}`. Same shape of problem, same +`/health`-still-green consequence. + +--- + +## 4. Weaviate connection + +### How it connects + +```python +client = weaviate.connect_to_local() +``` + +No host, no port, no API key, no gRPC configuration. `connect_to_local()` from +`weaviate-client` (pinned at 4.22.0 in [`uv.lock`](../uv.lock), declared as +`>=4.0.0` in [`pyproject.toml`](../pyproject.toml)) uses its own defaults — +HTTP on `localhost:8080` and gRPC on `localhost:50051` — and this service +overrides none of them. + +A connection is opened **per request** on `POST /index/message` and `GET /search`, +and closed in a `finally` block. There is no pooled or long-lived client, and no +connection is attempted at startup. + +**There is no Weaviate service in [`infra/docker-compose.yml`](../../../infra/docker-compose.yml).** +That file brings up Postgres, Redis, and MinIO only. A Weaviate instance must be +run separately, and it must be reachable on the loopback defaults above from the +perspective of the `ai_agent` process — which means a containerised `ai_agent` +talking to a Weaviate in another container will not connect without host +networking or a code change, since `localhost` inside the container is the +container itself. + +### Failure mode when Weaviate is unreachable + +Both Weaviate-backed endpoints wrap the connection in `try/except` and translate +any failure to **503**: + +```python +try: + client = weaviate.connect_to_local() +except Exception: + raise HTTPException(status_code=503, detail="Weaviate connection failed") +``` + +| Condition | Response | +| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Weaviate down / unreachable | **503** `{"detail": "Weaviate connection failed"}` | +| Weaviate up, later operation fails | **503** with the underlying exception text as `detail` — the second `except Exception as e` block re-raises as 503 with `str(e)` | +| Weaviate up, `Message` collection missing | `GET /search` → **200** `{"results": []}`; `POST /index/message` creates the collection and proceeds | +| Weaviate up, `OPENAI_API_KEY` missing | **500** — the embedding call fails after the connection succeeds | + +Two points that matter operationally: + +- **`/health` does not check Weaviate either.** A deployment with no Weaviate + reachable is green on its liveness probe and serves 503s on `/search` and + `/index/message`, exactly as with a missing API key. Chat and proposal + summarisation are unaffected. +- **The 503 detail can leak internals.** The second handler passes `str(e)` + straight into the response body, so a client can see raw client-library error + text. Treat that as an internal-facing detail, not a stable contract. + +`apps/ai_agent/tests/test_search.py::test_weaviate_connection_failure_returns_503` +pins the 503 behaviour, and `test_missing_collection_returns_empty_results` pins +the empty-result path, including that no query is issued. Note that the +equivalent failure on `POST /index/message` has no test of its own — the coverage +is on the `/search` side only. + +### Collection schema + +The collection is created on demand by `POST /index/message` with +`client.collections.create(name="Message")` — no property schema and no +vectoriser, because vectors are supplied explicitly from OpenAI embeddings. +Properties written per object are `conversationId`, `messageId`, `senderId`, and +`content`; searches filter on `conversationId` and return the top 5 by vector +similarity. See +[`contracts-weaviate-schema.md`](./contracts-weaviate-schema.md) for the full +shape and [`concepts-rag-search-architecture.md`](./concepts-rag-search-architecture.md) +for how it is used. + +--- + +## 5. Running the service + +From the repo root, as documented in the [main README](../../../README.md): + +```bash +cd apps/ai_agent && uv run fastapi dev main.py +``` + +With the API key set for the process: + +```bash +OPENAI_API_KEY=sk-... uv run fastapi dev main.py +``` + +Tests (Python 3.12+, per `requires-python` in `pyproject.toml`): + +```bash +cd apps/ai_agent +uv sync --group dev +uv run pytest +``` + +The suite sets `OPENAI_API_KEY=test-key` for every test through an autouse +fixture in [`tests/conftest.py`](../tests/conftest.py) and mocks both `OpenAI` +and `weaviate.connect_to_local`, so **no network access and no real key are +needed** — and, as a corollary, a real misconfiguration will never be caught by +running the tests. + +CI runs ruff, ruff format, mypy, and pytest with coverage via +[`.github/workflows/ai-agent-ci.yml`](../../../.github/workflows/ai-agent-ci.yml). + +--- + +## 6. Configuration checklist for a deployment + +- [ ] `OPENAI_API_KEY` is present in the process environment — verified by something other than `/health`. +- [ ] The `openai` package is installed in the runtime image (`uv sync`), or every request 500s. +- [ ] If `/search` or `/index/message` are used: a Weaviate instance is running and reachable at the client's local defaults **from the ai_agent process's own network namespace**. +- [ ] Bind host and port set on the server command line, not expected from env. +- [ ] Monitoring watches the 5xx rate on `/chat` and the 503 rate on `/search`, not just liveness. +- [ ] Model choice reviewed as a code change — `gpt-4o-mini` and `text-embedding-3-small` are compiled in. + +--- + +## 7. Related documents + +- [Repo-wide environment reference](../../../.env.example) — the single place every service's variables are declared, including the `OPENAI_API_KEY` this service shares with the backend. There is no separate env-reference document; `.env.example` is it. +- [Main README](../../../README.md) — how the AI service is started alongside the web and backend apps +- [`POST /chat`](./api-chat.md), [`POST /transfers/analyse`](./api-transfers-analyse.md), [`POST /proposals/summarise`](./api-proposals-summarise.md), [`/index` & `/search`](./api-index-search.md) — per-endpoint request/response contracts +- [Weaviate schema contract](./contracts-weaviate-schema.md) — the `Message` collection this service creates and queries +- [RAG search architecture](./concepts-rag-search-architecture.md) — how indexing and search fit together +- [Pydantic model contracts](./contracts-pydantic-models.md) — request/response models referenced above diff --git a/apps/ai_agent/docs/operations.md b/apps/ai_agent/docs/operations.md new file mode 100644 index 0000000..6f0e595 --- /dev/null +++ b/apps/ai_agent/docs/operations.md @@ -0,0 +1,291 @@ +# AI agent operations guide + +How to run the `ai_agent` service in production: the ASGI/uvicorn setup, what `/health` does and does not tell an orchestrator, the Weaviate dependency, scaling characteristics, and the state of cost controls on the LLM path. + +This document describes the service as implemented in [apps/ai_agent/main.py](../main.py). Where the current implementation lacks an operational control, this document says so explicitly rather than describing an intended design — see [Known operational risks](#known-operational-risks). + +## Overview + +`ai_agent` is a single-module FastAPI application. It exposes five endpoints: + +| Method | Path | External dependencies | +| ------ | ---------------------- | ----------------------------- | +| `GET` | `/health` | none | +| `POST` | `/chat` | OpenAI | +| `POST` | `/transfers/analyse` | OpenAI (conditionally — see below) | +| `POST` | `/proposals/summarise` | OpenAI | +| `POST` | `/index/message` | OpenAI **and** Weaviate | +| `GET` | `/search` | OpenAI **and** Weaviate | + +The application object is `app`, defined at module scope in `main.py`. It holds no background tasks, no scheduler, no connection pool, and no in-process cache. Every request is fully independent, which is what makes the scaling story simple (see [Scaling characteristics](#scaling-characteristics)). + +## Running the service + +### The production start command + +`main.py` ends with a `__main__` guard: + +```python +if __name__ == "__main__": + uvicorn.run(app, host="0.0.0.0", port=8000) +``` + +Running `python main.py` therefore starts a **single-process, single-worker** uvicorn server on port 8000. This is the development path. Do not use it in production: it gives you one worker, no process supervision, and no way to set worker counts or timeouts without editing source. + +For production, invoke uvicorn directly against the ASGI app path so that all server configuration lives in the command rather than in code: + +```bash +uvicorn main:app \ + --host 0.0.0.0 \ + --port 8000 \ + --workers 4 \ + --timeout-keep-alive 65 \ + --no-server-header +``` + +`main:app` resolves to the `app` object in `main.py`, so the process must start with `apps/ai_agent` as the working directory (or that directory on `PYTHONPATH`). This mirrors the test configuration, which sets `pythonpath = ["."]` in `[tool.pytest.ini_options]` in [pyproject.toml](../pyproject.toml). + +### Worker configuration + +The endpoints are defined with `def`, not `async def`. FastAPI runs synchronous path operation functions in a **thread pool** rather than on the event loop, so a single worker can serve multiple concurrent requests despite the blocking OpenAI and Weaviate calls. The default thread pool is bounded (40 threads in current AnyIO defaults), which sets the practical per-worker concurrency ceiling. + +Sizing guidance: + +- **Workers**: start at `2 × CPU cores`. The service is I/O-bound, not CPU-bound — nearly all wall-clock time is spent waiting on OpenAI or Weaviate — so worker count is governed by memory and by upstream rate limits, not by core count. +- **Memory**: each worker is a full Python process that imports `fastapi`, `openai`, and `weaviate-client`. Budget conservatively and measure; do not assume workers are cheap. +- **Keep-alive**: set `--timeout-keep-alive` above your load balancer's idle timeout so the balancer, not the server, closes idle connections. + +Because there is no shared state between requests, worker count can be changed freely without correctness consequences. + +### Required environment + +| Variable | Required by | Behaviour when absent | +| ---------------- | ------------------------------------------------------ | -------------------------------------------- | +| `OPENAI_API_KEY` | `/chat`, `/transfers/analyse`, `/proposals/summarise` | Those endpoints return `500`. `/health` still returns `200`. | +| `OPENAI_API_KEY` | `/index/message`, `/search` | Those endpoints return **`503`**, not `500` — see the note below. | + +**The two Weaviate endpoints report a missing API key as `503`, even when Weaviate is healthy.** `_openai_client()` raises `HTTPException(500, ...)` as usual, but on `/index/message` and `/search` the embedding call sits inside a `try` block whose `except Exception as e` catches the `HTTPException` and re-raises it as a `503`, folding the original message into the detail. The observed response is: + +``` +503 {"detail": "500: OPENAI_API_KEY is not configured"} +``` + +The operational consequence: **a `503` from these two endpoints does not reliably mean Weaviate is down.** It may equally be an OpenAI misconfiguration. Read the `detail` string before concluding which dependency has failed — the `"500: "` prefix is the tell. + +> **Correction.** [configuration.md](configuration.md#3-startup-and-missing-api-key-behaviour) states that these endpoints return `500` when Weaviate is reachable and only `503` when it is unreachable. That is not the observed behaviour: the broad `except Exception` catches `HTTPException` too, so the response is `503` in **both** cases. This was verified by exercising both endpoints with a mocked-healthy Weaviate and the key removed. The `503` is not a signal about which dependency failed. + +The key is read **per request** inside `_openai_client()`, not once at import time: + +```python +def _openai_client(): + api_key = os.environ.get("OPENAI_API_KEY") + if not api_key: + raise HTTPException(status_code=500, detail="OPENAI_API_KEY is not configured") + if OpenAI is None: + raise HTTPException(status_code=500, detail="openai package is not installed") + + return OpenAI(api_key=api_key) +``` + +Two operational consequences follow: + +1. **The service starts successfully with no API key.** Misconfiguration is not caught at boot; it surfaces as `500`s on first LLM traffic. There is no startup validation to rely on. +2. **A new `OpenAI` client is constructed on every LLM request.** There is no shared client and therefore no connection pooling or client-level reuse across requests. + +The Weaviate endpoints call `weaviate.connect_to_local()`, which targets `localhost:8080` (HTTP) and `localhost:50051` (gRPC) by default. There is no environment variable wired to override the Weaviate host in the current implementation — see [Known operational risks](#known-operational-risks). + +## Health checking + +```python +@app.get("/health") +def health_check(): + return {"status": "ok"} +``` + +`/health` returns `200` with body `{"status": "ok"}`. + +### Semantics for orchestrators + +**`/health` is a liveness probe, not a readiness probe.** It checks exactly one thing: that the ASGI process is running and able to serve a request. It performs no dependency checks whatsoever. + +Specifically, `/health` returns `200` when: + +- `OPENAI_API_KEY` is unset or invalid — the handler never calls `_openai_client()`, so the missing key is never detected. This behaviour is pinned by `test_health_works_without_api_key` in [tests/test_health.py](../tests/test_health.py). +- Weaviate is unreachable, unhealthy, or has never been started. +- The OpenAI API is down, rate-limiting the service, or the account is out of quota. + +This is a deliberate property, and it is the correct behaviour for a liveness probe: an orchestrator must not restart a process because a *third-party* dependency is degraded. Restarting fixes nothing and turns a partial outage into a crash loop. + +It also means **`/health` returning `200` does not mean the service can do useful work.** Do not treat it as a readiness signal, and do not gate a deploy on it alone. + +Recommended orchestrator wiring: + +```yaml +livenessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 +``` + +Do **not** configure `/health` as a `readinessProbe` if your intent is "can this pod serve LLM traffic" — it will report ready with a missing API key and a dead Weaviate. There is no endpoint in the current implementation that verifies dependency health. Until one exists, readiness must be inferred from error-rate monitoring on the real endpoints (see [Monitoring](#monitoring)). + +## The Weaviate dependency + +Two endpoints depend on Weaviate: `POST /index/message` (writes) and `GET /search` (reads). The other three do not touch it at all. + +Both follow the same connect/operate/close pattern: + +```python +try: + client = weaviate.connect_to_local() +except Exception: + raise HTTPException(status_code=503, detail="Weaviate connection failed") + +try: + ... +except Exception as e: + raise HTTPException(status_code=503, detail=str(e)) +finally: + client.close() +``` + +### Connection lifecycle + +A **fresh Weaviate connection is opened and closed on every request**. There is no connection pool and no persistent client. `connect_to_local()` performs a startup handshake including a gRPC health check, so each request pays that setup cost. This is the single largest fixed overhead on the Weaviate path and the first thing to change if `/search` latency becomes a problem. + +The `finally: client.close()` guarantees the connection is released on both the success and failure paths, so a Weaviate outage does not leak sockets. + +### Behaviour when Weaviate is down + +| Condition | Endpoint | Response | +| --------------------------------------------- | ---------------- | -------------------------------------------------------- | +| Weaviate unreachable (connect fails) | `/index/message` | `503` — `{"detail": "Weaviate connection failed"}` | +| Weaviate unreachable (connect fails) | `/search` | `503` — `{"detail": "Weaviate connection failed"}` | +| Connects, then fails mid-operation | `/index/message` | `503` — `detail` is the stringified exception | +| Connects, then fails mid-operation | `/search` | `503` — `detail` is the stringified exception | +| Connects, `Message` collection does not exist | `/search` | `200` — `{"results": []}` | +| Connects, `Message` collection does not exist | `/index/message` | `200` — the collection is **created**, then written to | + +Two asymmetries are worth internalising: + +**`/search` fails open on a missing collection.** If the `Message` collection does not exist, `/search` returns `200` with an empty result list rather than an error. From the caller's perspective a never-indexed corpus is indistinguishable from a genuine zero-hit query. An empty `/search` response is therefore *not* evidence that the index is healthy. + +**`/index/message` creates the collection on demand.** The first successful index call after a fresh Weaviate deployment implicitly creates `Message`. There is no separate schema migration step to run, but it also means an accidental point at an empty Weaviate silently starts building a new index rather than failing loudly. + +**Errors are surfaced verbatim.** On the mid-operation failure path, `detail=str(e)` returns the raw exception message to the caller. Treat `503` bodies from these endpoints as internal diagnostic data and do not render them directly in a user-facing surface. + +### Blast radius + +A Weaviate outage degrades search and indexing only. `/chat`, `/transfers/analyse`, and `/proposals/summarise` continue to serve normally because they never open a Weaviate connection. Availability of the two dependency groups should be tracked separately; a single aggregate error rate for the service will obscure which half is broken. + +Note that both Weaviate endpoints *also* call OpenAI, for embeddings. `/index/message` and `/search` therefore need **both** dependencies healthy — and because the embedding call happens inside the broadly-caught `try` block, an OpenAI failure on these two endpoints is also reported as `503`, not `500`. The status code alone cannot tell the two dependencies apart here; only the `detail` string can. + +## Scaling characteristics + +**The service is stateless.** No session state, no in-process cache, no cross-request coordination, no sticky-session requirement. Scale horizontally by adding replicas behind any load balancer; scale vertically with `--workers`. Both are safe. + +**Latency is dominated by upstream calls.** Per-request timeouts as configured in code: + +| Endpoint | Upstream call | Timeout | +| ---------------------- | -------------------------------- | --------- | +| `/chat` | chat completion (`gpt-4o-mini`) | 30s | +| `/transfers/analyse` | chat completion (`gpt-4o-mini`) | 10s | +| `/proposals/summarise` | chat completion (`gpt-4o-mini`) | 10s | +| `/index/message` | embedding + Weaviate write | **none** | +| `/search` | embedding + Weaviate query | **none** | + +The embedding calls in `/index/message` and `/search` pass no `timeout` argument, so they fall back to the OpenAI client default (10 minutes). Weaviate operations are likewise untimed. **A hung upstream on either Weaviate endpoint can occupy a thread-pool slot far longer than any of the chat endpoints can.** Under a partial OpenAI degradation, this is the failure mode most likely to exhaust worker capacity. Set an aggressive server-side or proxy-level request timeout in front of the service to bound it. + +**Capacity is bounded by the upstream rate limit, not by the service.** Because there is no in-process concurrency limit on outbound calls, adding replicas multiplies the request rate you present to the OpenAI API. Past a certain replica count, additional capacity converts a latency problem into a `429` problem. Size replicas against your OpenAI organisation's rate limits, and note that the service does not currently retry or back off on `429` — the error propagates to the caller as a `500`. + +**The rule-based short-circuit is the one free scaling win.** In `/transfers/analyse`, transfers above the threshold are resolved without an LLM call at all: + +```python +_HIGH_VALUE_THRESHOLD = 10_000.0 + +if request.amount > _HIGH_VALUE_THRESHOLD: + return TransferAnalyseResponse( + flagged=True, + reason=f"Amount {request.amount} XLM exceeds {_HIGH_VALUE_THRESHOLD} XLM threshold", + confidence=0.99, + ) +``` + +These requests are pure CPU, return in microseconds, cost nothing, and cannot be affected by an OpenAI outage. `test_high_value_transfer_is_flagged_without_llm_call` in [tests/test_transfers.py](../tests/test_transfers.py) asserts the LLM is never called on this path. Note the boundary is strict `>`: an amount of exactly `10_000.0` takes the LLM path. + +## Cost controls on the LLM path + +**There are no rate limits, quotas, spend caps, caching, or per-caller throttling anywhere in the service.** This is a known gap, recorded here so it can be planned for rather than discovered during an incident. + +What exists today: + +- **Per-request timeouts** on the three chat endpoints (30s / 10s / 10s), which bound the duration of a single call but not the number of calls. +- **The `/transfers/analyse` high-value short-circuit**, which avoids an LLM call for large transfers. +- **A cheap model choice** — `gpt-4o-mini` for all chat completions and `text-embedding-3-small` for all embeddings. See [concepts-prompts-and-models.md](concepts-prompts-and-models.md) for the per-endpoint model detail and how to change it safely. +- **Bounded response size** on `/search`, which caps results at `limit=5`. + +What does not exist: + +- No rate limiting of any kind — no per-IP, per-caller, per-conversation, or global request cap. Any client that can reach the service can issue unbounded LLM calls. +- No authentication or authorisation on any endpoint. There is no API key check, no bearer token, and no caller identity. Combined with the absence of rate limiting, **any party with network reach to the service can spend against the OpenAI account directly.** +- No `max_tokens` on any completion call, so output length — and therefore per-call cost — is bounded only by the model's own limit. +- No cap on input size. `ChatRequest.message`, `IndexMessageRequest.content`, and the `/search` `q` parameter are unbounded strings. A large payload becomes a large token bill. +- No caching. Identical `/search` queries re-embed on every call; identical `/chat` messages re-complete on every call. +- No retry or backoff, so a `429` from OpenAI surfaces as a `500` rather than being absorbed. +- No cost or token-usage metric is recorded. `response.usage` is available on every OpenAI response and is discarded. + +### Required mitigations + +Until controls exist in the application, they must be enforced in the deployment: + +1. **Do not expose the service to the public internet.** Bind it to an internal network and let only the backend reach it. This is the single most important control, because it is the only thing standing between an unauthenticated endpoint and an unbounded bill. +2. **Enforce rate limits at the ingress/gateway layer** — per-caller and global — since the application enforces none. +3. **Set a hard monthly spend cap and usage alerts in the OpenAI dashboard.** This is the only true backstop against runaway spend. +4. **Bound request body size at the proxy** to cap per-call token cost. +5. **Set an aggressive proxy request timeout** to cover the untimed embedding and Weaviate calls. + +## Monitoring + +There is no metrics endpoint, no structured logging, and no tracing in the service. Observability must come from the layer in front of it. Track at minimum: + +- **Rate of `500`s on `/chat`, `/transfers/analyse`, and `/proposals/summarise`** — the signal for a missing/invalid `OPENAI_API_KEY`, an OpenAI outage, or `429` rate limiting, none of which `/health` will show. +- **Rate of `503`s on `/index/message` and `/search`** — a combined Weaviate *and* OpenAI failure signal, again invisible to `/health`. Because both dependencies collapse into the same status code here, alert on the code but triage on the `detail` string. +- **p99 latency on `/index/message` and `/search`** separately from the chat endpoints, because these are the untimed paths. +- **OpenAI spend and token usage**, from the provider dashboard, since the service records neither. + +For the wider platform's metric conventions see [docs/observability.md](../../../docs/observability.md). + +## Known operational risks + +Recorded so they are tracked rather than rediscovered: + +| # | Risk | Impact | Suggested mitigation | +| - | ---- | ------ | -------------------- | +| 1 | No authentication on any endpoint | Anyone with network reach can spend against the OpenAI account | Network isolation now; auth in the application | +| 2 | No rate limiting or spend cap in the application | Unbounded cost exposure under abuse or a client bug | Gateway rate limits + OpenAI dashboard spend cap | +| 3 | `/health` never checks dependencies | A pod reports healthy while unable to serve any real request | Treat as liveness only; add a separate readiness endpoint | +| 4 | No startup validation of `OPENAI_API_KEY` | Misconfiguration deploys cleanly and fails on first traffic | Validate at startup and fail fast | +| 5 | Weaviate host is not configurable | `connect_to_local()` hardcodes localhost; Weaviate must be co-located | Wire host/port to environment variables | +| 6 | No timeout on embedding or Weaviate calls | A hung upstream holds a worker thread for up to 10 minutes | Pass explicit timeouts; bound at the proxy | +| 7 | New `OpenAI` client and new Weaviate connection per request | Avoidable per-request setup latency | Reuse clients across requests via app lifespan | +| 8 | `503` bodies leak raw exception text | Internal detail exposed to callers | Log the exception; return a generic message | +| 8a | The broad `except Exception` on both Weaviate endpoints swallows `HTTPException`, remapping a `500` to a `503` | A missing API key is indistinguishable from a Weaviate outage by status code | Re-raise `HTTPException` unchanged; catch only Weaviate errors | +| 9 | `/search` returns `200` `{"results": []}` when the collection is missing | An unindexed corpus is indistinguishable from no matches | Distinguish the two states in the response | +| 10 | No `max_tokens` and no input size cap | Per-call cost is unbounded | Set `max_tokens`; validate input length | + +## Related documents + +- [Configuration reference](configuration.md) — environment variables and hardcoded settings in full (but see the correction above on the missing-key status code) +- [Prompts and model configuration](concepts-prompts-and-models.md) — models per endpoint, prompts, and output parsing +- [Testing guide](testing.md) — running the suite without a real model or vector store +- [Chat API](api-chat.md) — `POST /chat` request/response detail +- [Transfers analyse API](api-transfers-analyse.md) — `POST /transfers/analyse` detail +- [Proposals summarise API](api-proposals-summarise.md) — `POST /proposals/summarise` detail +- [Index and search API](api-index-search.md) — `POST /index/message` and `GET /search` detail +- [Weaviate schema](contracts-weaviate-schema.md) — the `Message` collection +- [RAG search architecture](concepts-rag-search-architecture.md) — how indexing and retrieval fit together +- [Operator runbook](../../../docs/runbook.md) — platform-wide failure modes and incident response +- [Observability](../../../docs/observability.md) — platform metric and dashboard conventions diff --git a/apps/ai_agent/docs/testing.md b/apps/ai_agent/docs/testing.md new file mode 100644 index 0000000..c2751e3 --- /dev/null +++ b/apps/ai_agent/docs/testing.md @@ -0,0 +1,416 @@ +# AI agent testing guide + +How the `ai_agent` pytest suite is configured, what each fixture in [tests/conftest.py](../tests/conftest.py) patches, and how to add a test for a new endpoint without ever calling a real model or vector store. + +## Running the suite + +All commands run from `apps/ai_agent`. + +```bash +# install dev dependencies (pytest, pytest-mock, pytest-cov, httpx) +uv sync --extra dev + +# run everything +uv run pytest + +# one module +uv run pytest tests/test_chat.py + +# one test +uv run pytest tests/test_chat.py::test_correct_model_used + +# quieter, stop on first failure +uv run pytest -q -x +``` + +The suite is fully hermetic: **no test makes a network call, and none requires a running Weaviate or a real OpenAI key.** Every external dependency is patched. A test that hangs or hits the network is a bug in that test, not a missing service. + +## Test configuration + +From [pyproject.toml](../pyproject.toml): + +```toml +[tool.pytest.ini_options] +pythonpath = ["."] +testpaths = ["tests"] +addopts = "--cov=. --cov-report=term-missing --cov-config=pyproject.toml" + +[tool.coverage.run] +omit = ["tests/*"] +``` + +What each line does: + +- **`pythonpath = ["."]`** puts `apps/ai_agent` on `sys.path`, which is what makes the bare `from main import app` import work. Without it every test module fails at import. It is also why pytest must be invoked from `apps/ai_agent` rather than the repo root. +- **`testpaths = ["tests"]`** scopes collection to `tests/`, so a bare `pytest` never wanders into other packages. +- **`addopts`** applies coverage flags automatically — plain `pytest` already produces a coverage report; you never pass `--cov` by hand. + +Dev dependencies that matter to the suite: `pytest`, `pytest-mock` (provides the `mocker` fixture the conftest fixtures are built on), `pytest-cov`, and `httpx` (required by FastAPI's `TestClient`). + +## Fixtures + +[tests/conftest.py](../tests/conftest.py) defines four fixtures, available to every test module without import. + +### `set_openai_key` — autouse + +```python +@pytest.fixture(autouse=True) +def set_openai_key(monkeypatch: pytest.MonkeyPatch) -> None: + """Ensure OPENAI_API_KEY is always set so _openai_client() doesn't 500.""" + monkeypatch.setenv("OPENAI_API_KEY", "test-key") +``` + +- **Patches**: the `OPENAI_API_KEY` environment variable. +- **Default**: the literal string `"test-key"`. +- **Autouse**: applies to **every test in the suite** — you never request it by name. + +This exists because `_openai_client()` reads the key on every call and raises `HTTPException(500, "OPENAI_API_KEY is not configured")` when it is absent. Without this fixture, every LLM endpoint test would return `500` on a developer machine with no key set, and results would differ between local runs and CI. The value is never used to authenticate — the OpenAI client is always patched out — it only has to be non-empty. + +`monkeypatch` reverts the variable after each test, so there is no leakage between tests. + +**To test the missing-key path, delete the variable inside the test**, which overrides the autouse fixture for that test only: + +```python +def test_missing_api_key_returns_500(monkeypatch, client): + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + response = client.post("/chat", json=_BASE_BODY) + assert response.status_code == 500 +``` + +`raising=False` keeps the call safe when the variable is already gone. + +### `client` + +```python +@pytest.fixture() +def client() -> TestClient: + """FastAPI TestClient for the main app.""" + from main import app + + return TestClient(app) +``` + +- **Patches**: nothing. It constructs a real `TestClient` around the real `app`. +- **Returns**: a `fastapi.testclient.TestClient` bound to the app from `main.py`. + +`TestClient` dispatches requests in-process through the ASGI app — no socket is opened and no server is started. Routing, Pydantic request validation, `response_model` validation, and `HTTPException` handling all run exactly as in production, so status codes and bodies are trustworthy. + +The `from main import app` import is deliberately *inside* the fixture body rather than at module top level, so the app is imported when the fixture first runs rather than at conftest collection time. + +Note this fixture is function-scoped: each test gets a fresh `TestClient` over the same module-level `app` object. + +### `mock_openai` + +```python +@pytest.fixture() +def mock_openai(mocker): + """Patch the OpenAI client used inside main.py.""" + return mocker.patch("main.OpenAI") +``` + +- **Patches**: the `OpenAI` **class** as bound in `main`'s namespace (`main.OpenAI`). +- **Returns**: the `MagicMock` that replaced the class. + +The patch target is `main.OpenAI`, not `openai.OpenAI`, because `main.py` does `from openai import OpenAI` — the name must be patched where it is *used*, not where it is defined. + +Because the class is patched, `_openai_client()` still runs for real: it still checks `OPENAI_API_KEY` (supplied by the autouse fixture), still checks `OpenAI is None`, and then calls `OpenAI(api_key=api_key)` — which now returns `mock_openai.return_value`. **That means the instance your endpoint uses is `mock_openai.return_value`**, and that is the object to configure: + +```python +def test_valid_request_returns_reply(mock_openai, client): + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_chat_reply("Hello from Clicked AI!") +``` + +By default — with nothing configured — every attribute access returns a fresh `MagicMock`. That is enough to prove no real call was made, but not enough for an endpoint that reads `response.choices[0].message.content`: a bare `MagicMock` there fails `ChatResponse` validation. Configure a return value whenever the handler reads the response. + +The mock also records calls, which is how the suite asserts on prompts and model IDs without a real request: + +```python +call_args = mock_client.chat.completions.create.call_args +assert call_args[1]["model"] == "gpt-4o-mini" +messages = call_args[1]["messages"] +assert messages[0]["role"] == "system" +``` + +### `mock_weaviate` + +```python +@pytest.fixture() +def mock_weaviate(mocker): + """Patch weaviate.connect_to_local used inside main.py.""" + return mocker.patch("main.weaviate.connect_to_local") +``` + +- **Patches**: the `connect_to_local` function on the `weaviate` module as seen from `main`. +- **Returns**: the `MagicMock` that replaced the function. + +`main.py` does `import weaviate` and calls `weaviate.connect_to_local()`, so the attribute on the module object is the correct target. The connected client your endpoint receives is `mock_weaviate.return_value`. + +By default it returns a `MagicMock`, so `client.collections.exists(...)` returns a truthy `MagicMock` rather than a real boolean. For `/search`, whose branch on `collections.exists("Message")` decides between an empty result and a query, set it explicitly: + +```python +def test_search_returns_empty_when_collection_missing(mock_weaviate, mock_openai, client): + mock_weaviate.return_value.collections.exists.return_value = False + ... +``` + +To simulate an outage, give the mock a `side_effect` so the `try`/`except` around `connect_to_local()` fires: + +```python +mock_weaviate.side_effect = Exception("connection refused") +# endpoint now returns 503 {"detail": "Weaviate connection failed"} +``` + +The default `MagicMock` also satisfies the `finally: client.close()` in both Weaviate endpoints, so no extra setup is needed for teardown. + +### A note on existing modules + +The fixtures above are the intended approach for new tests, but **the suite is not uniform today**. Only [tests/test_chat.py](../tests/test_chat.py) uses `client` and `mock_openai` throughout. [tests/test_search.py](../tests/test_search.py), [tests/test_transfers.py](../tests/test_transfers.py), and [tests/test_health.py](../tests/test_health.py) instead build a module-level `TestClient(app)` and use `unittest.mock.patch` inline — often patching `main._openai_client` (the helper) rather than `main.OpenAI` (the class). + +Both styles work. Patching `main._openai_client` bypasses the real helper entirely, which is why `test_transfers.py` can assert `mock_openai.assert_not_called()` to prove the high-value rule never reaches the LLM. Prefer the conftest fixtures in new code — they are shorter and consistent — but do not treat the older modules as broken. + +## Worked example: adding a test for a new endpoint + +Suppose a `POST /proposals/classify` endpoint is added to `main.py`, calling `gpt-4o-mini` in JSON mode and returning a typed `category` plus `confidence`, with `category` clamped to `"treasury"` when the model returns something invalid. + +Create `tests/test_classify.py`. There is no fixture import and no conftest boilerplate — fixtures are injected by name. + +```python +"""Unit tests for POST /proposals/classify.""" + +import json +from unittest.mock import MagicMock + +_BASE_BODY = { + "title": "Fund the Q3 audit", + "description": "Engage an external auditor for the treasury contracts.", + "amount": 2500.0, +} + + +def _fake_json_response(payload: dict): + """Shape a MagicMock like an OpenAI chat completion carrying `payload` as JSON.""" + msg = MagicMock() + msg.content = json.dumps(payload) + choice = MagicMock() + choice.message = msg + resp = MagicMock() + resp.choices = [choice] + return resp + + +def test_returns_classification(mock_openai, client): + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_json_response( + {"category": "treasury", "confidence": 0.87} + ) + + response = client.post("/proposals/classify", json=_BASE_BODY) + + assert response.status_code == 200 + data = response.json() + assert data["category"] == "treasury" + assert data["confidence"] == 0.87 + + +def test_uses_expected_model_and_json_mode(mock_openai, client): + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_json_response( + {"category": "treasury", "confidence": 0.5} + ) + + client.post("/proposals/classify", json=_BASE_BODY) + + kwargs = mock_client.chat.completions.create.call_args[1] + assert kwargs["model"] == "gpt-4o-mini" + assert kwargs["response_format"] == {"type": "json_object"} + + +def test_prompt_includes_proposal_fields(mock_openai, client): + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_json_response( + {"category": "treasury", "confidence": 0.5} + ) + + client.post("/proposals/classify", json=_BASE_BODY) + + prompt = mock_client.chat.completions.create.call_args[1]["messages"][0]["content"] + assert "Fund the Q3 audit" in prompt + assert "2500.0" in prompt + + +def test_invalid_category_is_clamped(mock_openai, client): + """A category outside the allowed set must degrade, not leak through.""" + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_json_response( + {"category": "nonsense", "confidence": 0.4} + ) + + response = client.post("/proposals/classify", json=_BASE_BODY) + + assert response.status_code == 200 + assert response.json()["category"] == "treasury" + + +def test_missing_confidence_defaults_to_zero(mock_openai, client): + mock_client = mock_openai.return_value + mock_client.chat.completions.create.return_value = _fake_json_response( + {"category": "treasury"} + ) + + response = client.post("/proposals/classify", json=_BASE_BODY) + + assert response.status_code == 200 + assert response.json()["confidence"] == 0.0 + + +def test_missing_title_returns_422(client): + """Pydantic rejects the body before any model call — no mock needed.""" + response = client.post( + "/proposals/classify", + json={"description": "no title", "amount": 100.0}, + ) + assert response.status_code == 422 + + +def test_missing_api_key_returns_500(monkeypatch, client): + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + response = client.post("/proposals/classify", json=_BASE_BODY) + assert response.status_code == 500 +``` + +The pattern generalises to any endpoint: + +1. **Request `mock_openai` and/or `mock_weaviate`** for endpoints with external calls. Omit both for pure-validation tests — `test_missing_title_returns_422` never reaches a client, so mocking it would be noise. +2. **Never request `set_openai_key`** — it is autouse. +3. **Shape the mock response to what the handler reads.** For chat completions that is `response.choices[0].message.content`; for embeddings it is `response.data[0].embedding`. A local helper like `_fake_json_response` keeps this in one place. +4. **Assert on `call_args`** to pin the model, JSON mode, and prompt contents without a real call. +5. **Cover the degradation paths, not just the happy path.** Missing keys, invalid enum values, and malformed JSON are where the parsing layer earns its keep — see [concepts-prompts-and-models.md](concepts-prompts-and-models.md#output-parsing-and-validation) for what each endpoint does with bad output. +6. **Cover the `422` and `500` boundaries** — Pydantic rejection and missing API key. + +For a Weaviate-backed endpoint, add `mock_weaviate` and shape the embedding response too: + +```python +def test_search_returns_hits(mock_weaviate, mock_openai, client): + embedding = MagicMock() + embedding.embedding = [0.1] * 1536 + embed_result = MagicMock() + embed_result.data = [embedding] + mock_openai.return_value.embeddings.create.return_value = embed_result + + obj = MagicMock() + obj.properties = { + "messageId": "msg-1", + "conversationId": "conv-abc", + "senderId": "user-1", + "content": "payment sent", + } + query_result = MagicMock() + query_result.objects = [obj] + + wv = mock_weaviate.return_value + wv.collections.exists.return_value = True + wv.collections.get.return_value.query.near_vector.return_value = query_result + + response = client.get("/search", params={"q": "payment", "conversationId": "conv-abc"}) + + assert response.status_code == 200 + assert response.json()["results"][0]["messageId"] == "msg-1" +``` + +## Coverage + +Coverage runs automatically on every invocation via `addopts`: + +```toml +addopts = "--cov=. --cov-report=term-missing --cov-config=pyproject.toml" +``` + +- **`--cov=.`** measures everything under `apps/ai_agent` — in practice `main.py`, since it is the only application module. +- **`--cov-report=term-missing`** prints the report to the terminal *and* lists the specific line numbers not executed. This is the useful part: it names the untested lines rather than just a percentage. +- **`--cov-config=pyproject.toml`** reads coverage settings from `[tool.coverage.run]`, which sets `omit = ["tests/*"]` so the test files do not inflate their own numbers. + +### Reading the report + +Output appears after the test results. This is the report from the suite as it currently stands: + +``` +................................... [100%] +=============================== tests coverage ================================ +_______________ coverage: platform win32, python 3.12.5-final-0 _______________ + +Name Stmts Miss Cover Missing +--------------------------------------- +main.py 120 22 82% 83, 183-228, 267-268, 274 +--------------------------------------- +TOTAL 120 22 82% +35 passed +``` + +- **`Stmts`** — executable statements measured (not physical lines; blanks, comments, and most `def` bodies-as-declarations are excluded). +- **`Miss`** — statements never executed by any test. +- **`Cover`** — `(Stmts - Miss) / Stmts`. +- **`Missing`** — the line numbers that were missed, as individual lines and ranges. Open `main.py` at those lines to see exactly what is untested. + +Reading the current report: **`183-228` is the entire body of `index_message`**, which is the coverage gap described below. The rest is minor — `83` is the `OpenAI is None` guard in `_openai_client()` (unreachable while the package is installed), `267-268` is the exception branch in `/search`, and `274` is the `uvicorn.run(...)` line under `if __name__ == "__main__"`, which never executes under pytest. + +Closing the `/index/message` gap is worth roughly 38 statements, taking the module from 82% to around 97%. + +A useful additional report when hunting gaps interactively: + +```bash +uv run pytest --cov-report=html +open htmlcov/index.html +``` + +This renders each source line green (covered) or red (missed), which is faster to scan than a line-number list. + +Note that coverage is measured but **not enforced** — there is no `fail_under` setting, so a drop in coverage will not fail the build. Read the `Missing` column rather than relying on a gate. + +## Known gap: `/index/message` has no test module + +The `tests/` directory contains `test_chat.py`, `test_health.py`, `test_proposals.py`, `test_search.py`, and `test_transfers.py`. **There is no `test_index.py`, and no test anywhere in the suite exercises `POST /index/message`.** + +This is the single largest coverage gap in the service, and it is a good first contribution — the fixtures needed already exist, and `test_search.py` provides a close template since it patches the same two dependencies. + +`/index/message` is also the most complex endpoint to test properly, because it is the only one that both embeds *and* writes, with a branch on whether the object already exists. The cases worth covering: + +| Case | Setup | Expected | +| ---- | ----- | -------- | +| Weaviate unreachable | `mock_weaviate.side_effect = Exception(...)` | `503`, `{"detail": "Weaviate connection failed"}` | +| `Message` collection missing | `collections.exists.return_value = False` | Collection is **created** via `collections.create(name="Message")`, then written | +| New message (insert path) | `collection.data.exists.return_value = False` | `collection.data.insert` called; `replace` not called | +| Existing message (upsert path) | `collection.data.exists.return_value = True` | `collection.data.replace` called; `insert` not called | +| Correct embedding model | any success path | `embeddings.create` called with `model="text-embedding-3-small"` | +| Properties written correctly | any success path | `properties` carries `messageId`, `conversationId`, `senderId`, `content`; `uuid` is `messageId` | +| Mid-operation failure | make `data.insert` raise | `503` with the stringified exception as `detail` | +| Connection always closed | any path, success or failure | `mock_weaviate.return_value.close.assert_called_once()` | +| Missing required field | post a body without `content` | `422` | +| Missing API key | `monkeypatch.delenv("OPENAI_API_KEY", ...)` | **`503`**, not `500` — see below | + +Watch the last row: it is the one case here that does not behave the way the other endpoints do. `_openai_client()` raises `HTTPException(500, ...)`, but on this endpoint that call sits inside the `try` whose `except Exception as e` re-raises everything as `503`, with the original message carried through as `detail`. The assertion to write is: + +```python +def test_missing_api_key_returns_503(monkeypatch, mock_weaviate, client): + """The broad except on this endpoint remaps the helper's 500 to a 503.""" + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + mock_weaviate.return_value.collections.exists.return_value = True + + response = client.post("/index/message", json=_BODY) + + assert response.status_code == 503 + assert "OPENAI_API_KEY" in response.json()["detail"] +``` + +`GET /search` has the identical structure and the identical behaviour. + +The `close()` assertion is worth including on both a success and a failure case — it is the only guard against a Weaviate outage leaking connections, and it is currently unverified. See [operations.md](operations.md#the-weaviate-dependency) for the connection lifecycle this protects. + +## Related documents + +- [Operations guide](operations.md) — running the service, health checks, dependencies +- [Prompts and model configuration](concepts-prompts-and-models.md) — prompts, models, and the parsing behaviour these tests pin +- [Pydantic models](contracts-pydantic-models.md) — request/response shapes driving the `422` cases +- [Weaviate schema](contracts-weaviate-schema.md) — the `Message` collection the index tests would assert against +- [Index and search API](api-index-search.md) — the endpoint behaviour behind the coverage gap +- [Chat API](api-chat.md) · [Transfers analyse API](api-transfers-analyse.md) · [Proposals summarise API](api-proposals-summarise.md) diff --git a/apps/backend/docs/concepts-backpressure.md b/apps/backend/docs/concepts-backpressure.md new file mode 100644 index 0000000..7aeafe5 --- /dev/null +++ b/apps/backend/docs/concepts-backpressure.md @@ -0,0 +1,100 @@ +# Backpressure and slow-consumer handling + +How `services/backpressure.ts` watches per-socket send-buffer occupancy, what +the two thresholds do **today**, and how a client recovers after a hard +disconnect. + +Gateway architecture mentions this module in passing +([concepts-gateway-architecture.md](./concepts-gateway-architecture.md) §8). +That overview currently describes shed as "stop sending". This page is the +accurate account of what is wired. + +## Monitoring + +On connect, `index.ts` calls `registerForBackpressure(socket)`. The socket is +added to an in-process set. A single `setInterval(checkBuffers, 5000)` runs +while any socket is registered and is cleared when the last one +unregisters (disconnect). + +Each tick reads the engine.io transport's `bufferedAmount` (bytes waiting +in the WebSocket send buffer). If that field is missing or throws, occupancy +is treated as `0` — the socket is not shed or disconnected on a read error. + +This is **per socket**, not per user. Two devices of the same account are +independent. The sets live in process memory; they are not shared across +gateway instances. + +## Two tiers + +| Tier | Env | Default | What the code does today | +| --- | --- | --- | --- | +| Shed | `SOCKET_SHED_THRESHOLD` | `32768` bytes | Marks the socket id in `shedSockets`, increments `clicked_backpressure_events_total{action="shed"}`, logs a warning. **Does not change emit behaviour.** | +| Disconnect | `SOCKET_BUFFER_THRESHOLD` | `65536` bytes | Same mark + metric `{action="disconnect"}`, then `socket.disconnect(true)`. **This is the only tier that affects delivery.** | + +Both env values are parsed on every tick (`parseInt`, must be a positive +integer). Invalid or unset values fall back to the defaults above. + +### Which tier changes emit behaviour + +`isSocketShed(socketId)` exists and is exported. **Nothing in the backend +calls it.** No dispatcher, fan-out, or `socket.emit` path consults +`shedSockets` before sending. + +So: + +- **Disconnect** is enforced. Crossing `SOCKET_BUFFER_THRESHOLD` force-closes + the socket. +- **Shed** is telemetry and a flag only. Crossing `SOCKET_SHED_THRESHOLD` + does **not** stop new events being queued. A reader of the gateway + overview must not infer that the server sheds load today — the buffer + will keep growing until the disconnect threshold (or the client catches + up and the flag is cleared on a later tick). + +When occupancy drops back below the shed threshold, the id is removed from +`shedSockets`. That only matters for the metric/flag, not for emits. + +## Hard disconnect — what the client sees + +`socket.disconnect(true)` is a server-initiated close (`close` / disconnect +on the client, reason typically `io server disconnect`). The socket is +removed from rooms, presence, the device-delivery subscriber, and +backpressure monitoring. In-flight emits for that connection are gone. + +The client is not told "you were too slow". There is no dedicated +backpressure event. From the client's point of view this is the same shape +as any other unexpected drop: the connection is dead and it must open a +new one and authenticate again. + +## How resume recovers + +A new connection is a new socket. After auth it re-joins conversation / +user / device rooms and is registered for backpressure from scratch +(`shedSockets` does not survive the old socket id). + +Missed **durable** chat is not in the WebSocket buffer and is not in the +resume stream. The client recovers it with envelope sync +(`GET /sync`, `syncRequired: true` on `resume_complete`). See +[api-messages-sync.md](./api-messages-sync.md) and +[concepts-delivery-fanout.md](./concepts-delivery-fanout.md). + +Missed **ephemeral** events (read/delivery receipts, presence, system +notices) are recovered by the resume/replay path: + +1. Client emits `resume { lastEventId }` with the last Redis stream id it + persisted. +2. The gateway reads `resume:events:${userId}` after that id (exclusive) + and emits `ephemeral_replay` for each entry still within the 300s TTL / + 500-entry cap. +3. It finishes with `resume_complete { lastEventId, syncRequired: true }`. + +Implementation: `services/resumeStream.ts`, handler in +`socket/messaging.ts`. Protocol notes: +[concepts-gateway-architecture.md](./concepts-gateway-architecture.md) §5 +and [api-websocket-events.md](./api-websocket-events.md) (`resume` / +`resume_complete` / `ephemeral_replay`). + +Events that were only sitting in the killed socket's send buffer and were +never recorded to Postgres or the resume stream are not replayed. That is +the loss window a slow consumer accepts when it is hard-disconnected. + +Cross-referenced from [`IMPLEMENTATION_DOCS.md`](../../../IMPLEMENTATION_DOCS.md). diff --git a/apps/backend/docs/concepts-caching.md b/apps/backend/docs/concepts-caching.md new file mode 100644 index 0000000..377cf48 --- /dev/null +++ b/apps/backend/docs/concepts-caching.md @@ -0,0 +1,293 @@ +# Backend caching reference + +The backend caches exactly one thing in Redis: the conversation list a signed-in user sees +when the app opens. That is the query that runs on every cold start and every foreground +resume, it fans out into several joins and two aggregate subqueries, and its result changes +only when something in the conversation actually changes. Everything else the backend puts in +Redis — presence, rate-limit buckets, replay markers, pub/sub fan-out — is coordination state +rather than a cache, and is covered elsewhere. + +This document is the reference for that cache: what is stored, under what key, for how long, +every path that invalidates it, what happens when Redis is not there, and the one scoping +hazard the key format has to respect. + +## Contents + +- [What is cached](#what-is-cached) +- [The cache key](#the-cache-key) +- [Read and write paths](#read-and-write-paths) +- [Invalidation](#invalidation) +- [Degraded behaviour when Redis is unavailable](#degraded-behaviour-when-redis-is-unavailable) +- [The per-device scoping hazard](#the-per-device-scoping-hazard) +- [Other Redis key namespaces (not caches)](#other-redis-key-namespaces-not-caches) +- [Implementation references](#implementation-references) + +## What is cached + +| Property | Value | +| ----------- | -------------------------------------------------------------------------------- | +| Key | `conversations:{userId}` — built by `convCacheKey(userId)` in `src/lib/redis.ts` | +| Value | The JSON body of `GET /conversations`, as a string (`JSON.stringify(result)`) | +| TTL | `CONV_CACHE_TTL` = **30 seconds**, applied with `SETEX` | +| Written by | `GET /conversations` (default view only) | +| Read by | `GET /conversations` (default view only) | +| Invalidated | `invalidateConversationCaches(userIds)` and one direct `DEL` — see below | + +The payload is the complete response array, one entry per conversation the user belongs to: + +```jsonc +[ + { + "id": "…", + "type": "dm", + "name": null, + "avatarUrl": null, + "createdAt": "…", + "messages": [ + /* the single most recent message, with the ciphertext envelope + addressed to the *requesting device* */ + ], + "isMuted": false, + "isArchived": false, + "messageCount": 42, + "unreadCount": 3, + }, +] +``` + +That embedded `messages[0]` is the conversation-list preview, and it is the reason the rest of +this document is more careful than a 30-second cache would normally justify: the preview is +**device-specific**, because the envelope it carries is the one encrypted for the requesting +device and no other device can decrypt it. + +The TTL is deliberately short. Thirty seconds is long enough to absorb the burst of list +requests an app makes while starting up and reconnecting, and short enough that any +invalidation this document has missed self-heals within half a minute instead of persisting +until the user acts. + +## The cache key + +```ts +export function convCacheKey(userId: string): string { + return `conversations:${userId}`; +} +``` + +Everything that touches the cache goes through this function rather than formatting the +string inline, so the format has exactly one definition. Two consequences of the current +shape are worth stating explicitly: + +- **It is scoped to the user, not to the request.** The archived view is a different result + set, so it is neither read from nor written to the cache at all — `?archived=true` skips + both branches rather than using a second key. See + [the per-device scoping hazard](#the-per-device-scoping-hazard) for the dimension this key + does _not_ currently carry. +- **It is a plain string key with a TTL, not a hash or a set.** Invalidation is `DEL`, never a + partial update: no path rewrites part of a cached list. A change invalidates the whole entry + and the next read rebuilds it from Postgres. + +## Read and write paths + +Both live in `GET /conversations` (`src/routes/conversations.ts`) and both are guarded twice — +once on `redis` being non-null, and once by a `try`/`catch`: + +```ts +// Read — skipped entirely for the archived view +if (!showArchived && redis) { + try { + const cached = await redis.get(key); + if (cached) { + res.json(JSON.parse(cached) as unknown); + return; + } + } catch { + // Fall through to the database on any Redis error + } +} + +// … build `result` from Postgres … + +// Write — same two conditions +if (!showArchived && redis) { + try { + await redis.setex(key, CONV_CACHE_TTL, JSON.stringify(result)); + } catch { + // Ignore — the response is already computed + } +} +``` + +A cache miss, a Redis error, and a Redis that was never configured all converge on the same +path: query Postgres and answer from it. The write is best-effort and happens after the +response body exists, so a failure to cache cannot fail a request. + +## Invalidation + +`invalidateConversationCaches(userIds)` (`src/lib/conversationCache.ts`) is the single +invalidation helper: + +```ts +export async function invalidateConversationCaches(userIds: string[]): Promise { + if (!redis || userIds.length === 0) return; + const client = redis; + await Promise.allSettled([...new Set(userIds)].map((userId) => client.del(convCacheKey(userId)))); +} +``` + +Three properties: it de-duplicates the id list, it deletes in parallel, and it uses +`Promise.allSettled` so one failing `DEL` does not reject the whole call or abandon the +remaining users. Callers pass **every member of the affected conversation**, because a change +to one conversation changes every member's list. + +### Every call site + +| # | Trigger (the event a user would describe) | Call site | Users invalidated | +| --- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------- | +| 1 | Members added to a conversation (`POST /conversations/:id/members`) | `src/routes/conversations.ts:419` | All members after the change | +| 2 | Conversation metadata updated — name, avatar (`PATCH /conversations/:id`) | `src/routes/conversations.ts:529` | All members | +| 3 | The last member leaves, so the conversation row is deleted (`DELETE /conversations/:id/leave`) | `src/routes/conversations.ts:872` | The departing member | +| 4 | A member leaves a conversation that still has others (`DELETE /conversations/:id/leave`) | `src/routes/conversations.ts:900` | All members, including the one leaving | +| 5 | Per-member settings changed — mute, archive (`PATCH /conversations/:id/settings`) | `src/routes/conversations.ts:715` | **Direct `redis.del(convCacheKey(userId))`** — only the caller's own entry changes | +| 6 | A device is added or revoked, emitting a `device_added` / `device_revoked` system message into each of that user's conversations | `src/routes/devices.ts:735` (`emitDeviceChangeEvent`) | All members of every conversation the user belongs to | +| 7 | A message is sent over REST (`POST /messages`) | `src/routes/messages.ts:229` | All members | +| 8 | A message is deleted over REST (`DELETE /messages/:id`) | `src/routes/messages.ts:277` | All members | +| 9 | A message is sent over the socket (`send_message`) | `src/socket/messaging.ts:312` | All members | +| 10 | A message is edited over the socket (`edit_message`) | `src/socket/messaging.ts:441` | All members | +| 11 | A file message is sent over the socket (`send_file_message`) | `src/socket/messaging.ts:623` | All members | +| 12 | A conversation is created over the socket (`create_conversation`) | `src/socket/messaging.ts:1003` | Every member of the new conversation | +| 13 | The assistant replies (`ask_assistant`) | `src/socket/messaging.ts:1225` | All members | + +Case 5 is the one direct `DEL` outside the helper, and it is correct as written: mute and +archive are per-member columns, so no other member's list changes. Every other write path uses +the helper. + +The unifying rule, and the one to apply when adding a path: **anything that changes what +`GET /conversations` would return for a user must invalidate that user's key in the same +request.** In practice that means any write to `conversations`, `conversation_members`, or +`messages`, and any change to the device set that produces a system message. The usual shape +is a `findMany` over `conversationMembers` for the affected conversation followed by +`invalidateConversationCaches(members.map((m) => m.userId))`. + +A path that forgets to invalidate does not corrupt anything — it produces a list that is stale +for at most the 30-second TTL. That is a real bug (a sent message that does not appear in the +list preview for half a minute reads as data loss to a user) but a self-healing one, which is +why the TTL is short. + +## Degraded behaviour when Redis is unavailable + +`src/lib/redis.ts` creates the client only when `REDIS_URL` is set, with `lazyConnect: true` +and an `error` listener that deliberately swallows connection errors: + +```ts +export let redis: Redis | null = null; + +if (process.env['REDIS_URL']) { + redis = new Redis(process.env['REDIS_URL'], { lazyConnect: true }); + redis.on('error', () => { + // Graceful degradation: cache misses fall through to DB + }); +} +``` + +Without the listener, ioredis would emit an unhandled `error` event and crash the process on a +Redis blip. With it, every caching call site sees either `redis === null` or a rejected +promise, and both are already handled. + +**The cache is an optimisation, not a correctness dependency.** With Redis down or absent: + +- `GET /conversations` reads and writes nothing and answers from Postgres. The response is + byte-for-byte what the cached version would have been — the cache stores the finished body, + so there is no second code path that could diverge. +- `invalidateConversationCaches` returns immediately on the `!redis` guard, and per-user `DEL` + failures are absorbed by `allSettled`. Nothing upstream sees an error. +- No write is ever gated on the cache. No path reads the cache to make a decision; it is only + ever read to answer a `GET`. Nothing is stored in Redis that is not reconstructible from + Postgres. + +The cost of losing Redis is throughput on one endpoint, plus the effects on the _other_ +Redis-backed subsystems listed below — not correctness or data loss here. Local development +without a `REDIS_URL` is a supported configuration, and the test suite runs with `redis` mocked +to `null` for exactly this reason. + +## The per-device scoping hazard + +`GET /conversations` builds its preview through +`getConversationRelations(req.auth!.deviceId)`, whose message relation filters envelopes to the +requesting device: + +```ts +envelopes: { + where: eq(messageEnvelopes.recipientDeviceId, deviceId), + limit: 1, +} +``` + +**The response is therefore device-specific, not merely user-specific.** Each recipient device +gets its own envelope, encrypted for that device alone (see +[`concepts-protocol-negotiation.md`](./concepts-protocol-negotiation.md) for why one message +produces one ciphertext per device). Two devices belonging to the same user get _different_ +ciphertext for the same preview message, and neither can decrypt the other's. + +That makes the interaction between the payload and the key format the sharpest edge in this +subsystem: + +- A key that includes the device (`conversations:{userId}:{deviceId}`) is safe: each device + gets its own entry, and each entry holds the ciphertext addressed to that device. +- A key that omits the device serves whichever device populated the entry first to every other + device of the same user for the rest of the TTL. The second device receives an envelope it + has no key for. The failure does not look like a cache bug: it looks like a decryption + failure, or a preview stuck on an older message, on one device only, intermittently, for up + to 30 seconds after every send — the shape of bug that gets attributed to the crypto layer + and chased for days. + +**Today `convCacheKey` is `conversations:{userId}` and does not carry the device.** The +consequence above is a live hazard for a multi-device user, bounded by the 30-second TTL, and +`convCacheKey` is the single place a fix belongs — every read, write, and invalidation already +routes through it. Note that a device-scoped key changes invalidation too: +`invalidateConversationCaches` takes user ids, so it would have to delete every device's entry +for each user (a `SCAN`/`DEL` over `conversations:{userId}:*`, or a per-user set of that user's +device keys), rather than one `DEL` per user. + +The general rule this is an instance of: **a cache key must name every input the cached value +depends on.** Here the value depends on the user _and_ the device _and_ the archived flag. The +archived flag is handled by not caching that view at all; the device is the dimension to watch. +The same rule applies to any future per-device response — a sync cursor, a device-filtered +history page — that someone is tempted to cache. + +## Other Redis key namespaces (not caches) + +For orientation, so these are not mistaken for cache entries and cleared as if they were. Only +the first is invalidated by anything in this document. + +| Prefix | Purpose | Reference | +| ---------------------- | ------------------------------------------ | ------------------------------------------------------------------------ | +| `conversations:` | The conversation-list cache described here | this document | +| `replay:` | `eventId` replay markers, TTL'd | [`concepts-replay-protection.md`](./concepts-replay-protection.md) | +| `rl:` | Rate-limit buckets | [rate limits](../../../docs/security/rate-limits.md) | +| `presence:` | Device and user presence, socket mappings | [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md) | +| Socket.IO adapter keys | Cross-node pub/sub fan-out | [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md) | + +Deleting a `conversations:` key is always safe. Deleting keys in the other namespaces is not +equally harmless — dropping `rl:` keys resets live rate-limit budgets, and dropping `presence:` +keys makes online users appear offline until their next heartbeat. + +## Implementation references + +| Concern | File | +| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | +| Client, key format, TTL constant | [`src/lib/redis.ts`](../src/lib/redis.ts) | +| Invalidation helper | [`src/lib/conversationCache.ts`](../src/lib/conversationCache.ts) | +| Read, write, and the device-scoped preview | [`src/routes/conversations.ts`](../src/routes/conversations.ts) | +| Message-write invalidations | [`src/routes/messages.ts`](../src/routes/messages.ts), [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| Device-change invalidation | [`src/routes/devices.ts`](../src/routes/devices.ts) | +| Tests | `src/__tests__/conversations.cache.test.ts` | + +## Related documents + +- [Conversations API](./api-conversations.md) — the endpoint whose response is cached. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — the other + subsystem that treats Redis as an optimisation and fails open. +- [Gateway architecture](./concepts-gateway-architecture.md) — presence and pub/sub, the rest + of what Redis carries. +- [Backend testing guide](./testing.md) — mocking `lib/redis.js`, including the `null` form + that exercises the degraded path. diff --git a/apps/backend/docs/concepts-configuration.md b/apps/backend/docs/concepts-configuration.md new file mode 100644 index 0000000..50a51b5 --- /dev/null +++ b/apps/backend/docs/concepts-configuration.md @@ -0,0 +1,119 @@ +# Backend configuration loading and validation + +How the gateway reads environment variables: which ones fail boot, which ones +are read later, and how that interacts with the object-store singleton. + +This page is about **when** a variable is consumed. The catalog of names, +defaults, and meanings lives in the environment-variable reference +([`.env.example`](../../../.env.example)). Rate-limit buckets and override +syntax are in [`docs/security/rate-limits.md`](../../../docs/security/rate-limits.md). +Do not treat the tables below as a second copy of those lists. + +## Boot-validated set (`src/config.ts`) + +`index.ts` calls `loadEnv()` immediately after `dotenv.config()`. `loadEnv()` +parses `process.env` against `EnvSchema` (Zod). On success it returns the +typed object and prints nothing. On failure it logs the offending names and +**exits the process with code 1** — the HTTP server is never bound. + +Required (missing or empty → hard startup failure): + +| Variable | Constraint | +| --- | --- | +| `DATABASE_URL` | non-empty string | +| `REDIS_URL` | non-empty string | +| `JWT_SECRET` | non-empty string | +| `PORT` | positive integer (string is coerced) | +| `TOKEN_TRANSFER_CONTRACT_ID` | non-empty string | +| `OBJECT_STORE_ENDPOINT` | non-empty string | +| `OBJECT_STORE_BUCKET` | non-empty string | +| `OBJECT_STORE_ACCESS_KEY` | non-empty string | +| `OBJECT_STORE_SECRET_KEY` | non-empty string | +| `OBJECT_STORE_REGION` | non-empty string | +| `OBJECT_STORE_FORCE_PATH_STYLE` | `true` / `false` / `1` / `0` | + +Optional in the same schema (validated only when present; absence does not +fail boot): `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`, `VAPID_SUBJECT`, the +legacy `S3_*` aliases, and `IDEMPOTENCY_TTL_SECONDS`. + +`assertTransportSecurityConfig()` runs next. That is a separate check +(`ENFORCE_TLS`, `ALLOWED_ORIGINS`) and is not part of `EnvSchema`. + +### Failure output + +If `DATABASE_URL` is missing, stderr looks like this and the process exits 1: + +```text +Missing or invalid environment variables: DATABASE_URL + - DATABASE_URL: DATABASE_URL is required +``` + +An empty environment lists every required name, one `- name: message` line +each. A non-numeric `PORT` is reported the same way (`PORT must be an integer`). + +## Lazy vs eager + +**Eager (boot).** `EnvSchema` variables above. The process does not start +without them. `index.ts` then immediately calls `getObjectStore()`, so the +object-store singleton is constructed on the boot path too — but the +construction itself is still lazy at the *module* level (see below). + +**Lazy (first use, no boot failure).** Everything else is read from +`process.env` at the call site, with a hardcoded default when unset or +malformed: + +- Rate-limit buckets in `src/config/rateLimits.ts` via `getRateLimitRule()`. + Each call re-reads `RATE_LIMIT_` (and the legacy + `SOCKET_RATE_LIMIT_PER_SEC` for `socket_default`). A malformed override is + warned and ignored; the default stands. `RATE_LIMIT_DISABLED=true` is a + kill switch, also read on demand. +- Backpressure thresholds (`SOCKET_BUFFER_THRESHOLD`, `SOCKET_SHED_THRESHOLD`) + — see [concepts-backpressure.md](./concepts-backpressure.md). +- Stellar RPC wiring (`STELLAR_RPC_URL`, `GROUP_TREASURY_CONTRACT_ID`) — + missing values disable the listener rather than crashing; see + [concepts-stellar-listener.md](./concepts-stellar-listener.md). +- Sync/envelope knobs such as `ENVELOPE_TTL_SECONDS` and `SYNC_PAGE_SIZE`. + +Rate-limit rules are deliberately not cached at import time so tests and the +runbook can change a limit without restarting the module graph. + +## `loadEnv()` and the object-store singleton + +`lib/objectStore.ts` keeps a process-wide singleton behind `getObjectStore()`. +The client is **not** built at import time. The first call constructs it: + +- `NODE_ENV === 'production'` → `createObjectStore(loadEnv())` (real S3 / + MinIO / R2 client, credentials from the boot-validated `OBJECT_STORE_*` + fields). +- otherwise → `getLocalObjectStore()` (fs-backed store; no live S3 needed). + +Building the singleton in `objectStore.ts` (instead of exporting a client +from `index.ts`) avoids a circular import: + +```text +index.ts → app.ts → routes → lib/storage.ts → index.ts +``` + +`storage.ts` and `services/fileCleanup.ts` both call `getObjectStore()`, so +there is exactly one client/bucket pair per process. `loadEnv()` is invoked +again inside that first production call; by then boot validation has already +run, so it is a typed re-parse, not a second chance to start with a broken +env. + +`resetObjectStoreForTests()` clears the memo so tests can change env. + +## Production vs development in `lib/storage.ts` + +`generatePresignedPut` / `generatePresignedGet` branch on +`NODE_ENV === 'production'`: + +| Branch | Implementation | +| --- | --- | +| production | `getObjectStore()` — the S3-compatible client from `OBJECT_STORE_*` | +| any other `NODE_ENV` | `getLocalObjectStore()` — local-disk store, URLs served by `routes/localStorage.ts` | + +The same `NODE_ENV` split exists inside `getObjectStore()` itself. The +storage helpers short-circuit to the local store in non-production so +presigned-URL callers never touch the S3 constructor on a laptop or in CI. + +Cross-referenced from [`IMPLEMENTATION_DOCS.md`](../../../IMPLEMENTATION_DOCS.md). diff --git a/apps/backend/docs/concepts-crypto-invariants.md b/apps/backend/docs/concepts-crypto-invariants.md new file mode 100644 index 0000000..ae07e22 --- /dev/null +++ b/apps/backend/docs/concepts-crypto-invariants.md @@ -0,0 +1,140 @@ +# Server-side crypto invariants + +This note documents the guards that keep the backend from ever accepting or storing material it must not hold: [src/lib/ciphertextInvariant.ts](../src/lib/ciphertextInvariant.ts), [src/lib/signalInvariants.ts](../src/lib/signalInvariants.ts), the `.strict()` Zod schemas that enforce the same rule on the REST surface, and the [Security CI](../../../.github/workflows/security-ci.yml) job that regression-tests all of it. + +## 1. The invariants + +The backend is a relay and a durable queue for material it cannot read. Two rules follow from that, and both are enforced in code rather than left as a convention: + +1. **The server stores ciphertext only.** Message bodies are persisted as opaque ciphertext plus per-device envelopes. There is no column, schema field, or code path that accepts a plaintext message body. +2. **The server never accepts session state, ratchet state, or private keys on any inbound payload.** Double-Ratchet session state, root/chain/sender keys, and private key halves are client-local. Every client derives its own session state; the only key material that crosses the wire is _public_ — identity keys, prekeys, signed prekeys, and MLS key packages. + +The second rule is deliberately stronger than "we don't read it". Accepting a private key and ignoring it would still put it in a request log, an error report, or a crash dump. The guards reject the request outright so the value never reaches anything that persists. + +Related reading: [E2EE onboarding](e2ee-onboarding.md), [Signal migration](signal-migration.md), [Security hardening](security-hardening.md), and the repository [threat model](../../../docs/threat-model.md). + +## 2. `lib/signalInvariants.ts` + +### `FORBIDDEN_SESSION_STATE_FIELDS` + +The rejected field names are: + +| Field | Why it is forbidden | +| ---------------------- | -------------------------------------------------------- | +| `sessionState` | Serialized Signal session — client-local, never uploaded | +| `ratchetState` | Double-Ratchet state, including message-key chains | +| `rootKey` | Ratchet root key | +| `chainKey` | Sending or receiving chain key | +| `senderKey` | Group sender key | +| `privateKey` | Any private key half | +| `identityPrivateKey` | Long-term identity private key | +| `signedPreKeyPrivate` | Private half of a signed prekey | +| `oneTimePreKeyPrivate` | Private half of a one-time prekey | + +The list is exported as a `const` tuple and `ForbiddenSessionStateField` is its union type, so adding a name to the list is the only change needed to extend the guard. + +### `findForbiddenSessionStateField(payload)` + +Returns the name of the first forbidden field found, or `null` when the payload is clean. It checks two levels: + +- the top level of `payload`, and +- every entry of `payload.envelopes` when that is an array, because the per-device envelope array is the other place a client could attach key material. + +Two details matter: + +- It uses `Object.prototype.hasOwnProperty` rather than `in` or a truthiness check. Only own-enumerable keys are user-controlled input, and a field that is present but set to `null` or `""` is still a rejection — the presence of the key is the signal, not its value. +- It returns the field _name_, which the caller echoes back in the error so a client bug stays diagnosable without the server ever logging the value. + +### Where it runs, and when + +The guard is the enforcement point for the WebSocket paths, which parse the raw socket payload by hand rather than through Zod. It runs as the **first statement** of both handlers in [src/socket/messaging.ts](../src/socket/messaging.ts): + +- `send_message` — [messaging.ts:117](../src/socket/messaging.ts#L117) +- `edit_message` — [messaging.ts:321](../src/socket/messaging.ts#L321) + +```ts +dispatcher.register('send_message', async (payload) => { + const forbiddenField = findForbiddenSessionStateField(payload); + if (forbiddenField) { + socket.emit('error', { + event: 'send_message', + code: 400, + message: `Field "${forbiddenField}" is not permitted: the server never stores session or private-key state`, + }); + return; + } + // ... destructure payload, check membership, write to the database +}); +``` + +**Rejection happens before any database lookup.** The handler returns before it destructures the payload, before the conversation-membership check, and before any `db.query` call. This ordering is intentional and is asserted by [signalInvariants.socket.test.ts](../src/__tests__/signalInvariants.socket.test.ts). It matters for three reasons: + +- A forbidden payload never touches the database, so a rejected request cannot be used as an oracle for whether a conversation or a message id exists. +- Rejection cost is constant and independent of database state. +- Nothing partially validated is written, so there is no window in which a forbidden field is held inside a transaction. + +The client sees a `400` socket `error` event naming the offending field. There is no partial-accept mode: the message is not stored, not fanned out, and not acknowledged. + +## 3. `lib/ciphertextInvariant.ts` + +A second, broader list covering _stored or uploaded_ payloads — `FORBIDDEN_PERSISTED_OR_UPLOADED_FIELDS`. It extends the session-state list with plaintext body names (`content`, `body`, `plaintext`), MLS secrets (`mlsSecret`, `mlsSecrets`), and message keys (`messageKey`, `messageKeys`), and it carries both camelCase and snake_case spellings. + +Field names are compared after normalization — `field.replace(/[-_]/g, '').toLowerCase()` — so `ratchet_state`, `ratchetState`, `Ratchet-State`, and `RATCHETSTATE` all match the same entry. The explicit snake_case entries in the exported list keep the list itself readable; normalization is what makes the check spelling-insensitive. + +Two functions: + +- `findForbiddenCiphertextFields(payload): string[]` — returns every offending key on a plain object. Non-objects, `null`, and arrays return `[]`. +- `assertCiphertextOnlyPayload(payload): void` — throws when that list is non-empty, with the offending field names (not values) in the message. + +Covered by [ciphertextInvariant.test.ts](../src/__tests__/ciphertextInvariant.test.ts). + +## 4. Why the Zod schemas are `.strict()` + +REST endpoints do not call the guard functions directly. They get the same protection from Zod, but **only because the schemas are `.strict()`** — and this is the single most important detail in this document. + +Zod's default object mode **strips** unknown keys. A non-strict schema handed a payload carrying `ratchetState` would parse successfully, silently drop the field, and return a clean object. The request would be accepted, the client would receive `200`, and nobody would learn that a client had just tried to upload ratchet state. A regression that reintroduced a plaintext or key-bearing field would be invisible, because the schema would quietly absorb it forever. + +`.strict()` inverts that: an unrecognized key is a validation failure, and the request is rejected with `400` before the handler runs. Silent stripping hides a client bug; strict rejection surfaces it. + +The strict schemas that gate crypto-relevant input: + +| Schema | File | Gates | +| ------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | +| `EnvelopeSchema` | [schemas/message.schemas.ts](../src/schemas/message.schemas.ts) | Each per-device message envelope | +| `SendMessageSchema` | [schemas/message.schemas.ts](../src/schemas/message.schemas.ts) | `POST /messages` body | +| `VerifySchema` | [schemas/auth.schemas.ts](../src/schemas/auth.schemas.ts) | `POST /auth/verify` sign-in body | +| `DeviceLinkVerifySchema` | [schemas/auth.schemas.ts](../src/schemas/auth.schemas.ts) | `POST /devices/link/verify` — the only device-registration path | +| `PreKeyEntrySchema` | [lib/keys.ts](../src/lib/keys.ts) | Uploaded one-time prekeys | +| `SignedPreKeyEntrySchema` | [lib/keys.ts](../src/lib/keys.ts) | Uploaded signed prekeys | + +Note that `SendMessageSchema` has no plaintext field at all: the body carries `ciphertext` and `envelopes`, never a readable message. Content-type rules beyond the schema shape live in [lib/validateMessagePayload.ts](../src/lib/validateMessagePayload.ts), which rejects a text message that has no per-device envelopes. + +**When adding a schema that accepts client input touching keys, messages, or devices, make it `.strict()`.** Building a composite with `.extend()` produces a new schema whose strictness must be re-declared, which is why `SignedPreKeyEntrySchema` and `DeviceLinkVerifySchema` both re-apply `.strict()` explicitly. + +## 5. The Security CI regression job + +[.github/workflows/security-ci.yml](../../../.github/workflows/security-ci.yml) runs on every pull request and on every push to `main`. It has two jobs. + +### Job `regression` — "Ciphertext-only guard + secret-field scan" + +Runs [src/\_\_tests\_\_/security.regression.test.ts](../src/__tests__/security.regression.test.ts) from `apps/backend` with `pnpm test -- security.regression.test.ts`. Two suites: + +**Ciphertext-only guard.** Drives `validateMessagePayload` and asserts that a text message carrying a `plaintext` field is rejected, that ciphertext without per-device envelopes is rejected, that a message with envelopes is accepted, and that `SendMessageSchema.shape` contains neither `plaintext` nor `plainText`. + +**Secret-field source scan.** Walks every `.ts` file under `apps/backend/src` (skipping `__tests__` and `node_modules`) and fails if any file _declares_ a field named `plaintext`, `plainText`, `privateKey`, `private_key`, `sessionState`, `session_state`, `signalSession`, `identityPrivateKey`, or `preKeyPrivate`. The match pattern is `(^|[^A-Za-z0-9_])\s*[:?]\s*[^,]`, which catches Zod object keys and TypeScript interface or type members while leaving prose in comments alone. + +This scan is what makes the invariant durable. The unit tests prove the guards work on the paths that call them; the scan fails the build the moment _any_ schema, route, type, or service anywhere in the backend grows a field that could carry a private key or a session blob — including on a path nobody remembered to guard. + +Two supporting suites also live in the tree and run under the normal backend test job: [signalInvariants.messages.test.ts](../src/__tests__/signalInvariants.messages.test.ts) and [signalInvariants.devices.test.ts](../src/__tests__/signalInvariants.devices.test.ts) cover the REST message and device paths, and [signalInvariants.socket.test.ts](../src/__tests__/signalInvariants.socket.test.ts) covers the WebSocket handlers described above. + +### Job `dependency-audit` — "Crypto dependency CVE audit" + +Runs [scripts/audit-crypto-deps.mjs](../../../scripts/audit-crypto-deps.mjs), which scopes `pnpm audit` to the crypto-relevant backend dependencies (`ioredis`, `jsonwebtoken`, `web-push`, `@stellar/stellar-sdk`, `drizzle-orm`, `socket.io`) so CVEs in those surface without the job failing on advisories in unrelated transitive packages. + +### Changing the invariants + +If a change legitimately requires touching this surface: + +1. Update `FORBIDDEN_SESSION_STATE_FIELDS` or `FORBIDDEN_PERSISTED_OR_UPLOADED_FIELDS` rather than adding an ad-hoc check at a call site. +2. Keep new client-input schemas `.strict()`. +3. Expect the source scan to fail loudly on a new forbidden field name. That failure is the feature working — treat it as a design question, not as a test to relax. diff --git a/apps/backend/docs/concepts-e2ee-key-management.md b/apps/backend/docs/concepts-e2ee-key-management.md new file mode 100644 index 0000000..9af475d --- /dev/null +++ b/apps/backend/docs/concepts-e2ee-key-management.md @@ -0,0 +1,309 @@ +# E2EE device identity & key lifecycle (backend) + +This document explains the backend's role in the end-to-end-encryption key +lifecycle: how a device gets an identity, how it publishes key material, +how other clients fetch and consume that material, and how a device is +retired. It covers the full lifecycle as one coherent story and names the +exact routes and tables involved at each stage. + +**Scope.** This is a backend concepts document. It does not describe any +client-side cryptography (X3DH session establishment, Double Ratchet, MLS +epoch secrets) — only what the server stores, serves, validates, and +deletes along the way. See +[`../../apps/web/docs/concepts-e2ee-architecture.md`](../../apps/web/docs/concepts-e2ee-architecture.md) +for the client-side key model. + +## What the server can and cannot see + +Stated explicitly, because every section below depends on it: + +- **The server never sees a private key, of any kind.** Not an identity + private key, not a signed-prekey private key, not a one-time-prekey + private key, not an MLS HPKE init private key, not a session/ratchet key. + Every key-shaped column in `devices` and `device_prekeys` (and + `mls_key_packages`) stores **public** material only — + `identityPublicKey`, `publicKey` (signed and one-time), `keyPackage`. This + is a property of what the schema is defined to hold, not an access-control + claim about what a query *could* return if misused. +- **The server verifies signatures without needing the signer's private + key** — Ed25519 signature verification is a public-key operation. The + signed-prekey upload route (`POST /devices/:id/prekeys`) verifies that + `signedPreKey.signature` is a valid signature over `signedPreKey.publicKey` + made by the device's own `identityPublicKey`, entirely with public data. + This proves the signed prekey really was published by the device that owns + the identity key, without the server ever holding a signing key of its own + for that device. +- **The server cannot decrypt message ciphertext.** `messages.ciphertext` + and `message_envelopes.ciphertext` are opaque blobs to every backend code + path — nothing in `routes/` or `services/` ever attempts to decrypt them. + (See [`contracts-database-schema.md`](contracts-database-schema.md#messages) + for the column, and + [`../../apps/web/docs/concepts-e2ee-architecture.md`](../../apps/web/docs/concepts-e2ee-architecture.md) + for where decryption actually happens — exclusively on-device.) +- **What the server *can* see and act on**: which device belongs to which + user, when a device was last active, how many one-time prekeys a device + has left, which device fetched which other device's bundle and when + (`key_bundle_drained` audit events — see + [Audit trail](#audit-trail-what-gets-logged-and-why) below), and the + device capability set a device advertises (which protocols/ciphersuites it + supports — see + [`concepts-protocol-negotiation.md`](concepts-protocol-negotiation.md)). + All of this is metadata about key *distribution*, never about message + *content*. + +## The lifecycle, stage by stage + +### 1. Device identity & registration + +A device's identity is its Ed25519 `identityPublicKey`, stored once per +device in `devices.identity_public_key`. There are **two independent code +paths** that can create or resolve a `devices` row for an existing account, +and they behave differently on a revoked identity key — this distinction +matters and is easy to miss: + +| | `POST /auth/challenge` → `POST /auth/verify` (`routes/auth.ts`) | `POST /devices/link/challenge` → `POST /devices/link/verify` (`routes/devices.ts`) | +|---|---|---| +| Purpose | Sign in (first device, or any later device, via wallet proof) | Explicitly link an *additional* device to an already-authenticated account | +| Caller authentication | None required beforehand — this *is* the login step | Requires a valid JWT already (the caller must already be signed in on some device) | +| Proof required | Wallet signature over the sign-in challenge message | A **second, distinct** wallet signature over a device-link-specific message (`"Link device to Clicked\nUser: ...\nNonce: ..."`) — deliberately different wording from the login message so one signature can never be replayed as the other (see [`concepts-nonce-lifecycle.md`](concepts-nonce-lifecycle.md)) | +| Lookup key | `(userId, identityPublicKey)` | `(userId, identityPublicKey)` | +| New identity key, no existing row | Inserts a new `devices` row | Inserts a new `devices` row | +| Existing row, **not** revoked | Reuses it, updates `lastSeenAt`/name/platform/`registrationId`/capabilities | `409 Device already registered for this user` | +| Existing row, **revoked** | **Hard refusal**: `401 Device has been revoked` — sign-in is blocked outright, no reactivation | **Reactivates** the row in place (`revokedAt` cleared, name/platform/`registrationId` updated) — the comment in the source calls this out as a deliberately distinct audit event from a fresh link | +| Audit action recorded | `auth_failed` with `reason: device_revoked` (failure case only; success records nothing here) | `device_linked`, with `metadata.reactivatedRevokedDevice: true/false` | + +`POST /devices` itself (the bare, JWT-only registration endpoint) is +**retired** — it now unconditionally returns `403` directing the caller to +the link-challenge flow (tracked in-source as issue #333, closing a gap +where a stolen JWT alone was sufficient to attach an attacker-controlled +device). If you are reading +[`api-devices.md`](api-devices.md) and its `POST /devices` section describes +reactivation-on-registration as a single unauthenticated-by-wallet step, +that section predates #333 and no longer matches `routes/devices.ts` — this +document reflects the current two-step link flow; `api-devices.md` should be +read for the routes it still accurately describes (listing, revocation, +prekey upload). + +Either path writes the same columns: `identityPublicKey` (immutable per +row — there is no "update my identity key in place" operation; a client that +rotates its long-term identity key produces a **new** `devices` row, not a +mutated existing one — see [Key rotation](#7-key-rotation-and-the-unwritten-devicekeyhistory-table) +below), plus `deviceName`, `platform`, `registrationId` (the X3DH +registration id from the prekey bundle spec), and `capabilities` (the +protocol/ciphersuite/file-transfer versions the device advertises, defaulted +via `normalizeCapabilities` if omitted — see +[`concepts-protocol-negotiation.md`](concepts-protocol-negotiation.md)). + +### 2. Prekey publication + +`POST /devices/:id/prekeys` (ownership- and revocation-checked — see +[`api-devices.md`](api-devices.md#post-devicesidprekeys) for the full +request/response contract) writes into `device_prekeys`, discriminated by +`keyType`: + +- **Signed prekey** (`key_type = 'signed'`) — exactly one live row per + device, **upserted** on every upload (`ON CONFLICT` on the partial unique + index over `deviceId` where `keyType = 'signed'`). The server verifies + `signature` is a valid Ed25519 signature over `publicKey`, signed by the + device's `identityPublicKey`, **before** writing anything — an invalid + signature means nothing is persisted. This is the one point in the whole + lifecycle where the server actively validates a cryptographic property of + uploaded key material, rather than just storing it. +- **One-time prekeys** (`key_type = 'one_time'`) — many rows, added to the + existing pool (`ON CONFLICT ... DO NOTHING` on `(deviceId, keyType, + keyId)`, so a retried upload of the same `keyId` is a silent no-op, not an + error). Capped at **200 unconsumed** per device (`OTP_CAP`); a batch that + would exceed the cap is trimmed, not rejected outright, as long as some + headroom remains — see [`api-devices.md`](api-devices.md#the-200-key-cap-and-trimming-behavior) + for the exact trimming arithmetic. + +Nothing here is decrypted or re-derived — the server stores the public +key/signature bytes exactly as submitted and never generates key material on +a device's behalf. + +### 3. Key-bundle fetch & one-time-prekey consumption + +`GET /users/:userId/devices/:deviceId/key-bundle` (`routes/users.ts`) is how +a sender obtains a recipient device's X3DH bundle to start a session. Three +things happen atomically inside one DB transaction: + +1. The device's current **signed prekey** is read (required — `409` if the + device has never uploaded one). +2. **At most one** unconsumed one-time prekey row is selected + (`FOR UPDATE SKIP LOCKED`, oldest first) and flipped to `consumed = true` + — flipped, never deleted, so the fetch history stays auditable. Two + concurrent fetches for the same device can never be handed the same + one-time prekey, because the row lock excludes it from the other + transaction's candidate set. +3. The post-consumption remaining count is computed **inside the same + transaction**, so it can never race a concurrent fetch into reporting a + stale number. + +If no one-time prekey remains, the bundle is still returned — with +`oneTimePreKey: null` — rather than erroring, so the sender can fall back to +a 3-DH session using only the signed prekey. See +[`e2ee-onboarding.md`](e2ee-onboarding.md#c-low-prekey-warning-before-exhaustion) +for the exact JSON shapes of both the normal and exhausted responses, and +for the `prekeys_low` socket event that fires (debounced, once per threshold +crossing) to warn the *owning* device to replenish before it actually runs +dry. + +Every successful one-time-prekey consumption also increments a metric +(`prekeyConsumedTotal`) and records a `key_bundle_drained` audit event (see +[Audit trail](#audit-trail-what-gets-logged-and-why)) — the subject is the +device **owner** (whose prekey supply just shrank), the actor is whoever +fetched the bundle. + +### 4. Replenishment signal and background pruning + +Two independent mechanisms keep the one-time-prekey pool healthy without any +cross-service coordination: + +- **Low-supply push**: a `prekeys_low` event to the owning device's socket + room (`device:{deviceId}`) once a fetch drops the remaining count below a + threshold (default `20`, `PREKEY_LOW_THRESHOLD`). Re-arms once the device + tops back up at or above the threshold, so a later crossing warns again. +- **Background GC** (`services/deviceGc.ts`, hourly by default): prunes + `device_prekeys` rows where `key_type = 'one_time'` and either `consumed = + true` and older than `PREKEY_CONSUMED_RETENTION_DAYS` (default 30d), or + unconsumed and older than `PREKEY_UNCONSUMED_MAX_AGE_DAYS` (default 90d). + Signed prekeys are never touched by this pass — there is exactly one live + signed prekey per device, replaced in place on upload, not garbage + collected. See [`concepts-gc-jobs.md`](concepts-gc-jobs.md) for the full + job reference, including the MLS KeyPackage pass that mirrors this policy. + +### 5. Revocation + +A device is revoked via `DELETE /devices/:id` (single device, caller-owned +only, idempotent, refuses to revoke the caller's last active device) or +`POST /devices/logout-everywhere` (every other active device on the +account). Both route through one shared helper with **identical** per-device +side effects: + +1. `devices.revokedAt` is set (row kept, not deleted — revocation history is + permanent and queryable). +2. Every row in `device_prekeys` for that device — signed **and** one-time, + consumed or not — is **deleted outright**. A revoked device has no + prekeys left to hand out; re-registering the same identity key later (via + the link-verify reactivation path above) starts prekey-less and must + re-upload. +3. Live sockets for that device are force-disconnected, same-node + immediately and cross-node via a `device_revoked:{deviceId}` Redis + pub/sub publish (best-effort; a publish failure does not fail the HTTP + request). +4. A `device_revoked` system event is emitted into every conversation the + user is a member of, so other clients learn "this user's device set + changed" and know to refresh key bundles before encrypting to them again. + +Full request/response detail and the exact ownership/idempotency rules are +in [`api-devices.md`](api-devices.md#revocation-side-effects) — that section +of the doc is still accurate (only the *registration* section is affected by +the #333 change above). + +### 6. Post-revocation staleness flagging + +`services/deviceGc.ts`'s third pass flags (never deletes) devices that have +been revoked longer than `DEVICE_STALE_AFTER_DAYS` (default 180d), by +setting `devices.staleFlaggedAt`. This is purely informational — it marks a +row eligible for whatever downstream archival process an operator chooses +to run; revocation audit history is preserved indefinitely regardless. + +### 7. Key rotation and the unwritten `deviceKeyHistory` table + +`device_key_history` exists in the schema specifically to let clients detect +a silent identity-key swap and show a safety-number-style warning: it +records `(deviceId, userId, previousKey, newKey, changeReason, recordedAt)`, +and `GET /users/:id/key-history` (`routes/users.ts`) reads it back, ordered +oldest-first. + +**Verified against the current source: nothing writes to this table.** +There is no `db.insert(deviceKeyHistory, ...)` call anywhere in +`apps/backend/src`, and no test exercises one. The schema comment's claim +that it is "written whenever a device's `identityPublicKey` changes" does +not correspond to any implemented write path today — because, per +[stage 1](#1-device-identity--registration), **there is also no operation +that changes an existing device row's `identityPublicKey` in place.** A +client that rotates its long-term identity key does so by registering what +the server sees as a brand-new, unrelated `devices` row (a new `(userId, +identityPublicKey)` pair) — the old row is left exactly as it was, not +updated and not linked to the new one. `GET /users/:id/key-history` is +therefore a read endpoint over a table that the current implementation never +populates; treat it as reading back an empty history today, not as a working +key-transparency feed. Closing this gap would mean either (a) adding an +explicit identity-key-rotation endpoint that updates a row in place and logs +the transition, or (b) deriving history from the existing +add-new-row/revoke-old-row pattern after the fact — neither exists yet. + +### MLS key packages — a parallel, separate mechanism + +Group conversations run MLS (RFC 9420) instead of per-recipient X3DH +sessions, and MLS has its own public-key-material table, +`mls_key_packages`, populated by `POST /devices/:id/mls-key-packages` under +the same "public material only, idempotent upload, capped pool" shape as +`device_prekeys` — but it is a genuinely separate mechanism with its own +single-use-per-Add semantics, not an alternate code path over the same +table. This document's scope is the `devices`/`device_prekeys` pair +specifically; see [`mls-key-packages.md`](mls-key-packages.md) and +[`mls-group-membership.md`](mls-group-membership.md) for the MLS-side +lifecycle. + +## `devices` / `device_prekeys` at a glance + +Full column-by-column documentation (types, defaults, indexes, FKs) is in +[`contracts-database-schema.md`](contracts-database-schema.md#devices) — this +is just the subset load-bearing for the lifecycle above: + +| Column | Table | Role in the lifecycle | +|---|---|---| +| `identityPublicKey` | `devices` | The device's long-term public identity; immutable per row (see [stage 7](#7-key-rotation-and-the-unwritten-devicekeyhistory-table)); lookup key for both registration paths. | +| `registrationId` | `devices` | X3DH registration id published alongside the bundle. | +| `capabilities` | `devices` | Advertised protocol/ciphersuite support; defaults applied for pre-existing rows. | +| `revokedAt` | `devices` | `NULL` while active; set (never cleared except by link-verify reactivation) on revocation. | +| `staleFlaggedAt` | `devices` | Set by GC once revoked past the retention window; informational only. | +| `keyType` | `device_prekeys` | `'signed'` (one live row, upserted) or `'one_time'` (many rows, pooled and capped). | +| `signature` | `device_prekeys` | Required (DB check constraint) and server-verified only for `keyType = 'signed'`; never set for one-time keys. | +| `consumed` | `device_prekeys` | Flips `true` on bundle-fetch claim; one-time keys only; never un-set. | + +## Audit trail: what gets logged, and why + +Every stage above that touches a device's security posture writes to +`audit_logs` (see +[`../../../docs/security/audit-logging.md`](../../../docs/security/audit-logging.md) +for the full catalog and what is deliberately excluded from it): + +| Action | Fired by | Subject vs. actor | +|---|---|---| +| `device_linked` | `POST /devices/link/verify` | Actor = the authenticated caller; target = the new/reactivated device | +| `device_revoked` | `DELETE /devices/:id`, `logout-everywhere` (once per call, not per device) | Subject = device owner | +| `logout_everywhere` | `POST /devices/logout-everywhere` | Subject = the caller (revoking their own other devices) | +| `auth_failed` (`reason: device_revoked`) | `POST /auth/verify` rejecting a revoked device | Subject = the account being signed into | +| `key_bundle_drained` | `GET /users/:userId/devices/:deviceId/key-bundle` consuming the last (or any) one-time prekey | Subject = bundle/device **owner**; actor = the fetching caller | + +These rows never contain key material or message content — only +identifiers, counts, and outcomes, per the append-only audit log's design +constraint. + +## Related documents + +- [`api-devices.md`](api-devices.md) — full request/response reference for + listing, revocation, and prekey upload (registration section is + superseded by [stage 1](#1-device-identity--registration) above). +- [`e2ee-onboarding.md`](e2ee-onboarding.md) — the exact JSON shapes and + sequencing for first-time onboarding and first-DM bundle fetch, including + the offline-recipient and prekey-exhausted paths. +- [`contracts-database-schema.md`](contracts-database-schema.md) — full + column/index/FK reference for `devices`, `device_prekeys`, and every other + table. +- [`concepts-protocol-negotiation.md`](concepts-protocol-negotiation.md) — + how `capabilities` drives sender/recipient protocol agreement. +- [`concepts-nonce-lifecycle.md`](concepts-nonce-lifecycle.md) — the + sign-in and device-link nonce stores referenced in stage 1. +- [`concepts-gc-jobs.md`](concepts-gc-jobs.md) — the full background-job + reference, including the prekey/MLS-KeyPackage pruning and device + stale-flagging passes. +- [`mls-key-packages.md`](mls-key-packages.md) — the parallel MLS key + material lifecycle. +- [`../../../docs/security/audit-logging.md`](../../../docs/security/audit-logging.md) — + the audit log format and full action catalog. +- [`../../apps/web/docs/concepts-e2ee-architecture.md`](../../apps/web/docs/concepts-e2ee-architecture.md) — + the client-side key model this document deliberately does not cover. diff --git a/apps/backend/docs/concepts-gc-jobs.md b/apps/backend/docs/concepts-gc-jobs.md new file mode 100644 index 0000000..75237b1 --- /dev/null +++ b/apps/backend/docs/concepts-gc-jobs.md @@ -0,0 +1,249 @@ +# Background GC Jobs Reference + +This document is the operational reference for every scheduled cleanup pass the backend +runs: what each one deletes, how often, how long it waits before deleting it, and how to +retune that via environment variables. For the broader file-lifecycle and push-hygiene +picture these jobs sit inside, see +[`concepts-storage-push-jobs.md`](./concepts-storage-push-jobs.md). + +## Contents + +- [Summary table](#summary-table) +- [Device / key GC — `services/deviceGc.ts`](#device--key-gc--servicesdevicegcts) +- [Envelope GC — `services/envelopeGc.ts`](#envelope-gc--servicesenvelopegcts) +- [File cleanup — `services/fileCleanup.ts`](#file-cleanup--servicesfilecleanupts) +- [Push-subscription backoff re-enable](#push-subscription-backoff-re-enable) +- [Idempotency and retry-safety, in general](#idempotency-and-retry-safety-in-general) +- [Multi-node deployment implications](#multi-node-deployment-implications) + +## Summary table + +| Job | Interval (default) | Interval env var | What it removes | Retention window (default) | Retention env var(s) | +|---|---|---|---|---|---| +| Prekey GC | 1 hour | `DEVICE_GC_INTERVAL_MS` | Consumed or unclaimed one-time prekeys (`device_prekeys`, `key_type = 'one_time'`) | 30d consumed / 90d unconsumed | `PREKEY_CONSUMED_RETENTION_DAYS`, `PREKEY_UNCONSUMED_MAX_AGE_DAYS` | +| MLS key package GC | 1 hour | `DEVICE_GC_INTERVAL_MS` | Consumed or unclaimed MLS `KeyPackage`s (`mls_key_packages`) | 30d consumed / 90d unconsumed | `PREKEY_CONSUMED_RETENTION_DAYS`, `PREKEY_UNCONSUMED_MAX_AGE_DAYS` | +| Device stale-flag | 1 hour | `DEVICE_GC_INTERVAL_MS` | Nothing — flags (`stale_flagged_at`) devices revoked past the window | 180d since revocation | `DEVICE_STALE_AFTER_DAYS` | +| Envelope GC | 30 minutes | `ENVELOPE_GC_INTERVAL_MS` | `message_envelopes` rows (per-recipient-device delivery units) | 7d after delivery / 30d max age regardless | `ENVELOPE_DELIVERED_RETENTION_DAYS`, `ENVELOPE_MAX_AGE_DAYS` | +| File hard-delete | 5 minutes | `FILE_GC_INTERVAL_MS` | S3 objects (+ `files` row marked `hard_deleted_at`) for soft-deleted, fully-unreferenced files | 0ms grace by default, then immediate | `FILE_HARD_DELETE_GRACE_MS` | +| Pending-upload GC | 5 minutes (same tick as file hard-delete) | `FILE_GC_INTERVAL_MS` | S3 objects + `files` rows for uploads never confirmed | 24 hours since creation | `PENDING_UPLOAD_TTL_MS` | +| Push-subscription backoff re-enable | 5 minutes (same tick as file hard-delete) | `FILE_GC_INTERVAL_MS` | Nothing — clears `disabled_at` on `push_subscriptions` whose backoff has expired | n/a — driven by the 5-minute backoff set at send time, not a configurable window | none | + +All six passes are started once at process boot, in [`index.ts`](../src/index.ts), and +run on a plain `setInterval` for the lifetime of the process (`.unref()`'d so they never +keep the process alive on their own). + +--- + +## Device / key GC — `services/deviceGc.ts` + +Three independent passes share one hourly timer (`startDeviceGcJob`): + +### 1. One-time prekey pruning (`runPrekeyGcPass`) + +- **Removes**: rows in `device_prekeys` where `key_type = 'one_time'` and either + - `consumed = true` and `created_at` is older than `PREKEY_CONSUMED_RETENTION_DAYS` + (default **30 days**) — the recipient already claimed it, kept only for audit, or + - `consumed = false` and `created_at` is older than `PREKEY_UNCONSUMED_MAX_AGE_DAYS` + (default **90 days**) — nobody claimed it in time. +- Signed prekeys are never touched by this pass — a device has exactly one live signed + prekey and it's replaced in place on upload, not garbage-collected. + +### 2. MLS KeyPackage pruning (`runMlsKeyPackageGcPass`) + +- Same two-tier policy (consumed vs. unconsumed, same two env vars) applied to + `mls_key_packages` instead of `device_prekeys`. Kept as a separate pass because the two + tables are unrelated, but they're tuned by the same knobs since they represent the same + "unclaimed one-time key material" concept for two different protocols. + +### 3. Stale-device flagging (`runDeviceStaleFlagPass`) + +- **Flags, never deletes.** Sets `devices.stale_flagged_at` on rows where + `revoked_at IS NOT NULL`, `revoked_at` is older than `DEVICE_STALE_AFTER_DAYS` + (default **180 days**), and `stale_flagged_at IS NULL`. +- Revocation history is preserved indefinitely — this pass only marks a device eligible + for whatever downstream archival/audit process consumes the flag; it does not delete + the `devices` row or any of its history. + +**Env vars**: `DEVICE_GC_INTERVAL_MS` (tick interval, default 1h), +`PREKEY_CONSUMED_RETENTION_DAYS` (default 30), `PREKEY_UNCONSUMED_MAX_AGE_DAYS` +(default 90), `DEVICE_STALE_AFTER_DAYS` (default 180). All four are read via a shared +`envDays()`/interval helper that falls back to the default on a missing, non-numeric, or +non-positive value — a malformed env var degrades to "use the default", not a crash. + +--- + +## Envelope GC — `services/envelopeGc.ts` + +`message_envelopes` holds one row per **(message, recipient device)** — the actual +delivery unit. It is the only thing that keeps a delivered message's audit trail (the +delivered/read timestamps) around after the fact, and it grows with every message times +every recipient device, so it needs the tightest retention of any table in the system. + +`runEnvelopeGcPass` deletes a row when either is true: + +- **Delivered and aged out**: `delivered_at IS NOT NULL` and `delivered_at` is older + than `ENVELOPE_DELIVERED_RETENTION_DAYS` (default **7 days**) — the common case, the + recipient device picked it up. +- **Past the max-age ceiling regardless of delivery state**: `created_at` is older than + `ENVELOPE_MAX_AGE_DAYS` (default **30 days**) — a device that never comes back to + collect its envelope does not get to pin storage forever. + +**Schedule**: every 30 minutes by default (`ENVELOPE_GC_INTERVAL_MS`). + +**Env vars**: `ENVELOPE_GC_INTERVAL_MS`, `ENVELOPE_DELIVERED_RETENTION_DAYS`, +`ENVELOPE_MAX_AGE_DAYS`. + +--- + +## File cleanup — `services/fileCleanup.ts` + +Implements the soft-delete → hard-delete lifecycle from issue #231. `softDeleteFile()` +sets `files.deleted_at` synchronously when a message referencing it is retracted (it is +not part of the scheduled job — it's called inline from the message-delete path). The +scheduled job (`startFileCleanupJob`, tick interval `FILE_GC_INTERVAL_MS`, default +**5 minutes**) then does three things per tick, in order: + +### 1. Hard-delete pass (`runHardDeletePass`, first half) + +Candidates: `files` rows where `deleted_at IS NOT NULL`, `hard_deleted_at IS NULL`, and +`deleted_at` is older than `FILE_HARD_DELETE_GRACE_MS` (default **0** — no grace period, +eligible as soon as soft-deleted). + +For each candidate the job: + +1. **Re-checks the reference count** (see below) — if any live message still points at + the file, it's skipped this tick and picked up again on a later one once that last + reference clears. +2. Deletes the S3 object via `getObjectStore().deleteObject(storageKey)`. +3. Only after that delete succeeds, sets `hard_deleted_at = now()`. + +A failure at step 2 or 3 is caught, logged, and the file is simply left in its current +(soft-deleted, not-yet-hard-deleted) state for the next tick to retry — no exception +propagates out of the loop, so one bad object doesn't stop the rest of the batch. + +### 2. Pending-upload GC (`runHardDeletePass`, second half) + +Candidates: `files` rows with `status = 'pending'` and `created_at` older than +`PENDING_UPLOAD_TTL_MS` (default **24 hours**) — an upload slot was requested but the +client never confirmed it. The object is deleted from S3 and the `files` row is deleted +outright (there's no soft-delete step for a pending upload; nothing ever referenced it). + +### 3. Push-subscription backoff re-enable + +See [below](#push-subscription-backoff-re-enable) — runs as the last step of the same +tick, after the hard-delete pass, in `startFileCleanupJob`'s interval callback. + +### Reference-counting check + +Before either the initial candidate query or the hard delete itself, the job protects +against deleting a file another live message still points at: + +```sql +SELECT 1 FROM messages +WHERE file_id = + AND deleted_at IS NULL +LIMIT 1 +``` + +If this returns a row, the file is skipped for this tick. This matters because a file +can be attached to more than one message (e.g. forwarded), so retracting *one* message +that references a file must not delete the underlying object while another, +non-retracted message still needs it. `softDeleteFile()` itself runs the same +`NOT EXISTS (...)` check inline (as part of its `UPDATE ... WHERE`) before it will even +set `deleted_at`, so a file only becomes hard-delete-eligible once every referencing +message has been retracted. + +**Env vars**: `FILE_GC_INTERVAL_MS` (default 5min), `FILE_HARD_DELETE_GRACE_MS` (default +0), `PENDING_UPLOAD_TTL_MS` (default 24h). + +--- + +## Push-subscription backoff re-enable + +Not a standalone timer — it's called (`reenableExpiredBackoffs()`, defined in +`services/pushNotification.ts`) as the second step of every file-cleanup tick, so it +inherits `FILE_GC_INTERVAL_MS` as its effective schedule (default every 5 minutes). + +**What it removes**: nothing is deleted. It's a single `UPDATE`: + +```sql +UPDATE push_subscriptions +SET disabled_at = NULL +WHERE disabled_at IS NOT NULL AND disabled_at <= NOW() +``` + +**Where the backoff comes from**: a transient push-send failure (any error other than a +404/410 "gone" response, which prunes the subscription immediately instead) sets +`disabled_at = now() + 5 minutes` in `sendWebPush()`. This pass is what clears that flag +once the 5-minute window has elapsed, making the subscription eligible for delivery +again. There is no separate env var for the backoff duration itself — it's a fixed +5-minute constant at the call site — only the re-enable pass's cadence is configurable, +and only indirectly via `FILE_GC_INTERVAL_MS`. + +--- + +## Idempotency and retry-safety, in general + +Every pass above is safe to crash mid-run and simply pick back up on the next tick, +by construction: + +- **Prekey / MLS key package GC**: a plain `DELETE ... WHERE ` is naturally + idempotent — a row either still matches the cutoff (gets deleted again, a no-op if it + was already gone) or it doesn't (left alone). There's no two-step state to get half-done. +- **Device stale-flag pass**: only touches rows where `stale_flagged_at IS NULL`, so a + row that already got flagged is excluded from the next run's `WHERE` clause — + re-running against already-flagged devices is a no-op. +- **Envelope GC**: same reasoning as prekey GC — a single-statement `DELETE ... WHERE`. +- **File hard-delete — the one two-step case**: this is the pass where crash-safety + actually has to be designed for, because deleting the S3 object and marking the + database row are two separate operations that can't be wrapped in one transaction (the + object store isn't Postgres). The job orders them deliberately: + + > delete the S3 object first, and only set `hard_deleted_at` **after** that delete + > succeeds. + + If the process crashes (or the DB write fails) between those two steps, the object is + already gone from S3 but `hard_deleted_at` is still `NULL`. On the next tick, the file + is picked up as a candidate again; the re-check query finds no live reference, and the + code calls `deleteObject()` a second time. Object stores generally treat deleting an + already-absent key as a success (or the client's error is caught and logged and the row + is retried again next tick) rather than a hard failure, so the retry converges instead + of raising an inconsistency. The failure mode this ordering rules out is the opposite + one — flagging `hard_deleted_at` and then crashing *before* the S3 delete actually runs, + which would permanently orphan the object with no job left to clean it up. See + `fileCleanup.test.ts`'s `'does not mark hardDeletedAt when S3 delete throws (safe + retry)'` case for the regression test on this ordering. +- **Pending-upload GC**: same object-then-row ordering as hard-delete, same retry + argument — a crash between the S3 delete and the Postgres row delete just means the + row is deleted again (already-gone-in-S3, still-present-in-Postgres) on the next tick. +- **Push backoff re-enable**: a single `UPDATE ... WHERE disabled_at <= NOW()` — the + `WHERE` clause makes re-running it against already-cleared rows a no-op. + +## Multi-node deployment implications + +All six passes are started unconditionally in [`index.ts`](../src/index.ts) — +`startFileCleanupJob()`, `startDeviceGcJob()`, `startEnvelopeGcJob()` — with **no leader +election, no distributed lock, and no "only run on node 0" guard**. That means: + +- **Every gateway node runs every job, on its own timer, independently.** In an N-node + deployment, N processes are issuing the same `DELETE ... WHERE ` / + `UPDATE ... WHERE ` queries against the same Postgres database on overlapping + schedules. +- This is safe *because* every pass above is idempotent (see the previous section): a + row is only affected while it still matches the `WHERE` clause, so two nodes racing to + delete/flag the same row just means one of them does the work and the other's query + matches zero rows. Nothing double-decrements or double-processes. +- The cost is redundant work, not correctness risk: N nodes each executing the same scan + and (mostly no-op) `DELETE`/`UPDATE` every tick is wasted query load that scales + linearly with node count, and N nodes each calling `deleteObject()` for the same S3 key + around the same time is wasted object-store calls (again safe, since a repeat delete of + an already-gone key is not an error). +- **Timers are not synchronized across nodes.** Each node starts its own `setInterval` + from its own boot time, so in practice the N nodes' ticks are staggered, which spreads + the redundant load out somewhat rather than having every node hit the database in the + same instant — but this is incidental, not a designed-in stagger. +- If this redundant load ever becomes a real concern at higher node counts, the fix is to + gate `startDeviceGcJob()` / `startEnvelopeGcJob()` / `startFileCleanupJob()` behind a + single-designated-node check (e.g. lowest pod ordinal, or a Redis lock) rather than + changing the jobs' own logic — the jobs themselves don't need to change since they're + already safe to run from any single node. diff --git a/apps/backend/docs/concepts-local-object-store.md b/apps/backend/docs/concepts-local-object-store.md new file mode 100644 index 0000000..70c3cf3 --- /dev/null +++ b/apps/backend/docs/concepts-local-object-store.md @@ -0,0 +1,172 @@ +# Local Object Store (development/test) + +Source: [`src/lib/localObjectStore.ts`](../src/lib/localObjectStore.ts), +[`src/lib/objectStore.ts`](../src/lib/objectStore.ts), +[`src/routes/localStorage.ts`](../src/routes/localStorage.ts), +[`src/lib/storage.ts`](../src/lib/storage.ts), +[`src/routes/uploads.ts`](../src/routes/uploads.ts). + +See also: [File Uploads API](api-files-uploads.md) for the upload/confirm route contract this +store backs, and the [backend testing guide](testing.md) / +[cross-app testing strategy](../../../docs/testing.md) for the no-Docker-in-tests policy this +store exists to satisfy. + +## `ObjectStoreLike` — the shared interface + +Both the production `ObjectStore` (S3-compatible, via `@aws-sdk/client-s3`) and the +development-only `LocalDiskObjectStore` implement the same `ObjectStoreLike` interface, defined +in `objectStore.ts`: + +```ts +export interface ObjectStoreLike { + putObject(key: string, body: ..., contentType?: string): Promise; + getObject(key: string): Promise; + deleteObject(key: string): Promise; + headObject(key: string): Promise<{ exists: boolean; size?: number }>; + getPresignedPutUrl(key: string, contentType: string | undefined, ttlSeconds: number): Promise; + getPresignedGetUrl(key: string, ttlSeconds: number): Promise; +} +``` + +**Call sites only ever depend on `ObjectStoreLike`** (`getObjectStore()`'s return type, +imported by `routes/uploads.ts` and `lib/storage.ts`); nothing in the application imports +`ObjectStore` or `LocalDiskObjectStore` directly except the factory code itself. The backend +selected underneath is swapped without call-site changes — verified below. + +| Method | Purpose | Local (`LocalDiskObjectStore`) behavior | Production (`ObjectStore`) behavior | +|---|---|---|---| +| `putObject(key, body, contentType?)` | Store bytes at `key`. | Writes the body to a file under the local storage root, plus a sidecar `.meta.json` recording `contentType`. Creates parent directories as needed (`mkdir(..., { recursive: true })`). | `PutObjectCommand` against the S3-compatible bucket. | +| `getObject(key)` | Retrieve an object. | Reads the file and its sidecar meta file; returns `{ Body: Buffer, ContentType? }`. Throws (propagates the `fs` error) if the file doesn't exist. | `GetObjectCommand`; returns the raw SDK response. The two implementations' return shapes are intentionally left loosely typed (`unknown`) because nothing in the codebase consumes `getObject()` polymorphically today — callers that use it know which implementation they're calling. | +| `deleteObject(key)` | Remove an object. | `rm` on both the object file and its sidecar meta file, with `{ force: true }` (no error if already absent). | `DeleteObjectCommand`. | +| `headObject(key)` | Existence + size check, **used by upload-confirm verification** (`routes/uploads.ts`, `#356`). | `stat()`s the resolved path; returns `{ exists: true, size }` on success, `{ exists: false }` on any error (e.g. `ENOENT`). | `HeadObjectCommand`; maps `NotFound` / `NoSuchKey` names or a `404` status code to `{ exists: false }`, otherwise returns `{ exists: true, size: ContentLength }` (omitting `size` if the SDK didn't return `ContentLength`). Any other error is rethrown rather than swallowed. | +| `getPresignedPutUrl(key, contentType, ttlSeconds)` | Issue a time-limited upload URL. | Builds a URL back to this backend's own `/local-storage/` route (see below) with an HMAC signature and an `expires` timestamp `ttlSeconds` seconds out. `contentType` is accepted for interface parity but not used to constrain the signature. | Real S3 presigned URL via `getSignedUrl` from `@aws-sdk/s3-request-presigner`, expiring after `ttlSeconds`. | +| `getPresignedGetUrl(key, ttlSeconds)` | Issue a time-limited download URL. | Same local-URL scheme as the PUT case, signed for `GET`. | Real S3 presigned GET URL. | + +`headObject` matters specifically because `routes/uploads.ts`'s upload-confirm handler calls it +to prove an object actually landed in the store — at the expected size — before marking a file +row `ready`; both implementations must answer this the same way for that check to behave +identically in dev and production. + +## Local filesystem behavior + +- **Root directory:** `process.env.LOCAL_STORAGE_DIR`, falling back to `/.local-storage` + (`DEFAULT_ROOT_DIR` in `localObjectStore.ts`). This directory is expected to be gitignored; + this document does not assume any specific absolute path beyond that default. +- **Key → path mapping:** `resolvePath(key)` joins the root with `key` via `node:path`'s + `join`/`normalize`, then verifies the resolved path is still inside the root + (`resolved === root || resolved.startsWith(root + sep)`); otherwise it throws + `"Invalid storage key: "`. This is the path-traversal guard — a key like + `../../etc/passwd` cannot escape the root directory. +- **Directory creation:** `putObject` creates any missing parent directories with + `mkdir(dirname(path), { recursive: true })` before writing, so nested keys (e.g. + `uploads//`, the shape `generateStorageKey` in `lib/storage.ts` + produces) work without a separate provisioning step. +- **Metadata persistence:** yes, but only `contentType`, and only in a sidecar file + `.meta.json` next to the object (`metaPath(key)`), written by `putObject` and + read back by `getObject`. There is no metadata store beyond this — no size, no timestamps, + no arbitrary user metadata. +- **Missing objects:** `getObject` and `putObject`'s underlying `fs` calls throw/reject on a + missing file; `headObject` and the exported test helper `exists()` instead catch the error + and return `false`/`{ exists: false }`. Callers that need a non-throwing existence check use + `headObject`. +- **Concurrent writes:** no special handling — `writeFile` is used as-is, with no locking, + temp-file-then-rename, or write queuing. Two concurrent `putObject` calls to the same key race + at the filesystem level exactly as plain `fs.writeFile` calls would. + +## Signed local URLs + +Format (both PUT and GET use the same shape, differing only in HTTP method and which key each +verifies against): + +``` +http://localhost:/local-storage/?expires=&sig= +``` + +- **Base URL:** `process.env.STORAGE_ENDPOINT` if set (trailing slashes stripped), otherwise + `http://localhost:${process.env.PORT ?? 3001}/local-storage`. +- **Object identifier:** the storage `key`, appended as the path segment(s) after + `/local-storage/` (e.g. `uploads//`). +- **`expires`:** a Unix timestamp in seconds — `Math.floor(Date.now() / 1000) + ttlSeconds` at + signing time. +- **`sig`:** `HMAC-SHA256(processSecret, "::")`, hex-encoded. + `processSecret` is `randomBytes(32)` generated once per process at module load — it is not + persisted or shared across processes, which is fine because the same process that signs a URL + is the one that verifies it later in the same dev/test run. + +Example (placeholders, not a real signature): + +``` +http://localhost:3001/local-storage/uploads/abc123/def456?expires=1780000000&sig=4f3c...e9a1 +``` + +### Validation on the serving route + +`routes/localStorage.ts` mounts `PUT` and `GET` handlers on `/*splat` under `/local-storage` +(only outside production — see below). Both call `checkSignature(req, res, method)`, which: + +1. **Object/key:** extracts the key from the wildcard route params (`req.params['splat']`, + joined with `/` if Express gives it as an array). +2. **Signature:** reads `sig` from the query string and calls + `verifySignedRequest(method, key, expires, sig)` in `localObjectStore.ts`, which + recomputes the expected HMAC for `(method, key, expires)` and compares it to the supplied + signature using `timingSafeEqual` (constant-time, after confirming both buffers are the same + length — a length mismatch is treated as a non-match rather than passed to + `timingSafeEqual`, which throws on unequal lengths). +3. **Expiry:** `verifySignedRequest` also checks `Number.isFinite(expires)` and rejects if + `Date.now() / 1000 > expires`. +4. **No other validation** is performed by this route beyond signature + expiry — there is no + separate auth/session check, which is intentional: the route is mounted outside + `requireAuth` (see `app.ts`) because a valid signed URL is meant to be the credential, the + same way a real S3 presigned URL is. + +On failure, the route responds `403 { error: 'Invalid or expired signed URL' }` without +distinguishing "bad signature" from "expired" from "missing key/sig" in the response body. + +The `PUT` handler reads the raw body (`express.raw({ type: () => true, limit: '150mb' })`) and +the `content-type` header, then calls `getLocalObjectStore().putObject(key, body, contentType)`. +The `GET` handler calls `getObject(key)` and streams back `Body` with `Content-Type` set from +the stored metadata if present, returning `404 { error: 'Object not found' }` on any error +(e.g. the file doesn't exist). + +## Production/development selection + +Two independent switches exist in the codebase, both keyed on `process.env.NODE_ENV`: + +- **`objectStore.ts`'s `getObjectStore()`** (used by `routes/uploads.ts` for `headObject`, and + elsewhere for delete/head operations): a lazily-constructed singleton that picks + `createObjectStore(loadEnv())` (real S3 client) when `NODE_ENV === 'production'`, otherwise + `getLocalObjectStore()`. `resetObjectStoreForTests()` clears the singleton so tests can + change env and re-resolve it. +- **`lib/storage.ts`'s `generatePresignedPut` / `generatePresignedGet`**: each has its own + `isProduction()` check (`process.env.NODE_ENV === 'production'`) and calls + `getLocalObjectStore()` directly outside production, or `getObjectStore()` in production. This + is a second, independent branch rather than routing through `getObjectStore()`'s own switch — + both land on the same backend selection, but it's worth knowing these are two separate + `NODE_ENV` checks in the source rather than one shared decision point. +- **Route mounting:** `app.ts` only mounts `localStorageRouter` at `/local-storage` when + `process.env['NODE_ENV'] !== 'production'` — in production the route doesn't exist at all, so + a presigned local URL could never be dereferenced even if one were somehow issued. + +**Call-site agnosticism, verified:** `routes/uploads.ts` calls `getObjectStore().headObject(...)` +without any environment branching of its own — it only ever sees `ObjectStoreLike`. The +environment switch lives entirely in the two factory functions above, not in application code +that consumes the store. + +## Test architecture: why this exists + +The [cross-app testing strategy](../../../docs/testing.md) states the project-wide rule +directly: **no test in the repository may require Redis, Postgres, or an S3/MinIO server to be +running.** `LocalDiskObjectStore` is what makes that rule achievable for object-store-dependent +code: `getObjectStore()` resolves to it automatically whenever `NODE_ENV !== 'production'` +(the default for `vitest`), so upload/presign/download/head flows exercise a real, working +implementation — actual bytes on disk, actual signature verification — without a Docker +dependency, on a fresh clone, with Docker stopped. + +This does **not** mean every backend test is Docker-free for the same reason: the same +cross-app testing document also states that Postgres access goes through a hand-built `db` +mock rather than a real database in tests. `LocalDiskObjectStore` specifically replaces the +need for a MinIO/S3 dependency; it makes no claim about, and has no bearing on, how other +external dependencies (Postgres, Redis) are avoided in the test suite. Separately, the backend +CI workflow (`backend-ci.yml`) does start real Postgres, Redis, and MinIO containers — but per +the testing doc, that's to validate migrations against a real Postgres, not because the test +suite itself dials into any of those services. diff --git a/apps/backend/docs/concepts-logging.md b/apps/backend/docs/concepts-logging.md new file mode 100644 index 0000000..e9d4290 --- /dev/null +++ b/apps/backend/docs/concepts-logging.md @@ -0,0 +1,184 @@ +# Backend Structured Logging Conventions + +This document covers [`lib/logger.ts`](../src/lib/logger.ts) — the structured logger for +the encrypted pipeline (issue #393): its configuration, log levels, standard fields, and, +most importantly, the rule that message content, ciphertext, and key material never +appear in a log line. + +## Contents + +- [Configuration](#configuration) +- [Levels, and how level is set per environment](#levels-and-how-level-is-set-per-environment) +- [The no-content rule](#the-no-content-rule) +- [Standard correlation fields](#standard-correlation-fields) +- [Current adoption: `console.*` is still the norm](#current-adoption-console-is-still-the-norm) + +## Configuration + +`lib/logger.ts` exports a single shared [pino](https://getpino.io/) instance: + +```ts +export const logger = pino({ + level: process.env['LOG_LEVEL'] ?? 'info', + redact: { + paths: [ + 'ciphertext', '*.ciphertext', + 'envelopes', '*.envelopes', + 'payload', '*.payload', + 'plaintext', '*.plaintext', + ], + censor: '[redacted]', + }, + formatters: { + level: (label) => ({ level: label }), + }, + base: { service: 'clicked-backend' }, +}); +``` + +- **`level`** — read from `LOG_LEVEL`, defaulting to `'info'`. See + [below](#levels-and-how-level-is-set-per-environment). +- **`redact`** — a fixed list of field-name paths (`ciphertext`, `envelopes`, `payload`, + `plaintext`, each matched both at the top level and one level deep via the `*.` + wildcard) that pino will replace with the literal string `[redacted]` if they ever + appear in a logged object. The paths line up directly with the shapes actually flowing + through the system: `payload` is the field on `EventEnvelope` + (`lib/eventEnvelope.ts`'s `EventEnvelopeSchema`), `envelopes` is the array of + per-recipient-device ciphertext blobs on a `send_message` call, and + `ciphertext`/`plaintext` are the raw content fields on an individual envelope. This is + a **backstop**, not the primary guarantee — see the [no-content rule](#the-no-content-rule) + below for why you can't rely on it alone. +- **`formatters.level`** — pino's default is to log the numeric level; this overrides it + to log the string label (`"info"`, `"warn"`, etc.) instead, so log lines are readable + without a lookup table. +- **`base: { service: 'clicked-backend' }`** — every line this logger emits carries + `service: "clicked-backend"`, so log lines from this process are identifiable in a + shared/aggregated log stream (e.g. alongside the AI agent or other services) without + extra configuration at every call site. + +There is no per-request or per-socket child-logger factory defined yet — see +[Standard correlation fields](#standard-correlation-fields) for what's available to attach +manually, and [Current adoption](#current-adoption-console-is-still-the-norm) for why this +matters less today than it will once call sites migrate to it. + +## Levels, and how level is set per environment + +pino's standard level ordering applies: `fatal` > `error` > `warn` > `info` > `debug` > +`trace` (plus `silent` to disable entirely) — set `level` to a name and every level at or +above it (i.e. more severe or equal) is emitted; everything below it is a no-op with +near-zero overhead (pino checks the level before serializing). + +**Per-environment control is entirely via the `LOG_LEVEL` env var** — there's no +hardcoded environment-name branch (`if (NODE_ENV === 'production')` etc.) in +`lib/logger.ts` itself: + +- Unset → defaults to **`info`**. This is what local dev and any environment that doesn't + explicitly set `LOG_LEVEL` gets. +- Set `LOG_LEVEL=debug` (or `trace`) locally or in a staging environment to see verbose + output while debugging a specific issue — remember to unset it again rather than + leaving a deployment running at `debug` indefinitely, since `debug`/`trace` volume adds + up quickly on a busy gateway. +- Set `LOG_LEVEL=warn` (or stricter) in an environment where you only want actionable + signal and want to cut `info`-level noise (e.g. a high-throughput environment where + per-connection `info` lines aren't worth the log volume). +- `LOG_LEVEL` is read once at module load (`pino({ level: process.env['LOG_LEVEL'] ?? + 'info'] ... })`), so changing it requires a process restart — it is not something a + running process picks up live. + +## The no-content rule + +**Message content, ciphertext, and key material never appear in a log line — this is a +hard rule, not a style preference.** The whole point of the encrypted pipeline is that the +server never has plaintext to leak in the first place, but ciphertext, key material +(prekeys, signed prekeys, MLS `KeyPackage`s, session keys), and full envelope objects are +exactly as sensitive from a logging standpoint: they should never be the value of a +logged field, truncated preview or not, redacted-and-hope or not. + +**The redact list in `lib/logger.ts` is a backstop, not the mechanism you should rely on.** +It only fires if a call site accidentally passes a *whole object* that happens to contain +one of the redacted field names — it does nothing for a raw string interpolated into a +message (`` logger.info(`sending ${ciphertext}`) `` bypasses `redact` entirely, because +`redact` only inspects the structured fields of a log object, not free-text message +strings) and nothing for a field passed under a name the list doesn't happen to cover. +The actual guarantee has to come from what you choose to log in the first place. + +**The safe way to log about a message: ids and counts, never bodies.** + +```ts +// ❌ Never — even though `redact` will catch some of this, don't rely on it, +// and the string-interpolation case isn't caught at all. +logger.info(`delivering message ${JSON.stringify(envelope)}`); +console.log('sending', { ciphertext: envelope.ciphertext }); + +// ✅ Ids, counts, sizes, durations — never the content itself. +logger.info({ messageId, conversationId, recipientDeviceCount: envelopes.length }, + 'message fanout dispatched'); +logger.debug({ deviceId, eventId, envelopeByteLength: envelope.ciphertext.length }, + 'envelope accepted'); +logger.warn({ subscriptionId, statusCode }, 'push send failed, backing off'); +``` + +Concretely, safe fields to log about a message/envelope/key include: `messageId`, +`conversationId`, `deviceId`, `userId`, `eventId`, counts (`recipientDeviceCount`, +`prunedCount`), sizes/lengths (`envelopeByteLength`, never the bytes themselves), status +codes, durations, and timestamps. Never log: `ciphertext`, `plaintext`, the raw +`payload`/`envelopes` object, key material (`identityPublicKey`, prekey bytes, MLS +`KeyPackage` bytes), auth tokens, or `p256dh`/`auth` push subscription keys. When in +doubt, log the *shape* (a count, a length, a boolean) rather than the *value*. + +## Standard correlation fields + +There is no dedicated request-id/correlation-id middleware in this backend today — HTTP +access logging goes through `morgan('dev')` (`app.ts`), which logs method, path, status, +and response time, with no generated request id attached, and is separate from +`lib/logger.ts` entirely. For tracing a specific request or socket event through +structured log lines, use the identifiers already available at each call site: + +- **`deviceId`** — available on every authenticated socket via `socket.auth.deviceId` + (set by `socketAuthMiddleware`) and on every REST request via the equivalent JWT-derived + auth context. The most useful single field for tracing one device's activity across + connect/disconnect/send/receive. +- **`userId`** — `socket.auth.userId` / the REST auth context; ties multiple devices + belonging to the same account together. +- **`eventId`** — present on every dispatched socket envelope (`EventEnvelopeSchema.eventId`, + `lib/eventEnvelope.ts`), generated client-side or via `createEnvelope()`. This is what + the existing replay-protection debug log already keys on + (`dispatcher.ts`: `{ deviceId, eventId, type }`) — follow that shape for any new + per-event log line. +- **`conversationId`** / **`messageId`** — attach whichever is relevant to the operation; + both are safe to log (they're routing/addressing metadata, not content). +- **`socket.id`** — Socket.IO's own per-connection id; useful for correlating multiple + events from the same physical connection within one session, but don't use it as a + stand-in for `deviceId` across reconnects — a new connection gets a new `socket.id`. + +None of these are enforced by a shared child-logger or middleware yet — attach the +relevant subset by hand at each call site (as the redact-list field names already imply +you should for `payload`/`envelopes`/`ciphertext`/`plaintext`), following the pattern +shown in [the no-content rule](#the-no-content-rule) above. + +## Current adoption: `console.*` is still the norm + +**`lib/logger.ts` has no import sites anywhere else in `apps/backend/src` today.** Every +current log line in the backend — startup/shutdown messages in `index.ts`, the GC job +services (`deviceGc.ts`, `envelopeGc.ts`, `fileCleanup.ts`), the socket dispatcher and +messaging handlers, push notification hygiene, presence, backpressure, rate limiting, and +so on — goes through bare `console.log`/`console.warn`/`console.error`/`console.debug` +instead (roughly 60 call sites across the backend as of this writing). Even +`services/stellarListener.ts`, whose own doc comment says it "logs errors via the standard +backend logger," actually defaults its optional `log` dependency to a small hand-rolled +`consoleLogger` wrapper around `console.*`, not to `lib/logger.ts`. + +Practical implications: + +- **None of the level control, redaction backstop, or `service` base field described + above currently applies to the backend's actual log output** — `console.*` calls bypass + all of it. `LOG_LEVEL` has no effect on anything printed via `console.log`. +- **New code should use `lib/logger.ts`, not add more `console.*` calls.** The existing + ~60 call sites are the migration backlog, not the target state — don't grow that number. + When you're already touching a file that logs via `console.*`, prefer converting the + lines you touch to `logger` rather than leaving new logic logging through the old path. +- When you do add a `logger.*` call, follow the [no-content rule](#the-no-content-rule) + and attach the [correlation fields](#standard-correlation-fields) relevant to that + call site, the same way the codebase's existing `console.*` calls already do at their + best (e.g. `dispatcher.ts`'s `{ deviceId, eventId, type }` shape) — the goal is to bring + that same discipline under the structured logger, not to reinvent it. diff --git a/apps/backend/docs/concepts-nonce-lifecycle.md b/apps/backend/docs/concepts-nonce-lifecycle.md new file mode 100644 index 0000000..27dd37e --- /dev/null +++ b/apps/backend/docs/concepts-nonce-lifecycle.md @@ -0,0 +1,161 @@ +# Nonce and challenge store lifecycle + +This note documents [src/lib/nonce.ts](../src/lib/nonce.ts): how challenge nonces are minted, stored, consumed, and expired, for both the wallet sign-in challenge and the separate device-link challenge. + +Both flows follow the same shape — the server mints a random value, the client proves wallet ownership by signing a message containing it, and the server burns the value while verifying. What differs is the keyspace, the TTL, and what a successful verification grants. + +See also: [Auth API](api-auth.md), [Devices & Prekeys API](api-devices.md), [JWT auth contract](contracts-jwt-auth.md), and [Rate limits](../../../docs/security/rate-limits.md). + +## 1. The two nonce kinds at a glance + +| | Sign-in challenge | Device-link challenge | +| --------------------- | ------------------------------------ | --------------------------------------- | +| Mint | `createNonce(walletAddress)` | `createDeviceLinkNonce(userId)` | +| Consume | `consumeNonce(walletAddress, nonce)` | `consumeDeviceLinkNonce(userId, nonce)` | +| Store | `store` map | `deviceLinkStore` map | +| Key | Wallet address (`G...`) | `userId` | +| TTL | 5 minutes (`TTL_MS`) | 2 minutes (`DEVICE_LINK_TTL_MS`) | +| Issued by | `POST /auth/challenge` | `POST /devices/link/challenge` | +| Burned by | `POST /auth/verify` | `POST /devices/link/verify` | +| Caller authenticated? | No — this is the pre-login step | Yes — requires a valid JWT | +| Grants on success | A session (JWT) | Registration of one new device | + +## 2. Creation + +Both kinds are minted by the same internal helper: + +```ts +function issue(target, key, ttlMs) { + const nonce = randomBytes(16).toString('hex'); + target.set(key, { nonce, expiresAt: Date.now() + ttlMs }); + return nonce; +} +``` + +- **Value.** 16 bytes from Node's `crypto.randomBytes`, hex-encoded to a 32-character string. 128 bits of CSPRNG output — guessing a live nonce is not a practical attack, and collisions between concurrently outstanding nonces are not a concern. +- **Storage.** One entry per key, holding the nonce and an absolute `expiresAt` timestamp in epoch milliseconds. +- **Overwrite semantics.** `Map.set` replaces any existing entry for that key. Requesting a second challenge for the same wallet (or the same user) invalidates the first — only the most recently issued nonce for a key is live. A client that fires two challenge requests and then signs the first will fail verification. + +### Sign-in + +`POST /auth/challenge` ([routes/auth.ts](../src/routes/auth.ts)) takes a wallet address, mints a nonce, and returns it embedded in the message the client must sign: + +```text +Sign in to Clicked +Wallet: +Nonce: +``` + +The endpoint is unauthenticated — the wallet address is the only identity available at this point, and it is a public value. It is rate-limited per IP (`auth_challenge`: 10/minute). + +### Device link + +`POST /devices/link/challenge` ([routes/devices.ts](../src/routes/devices.ts)) requires a valid JWT, resolves the account's primary wallet, and mints a nonce keyed by `userId`: + +```text +Link device to Clicked +User: +Nonce: +``` + +The response carries the message, the nonce, and the wallet address the signature must come from. Rate-limited as `device_link_challenge` (10/minute). + +This is a re-authentication step, not a login: the caller already holds a session. The point is to prove wallet ownership _now_, because a JWT alone must not be enough to add a device to an account. + +## 3. Consumption + +Both kinds are burned by the same helper: + +```ts +function consume(target, key, nonce) { + const entry = target.get(key); + if (!entry) return false; + target.delete(key); // burned on read, valid or not + if (Date.now() > entry.expiresAt) return false; + return entry.nonce === nonce; +} +``` + +Three properties follow, and all three are deliberate: + +- **Single use.** The entry is deleted on read, whether or not it turns out to be valid. A second `consume` for the same key returns `false` because there is nothing left to read. +- **Burned even on failure.** A wrong nonce still deletes the stored entry. This is what stops an attacker from guessing repeatedly against one live challenge: each attempt destroys the target, so a guessing run must interleave a fresh `/challenge` call for every attempt, which puts it under the challenge rate limit rather than the (much cheaper) verify path. +- **Expiry checked after deletion.** An expired entry is removed and rejected in the same call. + +`consume` returns a plain boolean; the routes translate a `false` into `401` without revealing whether the nonce was missing, wrong, or expired. + +### Ordering at the call sites + +Both verify routes consume the nonce **before** doing any signature work: + +- `POST /auth/verify` calls `consumeNonce(walletAddress, nonce)` first, records an `auth_failed` audit event with reason `invalid_or_expired_nonce` on failure, and returns `401` before touching `Keypair.verify` or the database. +- `POST /devices/link/verify` calls `consumeDeviceLinkNonce(userId, body.nonce)` first, before resolving the wallet, before verifying the signature, and before any device lookup. + +Consuming first means a replayed request always fails on its second submission, and a caller cannot use the endpoint to run signature verifications against a nonce it keeps alive. + +## 4. Why device linking uses a separate namespace + +The two stores are separate maps, keyed differently on purpose. The header comment in `nonce.ts` states it directly: sharing the login store would let a device-link challenge and a login challenge for the same wallet silently overwrite each other. + +Concretely, with a shared store: + +- A user signed in on device A requests a device-link challenge. If that challenge landed in the sign-in store under the same key, a concurrent sign-in attempt from device B would overwrite it — and vice versa. Whichever flow finished second would fail with "invalid or expired nonce" for reasons the user could not see or fix. +- Because `Map.set` overwrites unconditionally and `consume` burns on any read, an attacker who can reach the unauthenticated `/auth/challenge` endpoint for a known wallet address could repeatedly overwrite the victim's outstanding device-link nonce, making device linking permanently fail. Separate namespaces mean the unauthenticated flow has no handle on the authenticated flow's state at all. + +The separation is reinforced by the key type: the sign-in store is keyed by wallet address, the device-link store by `userId`. Even a string that happens to be valid as both cannot cross over. [nonce.test.ts](../src/__tests__/nonce.test.ts) pins this with a test that issues both kinds under the identical key string and asserts that neither nonce satisfies the other's `consume`. + +The same "separate buckets" reasoning is applied one layer up, in [config/rateLimits.ts](../src/config/rateLimits.ts): `device_link_challenge` and `device_link_verify` mirror the `auth_challenge` and `auth_verify` limits but count in their own buckets, so hammering the device-link flow cannot exhaust a user's sign-in budget, and hammering sign-in cannot lock out device linking. Namespace isolation at the store level and bucket isolation at the rate-limit level are two halves of the same guarantee. + +## 5. TTLs + +- **Sign-in: 5 minutes.** Long enough to cover a wallet-extension prompt the user has to find, unlock, and approve — possibly on a phone. +- **Device link: 2 minutes.** Tighter, because the caller is already signed in and actively performing the flow; there is no unlock-from-cold path to accommodate. A shorter window narrows the period in which a captured challenge could be replayed after a signature is obtained by other means. + +Expiry is lazy. There is no sweeper: an entry sits in the map until someone calls `consume` for that key, at which point it is deleted and rejected. The practical consequences: + +- An abandoned challenge (user requests one and never verifies) leaves a ~90-byte entry in memory until the same key is used again. +- Memory is bounded in practice by the challenge rate limits and by the fact that there is at most one entry per key. It is not bounded by the TTL. On a long-running node under sustained challenge traffic from many distinct wallet addresses, the map grows with the number of distinct keys seen, not with the number of live nonces. If that ever becomes a concern, it is an argument for moving the stores to Redis with native key expiry rather than for adding a sweeper. + +## 6. Where nonces live: restarts and multi-node deployments + +Both stores are **in-process `Map` instances in the Node heap**. They are not in Postgres and not in Redis, unlike the rate-limit counters, which do use Redis so budgets are shared across gateway nodes. + +Two consequences follow, and both are operationally important: + +### Across a restart + +Every outstanding nonce is lost when the process restarts, is redeployed, or crashes. Any client that has a challenge in flight gets `401 Invalid or expired nonce` when it submits the signature. This fails safe — no nonce survives to be replayed against a new process — but it is user-visible: a deploy during a sign-in attempt makes that attempt fail. Clients should treat a nonce rejection as "request a fresh challenge and retry", not as a terminal auth error. + +### Across multiple nodes + +**The challenge and the verification must be served by the same process.** A nonce minted on node A does not exist on node B. In a multi-node deployment this means one of the following must hold: + +- the load balancer pins a client's `/auth/challenge` and `/auth/verify` (and the two `/devices/link/*` calls) to the same backend instance — sticky sessions or connection reuse; or +- the deployment runs a single gateway instance; or +- the stores are moved to a shared backend before scaling out. + +If none holds, sign-in and device linking fail intermittently at a rate that rises with the node count, and the failures look like spurious "invalid or expired nonce" errors rather than like a routing problem. This is the first thing to check when nonce rejections appear after a horizontal scale-out. + +Migrating the stores to Redis (see [lib/redis.ts](../src/lib/redis.ts)) would remove both limitations: `SET key value NX PX ` for `issue` and an atomic `GETDEL`-style consume preserve exactly the burn-on-read semantics described above, with expiry handled natively. + +## 7. Replay resistance: what this buys, and what it does not + +### What it provides + +- **Signature replay is prevented within a flow.** A captured `/auth/verify` body cannot be resubmitted: the nonce it carries was burned by the first submission. The same holds for `/devices/link/verify`. +- **Cross-flow replay is prevented.** The signed messages differ in prefix (`Sign in to Clicked` vs `Link device to Clicked`) and in body (wallet address vs user id), and the nonces live in separate keyspaces, so a signature captured from one flow is not valid for the other. +- **Freshness is bounded.** A signature is only useful inside the nonce's TTL — 5 minutes for sign-in, 2 for device linking. +- **Online guessing is expensive.** 128 bits of entropy, burn-on-read on a wrong guess, and rate limits on both the challenge and the verify endpoints. +- **Wallet signatures cannot be harvested for other purposes.** The signed string is scoped to this application and to a server-issued value, so a signature obtained elsewhere for a different message does not verify here. + +### What it does not provide + +- **No protection against an attacker who controls the wallet.** The nonce proves _freshness_ of a signature, not that the signer is the legitimate account holder. A compromised or malicious wallet passes every check here. Device linking is the mitigation for the account-level version of this: linking a new device requires a fresh wallet signature and is audited, and existing devices can be revoked. +- **No protection against a same-session in-flight attacker.** Anyone who can read the challenge response _and_ get the user to sign it (a malicious page, a compromised client, an attacker holding the wallet unlock) can complete the flow once. Single-use only stops the _second_ use. +- **Nothing is bound to a transport or a client.** The nonce is not tied to an IP, a TLS session, or a device fingerprint. A nonce obtained by client X can be verified by client Y if Y can produce the signature. Transport-level binding is handled separately — see [Transport security and pinning](../../../docs/security/tls-and-pinning.md). +- **No cross-node or cross-restart guarantee**, per section 6. The replay guarantee is per-process. It is not weakened by a restart (state is lost, not duplicated), but availability is. +- **The device-link nonce does not prove the new device is trustworthy.** It proves the wallet approved adding _a_ device at that moment. Verifying the identity of the device itself is the safety-number / fingerprint job — see [E2EE onboarding](e2ee-onboarding.md). + +## 8. Tests + +[src/\_\_tests\_\_/nonce.test.ts](../src/__tests__/nonce.test.ts) covers both stores: format (`/^[0-9a-f]{32}$/`), successful consume, single-use, wrong-nonce rejection, unknown-key rejection, cross-user rejection, keyspace separation between the two stores, and TTL boundaries on both sides using fake timers (just-before-expiry accepted, just-after rejected). diff --git a/apps/backend/docs/concepts-presence-heartbeat.md b/apps/backend/docs/concepts-presence-heartbeat.md new file mode 100644 index 0000000..536ce27 --- /dev/null +++ b/apps/backend/docs/concepts-presence-heartbeat.md @@ -0,0 +1,317 @@ +# Presence, heartbeat & connection liveness + +This document explains how the backend knows whether a user is online: the +Redis-backed per-device/per-user presence model in +[`services/presence.ts`](../src/services/presence.ts), the heartbeat watchdog +in [`services/heartbeat.ts`](../src/services/heartbeat.ts), and how the two +are wired together in the Socket.IO connection handler in +[`index.ts`](../src/index.ts). Verified against the current implementation of +all three files. + +For the WebSocket event contract (`heartbeat`, `presence_update`, +`user_online`/`user_offline`) from a client's point of view, see +[`api-websocket-events.md`](api-websocket-events.md). For the connection +lifecycle these services plug into, see +[`concepts-gateway-architecture.md`](concepts-gateway-architecture.md). For +the `users`/`devices` table columns referenced throughout (`presenceVisible`, +`lastSeenVisible`, `devices.lastSeenAt`, `devices.revokedAt`), see +[`contracts-database-schema.md`](contracts-database-schema.md). + +## Table of contents + +- [Two independent signals of "online"](#two-independent-signals-of-online) +- [Redis key structure](#redis-key-structure) +- [The device → user presence model](#the-device--user-presence-model) +- [Heartbeat timing contract](#heartbeat-timing-contract) +- [Connection lifecycle, step by step](#connection-lifecycle-step-by-step) +- [Multi-device aggregation](#multi-device-aggregation) +- [Transition-broadcast behavior](#transition-broadcast-behavior) +- [Privacy gating](#privacy-gating) +- [Known caveats, verified against the current code](#known-caveats-verified-against-the-current-code) + +## Two independent signals of "online" + +The backend actually has **two** ways to decide if a user is online, and it +prefers the first but falls back to the second: + +1. **Redis presence registry** (`services/presence.ts`) — live, socket-driven, + second-granularity. This is the fast path and the one that drives + real-time `presence_update`/`user_online`/`user_offline` broadcasts. +2. **Postgres `devices.lastSeenAt`** (`deriveDevicePresence`) — a + database-backed fallback with a 90-second recency window, used when Redis + is unreachable, and as the source for the `lastSeen` timestamp shown once + a user is confirmed offline. + +`GET /users/:id/presence` (`src/routes/users.ts`) tries Redis first +(`isOnline`) and only calls `deriveDevicePresence` if Redis says the user is +not online (or Redis is not configured at all) — see +[Known caveats](#known-caveats-verified-against-the-current-code) for what +this ordering does and does not guarantee. + +## Redis key structure + +Every key below is per-process-shared (visible to every gateway node), so +presence is correct in a horizontally-scaled deployment without any +in-process state. `PRESENCE_TTL = 90` (seconds) is the single TTL constant +used throughout `presence.ts`. + +| Key pattern | Type | Holds | TTL | Set/refreshed by | +|---|---|---|---|---| +| `presence:user:{userId}` | hash | `{ [deviceId]: lastSeenEpochMs }` — one field per currently-online device | **none** — not expired directly; see [caveat](#known-caveats-verified-against-the-current-code) | `setOnline`, `refreshPresence` (hset); `setOffline`/`markDeviceOffline` (hdel field, delete key once empty) | +| `presence:user:{userId}:device:{deviceId}` | hash | `{ lastSeen: epochMs }` | `PRESENCE_TTL` (90s), refreshed on every `setOnline`/`refreshPresence` | `setOnline`, `refreshPresence` | +| `presence:sockets:{userId}` | set | socket ids currently open for this user, across all their devices | `PRESENCE_TTL` (90s), refreshed on register | `registerPresenceSocket`/`refreshPresenceSocket`; drained by `unregisterPresenceSocket` | +| `presence:device_sockets:{userId}:{deviceId}` | set | socket ids for this specific `(user, device)` pair | `PRESENCE_TTL` (90s) | `registerPresenceSocket`; drained by `unregisterPresenceSocket` | +| `presence:device_sockets_by_device:{deviceId}` | set | socket ids for this device, keyed by **device id alone** (no `userId`) | `PRESENCE_TTL` (90s) | `registerPresenceSocket`; drained by `unregisterPresenceSocket` | +| `presence:socket:{socketId}` | hash | `{ userId, deviceId }` — reverse lookup from a raw socket id | `PRESENCE_TTL` (90s) | `registerPresenceSocket`; deleted by `unregisterPresenceSocket` | + +Why `presence:device_sockets_by_device:{deviceId}` exists as a *separate* +index from `presence:device_sockets:{userId}:{deviceId}`: some callers only +ever have a `deviceId` and no `userId` in hand — the device-revocation +pub/sub listener (`services/deviceRevocation.ts`) and the push-notification +online check are the two cited in the source comments — and this index lets +them resolve connected sockets without an extra DB round trip to look up the +owning user first. It replaced an in-process `Map` that was only visible to +sockets connected to the local gateway node (issue #341 in the source +comments). + +`cleanupStaleSockets` and `reconcileBoot` (called on gateway boot, after the +Redis adapter attaches) walk `presence:sockets:*`/`presence:sockets:{userId}` +and use `io.in(socketId).fetchSockets()` to detect socket ids that are no +longer connected anywhere in the cluster, removing them via the same +`removeStaleSocketMapping` helper used by normal unregistration. + +## The device → user presence model + +Presence is tracked **per device first**, then aggregated to a user: + +- `presence:user:{userId}` is a hash, not a boolean — one field per device + that is currently considered online, keyed by `deviceId`. +- `isOnline(redis, userId)` is simply `HLEN presence:user:{userId} > 0` — a + user is online if **any** device entry exists in that hash, regardless of + which device or how recently (within whatever the hash currently holds). +- A single device going offline (`setOffline`/`markDeviceOffline`) removes + only that device's field; the user stays online as long as at least one + other device's field remains. The user transitions to fully offline only + once the hash becomes empty, at which point the whole key is deleted. + +This is why the module is organized around **device-keyed** operations +(`setOnline(redis, userId, deviceId, ...)`, not `setOnline(redis, userId, +...)`) even though the externally-visible presence state (`isOnline`) is +per-user. + +## Heartbeat timing contract + +Two constants in `services/heartbeat.ts` define the contract, plus one in +`services/presence.ts`: + +| Constant | File | Value | Purpose | +|---|---|---|---| +| `HEARTBEAT_TIMEOUT_MS` | `heartbeat.ts` | **90,000 ms (90s)** | Server-side watchdog: if no `heartbeat` event arrives from a socket within this window, the server force-disconnects it and marks the device offline. | +| `LAST_SEEN_THROTTLE_MS` | `heartbeat.ts` | **30,000 ms (30s)** | Minimum spacing between `devices.lastSeenAt` DB writes for the same device — every heartbeat refreshes Redis, but the Postgres write is throttled to at most once per 30s per device. | +| `PRESENCE_TTL` | `presence.ts` | **90 seconds** | TTL on every per-device/per-socket Redis key (see table above); matches the heartbeat timeout so a device's Redis footprint expires at roughly the same time the watchdog would have fired anyway. | + +The client-side recommended send interval is **30 seconds** (documented in +[`api-websocket-events.md`](api-websocket-events.md#heartbeat)), giving a 3× +margin before the 90-second server timeout fires — a single missed or +delayed heartbeat does not disconnect the socket. + +> The module-level comment block at the top of `presence.ts` describes the +> per-device key's TTL as "60s" in prose. The actual constant in the same +> file is `PRESENCE_TTL = 90`. Treat the constant, not the comment, as +> authoritative — this documentation uses 90s throughout because that is +> what the code enforces. + +### What one `heartbeat` event does, server-side + +`handleHeartbeat(socket, userId, deviceId, redis)` runs, in order: + +1. Clears and does **not** immediately reschedule the existing disconnect + timer for this socket (the reschedule happens at the end, step 4). +2. If Redis is configured: `refreshPresence` (bumps the per-device hash entry + and its TTL) and `refreshPresenceSocket` (re-registers the socket mapping, + refreshing all four socket-related key TTLs from the table above). +3. Throttled DB write: if at least `LAST_SEEN_THROTTLE_MS` (30s) has elapsed + since the last write for this `deviceId` (tracked in an in-process `Map`, + per gateway node), updates `devices.lastSeenAt` and `devices.updatedAt`. + A failed write is swallowed (`catch { /* non-critical */ }`) — a heartbeat + is never allowed to fail the DB write path visibly to the client. +4. Reschedules the 90-second disconnect timer via the stored `schedule()` + closure (set up by `startHeartbeatTimer` at connect time). + +### What a timeout does, server-side + +If no heartbeat (and no other activity — the timer is reset by `schedule()`, +which is only called from `startHeartbeatTimer` and `handleHeartbeat`, so an +otherwise-active socket that never sends `heartbeat` will still time out) +arrives within 90 seconds, the scheduled callback: + +1. Logs the timeout. +2. If Redis is configured: `unregisterPresenceSocket` (remove this socket + from all socket-mapping keys) and, only if that was the device's **last** + remaining socket, `markDeviceOffline` (remove the device's field from + `presence:user:{userId}`). +3. If the device became fully offline and the socket is still marked + connected, emits `user_offline` and `presence_update` (volatile, i.e. + best-effort/no-buffering) to every room the socket was in. +4. Force-disconnects the socket (`socket.disconnect(true)`) if still + connected. + +## Connection lifecycle, step by step + +On a new Socket.IO connection (`index.ts`'s `io.on('connection', ...)`): + +1. `startHeartbeatTimer(socket, userId, deviceId, redis, io)` — arms the + 90-second watchdog described above. +2. `devices.lastSeenAt` is updated immediately (unthrottled — this is a + one-time write at connect, separate from the heartbeat throttle). +3. The socket joins its device room (`device:{deviceId}`), user room, and + every conversation room it's a member of. +4. If Redis is configured: + - `registerPresenceSocket` — writes the socket-mapping keys. + - `cleanupStaleSockets` — prunes any dead socket ids left over from a + previous session for this user. + - `cancelPendingOfflineBroadcast(userId)` — see + [Transition-broadcast behavior](#transition-broadcast-behavior). + - `setOnline(redis, userId, deviceId)` — returns `true` only if the user's + hash was empty before this call, i.e. this connection is what brought + the user online (as opposed to e.g. a second device connecting while + the user was already online via a first device). + - If the user *became* online (and the pending-offline broadcast wasn't + already cancelling a flap — see below) and `presenceVisible` is true, + emits `user_online`/`presence_update` to every conversation room the + user belongs to, and records the transition on co-members' resume + streams for offline replay (`recordPresenceForCoMembers`). + +On disconnect (`socket.on('disconnect', ...)`): + +1. `clearHeartbeatTimer(socket.id)` — cancels the watchdog; a clean + disconnect does not need the 90-second timeout to fire separately. +2. `devices.lastSeenAt` is updated again (unthrottled, same as connect). +3. **Gateway-shutdown short-circuit**: if the process is shutting down + (`SIGTERM`/`SIGINT` received, or Socket.IO's own shutdown reasons), the + handler returns immediately *without* touching presence state at all. The + comment in `index.ts` is explicit about why: "we must NOT wipe presence — + surviving devices re-assert via heartbeat and Redis TTLs." A rolling + restart should not cause every connected user to flicker offline. +4. Otherwise: `unregisterPresenceSocket`, `cleanupStaleSockets`, and — only + if this was the device's last socket — `setOffline`. If that leaves the + user's hash empty (`fullyOffline`), the offline broadcast is scheduled + (not fired immediately — see next section). + +## Multi-device aggregation + +Because `presence:user:{userId}` is a per-device hash, a user with two +connected devices (e.g. phone + laptop) has two fields in that one key. Each +device's connect/heartbeat/disconnect only ever touches its own field (and +its own `presence:user:{userId}:device:{deviceId}` key and own socket-mapping +keys) — devices never read or write each other's entries. + +The aggregation rule is purely "any field present ⇒ online": there is no +weighting, no "most recent device wins," and no per-device online/offline +event exposed externally — only the user-level transition (hash +empty↔non-empty) is ever broadcast. A client reconnecting a second device +while the first is already connected does **not** re-trigger a `user_online` +broadcast, since `setOnline` returns `false` (`wasOnline` was already `true`) +in that case. + +The Postgres fallback (`deriveDevicePresence`) aggregates differently: it +checks whether **any** non-revoked device for the user has `lastSeenAt` +within the last `DEVICE_PRESENCE_WINDOW_MS` (90,000 ms — the same 90-second +figure as `PRESENCE_TTL`, defined as its own local constant in `presence.ts` +rather than shared), and if none qualify, returns the single most recent +`lastSeenAt` across all of that user's non-revoked devices as `lastSeen`. + +## Transition-broadcast behavior + +Going offline is **debounced**, going online is not. This asymmetry exists +specifically to avoid a disconnect-then-immediately-reconnect blip (page +refresh, brief network hiccup, mobile app backgrounding) producing a visible +`user_offline` followed by `user_online` pair to every other member of a +shared conversation. + +- `scheduleOfflineBroadcast(userId, broadcastFn)` — called only once the + caller has already confirmed zero remaining live devices/sockets for the + user (the underlying Redis state removal via `setOffline` has already + happened synchronously; only the *broadcast* is deferred). It sets a timer + for `PRESENCE_OFFLINE_GRACE_MS` (env-configurable, **default 5,000 ms**, + falling back to the default on a missing/invalid/negative value), replacing + any previous pending timer for the same user. The timer is `.unref()`'d so + it never keeps the process alive on its own. +- `cancelPendingOfflineBroadcast(userId)` — called at the **start** of the + connect handler, before the new connection's own online-check runs. If a + reconnect lands within the grace window of a very recent disconnect (same + user, any device — not necessarily the same device that disconnected), the + pending offline broadcast is cancelled and the reconnect produces **no** + `presence_update` at all: nothing was ever announced offline, so there is + nothing to announce online again either (`becameOnline && !cancelledPendingOffline` + gates the online broadcast specifically to suppress this). + +When the offline broadcast does fire (grace window elapsed with no +reconnect), it re-reads conversation memberships fresh (not the stale +membership list captured at disconnect time) and calls +`deriveDevicePresence` to compute the `lastSeen` value to report, rather than +reusing any value cached from the live connection. + +## Privacy gating + +Two per-user boolean columns (`users.presenceVisible`, +`users.lastSeenVisible` — see +[`contracts-database-schema.md`](contracts-database-schema.md#users)) gate +what's ever broadcast or returned, independent of the Redis/DB mechanics +above: + +- `presenceVisible` (default `false`) gates **all** presence broadcasting for + that user — both the connect-time `user_online`/`presence_update` emission + and the disconnect-time scheduled offline broadcast check this flag before + emitting anything. `GET /users/:id/presence` returns `{ online: "unknown" }` + for a user with `presenceVisible: false`, regardless of actual connection + state. +- `lastSeenVisible` (default `false`) gates only whether a `lastSeen` + timestamp is **included** in an otherwise-permitted presence payload — + a user can be visibly online/offline without ever disclosing exactly when + they were last active. + +Both checks read the *current* value of these flags at the moment of +broadcast/query, not at connect time — a user who flips `presenceVisible` off +mid-session stops appearing in future presence broadcasts without needing to +reconnect (see the dedicated visibility-change broadcast in +`routes/users.ts`'s profile-update handler, which emits a synthetic +online/offline transition when `presenceVisible` itself changes). + +## Known caveats, verified against the current code + +- **`presence:user:{userId}` has no TTL of its own.** Every other + presence-related Redis key in the table above carries `PRESENCE_TTL`. + This one does not — it is only ever shrunk/deleted by an explicit + `hdel`/`del` inside `setOffline`, `markDeviceOffline`, or + `cleanupStaleSockets`'s indirect effects. If a gateway process crashes (as + opposed to a graceful shutdown or a normal socket disconnect) between + `setOnline` and any of those cleanup calls, that device's field can persist + in the hash indefinitely — there is no background sweep that expires it by + age. `isOnline()` would then report the user online based on a stale field + with no live socket or process behind it, until some other event + (`reconcileBoot` on the next gateway restart, a later `setOffline` call for + the same device, or a device revocation) happens to clear it. + `reconcileBoot` itself only rebuilds room membership from + `presence:sockets:*` and prunes *socket-mapping* keys for dead sockets — it + does not independently audit or repair the `presence:user:{userId}` + device-presence hash. +- **The per-device TTL (`PRESENCE_TTL`) is the only thing that would + eventually make a truly-dead device's own hash key disappear**, but that + key (`presence:user:{userId}:device:{deviceId}`) expiring does **not** + cascade into removing the corresponding field from + `presence:user:{userId}` — the two are updated independently by the same + call sites, not linked by Redis expiry notifications. A production + deployment that cares about this edge case exactly would need either a + periodic reconciliation pass or Redis keyspace-notification-driven cleanup, + neither of which exists in the current code. +- **`isOnline()` does not re-validate recency.** It is `HLEN > 0`, not a + check that any field's value is within `PRESENCE_TTL` of now — recency + enforcement is delegated entirely to the per-device key's TTL and the + explicit cleanup paths, not re-checked at read time. +- **The in-file header comment block in `presence.ts` is stale** — it + describes a 60-second TTL and a more simplified "add socketId to a set" + model closer to an earlier version of this module. The actual + implementation (device-hash + per-device key + three tiers of + socket-mapping keys, 90-second TTL) supersedes it; this document reflects + the code, not the comment. diff --git a/apps/backend/docs/concepts-protocol-negotiation.md b/apps/backend/docs/concepts-protocol-negotiation.md new file mode 100644 index 0000000..37ee061 --- /dev/null +++ b/apps/backend/docs/concepts-protocol-negotiation.md @@ -0,0 +1,309 @@ +# Device capability and E2EE protocol negotiation + +Clicked encrypts every message once per recipient device, and not every device speaks the +same encryption protocol. A device that shipped before the Signal work landed understands +only the Phase-1 sealed box; a current one also understands the Double Ratchet; an MLS +group member understands a third construction. Negotiation is how the system picks, for +each pair of devices, the strongest construction both of them can actually use — without a +flag day, and without a newer client silently breaking an older one. + +This document covers the negotiation layer itself: +[`src/lib/capabilities.ts`](../src/lib/capabilities.ts) (what a device advertises and how a +protocol is chosen) and [`src/services/e2eeProtocol.ts`](../src/services/e2eeProtocol.ts) +(what the server does when a sender claims a protocol on an envelope). + +The migration this machinery exists to serve — the rollout order, the per-pair cutover +timeline, and the client-side ratchet work — is documented separately in +[`signal-migration.md`](./signal-migration.md). Read this document for the mechanism, that +one for the plan. + +## Contents + +- [The capability payload](#the-capability-payload) +- [Advertising capabilities at registration](#advertising-capabilities-at-registration) +- [`normalizeCapabilities` and the baseline default](#normalizecapabilities-and-the-baseline-default) +- [Picking a mutually supported protocol](#picking-a-mutually-supported-protocol) +- [The per-envelope `protocol` column](#the-per-envelope-protocol-column) +- [`checkEnvelopeProtocols` — enforcement on the way in](#checkenvelopeprotocols--enforcement-on-the-way-in) +- [The `protocol_mismatch` rejection, and what a client does with it](#the-protocol_mismatch-rejection-and-what-a-client-does-with-it) +- [Why this enables a staged rollout](#why-this-enables-a-staged-rollout) +- [Implementation references](#implementation-references) + +## The capability payload + +`devices.capabilities` is a `jsonb` column holding a small document the device publishes +about itself. It is validated by `DeviceCapabilitiesSchema`, and every field is optional: + +```jsonc +{ + "protocols": ["sealed_box", "signal"], + "ciphersuites": ["MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519"], + "fileTransfer": ["file-v1"], +} +``` + +| Field | Type | Meaning | +| -------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `protocols` | `string[]` | Messaging encryption protocols this device can **decrypt**. Known values are `sealed_box`, `signal`, `mls` (`KNOWN_PROTOCOLS`). `sealed_box` is the Phase-1 ECDH + HKDF + AES-256-GCM envelope. | +| `ciphersuites` | `string[]` | MLS/Signal ciphersuite identifiers. Only consulted when `mls` is present in `protocols`. | +| `fileTransfer` | `string[]` | File-encryption scheme versions, e.g. `file-v1`. Independent of the messaging protocol; queried through `supportsFileTransfer(capabilities, version)`. | + +Two properties of the shape are deliberate and should be preserved: + +- **Everything is optional, and the whole document is optional.** A missing field is not an + error; it means "assume the default for this field". +- **Unrecognised values are preserved, not rejected.** A device may advertise a protocol + name this server has never heard of. `selectProtocol` ignores it — it only ever matches + against names in `PROTOCOL_PRIORITY` — but the value survives in the column. This is what + makes negotiation compatible in both directions: a newer client against an older server + degrades to a protocol they share instead of failing to register. + +`capabilities` is `NOT NULL` with the sealed-box baseline as its schema-level default +(`src/db/schema.ts`), so a device row can never exist without one. + +## Advertising capabilities at registration + +A device advertises its capabilities on the two paths that create or refresh a device row. +Both accept `capabilities` as an optional member of the device object (`DeviceSchema` in +`src/schemas/auth.schemas.ts`): + +| Path | Behaviour | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `POST /auth/verify` | New device → the value is normalised and inserted. Existing device (matched on identity key) → the value is normalised and **updated in place**. | +| `POST /devices/link/verify` | The device-linking registration path. Same treatment. | + +The update-in-place case is the **upgrade path**. A client that gains Signal support does +not re-register a new identity: it re-verifies with a larger `protocols` array, and the next +negotiation for that device picks the stronger protocol. Omitting `capabilities` entirely on +a re-verify leaves the stored document untouched — the update only sets the field when the +client actually sent one, so an older build of the same client cannot accidentally downgrade +a device's advertised capabilities to its own. + +Capabilities are read back by senders from three places, all of which run the stored value +through `normalizeCapabilities` first: + +| Endpoint | What it returns | +| ---------------------------------- | ------------------------------------------------------------------------------------- | +| `GET /devices` | The caller's own devices, each with its normalised `capabilities`. | +| `GET /user-devices/:id/public-key` | A single peer device's identity key plus its normalised capabilities. | +| `GET /conversations/:id/devices` | Every active member device, with `capabilities` **and** a `negotiatedProtocol` field. | + +`negotiatedProtocol` is `selectProtocol(callerDevice, thatDevice).protocol` computed +server-side. It exists so a client does not have to re-implement the preference order to +agree with the server about the answer — a client that computes its own and disagrees is +exactly the client whose sends get rejected. + +## `normalizeCapabilities` and the baseline default + +```ts +export const BASELINE_PROTOCOL: KnownProtocol = 'sealed_box'; +``` + +`normalizeCapabilities(raw)` turns anything — `null`, `undefined`, a partial object, a +malformed one — into a concrete `DeviceCapabilities`: + +1. The value is parsed with `DeviceCapabilitiesSchema.safeParse(raw ?? {})`. **A parse + failure is not an error**: it returns a copy of `DEFAULT_CAPABILITIES`. +2. An empty or missing `protocols` array becomes `[BASELINE_PROTOCOL]`. +3. `ciphersuites` and `fileTransfer` default to empty arrays. + +The result is that **every device supports `sealed_box`, whether it said so or not**. This is +the single assumption the whole design rests on, and it is what makes clients predating the +`capabilities` field work unchanged: + +- Rows written before the column existed carry the schema default, which is the baseline. +- A client that never sends `capabilities` gets the baseline. +- A client that sends a document this server cannot parse gets the baseline rather than a + `500` at registration. + +Because the baseline is universal, `selectProtocol` never has to return `null` and no send +path needs a "these two devices cannot talk to each other" branch. + +## Picking a mutually supported protocol + +`selectProtocol(a, b)` normalises both capability documents, then walks a fixed preference +order and returns the first protocol present in both sets: + +```ts +const PROTOCOL_PRIORITY = ['mls', 'signal', 'sealed_box']; +``` + +The order is strongest-first, so an overlap of `{sealed_box, signal}` resolves to `signal`, +not to the weaker option the two happen to share. If the loop finds nothing — only possible +when a device advertises a protocol set that excludes the baseline — the function falls back +to `{ protocol: 'sealed_box', ciphersuite: null }`. + +For `mls`, and only for `mls`, a ciphersuite is negotiated alongside the protocol: +`selectCiphersuite` returns the first entry of `a.ciphersuites` that also appears in +`b.ciphersuites`, preserving the caller's preference order, or `null` when there is no +overlap. + +**Negotiation is per device pair, not per conversation.** The envelope model already +encrypts once per recipient device, so a Signal-capable pair inside a group is not held back +by a third member still running an old client: that pair uses Signal today and the laggard +keeps the sealed box until it upgrades. +`protocolsForRecipients(senderDeviceId, recipientDeviceIds)` is the batch form used by +callers that need the answer for a whole fan-out at once; device ids that do not resolve map +to `BASELINE_PROTOCOL`. + +## The per-envelope `protocol` column + +`message_envelopes.protocol` is an `e2ee_protocol` enum column, +`NOT NULL DEFAULT 'sealed_box'`, whose values mirror `KNOWN_PROTOCOLS`. It is written by +`insertMessageEnvelopes` (`src/lib/messageFanout.ts`), which is shared by the REST send path +and both socket send paths so the default cannot drift between them. + +**Why per envelope rather than per device.** The two columns answer different questions, and +only one of them is stable over time: + +- `devices.capabilities.protocols` says what a device can decrypt **right now**. It is + mutable — that is the entire point of the upgrade path above. +- `message_envelopes.protocol` says what a particular ciphertext **was actually built + with**. It has to stay true forever. + +If the protocol were recorded only on the device, then the moment a device advertised +`signal`, every envelope ever written for it would look like a Signal ciphertext. All of its +sealed-box history would become undecryptable — not because the key material is gone, but +because the reader would pick the wrong construction. Recording it per envelope is what lets +pre-cutover history keep decrypting on the Phase-1 path indefinitely, and it is why the +migration that added the column used a defaulted `NOT NULL`: the default backfills every +pre-existing row with the construction those rows really used. + +It is also per envelope rather than per **message** because one message fans out to many +devices, and those devices are not all on the same side of the cutover. The same plaintext +can legitimately be a Signal ciphertext for one recipient device and a sealed box for +another, within a single send. + +The column is read back on the paths that hand a ciphertext to a client — the delivery +pipeline (`src/services/deliveryPipeline.ts`) and the sync endpoint (`src/routes/sync.ts`) — +so the recipient selects its decryption path from the data rather than guessing from the +bytes. + +## `checkEnvelopeProtocols` — enforcement on the way in + +Negotiation tells an honest sender what to use. It does not stop a patched, buggy, or +compromised one from claiming something else, so the server re-derives the answer on every +send. `checkEnvelopeProtocols(senderDeviceId, envelopes)` loads the sender device's +capabilities and every named recipient device's capabilities in two queries, then checks each +envelope for two distinct failures: + +| Reason | Status | Condition | +| -------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------- | +| `unsupported_by_recipient` | `400` | The declared protocol is **not** in the recipient's advertised `protocols`. The ciphertext would be undecryptable. | +| `downgrade` | `409` | The declared protocol is supported, but is not the one `selectProtocol` picks for this pair — both sides can do better. | + +Details worth knowing: + +- **Envelopes naming a device that does not resolve are skipped**, not rejected. The send + paths already drop those envelopes before persisting, and failing a whole batch because a + device was revoked between the client's device-list fetch and its send would turn a routine + race into a user-visible error. +- **Every violation in the batch is reported**, not just the first, so a client can fix one + send rather than discovering the problem one device at a time. +- **When a batch contains both kinds, the status is `400`.** Undecryptability is the more + specific and more serious failure, so it decides the code. +- The check runs **before anything is persisted**, on `POST /messages` + (`src/routes/messages.ts`) and on the `send_message` socket handler + (`src/socket/messaging.ts`). Envelopes with no explicit `protocol` default to + `BASELINE_PROTOCOL` in the schema (`EnvelopeSchema`), so an older client that never sends + the field is checked as a sealed-box sender — which is exactly what it is. + +## The `protocol_mismatch` rejection, and what a client does with it + +Over REST the failure is the status code from the table above with a body of: + +```jsonc +{ + "error": "Envelope protocol is not supported by the recipient device", + "violations": [ + { + "recipientDeviceId": "…", + "declared": "signal", + "expected": "sealed_box", + "reason": "unsupported_by_recipient", + }, + ], +} +``` + +Over the socket the same information arrives as an `error` event with +`event: 'protocol_mismatch'`, carrying `code` (`400` or `409`), `message`, and the identical +`violations[]` array. See [`contracts-error-catalog.md`](./contracts-error-catalog.md) for +where this sits among the other socket error events. + +**`protocol_mismatch` is not retryable as sent.** Resending the same envelope set produces +the identical rejection. The client should: + +1. **Re-fetch the recipient device list.** `GET /conversations/:id/devices` is the one call + that returns both `capabilities` and the server's own `negotiatedProtocol` per device. The + most common cause of a mismatch is a stale device list: the peer upgraded, or a new device + joined, after the sender last looked. +2. **Re-encrypt each envelope with the protocol the server names.** `violations[].expected` + already says what should have been used for that specific device, so a client can act on + the response without a second round trip if it trusts its own key state. +3. **Resend the whole message.** Nothing was persisted, so this is a fresh send rather than a + retry of a partial one. Reuse the same `messageId` — the send paths are idempotent on it, + see [`concepts-replay-protection.md`](./concepts-replay-protection.md). +4. **Do not fall back to a weaker protocol.** A `409 downgrade` means precisely that the + client already tried that. Downgrading again in response to a downgrade rejection is a + retry loop whose only successful outcome is the thing the check exists to prevent. + +A `downgrade` violation naming a device the client believes cannot do better is a signal that +the client's cached capability document is stale, not that the server is wrong — the server +read the row a moment ago. + +## Why this enables a staged rollout + +Together the four pieces give a sealed-box → Signal rollout with no coordinated cutover: + +1. **Old clients keep working with no change at all.** They never send `capabilities`, so + they are read as sealed-box-only. They never send `protocol`, so their envelopes default + to `sealed_box`. Both defaults are applied server-side; nothing about an old client has to + know this feature exists. +2. **A new client is useful before anyone else upgrades.** It advertises + `["sealed_box", "signal"]`. Against an old peer, `selectProtocol` finds only `sealed_box` + in common and the pair keeps working exactly as before. Against another new peer, the same + code picks `signal` — with no server config change, no feature flag, and no + per-conversation gate. +3. **Progress is monotonic per pair, and independent across pairs.** A pair cuts over the + moment both sides advertise the stronger protocol, and the `downgrade` check stops them + from silently sliding back. One user's laptop can be on Signal with one contact and on the + sealed box with another, in the same conversation, at the same time. +4. **History is never re-encrypted.** Every envelope carries the construction that produced + it, so a cutover changes what is written next and never what was written before. There is + no migration job to run, and no window during which old messages are unreadable. +5. **The same machinery carries the next protocol.** Adding MLS meant adding a name to + `KNOWN_PROTOCOLS`, a value to the enum, and an entry at the top of `PROTOCOL_PRIORITY`. The + negotiation, enforcement, and recording paths did not change. + +The cost of all this is one rule for contributors: **a new encryption construction must be +added to `KNOWN_PROTOCOLS`, to the `e2ee_protocol` enum, and to `PROTOCOL_PRIORITY` in the +same change.** A protocol missing from the enum cannot be recorded; a protocol missing from +the priority list can be advertised but will never be selected, which presents as "both +devices support Signal and it is still sending sealed boxes". + +## Implementation references + +| Concern | File | +| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Capability shape, defaults, selection | [`src/lib/capabilities.ts`](../src/lib/capabilities.ts) | +| Enforcement and batch negotiation | [`src/services/e2eeProtocol.ts`](../src/services/e2eeProtocol.ts) | +| `capabilities` and `protocol` columns | [`src/db/schema.ts`](../src/db/schema.ts) | +| Registration input schema | [`src/schemas/auth.schemas.ts`](../src/schemas/auth.schemas.ts) | +| Envelope input schema and `protocol` default | [`src/schemas/message.schemas.ts`](../src/schemas/message.schemas.ts) | +| Writing the per-envelope protocol | [`src/lib/messageFanout.ts`](../src/lib/messageFanout.ts) | +| REST send enforcement | [`src/routes/messages.ts`](../src/routes/messages.ts) | +| Socket send enforcement | [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| Capability read-back endpoints | [`src/routes/devices.ts`](../src/routes/devices.ts), [`src/routes/userDevices.ts`](../src/routes/userDevices.ts), [`src/routes/conversations.ts`](../src/routes/conversations.ts) | +| Tests | `src/__tests__/e2eeProtocol.test.ts`, `src/__tests__/signalMigration.routes.test.ts`, `src/__tests__/signalInvariants.socket.test.ts` | + +## Related documents + +- [Phase-1 → Signal migration](./signal-migration.md) — the rollout plan this mechanism + serves. +- [Error code and response catalog](./contracts-error-catalog.md) — `protocol_mismatch` + alongside every other error the backend returns. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — why resending + a rejected message with the same `messageId` is safe. +- [Delivery fan-out and receipts](./concepts-delivery-fanout.md) — how the envelopes this + document validates reach devices. diff --git a/apps/backend/docs/concepts-replay-protection.md b/apps/backend/docs/concepts-replay-protection.md new file mode 100644 index 0000000..aa7281a --- /dev/null +++ b/apps/backend/docs/concepts-replay-protection.md @@ -0,0 +1,287 @@ +# Replay protection and event idempotency + +A realtime client retries. It reconnects mid-send, it resumes after a suspend, it fires the +same action twice because a tap registered twice — and a hostile client replays a captured +frame on purpose. The backend answers this in **two independent layers**, at two different +levels of the stack, and confusing them is the usual source of "why did my duplicate still +create a row" questions. + +- **Transport level:** every enveloped socket event carries an `eventId`, and + [`src/services/replay-protection.service.ts`](../src/services/replay-protection.service.ts) + drops the second and later arrivals of that id from the same device inside a TTL window. +- **Message level:** every message carries a client-generated `messageId`, and the send paths + refuse to insert a second row for an id that already exists, acknowledging the original + instead. + +This document covers both, and the dispatcher path that ties them together. + +## Contents + +- [Why both layers exist](#why-both-layers-exist) +- [Layer 1 — transport-level `eventId` dedup](#layer-1--transport-level-eventid-dedup) + - [The device-scoped key](#the-device-scoped-key) + - [The TTL](#the-ttl) + - [Fail-open when Redis is unavailable](#fail-open-when-redis-is-unavailable) + - [`dispatch_ack`](#dispatch_ack) +- [Layer 2 — message-level `messageId` idempotency](#layer-2--message-level-messageid-idempotency) +- [Every event goes through the dispatcher](#every-event-goes-through-the-dispatcher) +- [Operational notes](#operational-notes) +- [Implementation references](#implementation-references) + +## Why both layers exist + +They protect different things and neither subsumes the other. + +| | Transport layer (`eventId`) | Message layer (`messageId`) | +| ------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------- | +| Scope | Every socket event type — reads, typing, receipts, joins, sends | Message-creating sends only | +| Identity | Per delivery attempt of an event | Per message, stable across attempts and across transports | +| Storage | Redis key with a TTL | The `messages` primary key — permanent | +| Window | `REPLAY_PROTECTION_TTL_SECONDS` (default 5 minutes) | Forever | +| Behaviour on repeat | Handler is not run at all; `dispatch_ack { duplicate: true }` | Handler runs, finds the row, re-acknowledges the original `createdAt` | +| Fails | Open (allows the event through when Redis is down) | Closed (the row either exists or it does not) | + +**The transport layer cannot be the only one.** Its window is finite and its state is in +Redis. A retry after the TTL has expired, a retry from a different device, a retry that +arrives over REST instead of the socket, or any retry at all while Redis is down, gets past +it. If duplicate suppression for messages lived only here, any of those would produce a +second copy of a message in the conversation. + +**The message layer cannot be the only one either.** Most events are not message sends. +`message_read`, `typing_start`, `join_room`, `heartbeat` and the rest have no durable id to +be idempotent on, and re-running them on a replayed frame costs real work — extra fan-out, +extra receipts, extra queries — even where the end state happens to be the same. The +transport layer is also the only one that sees a _deliberate_ replay of a captured frame, +because a captured frame replays the whole envelope, `messageId` included; the message layer +would treat that as a retry and cheerfully re-ack it. + +The practical rule: **`eventId` is per attempt and must be freshly generated for every +emit; `messageId` is per message and must be reused across retries of the same message.** +Getting these the wrong way round is what produces "my resend was silently ignored" +(reused `eventId`) or "my retry created a duplicate message" (fresh `messageId`). + +## Layer 1 — transport-level `eventId` dedup + +`isReplay(redis, deviceId, eventId)` performs an atomic check-and-mark: + +```ts +const result = await redis.set(key, '1', 'EX', ttl, 'NX'); +return result === null; // null → the key already existed → this is a replay +``` + +`SET … NX` is one round trip and it is atomic, so two frames racing on the same node — or on +two different gateway nodes sharing one Redis — cannot both observe "not seen yet". There is +no read-then-write window to lose. + +Return values are stated in the positive: `false` means _not_ a replay, process it; `true` +means drop it. The function also returns `false` — process it — when `deviceId` or `eventId` +is missing or blank, since there is nothing to key on. In practice the envelope schema +requires a non-empty `eventId`, so a blank one is rejected as a malformed envelope before the +replay check is reached. + +`markSeen(redis, deviceId, eventId)` exists to mark an id without consuming the check; it is +a testing and debugging affordance, not part of the live path. `isReplay` already marks +atomically. + +### The device-scoped key + +``` +replay:{deviceId}:{eventId} +``` + +`getReplayProtectionRedisKey()` builds it and is exported so tests and debugging can compute +the same string rather than duplicating the format. + +**The `deviceId` component is load-bearing, not decoration.** `eventId` is generated by the +client, and nothing forces two clients to agree on a generator. A user's phone and laptop can +legitimately produce the same id — a counter, a low-entropy uuid, a `test-evt-1` left in a +build. With a global `replay:{eventId}` key, whichever device emitted first would win and the +other device's genuine, first-time event would be silently dropped as a "replay", producing a +handler that never runs and no error anywhere. Scoping the key to the device makes each +device's id space its own: one device's replayed id cannot block another device's legitimate +event. + +It also matches the threat being defended. A replay attack is a frame captured from one +device and re-sent; the attacker cannot change the authenticated `deviceId` on the socket, +which is taken from the JWT (`socket.auth.deviceId`), not from the payload. Cross-device +collisions are noise, not attacks. + +### The TTL + +| Setting | Value | +| -------------- | --------------------------------------------------------- | +| Env var | `REPLAY_PROTECTION_TTL_SECONDS` | +| Default | `300` (5 minutes) | +| Accepted range | `1` … `86400` seconds (1 day) | +| Invalid values | Non-numeric, out-of-range, or empty → the default is used | +| Read | Per call, not cached at import | + +The TTL is what keeps the key space bounded: entries expire on their own, so there is no +sweeper job and no unbounded growth from a device that sends a million events. It also bounds +the protection — an event replayed after the window is not detected here, which is why the +message layer exists. + +The default is tuned against the dispatcher's own freshness check rather than picked +arbitrarily. `SOCKET_EVENT_MAX_AGE_MS` (default `300000`, also 5 minutes) rejects an envelope +whose `timestamp` is older than the window, so a frame old enough to have fallen out of the +replay set is already stale enough to be refused for that reason instead. **If you raise +`SOCKET_EVENT_MAX_AGE_MS`, raise `REPLAY_PROTECTION_TTL_SECONDS` with it**, or you open a gap +in which a frame is accepted as fresh but no longer remembered as seen. + +An out-of-range value degrades to the default instead of throwing, in keeping with how the +rest of the backend reads tuning env vars: a typo in the environment must not stop the +gateway from booting. + +### Fail-open when Redis is unavailable + +`isReplay` returns `false` — process the event — in two cases: + +- `redis` is `null`, meaning `REDIS_URL` was never configured, and +- the `SET` throws, which covers a connection loss, a timeout, or a Redis in a failed state. + The error is logged once per call at `warn` level and the event proceeds. + +This is deliberate. Replay protection is a **hardening** layer, not a correctness dependency: +the durable guarantees for messages come from the `messageId` primary key, which is in +Postgres and does not care what Redis is doing. Failing closed would mean that losing Redis +takes the entire realtime surface offline — nobody can send, read, or type — in exchange for +closing a replay window that requires an attacker to already hold a captured, authenticated +frame. Trading total availability for that is the wrong side of the trade for this system. + +The consequence to be aware of when reading tests: **with Redis mocked to `null`, the dedup +does nothing.** A test that reuses an `eventId` passes against a `null` Redis and starts +failing the moment someone gives the suite an `ioredis-mock` instance. Generate a fresh +`eventId` per emit regardless of what Redis the test has — see +[`testing.md`](./testing.md#driving-socket-handlers). + +## `dispatch_ack` + +Every event that reaches the dispatcher gets exactly one `dispatch_ack` back, and the flag +says which layer answered it: + +```jsonc +{ "eventId": "…", "duplicate": false } // first occurrence — the handler ran +{ "eventId": "…", "duplicate": true } // replay — the handler was not run +``` + +`duplicate: true` is **an acknowledgement, not an error.** The event was already processed, +so the client's intent is satisfied and it should clear the item from its outbox exactly as +it would on `duplicate: false`. Treating it as a failure and retrying produces a loop that +gets the same answer until the TTL expires, at which point the retry is processed for real — +which is the one outcome the client was trying to avoid. + +Note what does _not_ produce an ack: a malformed envelope, an unknown event type, a stale +timestamp, and an unauthenticated socket all emit an `error` envelope instead, and a handler +that throws is logged server-side with no ack at all. A client waiting on `dispatch_ack` +therefore needs a timeout, and must treat `error` as terminal for that event. + +## Layer 2 — message-level `messageId` idempotency + +`messageId` is generated by the client and is the primary key of the `messages` row. Every +send path performs the same check before inserting: + +```ts +const existing = await db.query.messages.findFirst({ + where: eq(messages.id, messageId), + columns: { createdAt: true }, +}); +if (existing) { + /* acknowledge the original, insert nothing */ +} +``` + +| Path | On an id that already exists | +| ---------------------------- | ------------------------------------------------------------- | +| `POST /messages` | `200` with `{ messageId, createdAt }` (a fresh send is `201`) | +| `send_message` (socket) | `message_ack` with the **original** `createdAt` | +| `edit_message` (socket) | `message_ack` with the original `createdAt` | +| `send_file_message` (socket) | `message_ack` with the original `createdAt` | + +Three properties matter: + +- **It spans transports.** A client that sends over the socket, loses the connection before + the ack, and retries over REST with the same `messageId` gets the original message back + rather than a duplicate. The transport layer cannot do this — the REST request has no + `eventId` at all. +- **It has no window.** The check is against the durable row, so it holds a day later as + readily as a second later. +- **The distinction between `200` and `201`, and the returned `createdAt`, are the client's + signal that the retry was absorbed** — the original timestamp is returned, not the retry's, + so message ordering never shifts because of a retry. + +Because the check is `SELECT` then `INSERT` rather than an upsert, two genuinely simultaneous +sends of the same `messageId` can both pass the check; the second insert then fails on the +primary key and the request returns a `500` rather than corrupting anything. That is a +tolerable outcome for a case that requires a client to race itself, and the row count stays +correct either way. + +## Every event goes through the dispatcher + +The order of operations in `EventDispatcher.listen()` is fixed, and the replay check sits +late in it on purpose — there is no point spending a Redis round trip on a frame that is not +going to be processed anyway: + +1. **Authenticated?** Otherwise `error` — the socket must be authenticated before any event. +2. **Valid envelope?** `EventEnvelopeSchema` requires `eventId`, `type`, and a positive + integer `timestamp`. Otherwise `error`. +3. **Known event type?** Unknown types are discarded with an `error`, so an unrecognised name + cannot reach a handler. +4. **Fresh timestamp?** Within `SOCKET_EVENT_MAX_AGE_MS` in the past and + `SOCKET_EVENT_MAX_FUTURE_SKEW_MS` (default `30000`) in the future. Otherwise `error`. +5. **Replay?** `isReplay(redis, socket.auth.deviceId, envelope.eventId)`. If so, emit + `dispatch_ack { duplicate: true }` and stop. +6. **Dispatch** to the registered handler, then `dispatch_ack { duplicate: false }`. + +`dispatcher.register(type, handler)` puts the handler into a map that only step 6 can reach. +**There is no raw `socket.on(type, …)` fallback for registered types** — `listen()` attaches +exactly one listener, for the `dispatch` event — so a handler cannot be reached without +passing every check above. That is why the dedup can be described as covering every event +rather than as something each handler opts into, and it is why a test must drive handlers by +emitting a `dispatch` envelope rather than by triggering a raw event name. + +**One handler is not on this path today:** `send_file_message` is still attached with a raw +`socket.on` in `src/socket/messaging.ts`. It therefore gets no envelope validation, no +timestamp freshness check, and no `eventId` dedup; its duplicate suppression comes entirely +from the `messageId` check in layer 2. Anything that moves it onto `dispatcher.register` +inherits all of the above for free, and any _new_ handler must be registered through the +dispatcher. + +## Operational notes + +- **Keys are ephemeral and safe to drop.** `replay:*` keys carry no data beyond "this id was + seen", and losing them fails open by construction. A Redis flush costs at most a window in + which a replayed frame would be accepted; it never loses a message. +- **Sizing.** One key per event per device for the TTL window. At `n` events per second + across the fleet and a 300-second TTL, the steady-state key count is roughly `300n`, each a + short string with a one-byte value. +- **A cluster shares the state.** All gateway nodes talk to the same Redis, so a frame + replayed against a different node than the original is still caught. This is the same + Redis the gateway uses for pub/sub fan-out and presence — see + [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md). +- **Debugging a dropped event.** A replay logs at `debug` with `deviceId`, `eventId`, and + `type`. If a handler "never ran" and no error came back, check for a `dispatch_ack` with + `duplicate: true`; the usual cause is a client reusing an `eventId` across retries rather + than generating a fresh one. + +## Implementation references + +| Concern | File | +| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `isReplay`, `markSeen`, key and TTL | [`src/services/replay-protection.service.ts`](../src/services/replay-protection.service.ts) | +| Dispatch order and `dispatch_ack` | [`src/socket/dispatcher.ts`](../src/socket/dispatcher.ts) | +| Envelope schema and event registry | [`src/lib/eventEnvelope.ts`](../src/lib/eventEnvelope.ts) | +| Socket send idempotency | [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| REST send idempotency | [`src/routes/messages.ts`](../src/routes/messages.ts) | +| Redis client and degradation | [`src/lib/redis.ts`](../src/lib/redis.ts) | +| Tests | `src/services/replay-protection.service.spec.ts`, `src/socket/dispatcher.spec.ts`, `src/__tests__/dispatcher.test.ts` | + +## Related documents + +- [Gateway architecture](./concepts-gateway-architecture.md) — the socket lifecycle the + dispatcher sits inside, and what else Redis carries. +- [WebSocket events](./api-websocket-events.md) — every event type, including `dispatch_ack`. +- [Error code and response catalog](./contracts-error-catalog.md) — the `error` envelopes the + checks above emit, and which are retryable. +- [Backend testing guide](./testing.md) — driving handlers through `dispatch` without + tripping the dedup. +- [Backend caching reference](./concepts-caching.md) — the other place Redis is treated as an + optimisation rather than a correctness dependency. diff --git a/apps/backend/docs/concepts-stellar-listener.md b/apps/backend/docs/concepts-stellar-listener.md new file mode 100644 index 0000000..16105bc --- /dev/null +++ b/apps/backend/docs/concepts-stellar-listener.md @@ -0,0 +1,124 @@ +# Stellar chain listener and on-chain reconciliation + +How `services/stellarListener.ts` watches Soroban contract events, writes +`token_transfers` / `treasury_proposals`, tracks its place in the ledger, and +behaves after downtime or an unreachable RPC. + +The listener is started from `index.ts` only when both `STELLAR_RPC_URL` and +`TOKEN_TRANSFER_CONTRACT_ID` are set. Missing either logs + +```text +[stellar-listener] STELLAR_RPC_URL or TOKEN_TRANSFER_CONTRACT_ID unset; listener disabled. +``` + +and leaves the API up. `GROUP_TREASURY_CONTRACT_ID` is optional: when set, a +second fetcher is attached for treasury events. + +`runForever` never rethrows. RPC and DB errors are logged inside the loop so +a dead chain cannot take the HTTP/WebSocket process down. + +## Watched events and database writes + +Two independent pollers share the same loop (default interval 5s). + +### `token_transfer` — topic `transfer` + +`buildRpcFetcher` calls Soroban RPC `getEvents` filtered to +`TOKEN_TRANSFER_CONTRACT_ID` and topic `transfer`. Each event is mapped to: + +| RPC field | Persisted column (`token_transfers`) | +| --- | --- | +| `txHash` | `tx_hash` (unique) | +| `value.to` | `recipient_address` | +| `value.amount` | `amount` (decimal string) | +| `value.memo` | `memo` (hex, optional) | + +`conversation_id` / `sender_id` are filled by decoding `memo` as a message +UUID and looking up `messages`. If that misses, the writer falls back to an +arbitrary existing conversation and user; if the database has neither, the +event is dropped. + +Write path: `INSERT … ON CONFLICT (tx_hash) DO UPDATE SET created_at = now()`. +A replayed ledger page refreshes `created_at` on the same row; it does not +insert a second transfer. + +### `group_treasury` — proposal lifecycle + +`buildTreasuryRpcFetcher` watches `GROUP_TREASURY_CONTRACT_ID` for: + +| Contract event | Row status | +| --- | --- | +| `proposal_created` | `active` | +| `proposal_approved` | `approved` | +| `proposal_rejected` | `rejected` | +| `proposal_executed` | `executed` | +| `proposal_expired` | `expired` | + +Persistence is an **update** on `(contract_id, proposal_id)` (unique index +`treasury_proposals_contract_proposal_idx`), copying `approvals` / +`rejections` when present. If no matching row exists, the event is ignored +(`if (!row) return`) — the listener does not insert treasury proposals. After +a successful update it emits `treasury_proposal_updated` to the linked +conversation Socket.IO room. + +## Cursor / position tracking + +Both pollers keep an **in-process** cursor (`pagingToken` from the last +successfully persisted event). There is no cursor table and nothing is +written to Redis or Postgres. + +- First poll after process start: `cursor` is `null`. `getEvents` is called + with `startLedger` and `cursor` both unset (the fetcher comment: "resume + on cursor only"). +- After each successful persist, the cursor advances to that event's + `pagingToken`. A persist failure leaves the cursor unmoved so the next + poll retries the same page. +- Token-transfer and treasury streams have separate cursors. + +Cursors die with the process. A restart is a cold start: both cursors are +`null` again. + +## Catch-up after downtime + +Two different "down" cases: + +1. **Process was down (restart / deploy).** Cursors reset. The next + `getEvents` page is whatever the RPC returns without a cursor. Events + still inside the RPC's retention window are re-delivered and upserted. + Events that have aged out of that window are not replayed by this + listener — those mirrored rows stay missing until something else writes + them. +2. **Process stayed up but a poll failed.** `consecutiveFailures` + increments. The loop waits `min(1000 * 2^(n-1), 30000)` ms and retries + **from the last good in-memory cursor**, so it continues where it left + off rather than rewinding. + +## Idempotency + +A replayed ledger event must not double-write: + +- Token transfers: unique `tx_hash` + `ON CONFLICT DO UPDATE`. The same + hash can be persisted any number of times; there is still one row. +- Treasury proposals: unique `(contract_id, proposal_id)`. A repeated + event updates status / counts on the existing row. + +Because the cursor only advances after a successful persist, a crash +mid-page can re-offer the same events; the unique keys absorb that. + +## RPC unreachable — drift from on-chain state + +When `getEvents` throws (timeout, 5xx, network partition): + +- The error is logged as `fetch failed; reconnecting after backoff`. +- The API and WebSocket server keep running. +- No new rows are written for the duration of the outage. + +**Yes, on-chain state can drift from the mirrored tables.** The listener is +a best-effort projection, not a consensus participant. During an RPC outage +the chain continues; `token_transfers` and `treasury_proposals` do not. +After reconnect, catch-up depends on RPC retention and on the treasury +update-only rule (a `proposal_*` event for a row that was never inserted +is silently skipped). There is no compensating backfill job in this +service. + +Cross-referenced from [`IMPLEMENTATION_DOCS.md`](../../../IMPLEMENTATION_DOCS.md). diff --git a/apps/backend/docs/contracts-database-schema.md b/apps/backend/docs/contracts-database-schema.md new file mode 100644 index 0000000..17e7b87 --- /dev/null +++ b/apps/backend/docs/contracts-database-schema.md @@ -0,0 +1,734 @@ +# Database schema reference + +Every table, column, index, check constraint, and relation defined in +[`db/schema.ts`](../src/db/schema.ts), the single source of truth for the +backend's Postgres schema (Drizzle ORM). Generated directly from the current +`schema.ts`; if you suspect it has drifted from the applied database, diff it +against a fresh `pnpm db:generate` output (see +[`migrations.md`](migrations.md) for that workflow) — this document does not +read `drizzle/` migration files directly. + +## Table of contents + +- [Enums](#enums) +- [Entity-relationship diagram](#entity-relationship-diagram) +- [Core identity & messaging](#core-identity--messaging) + - [`users`](#users) + - [`wallets`](#wallets) + - [`conversations`](#conversations) + - [`conversation_members`](#conversation_members) + - [`files`](#files) + - [`messages`](#messages) + - [`message_envelopes`](#message_envelopes) +- [Devices & E2EE key material](#devices--e2ee-key-material) + - [`devices`](#devices) + - [`device_prekeys`](#device_prekeys) + - [`device_key_history`](#device_key_history) +- [MLS group state](#mls-group-state) + - [`mls_groups`](#mls_groups) + - [`mls_group_members`](#mls_group_members) + - [`mls_commits`](#mls_commits) + - [`mls_welcomes`](#mls_welcomes) + - [`mls_key_packages`](#mls_key_packages) +- [Chain-synced data](#chain-synced-data) + - [`token_transfers`](#token_transfers) + - [`treasury_proposals`](#treasury_proposals) + - [`proposal_votes`](#proposal_votes) +- [Push & audit](#push--audit) + - [`push_subscriptions`](#push_subscriptions) + - [`audit_logs`](#audit_logs) + - [`group_control_events`](#group_control_events) +- [Drizzle `relations()` vs. actual foreign keys](#drizzle-relations-vs-actual-foreign-keys) +- [Notable schema observations](#notable-schema-observations) + +## Enums + +Every `pgEnum` defined in `schema.ts`, in declaration order: + +| Enum (Postgres type name) | Values | Used by | +|---|---|---| +| `conversation_type` | `dm`, `group` | `conversations.type` | +| `content_type` | `text`, `file`, `image`, `video`, `audio`, `system` | **Not currently applied to any column** — see [Notable schema observations](#notable-schema-observations) | +| `file_status` | `pending`, `ready`, `deleted` | `files.status` | +| `e2ee_protocol` | `sealed_box`, `signal`, `mls` | `message_envelopes.protocol` | +| `device_platform` | `web`, `ios`, `android` | `devices.platform` | +| `prekey_type` | `signed`, `one_time` | `device_prekeys.key_type` | +| `treasury_proposal_status` | `active`, `approved`, `rejected`, `executed`, `expired` | `treasury_proposals.status` | +| `proposal_vote_type` | `approve`, `reject` | `proposal_votes.vote` | +| `audit_action` | `device_linked`, `device_revoked`, `logout_everywhere`, `key_bundle_drained`, `auth_failed`, `file_access_denied`, `group_member_added`, `group_member_removed` | `audit_logs.action` | +| `group_control_event_type` | `member_added`, `member_removed`, `member_left`, `commit` | `group_control_events.event_type` | + +## Entity-relationship diagram + +Cardinalities reflect the actual `.references()` foreign keys in `schema.ts` +(not the separate `relations()` blocks — see +[below](#drizzle-relations-vs-actual-foreign-keys)). Split into two diagrams +for legibility; `devices` and `conversations` are the shared anchors between +them. + +**Core identity, messaging, devices, and chain-synced data:** + +```mermaid +erDiagram + users ||--o{ wallets : "owns" + users ||--o{ conversation_members : "joins" + users ||--o{ devices : "owns" + users ||--o{ messages : "sends" + users ||--o{ files : "uploads" + users ||--o{ token_transfers : "sends" + users ||--o{ proposal_votes : "casts" + users ||--o{ device_key_history : "has history for" + users ||--o{ group_control_events : "acts in" + + conversations ||--o{ conversation_members : "has" + conversations ||--o{ messages : "contains" + conversations ||--o{ files : "scopes" + conversations ||--o{ token_transfers : "scopes" + conversations |o--o{ treasury_proposals : "scopes (nullable)" + conversations ||--o| mls_groups : "runs (1:1)" + conversations ||--o{ group_control_events : "logs" + + messages |o--o{ conversation_members : "last-read ref (nullable, SET NULL)" + messages ||--o{ message_envelopes : "fans out to" + messages |o--o| messages : "edits (self-ref, nullable, SET NULL)" + files |o--o{ messages : "attached to (nullable, SET NULL)" + devices |o--o{ messages : "sender device (nullable, SET NULL)" + + devices ||--o{ device_prekeys : "publishes" + devices ||--o{ device_key_history : "rotates" + devices ||--o{ push_subscriptions : "registers" + devices ||--o{ message_envelopes : "receives" + + treasury_proposals ||--o{ proposal_votes : "receives" +``` + +**MLS group state** (all keyed through `mls_groups`, one row per +`conversations` row running MLS): + +```mermaid +erDiagram + conversations ||--o| mls_groups : "1:1" + mls_groups ||--o{ mls_group_members : "has leaves" + mls_groups ||--o{ mls_commits : "has commit log" + mls_groups ||--o{ mls_welcomes : "has pending welcomes" + devices ||--o{ mls_group_members : "holds leaf in" + devices |o--o{ mls_commits : "commits (nullable, SET NULL)" + devices ||--o{ mls_welcomes : "addressed to" + devices ||--o{ mls_key_packages : "publishes" +``` + +--- + +## Core identity & messaging + +### `users` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | Primary key. | +| `username` | `username` | `text` | yes | — | Unique. | +| `avatarUrl` | `avatar_url` | `text` | yes | — | | +| `presenceVisible` | `presence_visible` | `boolean` | no | `false` | Gates presence broadcast — see [`concepts-presence-heartbeat.md`](concepts-presence-heartbeat.md#privacy-gating). | +| `lastSeenVisible` | `last_seen_visible` | `boolean` | no | `false` | Gates whether a `lastSeen` timestamp is ever disclosed. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | +| `updatedAt` | `updated_at` | `timestamp` | no | `now()` | Not auto-updated by a DB trigger — the application must set it explicitly on update. | +| `sendReadReceipts` | `send_read_receipts` | `boolean` | no | `false` | Privacy setting: whether this user sends read receipts to others. | +| `allowDirectMessages` | `allow_direct_messages` | `boolean` | no | `true` | | +| `allowGroupInvites` | `allow_group_invites` | `boolean` | no | `false` | | + +**Indexes/constraints:** primary key on `id`; unique constraint on `username` +(no named index given — Drizzle's `.unique()` shorthand). No composite or +partial indexes. + +**Relations:** referenced by `wallets`, `conversation_members`, `devices`, +`messages.senderId`, `files.uploaderId`, `token_transfers.senderId`, +`proposal_votes`, `device_key_history`, `group_control_events` +(`actorUserId`/`targetUserId`) — all `ON DELETE CASCADE` except +`group_control_events`, which uses `ON DELETE SET NULL` for both user +references (see that table below). + +--- + +### `wallets` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `address` | `address` | `text` | no | — | Unique. Stellar wallet address. | +| `isPrimary` | `is_primary` | `boolean` | no | `false` | No DB-level constraint enforcing at most one primary wallet per user — enforced by application logic only (see `resolveAccountWallet` in `routes/devices.ts`, which falls back to the first row if none is flagged primary). | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id`; unique constraint on `address`. + +--- + +### `conversations` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `type` | `type` | `conversation_type` enum | no | `'dm'` | `dm` or `group`. | +| `name` | `name` | `text` | yes | — | | +| `avatarUrl` | `avatar_url` | `text` | yes | — | | +| `epoch` | `epoch` | `integer` | no | `0` | Incremented by every group-control event; this row also serializes event ordering, so a concurrent join and leave can never receive the same sequence number — see `group_control_events` below. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id`. No other indexes or constraints. + +**Relations:** referenced (all `ON DELETE CASCADE` unless noted) by +`conversation_members`, `messages`, `files`, `token_transfers`, +`treasury_proposals` (`ON DELETE SET NULL`), `mls_groups` (unique — one MLS +group per conversation), `group_control_events`. + +--- + +### `conversation_members` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `lastReadMessageId` | `last_read_message_id` | `uuid` | yes | — | FK → `messages.id`, `ON DELETE SET NULL`. | +| `isMuted` | `is_muted` | `boolean` | no | `false` | | +| `isArchived` | `is_archived` | `boolean` | no | `false` | | +| `joinedAt` | `joined_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id` only. **No index at all on +`(conversationId, userId)` or `conversationId` alone** — see +[Notable schema observations](#notable-schema-observations) for what this +means in practice. + +--- + +### `files` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `uploaderId` | `uploader_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `status` | `status` | `file_status` enum | no | `'pending'` | `pending` → `ready` → `deleted` lifecycle. | +| `size` | `size` | `integer` | no | — | | +| `mimeType` | `mime_type` | `text` | no | — | | +| `sha256` | `sha256` | `text` | no | — | Not unique — two uploads of identical bytes get two rows and two storage objects. | +| `storageKey` | `storage_key` | `text` | no | — | Unique. Object-store key. | +| `isThumbnail` | `is_thumbnail` | `boolean` | no | `false` | | +| `deletedAt` | `deleted_at` | `timestamp` | yes | — | Set when every referencing message is retracted (soft delete). | +| `hardDeletedAt` | `hard_deleted_at` | `timestamp` | yes | — | Set once the object-store object is actually deleted — see [`concepts-gc-jobs.md`](concepts-gc-jobs.md#file-cleanup--servicesfilecleanupts). | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id`; unique constraint on `storageKey`. Only +the `fileId` column *on `messages`* (not here) indexes the relationship back +to a file — there is no index on `files.conversationId` or +`files.uploaderId`. + +The `fileKey` (symmetric encryption key for the file's contents) is +deliberately **never** stored in this table or anywhere server-side — it +lives only inside the E2EE envelope ciphertext. See +[`concepts-e2ee-key-management.md`](concepts-e2ee-key-management.md#what-the-server-can-and-cannot-see). + +--- + +### `messages` + +Ordering is by `(createdAt, id)` — `id` (a `uuid`) is only a tiebreaker for +same-millisecond inserts, not a chronological signal. The source comment +notes a monotonic per-conversation sequence counter was considered (and +briefly implemented) but dropped, since the same counter couldn't coherently +serve both per-conversation ordering and the cross-conversation offline-sync +cursor (`/sync`). + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `senderId` | `sender_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `senderDeviceId` | `sender_device_id` | `uuid` | yes | — | FK → `devices.id`, `ON DELETE SET NULL`. | +| `contentType` | `content_type` | **plain `text`** | no | `'text'` | **Not** typed as the `content_type` enum despite that enum existing — see [Notable schema observations](#notable-schema-observations). Values used in practice: `text`, `file`, `image`, `video`, `audio`, `system`. | +| `ciphertext` | `ciphertext` | `text` | yes | — | Opaque E2EE ciphertext; `NULL` for `system` rows (enforced by the check constraint below). | +| `systemPayload` | `system_payload` | `jsonb` | yes | — | Typed as `{ userId: string; change: string } \| null`. Structured, server-generated metadata for `content_type = 'system'` rows only. | +| `fileId` | `file_id` | `uuid` | yes | — | FK → `files.id`, `ON DELETE SET NULL`. | +| `editsMessageId` | `edits_message_id` | `uuid` | yes | — | Self-referencing FK → `messages.id`, `ON DELETE SET NULL`. | +| `mlsEpoch` | `mls_epoch` | `bigint` (number mode) | yes | — | MLS epoch whose secrets encrypted `ciphertext`; `NULL` for non-MLS-group messages. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | +| `deletedAt` | `deleted_at` | `timestamp` | yes | — | Retraction marker. | + +**Indexes:** + +| Name | Columns | Notes | +|---|---|---| +| `messages_conversation_created_idx` | `(conversation_id, created_at)` | The primary read-path index — paginated history per conversation. | + +**Check constraints:** + +| Name | Expression | Enforces | +|---|---|---| +| `messages_system_payload_check` | `content_type <> 'system' OR (ciphertext IS NULL AND system_payload IS NOT NULL)` | A `system` row always carries structured metadata and never ciphertext; every other row never carries a system payload. The source comment notes this **supersedes** an earlier, looser constraint (`messages_system_payload_only_on_system_type`) that only forbade a payload on non-system rows without requiring a system row to actually have one. | + +--- + +### `message_envelopes` + +One row per **(message, recipient device)** — the actual per-device delivery +unit. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `messageId` | `message_id` | `uuid` | no | — | FK → `messages.id`, `ON DELETE CASCADE`. | +| `recipientDeviceId` | `recipient_device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `recipientUserId` | `recipient_user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `ciphertext` | `ciphertext` | `text` | no | — | Per-device-encrypted ciphertext — opaque to the server. | +| `protocol` | `protocol` | `e2ee_protocol` enum | no | `'sealed_box'` | Which construction produced this envelope's ciphertext. Defaults to `sealed_box` so every pre-existing row is correctly labelled by the migration's backfill. Recorded per envelope (not derived from the device's current capabilities) because a device's capability set changes over time but an envelope written months ago must still decrypt on whatever path it was actually encrypted with. | +| `deliveredAt` | `delivered_at` | `timestamp` | yes | — | | +| `readAt` | `read_at` | `timestamp` | yes | — | | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Notes | +|---|---|---| +| `me_recipient_device_created_idx` | `(recipient_device_id, created_at)` | Per-device delivery/sync queries. | +| `me_message_idx` | `(message_id)` | Fan-out lookups by message. | + +This table is the target of the tightest-retention background job in the +system — see +[`concepts-gc-jobs.md`](concepts-gc-jobs.md#envelope-gc--servicesenvelopegcts). + +--- + +## Devices & E2EE key material + +### `devices` + +The single canonical device registry — see +[`concepts-e2ee-key-management.md`](concepts-e2ee-key-management.md) for the +full lifecycle this table models. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `identityPublicKey` | `identity_public_key` | `text` | no | — | Base64 Ed25519 public key. Immutable per row — see [`concepts-e2ee-key-management.md`](concepts-e2ee-key-management.md#7-key-rotation-and-the-unwritten-devicekeyhistory-table). | +| `registrationId` | `registration_id` | `integer` | yes | — | X3DH/Signal registration id. | +| `deviceName` | `device_name` | `text` | yes | — | | +| `platform` | `platform` | `device_platform` enum | yes | — | `web`, `ios`, or `android`. | +| `lastSeenAt` | `last_seen_at` | `timestamp` | yes | — | Updated on auth, connect, heartbeat, and disconnect — see [`concepts-presence-heartbeat.md`](concepts-presence-heartbeat.md). | +| `pushEnabled` | `push_enabled` | `boolean` | no | `true` | | +| `revokedAt` | `revoked_at` | `timestamp` | yes | — | `NULL` while active. Row is never deleted on revocation — preserves audit history. | +| `staleFlaggedAt` | `stale_flagged_at` | `timestamp` | yes | — | Set by the device-GC job once revoked past the retention window; informational only. | +| `capabilities` | `capabilities` | `jsonb` | no | `DEFAULT_CAPABILITIES` | Typed as `DeviceCapabilities`; defaults to the sealed-box-only baseline so pre-existing rows (or clients that omit it) negotiate correctly — see [`concepts-protocol-negotiation.md`](concepts-protocol-negotiation.md). | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | +| `updatedAt` | `updated_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `devices_user_identity_idx` | `(user_id, identity_public_key)` | unique | At most one device row per `(user, identity key)` pair — the lookup key both device-creation paths (`POST /auth/verify`, `POST /devices/link/verify`) key off of. | +| `devices_user_id_active_idx` | `(user_id)` **WHERE `revoked_at IS NULL`** | partial, non-unique | Fast "this user's active devices" scans without touching revoked history. | + +--- + +### `device_prekeys` + +Signed and one-time prekeys in a single table, discriminated by `key_type`. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `keyType` | `key_type` | `prekey_type` enum | no | — | `signed` or `one_time`. | +| `keyId` | `key_id` | `integer` | no | — | Application-assigned, unique per `(device, type)`. | +| `publicKey` | `public_key` | `text` | no | — | Base64 public key. | +| `signature` | `signature` | `text` | yes | — | Base64 Ed25519 signature over `publicKey`, signed by the device's `identityPublicKey`. Required when `keyType = 'signed'` (enforced below). | +| `consumed` | `consumed` | `boolean` | no | `false` | One-time keys only in practice; flips to `true` on bundle-fetch claim rather than deleting the row, so fetch history stays auditable. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `device_prekeys_device_type_keyid_idx` | `(device_id, key_type, key_id)` | unique | No duplicate `keyId` within the same device + key type; also the `ON CONFLICT DO NOTHING` target for idempotent one-time-prekey re-uploads. | +| `device_prekeys_signed_device_idx` | `(device_id)` **WHERE `key_type = 'signed'`** | partial unique | At most one active signed prekey per device — the `ON CONFLICT` target the upload route upserts against. | +| `device_prekeys_one_time_available_idx` | `(device_id)` **WHERE `key_type = 'one_time' AND consumed = false`** | partial, non-unique | Fast bundle-assembly lookup of a device's unconsumed one-time pool. | + +**Check constraints:** + +| Name | Expression | Enforces | +|---|---|---| +| `device_prekeys_signed_requires_signature` | `key_type <> 'signed' OR signature IS NOT NULL` | A signed prekey row must carry a signature; one-time rows are unconstrained on this column. | + +--- + +### `device_key_history` + +Append-only log of identity-key changes per device — intended to let clients +detect silent key swaps. **Verified: no code path in `apps/backend/src` +currently writes to this table** — see +[`concepts-e2ee-key-management.md`](concepts-e2ee-key-management.md#7-key-rotation-and-the-unwritten-devicekeyhistory-table) +for the full explanation. Only `GET /users/:id/key-history` reads it. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `previousKey` | `previous_key` | `text` | yes | — | | +| `newKey` | `new_key` | `text` | no | — | | +| `changeReason` | `change_reason` | `text` | yes | — | | +| `recordedAt` | `recorded_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | +|---|---| +| `device_key_history_device_idx` | `(device_id, recorded_at)` | +| `device_key_history_user_idx` | `(user_id, recorded_at)` | + +--- + +## MLS group state + +Group conversations run MLS (RFC 9420). The server is a transport and +ordering service — it never holds group secrets, only the public ledger +every member needs to agree on. See +[`mls-group-membership.md`](mls-group-membership.md) and +[`mls-key-packages.md`](mls-key-packages.md) for the protocol-level detail +this section's tables support. + +### `mls_groups` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `groupId` | `group_id` | `text` | no | — | Base64 of the MLS group id chosen by the founding client. | +| `cipherSuite` | `cipher_suite` | `integer` | no | — | | +| `currentEpoch` | `current_epoch` | `bigint` (number mode) | no | `0` | Epoch of the most recent accepted commit. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | +| `updatedAt` | `updated_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `mls_groups_conversation_idx` | `(conversation_id)` | unique | One MLS group per conversation — the conversation *is* the group. | +| `mls_groups_group_id_idx` | `(group_id)` | unique | | + +--- + +### `mls_group_members` + +Device membership recorded as an **epoch interval**, not a boolean, so a +device's decryption window is derivable: it can read epochs in +`[joinedAtEpoch, removedAtEpoch)`. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `mlsGroupId` | `mls_group_id` | `uuid` | no | — | FK → `mls_groups.id`, `ON DELETE CASCADE`. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `joinedAtEpoch` | `joined_at_epoch` | `bigint` (number mode) | no | — | | +| `removedAtEpoch` | `removed_at_epoch` | `bigint` (number mode) | yes | — | `NULL` while the device is still in the group. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `mls_group_members_active_idx` | `(mls_group_id, device_id)` **WHERE `removed_at_epoch IS NULL`** | partial unique | A device can hold at most one *active* leaf in a group at a time. A device removed and later re-added gets a **second row** with a later `joinedAtEpoch` rather than reusing the first — this partial unique index is what makes that legal (the old row's `removedAtEpoch` is non-null, so it no longer competes for uniqueness). | +| `mls_group_members_device_idx` | `(device_id)` | non-unique | Cross-group lookups by device. | + +--- + +### `mls_commits` + +Append-only commit log. A device that was offline replays from its last +known epoch to catch up. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `mlsGroupId` | `mls_group_id` | `uuid` | no | — | FK → `mls_groups.id`, `ON DELETE CASCADE`. | +| `epoch` | `epoch` | `bigint` (number mode) | no | — | The epoch this commit *produces*. | +| `committerDeviceId` | `committer_device_id` | `uuid` | yes | — | FK → `devices.id`, `ON DELETE SET NULL`. | +| `commit` | `commit` | `text` | no | — | Base64 of the TLS-serialised MLS `Commit` message. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `mls_commits_group_epoch_idx` | `(mls_group_id, epoch)` | unique | Two members committing concurrently cannot both win — the epoch race resolves in the database via this constraint. | + +--- + +### `mls_welcomes` + +Welcome messages addressed to a device being added, held until it comes +online and claims them. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `mlsGroupId` | `mls_group_id` | `uuid` | no | — | FK → `mls_groups.id`, `ON DELETE CASCADE`. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `epoch` | `epoch` | `bigint` (number mode) | no | — | The epoch the recipient joins at. | +| `welcome` | `welcome` | `text` | no | — | Base64 of the TLS-serialised MLS `Welcome` message. | +| `claimedAt` | `claimed_at` | `timestamp` | yes | — | `NULL` while pending. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `mls_welcomes_group_device_epoch_idx` | `(mls_group_id, device_id, epoch)` | unique | | +| `mls_welcomes_pending_idx` | `(device_id)` **WHERE `claimed_at IS NULL`** | partial, non-unique | Fast "what's waiting for this device" lookup on reconnect. | + +--- + +### `mls_key_packages` + +Each device publishes a stock of MLS KeyPackages so any group member can add +it without the device being online. Public key material only. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `cipherSuite` | `cipher_suite` | `integer` | no | — | IANA MLS cipher suite id; a device may publish packages for several suites. | +| `keyPackage` | `key_package` | `text` | no | — | Base64 of the TLS-serialised MLS `KeyPackage`. Up to ~4 KiB. | +| `packageHash` | `package_hash` | `text` | no | — | SHA-256 (hex) of `keyPackage` — exists specifically because the package itself is too large for a btree unique index, but idempotent re-uploads still need a conflict target. | +| `expiresAt` | `expires_at` | `timestamp` | yes | — | | +| `consumed` | `consumed` | `boolean` | no | `false` | Single-use by spec — flips to `true` inside the same transaction that hands the package out (never deleted, preserving audit history), mirroring `device_prekeys.consumed`. | +| `consumedAt` | `consumed_at` | `timestamp` | yes | — | | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `mls_key_packages_device_hash_idx` | `(device_id, package_hash)` | unique | Dedupe target for idempotent re-uploads. | +| `mls_key_packages_available_idx` | `(device_id, cipher_suite, created_at)` **WHERE `consumed = false`** | partial, non-unique | Fast claim of the next available package for a device + cipher suite. | + +--- + +## Chain-synced data + +### `token_transfers` + +One row per Soroban `transfer` event the listener +(`services/stellarListener.ts`) pulls off the chain. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `senderId` | `sender_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `recipientAddress` | `recipient_address` | `text` | no | — | Raw Stellar address — not a FK to `wallets`, since the recipient need not be a Clicked user. | +| `amount` | `amount` | `text` | no | — | Stored as text, not a numeric type — avoids floating-point/precision loss for on-chain integer amounts. | +| `tokenContractId` | `token_contract_id` | `text` | no | — | | +| `txHash` | `tx_hash` | `text` | no | — | Unique — the upsert target that makes reconnects and replayed event pages idempotent. | +| `memo` | `memo` | `text` | yes | — | Correlates an on-chain transfer with a chat message. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id`; unique constraint on `txHash`. + +--- + +### `treasury_proposals` + +Synced from `GROUP_TREASURY_CONTRACT_ID` events by the Stellar listener. +Idempotent upsert on `(contractId, proposalId)`. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `contractId` | `contract_id` | `text` | no | — | The on-chain `group_treasury` contract address. | +| `proposalId` | `proposal_id` | `text` | no | — | The on-chain proposal id (`u32`, stored as text). | +| `conversationId` | `conversation_id` | `uuid` | yes | — | FK → `conversations.id`, `ON DELETE SET NULL`. | +| `status` | `status` | `treasury_proposal_status` enum | no | `'active'` | Mirrors the on-chain `ProposalStatus` — see [`../../../contracts/docs/contracts-group-treasury-storage.md`](../../../contracts/docs/contracts-group-treasury-storage.md) for the contract-side state machine this tracks. | +| `approvalsCount` | `approvals_count` | `integer` | no | `0` | | +| `rejectionsCount` | `rejections_count` | `integer` | no | `0` | | +| `recipient` | `recipient` | `text` | yes | — | | +| `amount` | `amount` | `text` | yes | — | Text, same precision rationale as `token_transfers.amount`. | +| `token` | `token` | `text` | yes | — | | +| `threshold` | `threshold` | `integer` | no | `3` | Mirrors the on-chain threshold at sync time — a default used only until the real event arrives. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | +| `updatedAt` | `updated_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `treasury_proposals_contract_proposal_idx` | `(contract_id, proposal_id)` | unique | The upsert target for idempotent chain-event syncing. | + +--- + +### `proposal_votes` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `treasuryProposalId` | `treasury_proposal_id` | `uuid` | no | — | FK → `treasury_proposals.id`, `ON DELETE CASCADE`. | +| `userId` | `user_id` | `uuid` | no | — | FK → `users.id`, `ON DELETE CASCADE`. | +| `vote` | `vote` | `proposal_vote_type` enum | no | — | `approve` or `reject`. | +| `signature` | `signature` | `text` | yes | — | | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `proposal_votes_proposal_user_unique` | `(treasury_proposal_id, user_id)` | unique | One recorded vote per user per proposal. | + +--- + +## Push & audit + +### `push_subscriptions` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `deviceId` | `device_id` | `uuid` | no | — | FK → `devices.id`, `ON DELETE CASCADE`. | +| `endpoint` | `endpoint` | `text` | no | — | Unique. The Web Push endpoint URL. | +| `p256dh` | `p256dh` | `text` | no | — | Web Push encryption key. | +| `auth` | `auth` | `text` | no | — | Web Push auth secret. | +| `lastUsedAt` | `last_used_at` | `timestamp` | yes | — | | +| `disabledAt` | `disabled_at` | `timestamp` | yes | — | Backoff marker set on a transient send failure; cleared by the re-enable pass once the backoff window elapses — see [`concepts-gc-jobs.md`](concepts-gc-jobs.md#push-subscription-backoff-re-enable). | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes/constraints:** PK on `id`; unique constraint on `endpoint`. + +--- + +### `audit_logs` + +Append-only record of security-relevant events. Nothing here may contain +message content — rows carry identifiers, counts, and outcomes only; the +writing service (`services/auditLog.ts`) strips anything content-shaped +before it reaches the database. + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `action` | `action` | `audit_action` enum | no | — | | +| `actorUserId` | `actor_user_id` | `uuid` | yes | — | **Not a foreign key** (deliberately — see below). | +| `actorDeviceId` | `actor_device_id` | `uuid` | yes | — | Not a foreign key. | +| `subjectUserId` | `subject_user_id` | `uuid` | yes | — | Not a foreign key. Nullable because a failed sign-in has no established actor. | +| `targetType` | `target_type` | `text` | yes | — | Free text: `'device'`, `'file'`, `'conversation'`, `'wallet'`, etc. | +| `targetId` | `target_id` | `text` | yes | — | | +| `ipAddress` | `ip_address` | `text` | yes | — | | +| `userAgent` | `user_agent` | `text` | yes | — | | +| `metadata` | `metadata` | `jsonb` | yes | — | Typed as `Record`; sanitised, bounded key/value context — never message content. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Why the actor/subject columns are plain `uuid`, not foreign keys:** an +audit row must record what was true when it was written and stay that way. +A `CASCADE` FK would delete audit history along with the account it +incriminates; `ON DELETE SET NULL` would issue an `UPDATE` that the +append-only trigger (next paragraph) correctly refuses. A responder resolves +these ids (or finds them already gone) at read time instead. + +**Indexes:** + +| Name | Columns | Notes | +|---|---|---| +| `audit_logs_subject_created_idx` | `(subject_user_id, created_at)` | Primary read path: "what happened *to* this account." | +| `audit_logs_actor_created_idx` | `(actor_user_id, created_at)` | "What did this account/device *do*." | +| `audit_logs_action_created_idx` | `(action, created_at)` | "Every failed auth in the last hour," incident-response style queries. | + +**Append-only enforcement is not visible in `schema.ts` at all** — it's a +database-level trigger (`audit_logs_no_mutation`, referenced by the schema +comment as living in the corresponding migration) that rejects `UPDATE`s and +`DELETE`s against this table outright, specifically so the log's integrity +does not depend on the same application credentials an attacker would +already have reached. See +[`../../../docs/security/audit-logging.md`](../../../docs/security/audit-logging.md). + +--- + +### `group_control_events` + +| Column | DB name | Type | Nullable | Default | Notes | +|---|---|---|---|---|---| +| `id` | `id` | `uuid` | no | `gen_random_uuid()` | PK. | +| `conversationId` | `conversation_id` | `uuid` | no | — | FK → `conversations.id`, `ON DELETE CASCADE`. | +| `sequence` | `sequence` | `integer` | no | — | Strictly increasing from 1, gap-free within a conversation. | +| `epoch` | `epoch` | `integer` | no | — | Group epoch after this event was applied — mirrors `conversations.epoch` at the time of writing. | +| `eventType` | `event_type` | `group_control_event_type` enum | no | — | | +| `actorUserId` | `actor_user_id` | `uuid` | yes | — | FK → `users.id`, `ON DELETE SET NULL`. | +| `targetUserId` | `target_user_id` | `uuid` | yes | — | FK → `users.id`, `ON DELETE SET NULL`. | +| `messageId` | `message_id` | `uuid` | yes | — | FK → `messages.id`, `ON DELETE SET NULL`. The system message emitted for this event, when one was created. | +| `payload` | `payload` | `text` | yes | — | Opaque client-supplied MLS commit/welcome material — never parsed server-side. | +| `createdAt` | `created_at` | `timestamp` | no | `now()` | | + +**Indexes:** + +| Name | Columns | Type | Enforces | +|---|---|---|---| +| `group_control_conversation_sequence_idx` | `(conversation_id, sequence)` | unique | Both the gap-free sequencing guarantee and the catch-up read path for a device replaying missed group-control events. | + +--- + +## Drizzle `relations()` vs. actual foreign keys + +`schema.ts` ends with a large block of `relations(table, ({ one, many }) => +...)` calls (`usersRelations`, `conversationsRelations`, +`messagesRelations`, etc.). **These are Drizzle ORM query-time conveniences +for the relational query API (`db.query.x.findMany({ with: {...} })`) — they +are not separate database constraints.** Every cascade/`SET NULL` behavior +documented in the per-table sections above comes from the `.references()` +call on the column itself, not from a `relations()` block. A handful of +`relations()` entries describe a join that has **no corresponding FK on +either side at all** — `users.transfers` (`many(tokenTransfers)`) is backed +by the real `tokenTransfers.senderId` FK, but if a future relation were added +without a matching `.references()`, it would type-check and run queries +successfully while enforcing nothing at the database level. When auditing +referential integrity, read the column definitions in each table's own +`pgTable(...)` block, not the `relations()` section. + +## Notable schema observations + +Verified against the current `schema.ts`, not speculative: + +- **`conversation_members` has no unique or composite index at all** beyond + its own primary key — specifically, nothing prevents two rows from + recording the same `(conversationId, userId)` pair, and there is no index + to accelerate "is this user a member of this conversation" or "list all + members of this conversation" beyond a full-table scan filtered by the FK + columns' implicit (non-indexed) equality. Every other membership-shaped + table in this schema (`mls_group_members`, `proposal_votes`) has an + explicit uniqueness constraint on its natural key; this one does not. +- **`content_type` (the enum) is defined but unused.** `messages.contentType` + is declared as a plain `text('content_type')` column, not + `contentTypeEnum('content_type')`, despite the enum existing immediately + above it in the file with exactly the values used in practice (`text`, + `file`, `image`, `video`, `audio`, `system`). The check constraint + `messages_system_payload_check` relies on string comparison + (`content_type <> 'system'`) rather than an enum comparison, which works + either way, but the column accepts any text value at the database level — + the enum provides no actual validation here. +- **`files.sha256` is not unique.** Two uploads of byte-identical content + produce two separate `files` rows (and two separate object-store objects) + rather than being deduplicated — `storageKey` is the only uniqueness + guarantee on this table. +- **`wallets.isPrimary` has no partial unique index** enforcing "at most one + primary wallet per user" — a user could have zero, one, or several wallets + flagged primary simultaneously as far as the database is concerned; the + single-primary invariant is maintained (if at all) by application code + such as `resolveAccountWallet`. +- **Chain-sourced monetary amounts (`token_transfers.amount`, + `treasury_proposals.amount`/`.token`) are stored as `text`, not a numeric + type** — this is deliberate (avoids floating-point representation of + on-chain integer amounts) but means the database itself performs no + numeric validation or arithmetic on these columns; all math happens in + application code or on-chain. +- **`audit_logs`'s actor/subject columns are plain UUIDs with no foreign + key**, by design — see the dedicated note under + [`audit_logs`](#audit_logs) above. diff --git a/apps/backend/docs/contracts-error-catalog.md b/apps/backend/docs/contracts-error-catalog.md new file mode 100644 index 0000000..4a543c3 --- /dev/null +++ b/apps/backend/docs/contracts-error-catalog.md @@ -0,0 +1,487 @@ +# Error code and error response catalog + +Every error shape the backend can return, across both transports, in one place. If a client +is handling a failure, the response it received is listed here. + +The backend has two error surfaces and they do **not** share a format: + +- **REST** — a JSON body `{ error: string }`, sometimes with extra fields, and an HTTP status + that carries most of the meaning. +- **Socket.IO** — an `error` event whose payload names the originating client event and + sometimes carries a `code`. There is no HTTP status, and — importantly — the payload comes + in [two different shapes](#socket-errors-come-in-two-shapes) depending on which layer + rejected the event. + +There is no machine-readable error code enum on the REST side. The `error` string is the +identifier, so **match on the status first and the string second**, and treat the string as +stable-ish rather than guaranteed: several are built with template literals and interpolate +runtime values. + +--- + +## REST errors + +### The common shape + +```jsonc +// The only field every REST error has: +{ "error": "Not a member of this conversation" } +``` + +Some errors add fields alongside `error` — never nested under it. The "Extra fields" column +below is exhaustive per error. + +Three things to know before reading the table: + +1. **There is no global Express error handler.** An exception that escapes a route handler + falls through to Express's built-in handler, which returns an HTML error page with a + `500` status, not `{ error }`. A client that assumes every non-2xx body is JSON will throw + while parsing. Always branch on `content-type`, or guard the `.json()` call. +2. **`GET /audit-logs` is not mounted.** `auditLogsRouter` exists in `src/routes/auditLogs.ts` + and is exercised in tests, but `src/app.ts` never mounts it, so its two errors are + currently unreachable over HTTP. They are listed for completeness and flagged. +3. **`4xx` bodies are safe to show to a developer, not to an end user.** Several leak internal + vocabulary (`sha256 mismatch`, `Commit epoch conflict`). Map them to user-facing copy on + the client — see the web app's [error handling guide](../../web/docs/concepts-error-handling.md). + +### Errors emitted by middleware + +These can accompany **any** authenticated route, so they are not repeated per-route in the +main table. + +| Status | `error` | Extra fields | Emitted by | +| ------ | ---------------------------------------------------------------------------- | --------------------------------- | ----------------------------------------------------------------------- | +| `401` | Missing or invalid Authorization header | — | `requireAuth` — no `Authorization: Bearer` header | +| `401` | Invalid or expired token | — | `requireAuth` — JWT signature or expiry check failed | +| `401` | Token missing deviceId | — | `requireAuth` — token predates device binding | +| `401` | Device not found or has been revoked | — | `requireAuth` — the token's device was revoked | +| `403` | `tls_required` | `message` | `transportSecurity` — plaintext HTTP against an https-only deployment | +| `403` | `origin_not_allowed` | `message` | `transportSecurity` — `Origin` not on the allow-list | +| `400` | Validation failed | `issues[]` (`{ field, message }`) | `validate(schema)` — Zod rejection on a schema-guarded route | +| `429` | Too many requests | `bucket`, `retryAfterSeconds` | `rateLimit(bucket)` — see [Rate-limit responses](#rate-limit-responses) | +| `503` | _(no `error` key)_ — `{ status: 'error', db: 'unreachable', node, version }` | — | `GET /health` when `SELECT 1` fails | + +Note the two odd ones out. `tls_required` and `origin_not_allowed` are the only REST errors +whose `error` value is a **machine code** rather than a sentence, and the only ones that pair +it with a separate human-readable `message`. `GET /health` is the only failure response with +no `error` key at all. + +### The catalog + +One row per distinct `(status, error)` pair, listing every route that emits it. + +| Status | `error` | Extra fields | Route(s) | +| ------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `400` | A device cannot be added and removed by the same commit | — | POST /conversations/:id/mls/commits | +| `400` | Added devices must be active devices of conversation members | `invalidDeviceIds` | POST /conversations/:id/mls/commits | +| `400` | addedDevices contains duplicate deviceIds | — | POST /conversations/:id/mls/commits | +| `400` | allowDirectMessages must be a boolean | — | PATCH /users/me | +| `400` | allowGroupInvites must be a boolean | — | PATCH /users/me | +| `400` | At least one of muted or archived is required | — | PATCH /conversations/:id/settings | +| `400` | At least one of name or avatarUrl must be provided | — | PATCH /conversations/:id | +| `400` | avatarUrl must be a string | — | PATCH /conversations/:id | +| `400` | cipherSuite must be a positive integer | — | GET /users/:userId/devices/:deviceId/mls-key-package | +| `400` | Conversation id is required | — | DELETE /conversations/:id/leave; GET /conversations/:id; GET /conversations/:id/devices; GET /conversations/:id/epoch; GET /conversations/:id/group-control; GET /conversations/:id/members; GET /conversations/:id/messages; GET /conversations/:id/transfers; PATCH /conversations/:id; PATCH /conversations/:id/settings; POST /conversations/:id/group-control; POST /conversations/:id/members; POST /conversations/:id/transfers; all `/conversations/:id/mls/*` | +| `400` | deviceId is required | — | GET /sync | +| `400` | DM conversations cannot add members | — | POST /conversations/:id/members | +| `400` | DM conversations cannot be left | — | DELETE /conversations/:id/leave | +| `400` | DM conversations cannot be updated | — | PATCH /conversations/:id | +| `400` | Endpoint is required | — | DELETE /push/subscriptions | +| `400` | expiresAt must be in the future | — | POST /devices/:id/mls-key-packages | +| `400` | File id is required | — | GET /files/:fileId | +| `400` | fileId is required | — | POST /uploads/:fileId/confirm | +| `400` | Invalid cursor | — | GET /conversations/:id/messages; GET /sync | +| `400` | Invalid membership change metadata | — | POST /messages | +| `400` | Invalid request | `details` (Zod issues) | POST /uploads | +| `400` | lastSeenVisible must be a boolean | — | PATCH /users/me | +| `400` | Message id is required | — | DELETE /messages/:id | +| `400` | Missing endpoint or keys | — | POST /push/subscriptions | +| `400` | MLS groups are only available for group conversations | — | POST /conversations/:id/mls/group | +| `400` | name must be a string | — | PATCH /conversations/:id | +| `400` | No wallet is associated with this account | — | POST /devices/link/challenge; POST /devices/link/verify | +| `400` | payload must be a non-empty string | — | POST /conversations/:id/group-control | +| `400` | presenceVisible must be a boolean | — | PATCH /users/me | +| `400` | Query parameter "q" is required | — | GET /users/search | +| `400` | recipientAddress, amount, tokenContractId, and txHash are required | — | POST /conversations/:id/transfers | +| `400` | sendReadReceipts must be a boolean | — | PATCH /users/me | +| `400` | Signed prekey signature is invalid | — | POST /devices/:id/prekeys | +| `400` | sinceEpoch must be a non-negative integer | — | GET /conversations/:id/mls/commits | +| `400` | sinceSequence must be a non-negative integer | — | GET /conversations/:id/group-control | +| `400` | Token is missing a deviceId | — | all `/conversations/:id/mls/*` | +| `400` | Unknown action filter | `allowed` | GET /audit-logs **(router not mounted)** | +| `400` | User id is required | — | GET /users/:id/key-history | +| `400` | userId is required | — | POST /conversations/:id/members | +| `400` | Username must be 3-30 alphanumeric characters and underscores only | — | PATCH /users/me | +| `401` | Device has been revoked | — | POST /auth/verify | +| `401` | Invalid or expired device link nonce | — | POST /devices/link/verify | +| `401` | Invalid or expired nonce | — | POST /auth/verify | +| `401` | Invalid signature or wallet address | — | POST /auth/verify | +| `401` | Signature verification failed | — | POST /auth/verify; POST /devices/link/verify | +| `403` | Device is not a member of this conversation MLS group | — | POST /uploads | +| `403` | Device is not a member of this MLS group | — | GET /conversations/:id/mls/commits | +| `403` | Device is revoked | — | POST /devices/:id/mls-key-packages; POST /devices/:id/prekeys | +| `403` | Device not found or not owned by this user | — | GET /sync | +| `403` | Device registration requires a fresh wallet signature. Use POST /devices/link/challenge then POST /devices/link/verify. | — | POST /devices | +| `403` | Invalid or expired signed URL | — | GET /local-storage/\*splat; PUT /local-storage/\*splat | +| `403` | No shared conversation with device owner | — | GET /user-devices/:id/public-key | +| `403` | Not a member of this conversation | — | GET /conversations/:id; GET /conversations/:id/devices; GET /conversations/:id/epoch; GET /conversations/:id/group-control; GET /conversations/:id/members; GET /conversations/:id/messages; GET /conversations/:id/transfers; PATCH /conversations/:id; PATCH /conversations/:id/settings; POST /conversations/:id/group-control; POST /conversations/:id/members; POST /conversations/:id/transfers; POST /messages; POST /uploads; all `/conversations/:id/mls/*` | +| `403` | Not authorized to access this file | — | GET /files/:fileId | +| `403` | Not authorized to confirm this upload | — | POST /uploads/:fileId/confirm | +| `403` | Only an active group member device may publish a commit | — | POST /conversations/:id/mls/commits | +| `403` | Only the device owner may upload MLS key packages | — | POST /devices/:id/mls-key-packages | +| `403` | Only the device owner may upload prekeys | — | POST /devices/:id/prekeys | +| `403` | system messages are reserved for the server | — | POST /messages | +| `403` | This device has no key for this file | `reason` | GET /files/:fileId | +| `403` | User is not accepting group invites | — | POST /conversations/:id/members | +| `403` | You can only delete your own messages | — | DELETE /messages/:id | +| `404` | Conversation has no MLS group | — | GET /conversations/:id/mls/commits; GET /conversations/:id/mls/group; GET /conversations/:id/mls/pending-devices; GET /conversations/:id/mls/welcome; POST /conversations/:id/mls/commits | +| `404` | Conversation membership not found | — | DELETE /conversations/:id/leave | +| `404` | Conversation not found | — | DELETE /conversations/:id/leave; GET /conversations/:id; GET /conversations/:id/epoch; GET /conversations/:id/group-control; PATCH /conversations/:id; POST /conversations/:id/members | +| `404` | Device not found | — | DELETE /devices/:id; POST /devices/:id/mls-key-packages; POST /devices/:id/prekeys | +| `404` | Device not found or has been revoked | — | GET /users/:userId/devices/:deviceId/key-bundle; GET /users/:userId/devices/:deviceId/mls-key-package | +| `404` | Device not found or revoked | — | GET /user-devices/:id/public-key | +| `404` | File not found | — | GET /files/:fileId; POST /uploads/:fileId/confirm | +| `404` | File not referenced by any message | — | GET /files/:fileId | +| `404` | Message not found | — | DELETE /messages/:id | +| `404` | No active devices found for this user | — | GET /users/:id/key-fingerprint | +| `404` | No pending Welcome for this device | — | GET /conversations/:id/mls/welcome | +| `404` | Object not found | — | GET /local-storage/\*splat | +| `404` | Proposal not found | — | POST /treasury/proposals/:id/approve; POST /treasury/proposals/:id/reject | +| `404` | User not found | — | GET /users/:id; GET /users/:id/key-fingerprint; GET /users/:id/key-history; GET /users/:id/presence; GET /users/me; PATCH /users/me | +| `409` | Already voted on this proposal | — | POST /treasury/proposals/:id/approve; POST /treasury/proposals/:id/reject | +| `409` | Cannot revoke your only active device | — | DELETE /devices/:id | +| `409` | Commit epoch conflict — another commit was applied first | `currentEpoch`, `expectedEpoch` | POST /conversations/:id/mls/commits | +| `409` | Conversation already has an MLS group | `groupId`, `currentEpoch` on the pre-check path; absent on the insert-race path | POST /conversations/:id/mls/group | +| `409` | Database conflict or validation error | — | POST /conversations/:id/members; POST /conversations/:id/transfers | +| `409` | Device already registered for this user | — | POST /devices/link/verify | +| `409` | Device has not uploaded a signed prekey yet | — | GET /users/:userId/devices/:deviceId/key-bundle | +| `409` | Envelope protocol is weaker than both devices support | `violations` | POST /messages | +| `409` | File has been deleted | — | POST /uploads/:fileId/confirm | +| `409` | File is already ready | — | POST /uploads/:fileId/confirm | +| `409` | No MLS key packages available for this device | `remaining` | GET /users/:userId/devices/:deviceId/mls-key-package | +| `409` | Proposal is no longer active | — | POST /treasury/proposals/:id/approve; POST /treasury/proposals/:id/reject | +| `409` | Transaction hash already exists | — | POST /conversations/:id/transfers | +| `409` | User is already a member | — | POST /conversations/:id/members | +| `409` | Username conflict or database error | — | PATCH /users/me | +| `409` | Username is already taken | — | PATCH /users/me | +| `410` | Server-side search removed; search is now client-side over decrypted messages | `docs` | GET /conversations/:id/search | +| `413` | payload exceeds `${MAX_GROUP_CONTROL_PAYLOAD_BYTES}` bytes | — | POST /conversations/:id/group-control | +| `415` | Unsupported media type | `mimeType` | POST /uploads | +| `422` | File integrity verification failed | `expectedHash`, `computedHash` | POST /uploads/:fileId/confirm | +| `422` | MLS key package cap of `${MLS_KEY_PACKAGE_CAP}` reached. … | — | POST /devices/:id/mls-key-packages | +| `422` | Object not found in storage | `storageKey` | POST /uploads/:fileId/confirm | +| `422` | Object size mismatch | `expectedSize`, `actualSize` | POST /uploads/:fileId/confirm | +| `422` | One-time prekey cap of `${OTP_CAP}` reached. … | — | POST /devices/:id/prekeys | +| `422` | sha256 is required | — | POST /uploads/:fileId/confirm | +| `422` | sha256 mismatch | — | POST /uploads/:fileId/confirm | +| `429` | Daily upload quota exceeded | `bucket`, `retryAfterSeconds` | POST /uploads | +| `429` | Too many group invites. Please try again later. | — | POST /conversations/:id/members | +| `500` | Failed to add conversation member | — | POST /conversations/:id/members | +| `500` | Failed to append group control event | — | POST /conversations/:id/group-control | +| `500` | Failed to claim MLS key package | — | GET /users/:userId/devices/:deviceId/mls-key-package | +| `500` | Failed to compute key fingerprint | — | GET /users/:id/key-fingerprint | +| `500` | Failed to create MLS group | — | POST /conversations/:id/mls/group | +| `500` | Failed to create user | — | POST /auth/verify | +| `500` | Failed to delete subscription | — | DELETE /push/subscriptions | +| `500` | Failed to fetch device public key | — | GET /user-devices/:id/public-key | +| `500` | Failed to generate download URL | — | GET /files/:fileId | +| `500` | Failed to issue device link challenge | — | POST /devices/link/challenge | +| `500` | Failed to list devices | — | GET /devices | +| `500` | Failed to persist message | — | POST /messages | +| `500` | Failed to read audit log | — | GET /audit-logs **(router not mounted)** | +| `500` | Failed to register device | — | POST /auth/verify; POST /devices/link/verify | +| `500` | Failed to register subscription | — | POST /push/subscriptions | +| `500` | Failed to retrieve transfers | — | GET /conversations/:id/transfers | +| `500` | Failed to store object | — | PUT /local-storage/\*splat | +| `500` | Failed to update conversation | — | PATCH /conversations/:id | +| `500` | Search failed | — | GET /users/search | + +### Message payload validation (`POST /messages`) + +`POST /messages` returns the status **chosen by the validator**, so these do not appear as +literal `res.status(400)` calls in the route. All are `{ error: }` with no extra +fields, and all come from `validateMessagePayload` (`src/lib/validateMessagePayload.ts`): + +| Status | `error` | Condition | +| ------ | --------------------------------------------------------------------------------- | ---------------------------------------------------- | +| `403` | system messages are reserved for the server | `contentType: 'system'` from a client | +| `400` | unsupported contentType: "…" | Not one of `text`, `file`, `image`, `video`, `audio` | +| `400` | MLS group messages carry a single group ciphertext, not per-device envelopes | `mlsEpoch` set _and_ `envelopes` non-empty | +| `400` | ciphertext is required for MLS group messages | `mlsEpoch` set, `ciphertext` empty | +| `400` | fileId is required for file-type messages | `file`/`image`/`video`/`audio` with no `fileId` | +| `400` | envelopes are required for file-type messages (they carry the encrypted file key) | File-type message with no envelopes | +| `400` | text messages require at least one envelope with an encrypted key | Text message with no envelopes | + +The same validator backs the socket `send_message` and `send_file_message` handlers, where +its `code` is surfaced as the payload's `code` field rather than an HTTP status. + +### Protocol enforcement (`POST /messages` and `send_message`) + +`checkEnvelopeProtocols` (`src/services/e2eeProtocol.ts`) rejects envelopes that name a +protocol the recipient cannot use. Both errors carry a `violations` array: + +```jsonc +{ + "error": "Envelope protocol is not supported by the recipient device", + "violations": [ + { + "recipientDeviceId": "…", + "declared": "…", // what the envelope claimed + "expected": "…", // what the two devices should have negotiated + "reason": "unsupported_by_recipient", // or "downgrade" + }, + ], +} +``` + +| Status | `error` | When | +| ------ | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| `400` | Envelope protocol is not supported by the recipient device | At least one violation has `reason: 'unsupported_by_recipient'` — the recipient literally could not decrypt it | +| `409` | Envelope protocol is weaker than both devices support | Every violation is a `downgrade` — decryptable, but weaker than negotiated | + +An undecryptable envelope is the more specific failure, so `400` wins when a batch contains +both kinds. + +--- + +## Rate-limit responses + +`429` does not have one shape. There are **three** rate limiters with three different bodies +and three different header sets — worth knowing, because a client that reads `Retry-After` +unconditionally will get `null` from one of them. + +| Source | Body | `Retry-After` | `RateLimit-*` | +| ------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------- | ------------- | +| `rateLimit(bucket)` middleware — the standard path | `{ error: 'Too many requests', bucket, retryAfterSeconds }` | ✅ | ✅ | +| `POST /uploads` daily byte quota | `{ error: 'Daily upload quota exceeded', bucket: 'upload_bytes_daily', retryAfterSeconds }` | ✅ | ❌ | +| `POST /conversations/:id/members` group-invite throttle | `{ error: 'Too many group invites. Please try again later.' }` | ❌ | ❌ | + +The standard middleware path sets all four headers: + +```http +HTTP/1.1 429 Too Many Requests +RateLimit-Limit: 30 +RateLimit-Remaining: 0 +RateLimit-Reset: 42 +Retry-After: 42 +``` + +```jsonc +{ "error": "Too many requests", "bucket": "auth_challenge", "retryAfterSeconds": 42 } +``` + +`RateLimit-Reset` and `Retry-After` are both **seconds until the window rolls over**, not a +timestamp, and they always carry the same value. The `RateLimit-*` headers are set on +**successful** responses too, reporting the tightest remaining budget across every bucket +checked — so a client can back off before being rejected rather than after. + +**The safe client rule:** read `retryAfterSeconds` from the body when present, fall back to +the `Retry-After` header, and fall back to a local backoff constant when neither exists. Only +the first row above guarantees both. + +Per-bucket thresholds are in [`docs/security/rate-limits.md`](../../../docs/security/rate-limits.md); +the mechanism is in `src/services/rateLimiter.ts`. + +--- + +## Socket errors + +### Socket errors come in two shapes + +Both arrive on the `error` event, and clients must handle both. + +**Shape A — bare payload.** Emitted by the per-socket security middleware (`src/index.ts`) +and every handler in `src/socket/messaging.ts`: + +```jsonc +{ + "event": "send_message", // which client event failed + "message": "Not a member of this conversation", + "code": "envelope_too_large", // sometimes; see the table +} +``` + +**Shape B — enveloped.** Emitted by the dispatcher itself (`src/socket/dispatcher.ts`), which +wraps the payload in the standard event envelope: + +```jsonc +{ + "eventId": "…", + "type": "error", + "timestamp": 1730000000000, + "payload": { "message": "Malformed envelope", "details": { … } } +} +``` + +The difference is _which layer_ rejected the event. The dispatcher rejects before dispatch — +unauthenticated socket, unparseable envelope, unknown event type, stale timestamp — and it +speaks envelopes. Everything past that point emits the bare form. So: + +```ts +socket.on('error', (raw) => { + const payload = raw?.type === 'error' && raw.payload ? raw.payload : raw; + // payload.message is always present; payload.event and payload.code may not be +}); +``` + +Note also that Shape B carries **no `event` field** for three of its four cases — only the +`Unauthenticated` rejection names one (`event: 'dispatch'`). Do not key error handling on +`payload.event` being present. + +### Dispatcher errors (Shape B) + +Raised before any handler runs, so no client event has been processed. + +| `payload.message` | Extra fields | Cause | +| ----------------------------------- | ------------------------------- | ----------------------------------------------------- | +| Unauthenticated | `event: 'dispatch'` | Socket has no `auth` — the handshake did not complete | +| Malformed envelope | `details` (flattened Zod error) | The `dispatch` payload failed `EventEnvelopeSchema` | +| Unknown event type: "…" | `eventId` | Envelope parsed, but `type` is not a known event | +| Stale or invalid envelope timestamp | `eventId` | `timestamp` outside the accepted window | + +A duplicate `eventId` is **not** an error: the dispatcher replies `dispatch_ack` with +`{ eventId, duplicate: true }` and does not re-run the handler. Replay protection fails open +when Redis is unavailable. + +### Socket errors keyed by `event` (Shape A) + +The `event` value is the routing key. Most name the client event that failed; four name a +_condition_ instead, and those are the non-obvious ones a client is most likely to mishandle. + +#### Condition-valued `event` — the non-obvious ones + +| `event` | `message` | Extra fields | Meaning | +| --------------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `device_set_mismatch` | Missing envelopes for N sibling device(s) | `missingDeviceIds[]` | The send omitted envelopes for some of the **sender's own** other devices. Those devices would never be able to read the message. Re-fetch the device list, re-encrypt for the full set, resend. Emitted from `send_message`, `edit_message`, and `send_file_message` — the `event` field does **not** tell you which. | +| `protocol_mismatch` | Envelope protocol is not supported by the recipient device / Envelope protocol is weaker than both devices support | `code` (`400` or `409`), `violations[]` | An envelope named a protocol the recipient cannot use, or downgraded below what both devices support. `violations[]` is the same shape as the REST version above. | +| `rate_limited` | Rate limit exceeded | `limitedEvent`, `limit`, `retryAfterSeconds` | Per-event socket rate limit. **`limitedEvent`, not `event`, names what was throttled** — `event` is the literal string `rate_limited`. Three violations on one socket force a disconnect. | +| `envelope_too_large` | Envelope for device … exceeds size limit | — | Arrives as `code: 'envelope_too_large'` **inside** a normal `event: 'send_message'` / `'edit_message'` / `'send_file_message'` payload — it is a `code` value, not an `event` value. Listed here because it reads like a peer of the others. Distinct from `payload_too_large`, which is the whole-frame limit. | + +Two further condition-valued events come from the connection middleware in `src/index.ts`: + +| `event` | `message` | Extra fields | Meaning | +| ------------------- | ---------------------------- | ------------ | ------------------------------------------------------------------------------------------------ | +| `device_revoked` | Device has been revoked | — | The device was revoked mid-session. The socket is disconnected immediately afterwards. Terminal. | +| `payload_too_large` | Payload size N exceeds limit | — | The whole event frame exceeded `MAX_PAYLOAD_SIZE`. The event is dropped; the socket stays open. | + +#### Handler-valued `event` + +| `event` | `message` | Extra fields | +| --------------------- | -------------------------------------------------------------------------------- | ---------------------------- | +| `join_room` | Not a member of this conversation | — | +| `send_message` | messageId is required | — | +| `send_message` | Field "…" is not permitted: the server never stores session or private-key state | `code: 400` | +| `send_message` | _(from `validateMessagePayload`)_ | `code` (`400`/`403`) | +| `send_message` | Envelope for device … exceeds size limit | `code: 'envelope_too_large'` | +| `send_message` | Not a member of this conversation | — | +| `send_message` | Failed to persist message | — | +| `edit_message` | Field "…" is not permitted: the server never stores session or private-key state | `code: 400` | +| `edit_message` | originalMessageId and messageId are required | — | +| `edit_message` | Content (envelope ciphertext) must not be empty | — | +| `edit_message` | Envelope for device … exceeds size limit | `code: 'envelope_too_large'` | +| `edit_message` | Original message not found | — | +| `edit_message` | Only the original sender can edit this message | — | +| `edit_message` | Failed to persist message edit | — | +| `send_file_message` | messageId is required | — | +| `send_file_message` | Content (envelope ciphertext) must not be empty | — | +| `send_file_message` | _(from `validateMessagePayload`)_ | `code` (`400`/`403`) | +| `send_file_message` | Envelope for device … exceeds size limit | `code: 'envelope_too_large'` | +| `send_file_message` | Not a member of this conversation | — | +| `send_file_message` | File not found | — | +| `send_file_message` | File is not ready for use | — | +| `send_file_message` | File does not belong to this conversation | — | +| `send_file_message` | Access denied: you are not the uploader of this file | — | +| `send_file_message` | Failed to persist file message | — | +| `message_history` | Not a member of this conversation | — | +| `delete_message` | Message not found or not sender | — | +| `message_read` | Not a member of this conversation | — | +| `message_read` | Message not found in conversation | — | +| `message_delivered` | conversationId and messageId are required | — | +| `message_delivered` | Not a member of this conversation | — | +| `create_conversation` | One or more recipients are not accepting direct messages / … group invites | `blockedUserIds[]` | +| `create_conversation` | Too many new conversations. Please wait before starting another. | — | +| `create_conversation` | Failed to create conversation | — | +| `typing_start` | Invalid conversationId | — | +| `typing_start` | Not a member of this conversation | — | +| `typing_stop` | Invalid conversationId | — | +| `typing_stop` | Not a member of this conversation | — | +| `ask_assistant` | Not a member of this conversation | — | +| `ask_assistant` | Failed to get AI reply | — | + +`ask_assistant` also has its own `rate_limited` emission with `limitedEvent: 'ask_assistant'`, +separate from the generic per-event limiter. + +### Handshake failures + +These are **not** `error` events. They reject the connection during the Socket.IO handshake +and surface client-side on `connect_error` as a plain `Error` with only a message: + +| Message | Source | +| -------------------------------------------------- | ---------------------------------------------------------------- | +| Authentication token required | `socketAuth` — no token in the handshake | +| Invalid or expired token | `socketAuth` — JWT rejected | +| Device not found or has been revoked | `socketAuth` — token's device is gone or revoked | +| Insecure transport: connect over wss:// | `socketSecurity` — plaintext ws against an https-only deployment | +| Origin is not permitted to open a socket | `socketSecurity` — `Origin` not on the allow-list | +| Cookies may not be sent over an insecure handshake | `socketSecurity` — cookies on a non-TLS handshake | + +A client that only listens on `error` will see nothing at all for any of these. + +--- + +## Retryable vs terminal + +"Retryable" here means _the identical request may succeed later without the user changing +anything_. It is not the same as "recoverable" — most terminal errors are fixable, but only by +changing the request, the state, or the session. + +### Terminal — do not retry + +| Errors | Why | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| Every `400`, plus socket `code: 400` | The request is malformed. Retrying sends the same malformed request. Fix the payload. | +| `401` on `/auth/*` and `/devices/link/verify` (`Invalid or expired nonce`, `Signature verification failed`, `Invalid signature or wallet address`) | The challenge is single-use and consumed. **Restart from `POST /auth/challenge`** — never resubmit the same nonce. | +| `403` authorization errors (`Not a member of this conversation`, `Only the device owner may …`, `Not authorized to access this file`) | Membership or ownership does not change because you asked twice. | +| `403` `system messages are reserved for the server` | The client may never send this. | +| `403` `tls_required`, `origin_not_allowed` | Deployment configuration. Reconnect over https/wss or from an allowed origin. | +| `404` on a resource id the client supplied | The row does not exist. `Conversation has no MLS group` is the one worth special-casing: create the group rather than retrying. | +| `409` uniqueness conflicts (`Username is already taken`, `User is already a member`, `Transaction hash already exists`, `Already voted on this proposal`, `Device already registered for this user`) | The conflicting state is durable. Several are effectively idempotency signals — `Transaction hash already exists` means your transfer already landed. | +| `409` `File is already ready` | The upload was already confirmed. Treat as success. | +| `410` `Server-side search removed` | The endpoint is gone permanently. Use client-side search. | +| `413`, `415`, and `422` integrity failures (`sha256 mismatch`, `Object size mismatch`, `File integrity verification failed`) | The bytes are wrong or too big. Re-encrypt or re-upload; do not resend the same request. | +| `422` prekey / key-package caps | Retrying fails identically until existing keys are consumed. | +| Socket `device_set_mismatch` | Re-fetch the device list and re-encrypt first. Resending the same envelope set fails identically. | +| Socket `protocol_mismatch` | Re-negotiate the protocol first. | +| Socket `device_revoked` | The session is over and the socket is disconnected. Re-authenticate. | +| Socket `payload_too_large`, `envelope_too_large` | Shrink the payload. | +| All handshake failures | Fix the token, transport, or origin. | + +### Retryable — with backoff + +| Errors | How | +| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `429`, all three variants | Wait `retryAfterSeconds` / `Retry-After`, then retry once. Do not retry immediately — three socket violations force a disconnect. | +| `500` `Failed to …` | Transient server or database failure. Exponential backoff, bounded attempts. **`Failed to persist message` is the one to be careful with**: retry with the _same_ `messageId`, since the send path is idempotent on it — a new id would duplicate the message if the first attempt actually committed. | +| `503` from `GET /health` | The database is unreachable. Poll until healthy. | +| `409` `Commit epoch conflict` | A genuine race: another commit landed first. **Rebuild the commit against `currentEpoch` from the response**, then resend. Retrying the same commit body fails identically. | +| `409` `Database conflict or validation error`, `Username conflict or database error` | Ambiguous by design — these wrap both real conflicts and transient database errors. Retry once; if it repeats, treat as terminal. | +| Socket `Failed to persist message` / `Failed to persist message edit` / `Failed to create conversation` | Same as `500` above, with the same idempotency caution. | + +### Neither — re-authenticate + +`401` from `requireAuth` (`Missing or invalid Authorization header`, `Invalid or expired +token`, `Token missing deviceId`, `Device not found or has been revoked`) means the session is +no longer valid. Refresh or re-authenticate, then replay the original request once. +`Device not found or has been revoked` specifically means the device is gone — a token refresh +will not help; the user must link the device again. + +--- + +## Related documents + +- [WebSocket events](api-websocket-events.md) — every event the gateway accepts and emits. +- [WebSocket payloads](contracts-websocket-payloads.md) — success-path payload shapes. +- [REST schemas](contracts-rest-schemas.md) — success-path request and response bodies. +- [Rate limits](../../../docs/security/rate-limits.md) — every bucket and its threshold. +- [Frontend error handling](../../web/docs/concepts-error-handling.md) — how the client maps + these onto user-facing messages. diff --git a/apps/backend/docs/migrations.md b/apps/backend/docs/migrations.md new file mode 100644 index 0000000..538299a --- /dev/null +++ b/apps/backend/docs/migrations.md @@ -0,0 +1,268 @@ +# Database migration workflow + +How schema changes reach a database in this repository. Everything here is `drizzle-kit` +driven and lives under `apps/backend`; run every command from that directory (or with +`pnpm --filter backend`). + +--- + +## The one-sentence version + +Edit `src/db/schema.ts`, run `pnpm db:generate`, **read the SQL it emitted**, commit both +the schema change and the generated files together, and apply it with `pnpm db:migrate`. + +--- + +## `schema.ts` is the source of truth + +`src/db/schema.ts` is the single declarative description of the database. The files in +`drizzle/` are _output_: drizzle-kit diffs the schema against the snapshot of the last +generated state and writes the SQL needed to close the gap. + +**Never hand-write a migration first and then update the schema to match.** Doing it in that +order means the next `pnpm db:generate` diffs against a snapshot that does not reflect what +your SQL did, and it emits a second migration trying to re-apply — or worse, to undo — the +same change. The snapshot, not the database, is what drizzle-kit compares against, so it has +no way to notice that your SQL already handled it. + +There is one legitimate exception: a change drizzle-kit cannot express (a data backfill, a +`CREATE INDEX CONCURRENTLY`, a multi-step column rewrite). Handle it by generating the +migration normally first, then editing the emitted `.sql` file to add the extra statements — +so the journal entry and the snapshot still describe the change. Keep the edits inside the +generated file; do not add a `.sql` file that drizzle-kit did not create. + +Config lives in `drizzle.config.ts`: schema `./src/db/schema.ts`, output `./drizzle`, +dialect `postgresql`, and `DATABASE_URL` for credentials. + +--- + +## The workflow + +### 1. Edit the schema + +```ts +// src/db/schema.ts +export const users = pgTable('users', { + id: uuid('id').primaryKey().defaultRandom(), + username: text('username').unique(), + // ... new column here +}); +``` + +### 2. Generate + +```bash +pnpm --filter backend db:generate +``` + +This writes three things: + +- `drizzle/NNNN_.sql` — the DDL, +- `drizzle/meta/NNNN_snapshot.json` — the full schema state after this migration, +- a new entry appended to `drizzle/meta/_journal.json`. + +The `` suffix is generated (`0000_lean_scrambler`). You may rename the file to +something descriptive, but if you do you **must** update the matching `tag` in +`_journal.json` — the tag is how the runner finds the file. + +### 3. Review the emitted SQL + +This is the step people skip, and it is the one that matters. drizzle-kit infers intent from +a diff, and a diff is ambiguous in ways that lose data: + +- **A rename looks like a drop plus an add.** If you renamed a column, drizzle-kit will + usually prompt; if it guesses wrong you get `DROP COLUMN` followed by `ADD COLUMN`, and + every existing value is gone. Rewrite it as `ALTER TABLE ... RENAME COLUMN ...`. +- **A new `NOT NULL` column with no default fails on a non-empty table.** Add a default, or + split it into add-nullable, backfill, then set `NOT NULL`. +- **A type change may need a `USING` clause** that drizzle-kit will not write for you. +- **Dropping a column is silent and irreversible.** Confirm every `DROP` in the diff is one + you meant. + +If the SQL is wrong, fix the schema and regenerate rather than patching the SQL — unless it +is the "cannot be expressed" case above. + +### 4. Apply + +```bash +pnpm --filter backend db:migrate +``` + +`db:migrate` walks `_journal.json` in order and runs each migration that the target database +has not recorded yet, inside a transaction, tracking applied migrations in drizzle's own +bookkeeping table. It is the only command that should ever touch a shared database. + +`pnpm db:push` also exists. It diffs the schema straight onto a database with **no** +migration file and no journal entry, which puts that database into a state no migration +history describes. Use it for throwaway local experiments only, never against a shared or +production database, and never as a substitute for generating a migration. + +### 5. Commit together + +The schema change, the `.sql` file, the new snapshot, and the `_journal.json` change belong +in one commit. Splitting them produces a revision where the schema and the migration history +disagree. + +--- + +## The `drizzle/` directory + +``` +apps/backend/drizzle/ +├── 0000_lean_scrambler.sql # DDL for migration 0 +├── meta/ +│ ├── _journal.json # ordered list of migrations to run +│ └── 0000_snapshot.json # full schema state after migration 0 +``` + +**`.sql` files** are the migrations themselves, prefixed with a zero-padded index in apply +order. Statements are separated by drizzle's `--> statement-breakpoint` marker, which is what +`breakpoints: true` in the journal refers to; it tells the runner where one statement ends so +it can run them individually. + +**`meta/_journal.json`** is the index. Each entry carries `idx` (apply order), `version`, +`when` (generation timestamp), `tag` (the `.sql` filename without its extension), and +`breakpoints`. + +> **The journal decides what runs.** `db:migrate` reads `_journal.json`, not the directory +> listing. A `.sql` file sitting in `drizzle/` with no entry in the journal is **silently +> skipped** — no error, no warning, and the migration simply never happens. Because +> drizzle-kit rewrites the journal on every generate, this is exactly what a mishandled merge +> produces, and the failure surfaces much later as a "column does not exist" error in an +> environment where nobody remembers what changed. + +**`meta/NNNN_snapshot.json`** is the complete schema state after that migration. drizzle-kit +diffs the current `schema.ts` against the latest snapshot to work out what the next migration +should contain. A stale, missing, or hand-edited snapshot makes the _next_ contributor's +generate wrong, not yours — which is why snapshot conflicts must never be resolved by +guessing. + +`drizzle/meta/` is listed in `.prettierignore`. It is generated output, not hand-edited +source: reformatting it produces enormous noisy diffs, and drizzle-kit rewrites it in its own +format on the next generate anyway. Do not run Prettier over it, and do not "tidy" it by +hand. + +--- + +## The merge hazard + +This is the one thing to internalise, because **it has already broken this repository's +migration history once.** + +drizzle-kit numbers migrations by "one past the highest index I currently see" and appends to +the journal. It has no knowledge of other branches. So when two branches each add a +migration off the same base: + +- Branch A generates `0001_audit_logs.sql` and appends journal entry `idx: 1`. +- Branch B generates `0001_mls_key_packages.sql` and appends journal entry `idx: 1`. + +Both are correct in isolation. Merged, the result is two different migrations claiming index +`0001`, two snapshots claiming to be `0001_snapshot.json`, and two journal entries claiming +`idx: 1`. Git resolves the `.sql` files trivially — they have different names, so both are +simply kept — while the real damage happens inside `_journal.json`, where a careless +conflict resolution merges the two entries into one malformed object. + +That is precisely what happened here. Before commit `d60b648`, `drizzle/` held **seven** +distinct `0001_*.sql` files (`audit_logs`, `device_key_history`, `gc_background_jobs`, +`group_control_events`, `mls_group_state`, `mls_key_packages`, +`add_system_payload_to_messages`) — and a journal whose entry for `idx: 1` looked like this: + +```json +{ + "idx": 1, + "version": "7", + "when": 1785395646991, + "tag": "0001_mls_group_state", + "when": 1785395076224, + "tag": "0001_mls_key_packages", + "when": 1785129818340, + "tag": "0001_add_system_payload_to_messages", + "breakpoints": true +} +``` + +Duplicate keys in a JSON object are not an error — the last one wins. So that entry ran +`0001_add_system_payload_to_messages` and nothing else. Across the whole directory, thirteen +`.sql` files were present and four journal entries existed: **nine migrations were silently +skipped**, and the journal even referenced `0004_envelope_protocol` at `idx: 3` while +`0003_ciphertext_only_messages.sql` was never listed at all. The history was unrecoverable +and had to be squashed back to a single `0000_lean_scrambler.sql` baseline. + +Nothing warns you about this. `pnpm db:migrate` reports success, because from the runner's +point of view it did exactly what the journal asked. + +### Resolving a migration conflict during a merge + +When a merge touches `drizzle/`, do **not** hand-resolve `_journal.json` or the snapshots. +Regenerate instead: + +1. **Merge everything else first.** Resolve `src/db/schema.ts` on its own terms — both + branches' table and column definitions must survive, and this is the only file where a + real semantic decision is needed. +2. **Take the other branch's `drizzle/` wholesale** — the branch you are merging _into_, + normally `dev`: + + ```bash + git checkout --theirs apps/backend/drizzle # during a merge into your branch + git checkout dev -- apps/backend/drizzle # or, explicitly, from the base branch + ``` + +3. **Delete your own branch's migration files** — the `.sql` and its `meta/NNNN_snapshot.json` + — and make sure they are gone from `_journal.json`. Your change now exists only in + `schema.ts`, which is where it belongs. +4. **Regenerate:** + + ```bash + pnpm --filter backend db:generate + ``` + + drizzle-kit diffs your merged `schema.ts` against the base branch's latest snapshot and + emits a single migration at the correct next index, with a clean journal entry. + +5. **Review the new SQL.** It should contain your change and nothing from the other branch — + if it tries to re-create something the other branch's migration already made, the + snapshot you kept in step 2 was the wrong one. +6. **Verify before pushing**, against a scratch database: + + ```bash + pnpm --filter backend db:migrate + ``` + +Checks that catch the problem before it lands: + +- The number of `.sql` files in `drizzle/` equals the number of entries in `_journal.json`. +- Every `tag` in the journal names a file that exists, and every file is named by a tag. +- `idx` values are unique and contiguous from `0`. +- No index prefix appears on two files. +- `_journal.json` contains no duplicate keys within an entry. + +If you have already pushed a colliding migration, fix it by regenerating on a follow-up +commit as above. Do not renumber files by hand — the snapshots encode the chain, and renaming +a file without rebuilding its snapshot corrupts the next generate. + +--- + +## Rolling back + +There is no `db:rollback`. drizzle-kit generates forward migrations only. To undo an applied +change, generate a new forward migration that reverses it — which means destructive +migrations deserve extra review, since "revert the PR" does not revert the database. + +Where a destructive change has needed a documented undo path, this repository has used a +`drizzle/rollback/` directory holding `NNNN_.down.sql` files. Those are operator-run: +they are never listed in `_journal.json` and `db:migrate` never executes them. The directory +is absent whenever no migration currently needs one; recreate it if yours does. + +--- + +## Migrations and the test suite + +Backend tests never run migrations and never connect to Postgres — the database client is +mocked. See [Testing strategy and conventions](../../../docs/testing.md). Migrations are +validated in CI instead: the backend workflow starts a real Postgres service container and +runs `pnpm db:migrate` against it before the test step, so a migration that does not apply +cleanly to an empty database fails the build. + +That check is only as good as the journal, though. A migration missing from `_journal.json` +is skipped in CI exactly as silently as it is skipped everywhere else, so the review checks +above are not optional. diff --git a/apps/backend/docs/testing.md b/apps/backend/docs/testing.md new file mode 100644 index 0000000..2c7d0da --- /dev/null +++ b/apps/backend/docs/testing.md @@ -0,0 +1,472 @@ +# Backend testing guide + +The backend-specific companion to the cross-app +[testing strategy and conventions](../../../docs/testing.md). That document sets the rules +every suite in the repository follows — no test starts Redis, Postgres, or S3; the approved +substitutes; the runners and commands. This one is the working reference for the backend +suite specifically: the mock set a route test needs, the socket pattern, and the four or five +ways a test in this suite fails confusingly rather than clearly. + +Read the cross-app document first if you have not. Everything here assumes it. + +The conventions below are not stylistic. Each one exists because the alternative produces a +failure that does not name its cause: `undefined is not a function` from a mock that is missing +one export, a handler that "never ran" because an id was reused, a `429` that only appears when +the file runs in a particular order. Getting them right is mostly a matter of copying the +skeleton and knowing which traps are there. + +## Contents + +- [Commands and layout](#commands-and-layout) +- [The standard route-test skeleton](#the-standard-route-test-skeleton) +- [Why `await import`, always](#why-await-import-always) +- [Drizzle chain mocking, and its traps](#drizzle-chain-mocking-and-its-traps) +- [Driving socket handlers](#driving-socket-handlers) +- [Service mocks: add one whenever a route gains a dependency](#service-mocks-add-one-whenever-a-route-gains-a-dependency) +- [Resetting shared in-process state](#resetting-shared-in-process-state) +- [Checklist for a new backend test](#checklist-for-a-new-backend-test) + +## Commands and layout + +```bash +pnpm --filter backend test # the whole suite +pnpm --filter backend test -- messages.routes # one file, by substring +pnpm --filter backend test:watch +pnpm --filter backend test:coverage +``` + +| Fact | Value | +| ---------------- | ---------------------------------------------------------------------------------------------------------------- | +| Runner | Vitest, `environment: 'node'`, `testTimeout: 15000` (`vitest.config.ts`) | +| Collected | `src/**/*.{test,spec}.ts`; `node_modules` and **`dist`** excluded | +| Where tests live | `src/__tests__/*.test.ts`, plus a few `*.spec.ts` co-located with their module (`src/socket/dispatcher.spec.ts`) | +| Global setup | `src/__tests__/setup.ts` — sets `JWT_SECRET`, `DATABASE_URL`, and the `OBJECT_STORE_*` placeholders | +| Lint and format | `pnpm --filter backend lint`, `pnpm --filter backend format:check` — both cover `src/`, tests included | + +`dist` is excluded deliberately: `pnpm build` emits a compiled copy of every spec, and without +the exclusion each test would run twice, the second time against stale output. + +If a new module validates a new required environment variable at import time, add a +placeholder to `src/__tests__/setup.ts`. Do not mock the config module in each test file that +happens to import the new one transitively. + +## The standard route-test skeleton + +Almost every route test mocks the same five modules. Copy this, delete what the route under +test does not touch, and add service mocks as needed. + +```ts +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import request from 'supertest'; +import express from 'express'; + +// ── 1. Database ───────────────────────────────────────────────────────────── +// Only the query methods the route actually calls need to exist. Anything the +// route calls that is missing here fails as "not a function", which reads as a +// bug in the route rather than a gap in the mock — so add the method, do not +// change the route. +const mockFindFirst = vi.fn(); +const mockFindMany = vi.fn(); +const mockInsert = vi.fn(); +const mockUpdate = vi.fn(); +const mockDelete = vi.fn(); + +vi.mock('../db/index.js', () => ({ + db: { + query: { + conversationMembers: { findFirst: mockFindFirst, findMany: mockFindMany }, + messages: { findFirst: vi.fn() }, + }, + insert: mockInsert, + update: mockUpdate, + delete: mockDelete, + }, +})); + +// ── 2. Schema ─────────────────────────────────────────────────────────────── +// Column references become inert sentinels. Their only job is to be +// distinguishable inside an assertion — the mocked operators below never +// interpret them. Every table the router imports must appear here, even one it +// only references in a code path this test does not exercise: the import itself +// is what fails otherwise. +vi.mock('../db/schema.js', () => ({ + conversations: { id: 'id', type: 'type' }, + conversationMembers: { conversationId: 'conversationId', userId: 'userId' }, + messages: { id: 'id', conversationId: 'conversationId', createdAt: 'createdAt' }, + messageEnvelopes: { messageId: 'messageId', recipientDeviceId: 'recipientDeviceId' }, + tokenTransfers: {}, +})); + +// ── 3. Drizzle operators ──────────────────────────────────────────────────── +// Identity-ish stubs, so a test asserts on the shape a call site built rather +// than on generated SQL. Every operator the module under test imports must be +// exported here or the import throws. +vi.mock('drizzle-orm', () => ({ + and: vi.fn((...args: unknown[]) => args), + eq: vi.fn((col: unknown, val: unknown) => ({ col, val })), + ne: vi.fn(), + desc: vi.fn(), + lt: vi.fn(), + inArray: vi.fn(), + count: vi.fn(), + sql: vi.fn(), +})); + +// ── 4. Redis ──────────────────────────────────────────────────────────────── +// The getter form matters: it lets a test swap the instance (or set it to null) +// between cases without re-registering the mock. `null` exercises the degraded +// path, which is the default choice unless the test is about caching itself. +vi.mock('../lib/redis.js', () => ({ + get redis() { + return null; + }, + CONV_CACHE_TTL: 30, + convCacheKey: (userId: string) => `conversations:${userId}`, +})); + +// ── 5. Auth middleware ────────────────────────────────────────────────────── +// Inject `req.auth` rather than minting a real JWT. Include `deviceId`: several +// routes read it, and a missing one surfaces as a confusing 500 or an empty +// result set rather than a 401. +vi.mock('../middleware/auth.js', () => ({ + requireAuth: (req: express.Request, _res: express.Response, next: express.NextFunction) => { + (req as express.Request & { auth: { userId: string; deviceId: string } }).auth = { + userId: 'user-1', + deviceId: 'device-1', + }; + next(); + }, +})); + +// ── Import the module under test AFTER the mocks ──────────────────────────── +const { conversationsRouter } = await import('../routes/conversations.js'); + +function makeApp() { + const app = express(); + app.use(express.json()); + app.use('/conversations', conversationsRouter); + return app; +} + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe('GET /conversations', () => { + it('…', async () => { + const res = await request(makeApp()).get('/conversations'); + expect(res.status).toBe(200); + }); +}); +``` + +Notes on the pieces that most often go wrong: + +- **Mount only the router under test**, not `app.ts`, unless the test is specifically about + middleware ordering. A `supertest` app built from one router keeps the failure local. +- **Do not mock `../middleware/validate.js`.** Request-body validation is behaviour a route + test should exercise; mocking it hides a whole class of `400`s the real route returns. +- **Auth tests use the real middleware.** The mock above is for tests that are about something + else. +- **`vi.clearAllMocks()` in `beforeEach` clears calls and implementations of mock functions + only.** It does not touch module-level state inside the code under test — see + [resetting shared in-process state](#resetting-shared-in-process-state). + +## Why `await import`, always + +`vi.mock` calls are hoisted to the top of the file, but the factories close over `vi.fn()` +handles declared below them. A static `import` of the module under test binds the real `db` +before those handles exist. Every backend suite that mocks the database therefore imports the +subject with a top-level `await import(...)`: + +```ts +const { messagesRouter } = await import('../routes/messages.js'); +``` + +The symptom of getting this wrong is a test that tries to open a real Postgres connection and +either hangs to the 15-second timeout or fails inside a driver stack trace with no mention of +your route. + +## Drizzle chain mocking, and its traps + +The cross-app guide covers the general pattern. These are the specific chains this suite hits. + +### `.values()` must be both thenable and expose `.returning()` + +Drizzle's insert builder is a thenable. `db.insert(t).values(rows)` is itself awaitable and +executes the statement, **and** `.returning()` can be chained onto it to execute and get the +inserted rows back. Both forms are used in this codebase, sometimes inside a single +transaction: the message insert needs the generated `id` and `createdAt` so it calls +`.returning()`, while the envelope batch insert (`insertMessageEnvelopes`) only cares that the +rows landed and just awaits `.values(...)`. + +The two wrong stubs fail in opposite, equally unhelpful ways: + +| Stub returns | What breaks | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `{ returning }` alone | The awaited `.values(...)` resolves to a plain object and records nothing. **The test passes** while asserting on an insert that never happened. | +| A bare promise | `.returning is not a function`, thrown from inside the route. | + +So a shared insert stub has to satisfy both shapes: + +```ts +function insertStub(table: string) { + return { + values: (vals: unknown) => ({ + returning: async () => recordInsert(table, vals), + then: (resolve: (value: unknown) => void) => resolve(recordInsert(table, vals)), + }), + }; +} +``` + +`src/__tests__/e2ee.integration.test.ts` is the canonical version. Two cautions: + +- **Do not add `then` to a stub whose call sites never await the builder directly.** A thenable + is awaited implicitly whenever it is returned from an `async` function, which can fire the + recording side effect a second time. +- **If both forms run against the same stub, make the recorder idempotent** — or assert on call + counts you have actually verified, rather than assuming one insert equals one recorded row. + +### Transactions + +A route that writes more than one table wraps the work in `db.transaction`. Mock it as a +function that invokes its callback with an object exposing the same stubbed builders, plus +whatever `query.*` methods the transaction body uses (`insertMessageEnvelopes`, for example, +calls `tx.query.devices.findMany`): + +```ts +const mockTransaction = vi.fn(async (cb: (tx: unknown) => Promise) => + cb({ + insert: insertStub, + query: { devices: { findMany: mockDeviceFindMany } }, + }), +); +``` + +A transaction body that throws must propagate, so do not wrap the callback in a `try`/`catch` +in the mock — the route's own error handling is usually the thing under test. + +### Read chains and raw SQL + +`db.select().from().where().groupBy()` needs each link to return the next one, innermost first: + +```ts +const mockGroupBy = vi.fn().mockResolvedValue([]); +const mockWhere = vi.fn(() => ({ groupBy: mockGroupBy })); +const mockFrom = vi.fn(() => ({ where: mockWhere })); +const mockSelect = vi.fn(() => ({ from: mockFrom })); +``` + +`db.execute(sql\`…\`)`is used for the aggregate subqueries (unread counts). Mock it as a plain`vi.fn()`resolving to the row array; the route spreads the result, so resolve to an array and +not to a`{ rows }` object. + +When a module uses `sql` as both a tag and a namespace (`sql.join(...)`), the `drizzle-orm` +mock has to provide both: + +```ts +const sqlMock = Object.assign( + vi.fn(() => 'sql'), + { join: vi.fn(() => 'joined') }, +); +``` + +### The two import-time failures + +Both present as an error in a file you did not touch, so recognise them by shape: + +- **A table missing from the `db/schema.js` mock** → the router's import of that name is + `undefined`, and the first use is a `Cannot read properties of undefined` far from the cause. + Add the table to the mock with the columns the route names. +- **An operator missing from the `drizzle-orm` mock** → an ESM named-export error naming the + operator (`ne`, `isNull`, `inArray`, `count`, `desc`, `lt`, `gte`, `sql`). Add it as + `vi.fn()`. + +Both happen when a route gains a new query, which is the same trigger as the service-mock rule +below. + +## Driving socket handlers + +Every client-to-server socket event goes through one enveloped `dispatch` event +(`src/socket/dispatcher.ts`). `dispatcher.register(type, handler)` stores the handler in a map +and `listen()` attaches exactly one `socket.on('dispatch', ...)` listener, which checks +authentication, validates the envelope, rejects unknown types and stale timestamps, and applies +`eventId` replay protection before any handler is reached. + +**Grabbing a raw listener no longer works.** A test that reaches into the emitter looking for +a listener registered for `'send_message'` finds nothing, because none is registered. Worse, a +test written that way against an older revision _passed_ — while bypassing envelope validation, +idempotency, and the auth gate, which are exactly the checks those events need to be tested +through. `src/socket/dispatcher.spec.ts` has a test asserting that a raw emit does **not** reach +the handler; treat it as the specification. + +Drive handlers by emitting a well-formed envelope: + +```ts +let envelopeSeq = 0; + +function dispatchEvent(socket: EventEmitter, type: string) { + return async (payload: unknown) => { + envelopeSeq += 1; + // EventEmitter.prototype.emit.call bypasses the fake socket's emit override, + // so this delivers to the listener instead of being captured as an outbound + // server -> client emit. + EventEmitter.prototype.emit.call(socket, 'dispatch', { + eventId: `test-evt-${envelopeSeq}`, + type, + timestamp: Date.now(), + payload, + }); + await new Promise((resolve) => setTimeout(resolve, 10)); + }; +} +``` + +Preserve all five of these when copying it: + +1. **A fresh `eventId` per emit.** Replay protection is keyed by + `replay:{deviceId}:{eventId}`, so a reused value turns the second and later events into + no-ops and produces a "the handler never ran" failure with no error attached. The check + _fails open_ when Redis is `null`, so a suite that mocks Redis away will not catch a reused + id — and the same test starts failing the day someone gives the suite an `ioredis-mock` + instance. Generate a fresh id regardless. See + [`concepts-replay-protection.md`](./concepts-replay-protection.md). +2. **A current `timestamp`.** Envelopes outside `SOCKET_EVENT_MAX_AGE_MS` (5 minutes) in the + past or `SOCKET_EVENT_MAX_FUTURE_SKEW_MS` (30 seconds) in the future are rejected before the + handler runs. With `vi.useFakeTimers()`, keep the envelope timestamp inside the window. +3. **Set `socket.auth` first** (`{ userId, deviceId }`). An unauthenticated socket gets an + `error` envelope back and the handler is never reached. +4. **Await a tick.** The dispatch listener is `async` and `emit` returns synchronously, so + assert after the `setTimeout` above. +5. **Register through the real registrar** — `registerMessagingHandlers(io, socket)` — rather + than pulling a handler function out of the module, so whatever the registrar installs stays + in the path. + +Assertions usually read the socket's captured emits. `dispatch_ack` is the dispatcher's own +acknowledgement: `{ eventId, duplicate: false }` on a first occurrence and +`{ eventId, duplicate: true }` on a replay, which is the cleanest way to assert that an event +was processed exactly once. + +**The one exception:** `send_file_message` is still attached with a raw +`socket.on('send_file_message', ...)` in `src/socket/messaging.ts`, so it is triggered by +emitting that event name directly. It is the only handler for which that is correct. If it +moves onto the dispatcher, its tests move with it. + +Worked examples: `src/__tests__/dispatcher.test.ts` and `src/socket/dispatcher.spec.ts` for the +dispatcher itself, `src/__tests__/askAssistant.test.ts` for a handler driven through +`dispatch`. + +## Service mocks: add one whenever a route gains a dependency + +Route tests mock the database, but a route also calls services — and an unmocked service runs +for real, which means it reaches for the database _it_ imported, or Redis, or the object store. +The failures are indirect: a timeout, a `500` from a service several frames down, or a passing +test that quietly performed a real side effect. + +**When a route or handler gains a new service dependency, every existing test file for that +route needs the new `vi.mock`, in the same change.** This is the single most common way a +change breaks unrelated backend tests, and the error message never names the new import. + +Frequently mocked services, with what they stand in for: + +| Module | Why a route test mocks it | +| --------------------------------- | ------------------------------------------------------------------------------------------------- | +| `../services/pushNotification.js` | Would attempt a real web-push send | +| `../services/deliveryPipeline.js` | Envelope delivery and per-device fan-out | +| `../services/deviceDelivery.js` | Opens a duplicated Redis subscriber connection | +| `../services/e2eeProtocol.js` | Queries `devices` for capabilities; mock when the test is not about protocol negotiation | +| `../services/mlsGroups.js` | Group epoch state | +| `../services/presence.js` | Redis presence keys | +| `../services/deviceRevocation.js` | Revocation broadcast | +| `../services/roomManager.js` | Socket room membership | +| `../services/auditLog.js` | Writes an audit row on many routes | +| `../services/fileCleanup.js` | Soft-delete plus object-store work | +| `../lib/socket.js` | `getSocketServer()` — return `null`, or `{ to: () => ({ emit }) }` when the test asserts on emits | + +Mock at the seam the route imports, and give the mock the same shape the real export has +(`vi.fn().mockResolvedValue(undefined)` for a fire-and-forget async service). A service mocked +as a synchronous `vi.fn()` where the route awaits it works by accident and stops working the +moment the route checks the result. + +When the test _is_ about the service, mock the service's own dependencies instead and exercise +the real thing — `src/__tests__/e2eeProtocol.test.ts` mocks only `db`, `db/schema`, and +`drizzle-orm`, and runs the real `checkEnvelopeProtocols`. + +## Resetting shared in-process state + +Several modules keep module-level state that survives across tests inside a Vitest worker. +`vi.clearAllMocks()` does not touch it, because it is not a mock — it is the real module's +memory. + +Rate limiting is the one that bites most often. `services/rateLimiter.ts` keeps a +`localCounters` map used whenever Redis is unavailable — which, in a suite that mocks `redis` +to `null`, is always. The map is keyed by bucket, window, and subject, and the window comes +from wall-clock time, so several tests in one file hitting the same endpoint as the same +subject are all charged against **one** budget. The symptom is a test that passes alone and +returns `429` when the file runs in order, or a failure that moves when you reorder the file. + +```ts +const { resetRateLimitBucket, clearLocalRateLimitCounters } = + await import('../services/rateLimiter.js'); + +beforeEach(async () => { + vi.clearAllMocks(); + clearLocalRateLimitCounters(); // the process-local fallback map + await resetRateLimitBucket('auth_challenge'); // that bucket's Redis keys too + await resetRateLimitBucket('auth_verify'); + await resetRateLimitBucket('global_ip'); +}); +``` + +`clearLocalRateLimitCounters()` clears only the in-process map. `resetRateLimitBucket(bucket)` +clears the matching local keys **and** scans and deletes the bucket's keys in Redis, real or +`ioredis-mock`. Where a suite shares one `ioredis-mock` instance, `await sharedRedis.flushall()` +in `beforeEach` is the blunter equivalent for the Redis half. + +The same shape of leak exists elsewhere. Each affected module exports its own hook — use it +rather than reaching into the module: + +| Module | State it keeps | Reset | +| ----------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------- | +| `services/rateLimiter.ts` | Fallback counters plus Redis buckets | `clearLocalRateLimitCounters()`, `resetRateLimitBucket(bucket)` | +| `services/rateLimit.ts` | Per-socket repeat-violation counts | `clearViolations(socketId)` | +| `services/prekeyLowSignal.ts` | One-shot alert latches, so a second low-prekey event does not re-alert | `__resetPrekeyLowLatches()` | +| `services/presence.ts` | Offline-broadcast dedupe set | `__resetOfflineBroadcastsForTesting()` | +| `services/heartbeat.ts` | Per-socket heartbeat timers | `clearHeartbeatTimer(socketId)` | +| `lib/objectStore.ts` | Memoised S3 client | `resetObjectStoreForTests()` | + +Timers are a second form of leaked state: a test that leaves a `setTimeout` pending (typing +indicators, heartbeats) can fire it during a later test. Clear timers the handler created, or +use `vi.useFakeTimers()` and dispose of them in `afterEach`. + +The general rule: **if a module keeps state outside a function so that production behaves +correctly across requests, it needs a test-visible reset, and every suite that touches it calls +that reset in `beforeEach`.** When you add such state, export the reset in the same commit. + +## Checklist for a new backend test + +1. File in `src/__tests__/` as `*.test.ts` (or co-located `*.spec.ts` for a single module). +2. Mocks declared before the subject; subject imported with `await import(...)`. +3. `db/index.js`, `db/schema.js`, `drizzle-orm`, `lib/redis.js`, `middleware/auth.js` mocked — + with every table and operator the module imports present. +4. Every service the route calls mocked, unless the service is what is under test. +5. Socket events driven through `dispatch`, with a fresh `eventId` and a current `timestamp`. +6. Shared in-process state reset in `beforeEach`. +7. Assertions on behaviour — status, body, emitted events, rows handed to `insert`/`update` — + never on generated SQL. +8. No security invariant weakened to make a test pass. The guards in + `src/__tests__/security.regression.test.ts` have their own CI job; if a change trips them, + the change is wrong. +9. `pnpm --filter backend test`, `lint`, and `format:check` all clean. + +## Related documents + +- [Testing strategy and conventions](../../../docs/testing.md) — the cross-app rules this + document sits under. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — why `eventId` + must be fresh per emit and `messageId` must not be. +- [Backend caching reference](./concepts-caching.md) — what the `lib/redis.js` mock is standing + in for, and the degraded path a `null` Redis exercises. +- [Gateway architecture](./concepts-gateway-architecture.md) — the socket lifecycle the + dispatcher tests model. +- [Database migration workflow](./migrations.md) — why the suite never runs migrations. diff --git a/apps/web/docs/accessibility.md b/apps/web/docs/accessibility.md new file mode 100644 index 0000000..c5e95fd --- /dev/null +++ b/apps/web/docs/accessibility.md @@ -0,0 +1,231 @@ +# Frontend accessibility guide + +The accessibility standard `apps/web` targets, and the patterns required to meet it: +keyboard navigation through the conversation list and message composer, focus management +in modals and the safety-number panel, screen-reader announcement of incoming messages, +and colour contrast. + +This document describes both the pattern to follow and, honestly, where the current code +already meets it and where it does not yet. Where a gap is called out, treat it as +something to fix when you touch that area, not as the intended design. + +--- + +## Target standard + +**WCAG 2.1 Level AA.** This is the conventional baseline for a web application handling +real user-to-user communication, and is the standard assumed throughout this document — +there is no stricter internal bar and no formal deviation from it. + +**How it is checked today: manually, not automatically.** There is no `eslint-plugin-jsx-a11y`, +no automated axe/Lighthouse run in CI, and no accessibility test suite in this repo as of +this writing. Conformance currently depends entirely on the patterns below being followed +by hand and reviewed in PRs. If you are adding accessibility tooling, wiring an automated +check (axe-core in CI, or `eslint-plugin-jsx-a11y` at minimum) closes a real gap rather +than adding redundant coverage — until then, the [pre-merge checklist](#pre-merge-checklist-for-a-new-interactive-component) +below is the actual enforcement mechanism. + +--- + +## Keyboard navigation + +### Conversation list + +`components/conversations/ConversationListSidebar.tsx` renders each conversation as a +plain ``: + +```tsx + + {/* avatar, title, preview */} + +``` + +There is no custom keyboard handling — no roving `tabindex`, no arrow-key list navigation. +Keyboard support comes entirely from using a real `` (an anchor): it is naturally +focusable, appears in Tab order, and activates on Enter, all for free. **This is the +required pattern for list items in this app** — a clickable row must be a real +link/button, never a `
`, precisely so keyboard support does not have to be +hand-built. Arrow-key roving-tabindex navigation (a full ARIA `listbox`/`menu` pattern) is +not implemented and is not required — sequential Tab order through the list is the +supported navigation model. + +### Message composer + +The composer (`app/app/conversations/[id]/page.tsx`) supports Enter-to-send / +Shift+Enter-for-newline on the message input: + +```tsx + setSendText(e.target.value)} + onKeyDown={(e) => { + if (e.key === 'Enter' && !e.shiftKey) { + e.preventDefault(); + void handleSendEncrypted(); + } + }} +/> +``` + +When adding a keyboard shortcut like this, always `preventDefault()` only on the branch +that consumes the key (here, plain Enter) and let every other key (including Shift+Enter) +fall through untouched — do not swallow keys you are not handling. + +The composer's send and attach-file buttons are real `