diff --git a/AGENTS.md b/AGENTS.md index 2434ab8fa..40d25bf9f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,6 +5,22 @@ This file defines repository-level operating assumptions for automated agents, including mandatory quality and security merge gates. +## Canonical product and architecture authority + +Before selecting or changing product work, read the current canonical graph: + +- `docs/PRD.md` for product requirements and maturity; +- `docs/TRD.md` for technical requirements and evidence boundaries; +- root `ARCHITECTURE.md` for runtime, trust, deployment, and document authority; +- `docs/adr/README.md` for accepted, proposed, active, and planned decisions; +- `docs/TRACEABILITY.md` for requirement-to-code/test/issue/PR evidence; +- `docs/DOCUMENTATION_ASSESSMENT.md` for documentation fitness and open gaps. + +Repository documents are the detailed authority. Scheduler prompts and PR prose +are execution/control evidence, not a second independently maintained product +specification. Active-PR behavior must never be described as protected-main +shipped behavior. + ## Mandatory merge gates - `mvn -B --no-transfer-progress verify` is the authoritative local and CI diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 13dbeda7e..6744bf506 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,89 +1,289 @@ -# Architecture Map - -Last updated: 2026-02-23 - -## System Purpose - -Clearfolio Viewer is an MVP backend that accepts document uploads, processes conversion asynchronously, exposes conversion status, and serves viewer bootstrap metadata when conversion succeeds. - -Runtime stance: Spring WebFlux is adopted as the current non-blocking web runtime (Servlet/MVC path is not the selected implementation for this repo). - -S2S chain (target integration path): `Clearfolio Viewer <-> internal WAS -> Azure On-premise Gateway -> Power Platform -> mobile/tablet`. -Current state: viewer/state API is implemented in this repository; downstream S2S orchestration remains planned and documented. - -## Runtime Components - -- `ConversionController` (`src/main/java/com/clearfolio/viewer/controller/ConversionController.java`) - - `POST /api/v1/convert/jobs`: async submit contract. - - `POST /api/v1/convert/jobs/{jobId}/retry`: operator retry for dead-lettered jobs. - - `GET /api/v1/convert/jobs/{jobId}`: status polling. - - `GET /api/v1/viewer/{docId}` (+ alias): viewer bootstrap JSON/state-gated responses. -- `ViewerUiController` (`src/main/java/com/clearfolio/viewer/controller/ViewerUiController.java`) - - `GET /viewer/{docId}`: HTML viewer UI entrypoint (loading/failed/ready) that embeds PDF.js. -- `ArtifactController` (`src/main/java/com/clearfolio/viewer/controller/ArtifactController.java`) - - `GET /artifacts/{docId}.pdf`: serves PDF bytes for SUCCEEDED jobs with basic HTTP Range support. -- `DefaultDocumentConversionService` (`src/main/java/com/clearfolio/viewer/service/DefaultDocumentConversionService.java`) - - Validation, content hash generation, dedupe lookup, repository persistence, worker enqueue. - - PDF passthrough: uploads that declare PDF (extension/content type) and carry the `%PDF-` magic header are seeded into the artifact store as-is, so the original bytes are served instead of a generated placeholder. -- `DefaultDocumentValidationService` (`src/main/java/com/clearfolio/viewer/service/DefaultDocumentValidationService.java`) - - Enforces extension blocklist and size limits, including auditable policy-override exception lane. -- `DefaultConversionWorker` (`src/main/java/com/clearfolio/viewer/service/DefaultConversionWorker.java`) - - Runs conversion on a bounded executor with retry scheduling and dead-letter fallback. -- `ArtifactStore` (`src/main/java/com/clearfolio/viewer/artifact/ArtifactStore.java`) - - Stores converted PDF bytes by docId. -- `FileSystemArtifactStore` (`src/main/java/com/clearfolio/viewer/artifact/FileSystemArtifactStore.java`) - - Default disk-backed artifact store; persists bytes plus minimal metadata under `clearfolio.artifact-store.root-dir` (default `data/artifacts`) so artifacts survive restarts, with an in-memory cache on the read path. -- `InMemoryArtifactStore` (`src/main/java/com/clearfolio/viewer/artifact/InMemoryArtifactStore.java`) - - In-memory artifact store implementation used by tests and `clearfolio.artifact-store.mode=in-memory`. -- `ArtifactStoreConfig` (`src/main/java/com/clearfolio/viewer/config/ArtifactStoreConfig.java`) - - Selects the artifact store implementation from `ArtifactStoreProperties`. -- `PdfBoxArtifactGenerator` (`src/main/java/com/clearfolio/viewer/artifact/PdfBoxArtifactGenerator.java`) - - Generates a placeholder one-page PDF via PDFBox for non-PDF sources; real docx/hwp conversion remains future work. -- `InMemoryConversionJobRepository` (`src/main/java/com/clearfolio/viewer/repository/InMemoryConversionJobRepository.java`) - - In-memory job store and content-hash dedupe index. -- `ConversionJob` (`src/main/java/com/clearfolio/viewer/model/ConversionJob.java`) - - Domain lifecycle and retry metadata (`attemptCount`, `maxAttempts`, `retryAt`, `deadLettered`) plus manual dead-letter retry transition. -- `ViewerBootstrapResponse` (`src/main/java/com/clearfolio/viewer/api/ViewerBootstrapResponse.java`) - - Includes deterministic `sourceExtension` and `rendererAdapter` metadata for viewer adapter bootstrap. - -## State Model - -- Status values: `SUBMITTED`, `PROCESSING`, `SUCCEEDED`, `FAILED`. -- Retry-exhausted terminal state remains `FAILED` and is identified by `deadLettered=true`. - -## Operational Gates - -- Build and test gates are defined in `AGENTS.md` and include: - - `mvn -DskipTests compile` - - `mvn test` - - JaCoCo line/branch 100% for `com.clearfolio.viewer.*` - - JavaDoc gate: `mvn -q -DskipTests javadoc:javadoc` - - Markdown lint for changed docs - -Mandatory AC list (exact): - -1. coverage -2. docstring -3. non-blocking web -4. lightweight queue -5. warning 0 -6. deprecated 0 -7. 1-day schedule+security verification - -Optional tracks: - -- client DB pooler -- PostgreSQL 17 - -## Detailed Design Docs - -- `docs/architecture.md` -- `docs/prd-integrated-document-viewer-platform.md` -- `docs/trd-integrated-document-viewer-platform.md` -- `docs/diagrams/submit-flow.md` -- `docs/diagrams/submit-policy-adapter-flow.md` -- `docs/diagrams/status-flow.md` -- `docs/diagrams/preview-flow.md` -- `docs/diagrams/retry-deadletter-flow.md` -- `docs/engineering/acceptance-criteria.md` -- `docs/workflow/one-day-delivery-plan.md` +# Clearfolio Architecture + +Last updated: 2026-08-10 +Canonical documentation baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +## Purpose and status vocabulary + +Clearfolio is an independently deployable secure document-conversion and viewing service that can also compose with naruon and other ContextualWisdomLab hosts through explicit versioned interfaces. + +This architecture distinguishes: + +- `IMPLEMENTED_ON_MAIN`: protected default-branch behavior; +- `ACTIVE_PR`: implemented only in an open PR and not yet shipped; +- `PARTIAL`: useful protected-main subset exists but the product contract is incomplete; +- `ACCEPTED_ARCHITECTURE`: agreed boundary without complete protected-main implementation; +- `PLANNED`: backlog work. + +Historical MVP PRD/TRD/plans remain useful provenance but do not override this map, `docs/PRD.md`, `docs/TRD.md`, or live protected code. + +## Product and trust topology + +```mermaid +flowchart LR + User[Document user / API client] + Operator[Tenant operator] + Gateway[Trusted identity / gateway] + + subgraph CF[Clearfolio] + HTTP[WebFlux controllers] + Auth[Tenant authorization] + Convert[Conversion orchestration] + Worker[Bounded async worker] + Repo[Job repository/state boundary] + Artifact[Artifact store] + Link[Signed artifact delivery] + Evidence[Audit / ledger / QA evidence] + Viewer[Viewer / PDF.js] + Availability[Health / readiness] + end + + User --> HTTP + Operator --> HTTP + Gateway --> Auth --> HTTP + HTTP --> Convert --> Repo + Convert --> Worker + Worker --> Artifact + HTTP --> Link --> Artifact + Link --> Evidence + HTTP --> Viewer + HTTP --> Availability + + Naruon[naruon / CWL host] -. versioned API .-> HTTP + Central[ContextualWisdomLab/.github] -. development governance .-> CF +``` + +Uploaded documents, document-derived strings, browser data, model output and PR text are untrusted input. Clearfolio server-side deterministic code remains authority for conversion validation, tenant authorization, artifact delivery, fidelity acceptance and release gates. + +## Protected-main runtime components + +### HTTP layer + +- `ConversionController` + - asynchronous submit; + - job status; + - dead-letter retry; + - delete/direct-download/viewer bootstrap surfaces according to the current code; + - tenant and permission checks before protected state access. +- `ArtifactController` + - signed artifact-link issuance and revocation; + - artifact-read audit retrieval; + - signed PDF delivery with token/checksum/revocation validation; + - zero-or-one HTTP byte range support. +- `ViewerUiController` + - HTML/PDF.js viewer shell. +- `ApiExceptionHandler` + - controlled API failure mapping. +- health/availability controller(s) + - protected main currently exposes the existing health surface; explicit liveness/readiness split is `ACTIVE_PR` #295. + +### Authorization and security + +- `TenantContext`, `TenantAccessService`, `TenantPermissions` + - request tenant/subject/permission boundary and same-tenant checks; + - HMAC-signed claim configuration exists in the current scaffold; active stack hardens production fail-closed behavior. +- `ArtifactLinkService` + - short-lived signed artifact-read claims tied to token id, tenant, subject, document, scope, purpose, checksum, issue and expiry times; + - issued-token ledger, revocation, checksum binding and read audit. + +`IMPLEMENTED_ON_MAIN` through protected #270 for signed claims, dedicated least privilege, key separation, and HMAC audit pseudonymization. #268 is `SUPERSEDED`; immutable lifecycle identity and admin/deletion boundaries remain `PARTIAL` in unintegrated bounded successors. + +### Conversion and lifecycle + +- `DefaultDocumentConversionService` + - validation, content identity/dedupe, job state, enqueue and artifact orchestration. +- `DefaultDocumentValidationService` + - blocked-format and size/policy boundary. +- `DefaultConversionWorker` + - bounded async execution, retry/dead-letter behavior and recovery against state available in the configured repository. +- `ConversionJob` / `ConversionJobStatus` + - public lifecycle uses `SUBMITTED`, `PROCESSING`, `SUCCEEDED`, `FAILED`; dead-letter exhaustion is represented with evidence/flag rather than a new public terminal enum. + +### State and artifacts + +- `InMemoryConversionJobRepository` + - protected-main default job state and content-hash dedupe boundary; process-local by default. +- `FileSystemArtifactStore` + - configurable local filesystem artifact persistence plus read cache. +- `InMemoryArtifactStore` + - test/development artifact mode. +- append-only/file-backed ledgers + - selected artifact-link and analytics evidence can survive restart when configured. + +These persistence choices do **not** establish a durable distributed job system. SQL-backed lifecycle state, distributed idempotency/backpressure/cancellation and remote-object-store transaction/fencing remain `PLANNED`; #268 is `SUPERSEDED`, and deletion-receipt hardening remains a `PARTIAL` target across unintegrated bounded successors. + +## Conversion truth and fidelity boundary + +### `IMPLEMENTED_ON_MAIN` + +- validated PDF passthrough can preserve original PDF bytes in the artifact path; +- PDF.js renders controlled PDF artifacts. + +### `PARTIAL` + +`PdfBoxArtifactGenerator` can generate a placeholder one-page PDF for non-PDF sources. This is development/demo behavior and **not** a production-fidelity DOCX/HWP/Office conversion implementation. + +### `ACTIVE_PR` / `PLANNED` + +Draft #306 defines a provider-neutral `OfficeConversionAdapter`, source preflight and post-provider PDF publication checks. The production Office runtime remains `PLANNED`: a sandboxed sidecar or independently operated remote service must own Office-process execution outside the API-container trust boundary. The deterministic fixture adapter is an offline contract oracle, not a production Office engine. + +Each advertised transformed source format must have a qualified deterministic converter and realistic authorized fixture suite proving structure/rendering/security/failure expectations. Macro/active-content execution and implicit external resource fetching are not accepted production authority paths. Unsupported or unverifiable inputs fail closed instead of silently producing a placeholder branded as successful conversion. + +See ADR-0005, `docs/FIDELITY_ACCEPTANCE.md`, issue #5, `docs/UML.md`, `docs/DATA_MODEL.md`, and the fidelity requirements in `docs/PRD.md` / `docs/TRD.md`. + +## Artifact delivery authority + +Canonical protected-main artifact delivery uses: + +```text +tenant/job state +→ artifact bytes +→ signed token resolution +→ signature / expiry / scope / doc checks +→ issued-token ledger / revocation +→ tenant + current artifact checksum binding +→ single-range resolution +→ controlled read audit +→ 200 / 206 / controlled failure +``` + +Dedicated tenant permission and signed artifact-token authority are separate controls where both apply. Protected #270 aligned the direct conversion-job download endpoint with this contract (`IMPLEMENTED_ON_MAIN`); permission-only direct byte delivery is not the accepted architecture. + +## Lifecycle and recovery + +Current protected-main state machine: + +```mermaid +stateDiagram-v2 + [*] --> SUBMITTED + SUBMITTED --> PROCESSING + PROCESSING --> SUCCEEDED + PROCESSING --> SUBMITTED: retry / recover + PROCESSING --> FAILED: terminal or exhausted + FAILED --> SUBMITTED: authorized dead-letter retry +``` + +#268 is `SUPERSEDED`. Bounded successors preserve restart-safe artifact deletion receipts, immutable lifecycle generations, generation-fenced cleanup, durable failed-attempt evidence, replay validation, and recovery fairness as `PARTIAL` target semantics; do not treat them as protected-main behavior until integration. + +`docs/MIGRATION_ROLLBACK.md` is the canonical compatibility/recovery authority for file-ledger, artifact-store, converter/runtime, future SQL and public-contract changes. It explicitly distinguishes current process-local job state from future persistent architecture and forbids rollback that re-enables a known authorization bypass. + +## Accessibility + +Protected main has a responsive viewer/demo shell and PDF.js path. `ACTIVE_PR` #264 adds contextual names for repeated actions and reusable nested/reentrant busy-state semantics that preserve original DOM node identity, disabled state and ARIA attributes. Broader keyboard/screen-reader/print/export acceptance remains product work. + +## Availability + +Protected main has an existing health endpoint. `ACTIVE_PR` #295 separates process liveness from traffic readiness so routing can stop without unnecessary process restart and probe responses remain low-information/no-store. Shared dependency health must not turn liveness into a restart cascade. + +## Standalone / MSA ownership + +Clearfolio owns: + +- document validation and conversion contract; +- conversion-job and artifact interfaces; +- local tenant authorization enforcement; +- artifact token/delivery semantics; +- product-specific audit/evidence contracts; +- its own migrations/state adapters when durable persistence is introduced. + +A host such as naruon may own: + +- higher-level user/product workflow; +- upstream identity/federation and trusted claim exchange; +- integration orchestration and deployment composition. + +A host does not write Clearfolio application persistence directly. `contextual-orchestrator` may be used for explicitly model-backed features outside deterministic conversion/authorization authority; Clearfolio remains usable without it. + +## Development automation architecture + +### Central PR maintenance + +`ACCEPTED_ARCHITECTURE`: privileged PR review/repair/exact-head gate/protected-merge automation belongs to `ContextualWisdomLab/.github`. Clearfolio should use thin contracts/callers instead of copying central reviewer/merge authority. + +### Local product development (`ACTIVE_PR` #271) + +The local OpenCode path: + +1. refetches open PR/path/base state; +2. performs RCA → distinct remedies → real-world feasibility; +3. proposes one bounded path-disjoint change; +4. treats model output as untrusted patch data; +5. runs full verification in a fresh credential-free environment; +6. publishes only a Draft with short-lived scoped authority; +7. never self-approves, merges, releases or deploys. + +Model-backed development uses GitHub Secret `NVIDIA_NIM_API_KEY`, never `COPILOT_GITHUB_TOKEN` as a development-model credential, and preserves independent reviewer credentials. + +The loop is work-conserving: queued checks, review latency, external approval, a failed first remedy, one completed documentation artifact or one green PR blocks only that action. It rotates to the next safe item and requires two fresh exit sweeps before a normal no-work termination. + +## Evidence and merge authority + +The following are distinct and never collapsed: + +- exact source revision; +- independently resolved current base revision; +- CI/security/fuzz/check evidence; +- commit status evidence; +- model/advisory review evidence; +- formal GitHub review/approval evidence; +- synthetic merge compatibility; +- protected-main operational evidence. + +PR-body SHAs and old run IDs are historical narrative. Any head movement invalidates predecessor-head assumptions. See ADR-0007. + +## Quality / release gates + +Canonical full local acceptance begins with: + +```bash +mvn -B --no-transfer-progress verify +python3 scripts/verify_maven_test_reports.py +``` + +Required policy includes: + +- zero skipped/failing/error test-report evidence and positive executed count; +- exact 100% owned production line/branch acceptance; +- warning-free public Javadocs; +- required CI/security/SAST/fuzz checks for the unchanged candidate revision; +- no unresolved valid review findings; +- qualifying independent approval when policy requires it; +- realistic fidelity and accessibility evidence for release claims; +- SBOM/attribution/provenance/reproducibility; +- migration/rollback/recovery and protected-main operational evidence when applicable. + +Queued, pending, skipped-required, cancelled, absent, stale-head, predecessor-head, synthetic-only, status-only and model-only outcomes are not passing release evidence. + +`docs/RELEASE_ACCEPTANCE.md` is the integrated release authority. It requires one exact protected source/artifact identity and keeps source-head, live-base, workflow, review, security, fidelity/recovery and SBOM/provenance evidence separate instead of inferring release readiness from one green PR or status. + +## Canonical documentation spine + +Core product/architecture graph: + +- `docs/PRD.md` — product users, outcomes, scope, fidelity and release requirements. +- `docs/TRD.md` — current/active/target technical contracts. +- `ARCHITECTURE.md` — this system/trust/ownership map. +- `docs/adr/README.md` — status-bearing durable decisions. +- `docs/DATA_MODEL.md` — logical ERD and persistence ownership. +- `docs/UML.md` — component, sequence, lifecycle, failure and deployment diagrams. +- `docs/API_CONTRACT.md` — API authority/versioning/integration boundary. +- `docs/THREAT_MODEL.md` — current threat/trust model. +- `docs/TEST_STRATEGY.md` — TDD, security, fidelity, recovery and release test strategy. +- `docs/OPERABILITY.md` — startup, degraded behavior, recovery, backup/restore and release operations. +- `docs/TRACEABILITY.md` — requirement/ADR → implementation/test/PR/evidence map. +- `docs/RESEARCH_TRACEABILITY.md` — standards, primary technical sources and peer-reviewed methodological evidence. +- `docs/DOCUMENTATION_ASSESSMENT.md` — dated completeness and drift assessment. +- `docs/ACQUISITION_DILIGENCE.md` — current product/engineering/security/operability/license/IP/acquisition evidence and external-diligence boundary. +- `docs/engineering/acceptance-criteria.md` — executable engineering acceptance-gate authority and evidence semantics. + +Release and change-acceptance authorities: + +- `docs/FIDELITY_ACCEPTANCE.md` — transformed-format support/fidelity qualification and evidence taxonomy. +- `docs/MIGRATION_ROLLBACK.md` — state/ledger/artifact/converter/API migration, rollback and recovery contract. +- `docs/RELEASE_ACCEPTANCE.md` — exact integrated protected-head release, provenance and publication verification gate. + +Supporting policy: + +- root `SECURITY.md` — vulnerability reporting policy plus linked detailed security docs. + +Legacy/detailed documents, including dated buyer-diligence scorecards and data-room snapshots, remain supporting historical evidence but must not contradict this canonical spine without an explicit status/supersession note. diff --git a/CLAUDE.md b/CLAUDE.md index c2207fdee..221492afd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,144 +1,169 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file provides repository-specific guidance for Claude Code and other assistants that read it. -## Read AGENTS.md first +## Authority order -`AGENTS.md` at the repo root is the canonical agent operating guide. It defines the -mandatory quality and security merge gates (compile with zero warnings, tests, -100% JaCoCo coverage, JavaDoc, markdown lint, license/attribution/diligence drift -checks) and the change-management rule that any new gate must be added to -`AGENTS.md` in the same PR. Follow those gates before claiming any change complete. -This file complements `AGENTS.md` with commands and architecture context; when in -doubt, `AGENTS.md` wins. +Read `AGENTS.md` first. It defines mandatory merge, security, evidence, documentation and autonomous-writer rules. Then use the canonical documentation spine: -Related canonical docs: +- `docs/PRD.md` +- `docs/TRD.md` +- `ARCHITECTURE.md` +- `SECURITY.md` +- `docs/adr/README.md` +- `docs/DATA_MODEL.md` +- `docs/UML.md` +- `docs/API_CONTRACT.md` +- `docs/THREAT_MODEL.md` +- `docs/TEST_STRATEGY.md` +- `docs/OPERABILITY.md` +- `docs/TRACEABILITY.md` +- `docs/DOCUMENTATION_ASSESSMENT.md` +- `docs/engineering/acceptance-criteria.md` -- `ARCHITECTURE.md` — root architecture map (components, state model, gate list). -- `docs/architecture.md` — detailed runtime flows and component boundaries. -- `docs/engineering/acceptance-criteria.md` — canonical acceptance policy with exact repro commands and evidence pointers. -- `README.md` — API scope, tenant-header contract, and compatibility notes. +Supplemental release and research authorities: -## Common commands +- `docs/FIDELITY_ACCEPTANCE.md` +- `docs/MIGRATION_ROLLBACK.md` +- `docs/RELEASE_ACCEPTANCE.md` +- `docs/RESEARCH_TRACEABILITY.md` -Toolchain: Java 21, Maven (Spring Boot parent 3.5.x). Python 3 for `scripts/`. +Historical plans, evidence snapshots, PR bodies, `.jules/` notes and old MVP PRD/TRD documents are supporting provenance, not current source of truth. Never infer live head/base/check/review state from them. + +## Canonical verification + +Toolchain: Java 21, Maven, Python 3 for repository helper scripts, and Node where viewer JavaScript tests require it. ```bash -# Compile (gate: warning/deprecated budget = 0; -Xlint:all -Werror is enforced) -mvn -DskipTests compile +# Full Maven acceptance: compile/tests/coverage/Javadocs and configured verification gates +mvn -B --no-transfer-progress verify -# Full test suite (gate) -mvn test +# Fail closed if Maven XML evidence is missing, malformed, empty, skipped or failing +python3 scripts/verify_maven_test_reports.py + +# Repository helper and documentation contract tests +python3 -m unittest discover -s scripts -# Single test class / single test method (standard Surefire selection) +# Focused test examples during TDD mvn test -Dtest=ConversionControllerTest mvn test -Dtest=ConversionControllerTest#methodName +``` -# Coverage gate: JaCoCo line/branch missed must be 0 for com.clearfolio.viewer.* -mvn -q -Djacoco.includes=com.clearfolio.viewer.* \ - org.jacoco:jacoco-maven-plugin:0.8.13:prepare-agent test \ - org.jacoco:jacoco-maven-plugin:0.8.13:report -# Report lands in target/site/jacoco/jacoco.csv - -# JavaDoc gate (must produce no warnings/errors) -mvn -q -DskipTests javadoc:javadoc +Local app smoke uses two terminals so the foreground Spring process remains running: -# Run the app locally, then probe readiness +```bash +# Terminal 1 mvn spring-boot:run -curl -sS http://localhost:8080/healthz +``` -# Python helper-script unit tests (unittest, no pytest dependency) -python3 -m unittest discover -s scripts +```bash +# Terminal 2, after Terminal 1 reports the application started +curl -fsS http://localhost:8080/healthz ``` -The license-policy, third-party-attribution, buyer data-room manifest, buyer -readiness scorecard, and Figma deck payload drift checks are Python scripts under -`scripts/`; the exact invocations (with the current evidence/policy file paths) -are listed in `AGENTS.md` and must pass for doc/dependency changes that touch them. +Focused tests accelerate RED/GREEN iteration but never replace final `mvn verify` and exact-head GitHub evidence. -There are no repo-local CI workflows in `.github/workflows`; gates are reproduced -locally and evidence is committed under `docs/qa/evidence/`. CodeQL runs through -GitHub default setup (do not add a repo-local advanced CodeQL workflow), and -Dependabot config lives in `.github/dependabot.yml`. +## What Clearfolio is now -## What this repository is +Clearfolio is a secure document-conversion and viewing service, independently deployable and composable with naruon and other CWL hosts through explicit versioned contracts. The runtime is Spring WebFlux. Heavy conversion work remains outside request completion. The service owns document validation/conversion contracts, conversion lifecycle interfaces, local tenant authorization enforcement, signed artifact-delivery semantics, artifact/state adapters, and product-specific evidence. -Clearfolio Viewer (`com.clearfolio` / `clearfolio-viewer`) is the MVP backend for -an integrated document viewer platform: non-blocking upload submit, async -conversion with retry/dead-letter, job status polling, and a PDF.js viewer served -from the same app. The runtime is Spring WebFlux (Servlet/MVC is explicitly not -the selected stack), logging is Log4j2 (the default Logback starter is excluded), -and the default job/artifact stores are in-memory (a SQL repository profile is -planned but not implemented). +The default conversion-job repository on protected main remains process-local. `FileSystemArtifactStore` provides configurable local artifact persistence; selected append-only/file-backed ledgers provide only their documented evidence durability. Do not describe those as a durable distributed job system. Entry point: `src/main/java/com/clearfolio/viewer/ClearfolioViewerApplication.java`. -Configuration: `src/main/resources/application.yml` (`conversion.*` queue/retry/ -upload limits, `viewer.security.frame-ancestors`) plus the `buyer-demo` Spring -profile (`application-buyer-demo.yml`) for buyer sandbox deployments. - -## High-level architecture - -All production code lives in `src/main/java/com/clearfolio/viewer/`: - -- `controller/` — HTTP endpoints and exception mapping. `ConversionController` - (submit `POST /api/v1/convert/jobs`, status polling, operator retry, viewer - bootstrap JSON), `ViewerUiController` (`GET /viewer/{docId}` HTML shell), - `ArtifactController` (`GET /artifacts/{docId}.pdf` with token verification and - single-range support), `AdminController`, `AnalyticsController` (KPI snapshot), - `HealthController` (`/healthz`), `ApiExceptionHandler` (shared error shape: - `errorCode`, optional `code`, `message`, `traceId`, `details`). -- `service/` — `DefaultDocumentValidationService` (extension blocklist — HWP/HWPX - blocked by default — size limits, HMAC-verified policy-override lane), - `DefaultDocumentConversionService` (validation, content-hash dedupe, persist, - enqueue), `DefaultConversionWorker` (bounded executor, retry backoff, - dead-letter fallback, startup recovery sweep for stale leases). -- `repository/` — `ConversionJobRepository` (read/dedupe boundary, in-memory - implementation) and `ConversionJobStateStore` (explicit lifecycle-transition - boundary with a process-local event trail; designed as the seam for the future - SQL implementation). -- `model/` — `ConversionJob` lifecycle (`SUBMITTED`, `PROCESSING`, `SUCCEEDED`, - `FAILED`; retry-exhausted jobs stay `FAILED` with `deadLettered=true`). -- `artifact/` — `ArtifactStore` (in-memory PDF bytes), `PdfBoxArtifactGenerator` - (PDFBox conversion stub), `ArtifactLinkService`/`ArtifactLinkLedger` - (short-lived signed artifact tokens, revocation, read-audit ledger). -- `auth/` — tenant enforcement scaffold: protected JSON APIs require - `X-Clearfolio-Tenant-Id`, `X-Clearfolio-Subject-Id`, `X-Clearfolio-Permissions` - headers (optionally gateway-HMAC-signed); cross-tenant jobs are hidden as 404. - This is not production OIDC/JWT validation. -- `analytics/` — KPI snapshot counters and export ledger. -- `api/` — response/request DTOs; `config/` — `ConversionProperties`, executor, - security-headers WebFilter; `exception/` — domain exceptions. - -Request flow: controller validates and enqueues (returns `202` fast; the request -path must never run conversion inline), the worker converts and stores the PDF, -clients poll status, then fetch viewer bootstrap JSON and the artifact via a -signed link. Static viewer assets (PDF.js shell, demo fixtures) live under -`src/main/resources/static/assets/viewer/`. - -Tests mirror the package tree under `src/test/java/`, including -`config/DependencyPolicyTest` (blocks reintroducing `tika-parsers-standard-package`, -the default Logback starter, or the excluded Jakarta annotation dependency) and -the Jazzer fuzz target `controller/FuzzDownloadFilename` (ClusterFuzzLite marker -in `.clusterfuzzlite/`). - -## Key conventions - -- Coverage is absolute: JaCoCo line/branch missed must stay 0 for - `com.clearfolio.viewer.*`, so every production change ships with tests covering - all new lines and branches. -- Every public type/member needs JavaDoc; the JavaDoc gate fails on any warning. -- The compiler runs with `-Xlint:all -Werror` (`showWarnings`/`showDeprecation` - on), so any warning or deprecated usage breaks the build. -- `checkstyle-suppressions.xml` at the repo root relaxes strict style checks - (Javadoc, magic numbers, line length, etc.) for `src/test/java` only; production - sources get no suppressions. -- Markdown lint applies to changed docs; rule overrides are in - `.markdownlint.yaml`. -- Dependency changes are policy-sensitive: security pins and exclusions in - `pom.xml` carry CVE-annotated comments, `osv-scanner.toml` holds narrowly - scoped, time-boxed ignores, and license/SBOM/attribution evidence under `docs/` - must be updated together (see `AGENTS.md` and `DependencyPolicyTest`). -- Gate evidence is committed under `docs/qa/evidence/`; plans and design docs - under `docs/` use dated filenames (`YYYY-MM-DD-...`). -- `.jules/` holds accumulated lessons (performance, XSS, HMAC canonicalization); - worth scanning before touching validation, viewer JS, or token signing code. + +## Current runtime map + +Production code lives under `src/main/java/com/clearfolio/viewer/`. + +- `controller/` — HTTP ingress, viewer/artifact/admin/analytics/availability surfaces and exception mapping. +- `auth/` — tenant/subject/permission claims and same-tenant enforcement. +- `service/` — validation, conversion orchestration, worker/retry/recovery and lifecycle coordination. +- `repository/` — job read/dedupe and lifecycle-state interfaces; current default implementation is in memory. +- `artifact/` — artifact stores, placeholder generator, signed artifact link/token/ledger logic and artifact evidence. +- `model/` — conversion lifecycle and immutable business state. +- `api/` — request/response records. +- `analytics/` — KPI/evidence surfaces. +- `config/` — bounded executor, artifact-store, security and runtime configuration. +- `exception/` — controlled domain failures. + +Static viewer assets and PDF.js integration live under `src/main/resources/static/assets/viewer/`. + +## Conversion truth + +Protected main supports validated PDF passthrough and PDF viewing. `PdfBoxArtifactGenerator`'s non-PDF one-page PDF is a development/demo placeholder, **not a faithful DOCX/HWP/Office converter**. Never expand supported-format claims without a deterministic converter plus authorized realistic fixture-based fidelity/security/accessibility evidence. + +Uploaded documents and every document-derived filename/string are untrusted. Production conversion must not grant macro/script execution or implicit external network access. Unsupported or unsafe input fails closed. + +## Tenant and artifact authority + +Cross-tenant resources are concealed according to server-side policy. Metadata permission and document-byte authority are not interchangeable. + +The canonical artifact path uses `ArtifactLinkService` to bind signed reads to token id, tenant, subject, document, scope, purpose, artifact checksum, issue time, expiry, issuance ledger and revocation state; verified reads follow the controlled single-range/read-audit contract. A convenience download endpoint must not bypass that authority merely because the caller has `artifact:read`. + +Signed tenant-claim, admin authorization, HMAC audit pseudonymization, immutable lifecycle-generation and deletion-recovery hardening may live on active PRs; check `docs/TRACEABILITY.md` and live GitHub before calling them protected-main functionality. + +## Lifecycle + +Public lifecycle values are currently: + +- `SUBMITTED` +- `PROCESSING` +- `SUCCEEDED` +- `FAILED` + +Retry exhaustion remains `FAILED` with dead-letter evidence rather than a separate public terminal enum. Do not invent lifecycle states in docs or clients without a versioned contract change. + +## Availability + +Protected main currently has its existing health surface. A separate `/readyz` traffic-readiness endpoint is active-PR work until integrated. Never describe active-PR availability semantics as shipped; always refetch the relevant PR/head. + +## Exact evidence rules + +Before reviewing, editing or merging: + +1. refetch exact PR head; +2. independently resolve current base-branch tip; +3. refetch formal reviews and unresolved threads; +4. refetch exact-head checks/workflow results and actual security gates; +5. refetch target blob/ref before writing; +6. if any refetched target identity differs from the identity the write was prepared against, stop and freeze that write, preserve the fresh evidence, and rotate to a newly revalidated exact-head work item instead of writing against stale state. + +PR-body SHAs and remembered run IDs are historical. Queued, pending, skipped-required, cancelled, absent, stale-head, predecessor-head, synthetic-only, model-only and failed results are not passing evidence. Commit statuses, automated-review comments and formal GitHub approvals are different authority classes. + +Before merging, revalidate the protected `main` branch rules, **all required checks** and security gates, the required reviewer count, and the qualifying independent approval for the unchanged exact head. **Automated comments, check results, and model output do not count as approval.** If the source head or supported base tip moves, abort the stale merge attempt and restart from fresh evidence. + +## TDD and coverage + +Production source defects use strict RED → narrow root-cause implementation → GREEN → full relevant validation. A RED is valid only if it reaches the intended product boundary; a setup/import/fixture failure is not the desired regression. + +Owned production line/branch coverage remains exactly 100%, and public production APIs require beginner-readable warning-free Javadocs. Coverage is not sufficient by itself: realistic security, lifecycle, concurrency, fidelity, accessibility, crash/restart and release assertions are required. + +## Dependency and supply-chain changes + +Dependency changes must update and verify security policy, license policy, SBOM/third-party attribution and buyer-diligence evidence as applicable. Do not add a duplicate repo-local advanced CodeQL workflow while the repository/organization's current CodeQL authority is already active. Keep GitHub Actions sources immutably pinned where practical. + +## Standalone and MSA composition + +Clearfolio remains independently usable. naruon or another host may own higher-level user workflow, upstream identity/federation and deployment composition, but communicates through explicit APIs/contracts and does not write Clearfolio application persistence directly. + +`contextual-orchestrator` is optional for explicitly model-backed features and is not authority for deterministic conversion, authorization, fidelity or release acceptance. + +## Autonomous development + +The dedicated Clearfolio commercial loop is the writer automation while enabled. Central `.github` owns privileged PR-maintenance/review/merge control-plane behavior; local product development must not duplicate those credentials or authorities. + +Model-backed autonomous development uses an immutably pinned OpenCode Agent and GitHub Secret `NVIDIA_NIM_API_KEY`, never `COPILOT_GITHUB_TOKEN` as a development-model credential. Model output is an untrusted bounded proposal. It cannot self-approve, merge, release or deploy. + +A blocker, one commit, one documentation update, one green test, a pending review or an external-approval wait is intermediate while safe work remains. Use `exact evidence → RCA → distinct remedies → empirical feasibility → test-first action → proof → next queue item`. Branch conflicts are branch-local. Normal no-work termination requires two fresh repository-wide exit sweeps. + +## Documentation change rule + +A material change to product ownership, public API/lifecycle, tenant/security authority, logical/persistent entities, conversion support/fidelity, automation authority, recovery or release acceptance updates the corresponding canonical PRD/TRD/Architecture/ADR/UML/ERD/API/Threat/Test/Operability/Traceability documents in the same reviewed change or explicitly proves no impact. + +Supplemental release and research authorities must also remain synchronized when applicable: `docs/FIDELITY_ACCEPTANCE.md`, `docs/MIGRATION_ROLLBACK.md`, `docs/RELEASE_ACCEPTANCE.md`, and `docs/RESEARCH_TRACEABILITY.md`. + +`python3 -m unittest discover -s scripts` must keep the canonical documentation contract executable. + +## Research and standards + +Use current authoritative standards, primary technical documentation and peer-reviewed primary research for material security, document-processing, accessibility, testing, provenance, interoperability and governance decisions. Record full APA 7 references with stable links. Commit a paper PDF only when redistribution is legally permitted; otherwise cite/link/summarize it without copying the PDF. diff --git a/README.md b/README.md index 39584e169..479d36ce7 100644 --- a/README.md +++ b/README.md @@ -1,146 +1,121 @@ # Clearfolio Viewer -This repository contains the MVP backend for an integrated document viewer platform. -The current implementation includes non-blocking submit, job status polling, and -asynchronous conversion that produces an in-memory PDF artifact for preview. - -## Quick Start - -1. Build and verify compilation: - - `mvn -DskipTests compile` -2. Run tests: - - `mvn test` -3. Start the app locally: - - `mvn spring-boot:run` -4. Check readiness: - - `curl -sS http://localhost:8080/healthz` - -## Scope - -- `GET /`: buyer-demo upload, status, KPI, KPI snapshot evidence, - operator recovery evidence, session-history shell, and a static - `Load demo story` seed for buyer screenshots and Figma review. -- `POST /api/v1/convert/jobs`: upload document and receive async job id. -- `GET /api/v1/convert/jobs/{jobId}`: poll conversion status and lifecycle fields. -- `POST /api/v1/convert/jobs` response includes `jobId`, `status`, and `statusUrl`. -- `POST /api/v1/convert/jobs/{jobId}/retry`: operator-triggered retry for dead-lettered jobs. -- `GET /viewer/{docId}`: canonical HTML viewer UI entrypoint (mobile-safe loading/failed/ready states). -- `GET /api/v1/viewer/{docId}` and `GET /api/v1/convert/viewer/{docId}`: viewer bootstrap JSON with a short-lived signed artifact URL. -- `POST /api/v1/viewer/{docId}/artifact-links`: create a tenant-bound signed artifact URL for succeeded jobs. -- `GET /api/v1/analytics/kpi-snapshot`: current conversion KPI counters and optional snapshot evidence for demo and diligence evidence. -- `GET /api/v1/analytics/kpi-snapshot-exports`: tenant-scoped exported KPI snapshot evidence. -- `GET /artifacts/{docId}.pdf`: serves converted PDF bytes (SUCCEEDED jobs only) with single-range support after artifact token verification. -- Errors follow shared shape (`errorCode`, optional `code`, `message`, `traceId`, `details`) for 404/409/400/500 paths. -- `GET /healthz`: readiness probe. -- HWP/HWPX are blocked by configuration. - -Protected JSON APIs require Clearfolio tenant headers in the current buyer-demo -runtime: `X-Clearfolio-Tenant-Id`, `X-Clearfolio-Subject-Id`, and -`X-Clearfolio-Permissions`. The built-in demo shell sends `buyer-demo` headers -automatically. These headers are a runtime enforcement scaffold, not production -OIDC/JWT validation. Deployments can set -`clearfolio.tenant-claims.hmac-secret` to require gateway-signed tenant headers -with `X-Clearfolio-Claims-Issued-At` and `X-Clearfolio-Claims-Signature`; -validated OIDC/JWT issuer, audience, expiry, revocation, and role mapping remain -production gaps. Buyer sandbox deployments should use the `buyer-demo` Spring -profile and follow -`docs/deployment/2026-07-02-buyer-deployment-integration-playbook.md`. - -## Compatibility notes - -- API contract has been kept backward-compatible with the existing jobs + viewer flow. -- `GET /viewer/{docId}` remains the canonical entry route, but now serves HTML (PDF.js viewer). -- Alias endpoints remain stable, with signed artifact link fields added to - viewer bootstrap responses. -- Dead-letter terminal cases keep `status=FAILED` in API payloads and set - `deadLettered=true` when retries are exhausted. -- Dead-lettered jobs can be re-queued by an operator with - `X-Clearfolio-Operator-Id` via `/api/v1/convert/jobs/{jobId}/retry`. -- The buyer-demo shell summarizes session-scoped operator recovery evidence: - needs-action documents, retry-ready dead-lettered jobs, last accepted retry, - and latest inspected job detail. This is demo evidence, not a production admin - console. -- The `Load demo story` control loads - `src/main/resources/static/assets/viewer/demo-fixtures.json` into the current - browser session. The fixture covers succeeded, processing, unsupported-format, - and dead-lettered states plus KPI snapshot/export evidence for screenshot and - FigJam handoff work. It is static local demo data, not production seed data. -- Status, viewer bootstrap, retry, and KPI JSON APIs enforce tenant permission - headers and hide cross-tenant jobs as `404`. -- Tenant headers can be HMAC-signed by a trusted gateway when - `clearfolio.tenant-claims.hmac-secret` is configured; unsigned local demo mode - should not be exposed as a production internet boundary. -- `GET /viewer/{docId}` returns an HTML shell without checking job existence; - the protected JSON APIs determine visible state. -- Artifact reads now require a signed `artifactToken` query parameter or bearer - token. Issued tokens are recorded in a runtime ledger, can be revoked by - `tokenId`, and successful artifact reads are exposed as tenant-scoped audit - evidence. Deployments can set `clearfolio.artifact-link-ledger.path` to - replay issued-link, revocation, and read-audit metadata from a local - append-only ledger file after restart. Centralized durable persistence and - object-store metadata remain open. -- Deployments can set `clearfolio.analytics-snapshot-ledger.path` to append - exported KPI snapshots to a local file and replay them after a single-process - restart. `GET /api/v1/analytics/kpi-snapshot-exports` exposes the same - evidence to authorized tenant callers, and the buyer-demo shell renders the - latest export count, subject, export time, and runtime job count without - showing tenant ids. This is buyer-demo evidence continuity, not the full - durable metrics event stream described in - `docs/analytics/2026-07-02-durable-metrics-event-model.md`. -- `DefaultConversionWorker` runs a startup recovery sweep using - `conversion.processing-lease-timeout-ms` to re-enqueue due `SUBMITTED` jobs - and stale retryable `PROCESSING` jobs while repository state is available. - The default in-memory runtime still loses conversion jobs across process - restart until the planned SQL repository profile is implemented. - -## Acceptance gates (current) - -- Mandatory: test coverage 100%, docstring 100%, non-blocking request path, - lightweight event queue, warning count 0, deprecated usage 0, - and one-day delivery schedule with security verification evidence. -- Optional: DB pooler client path (when DB is introduced), PostgreSQL 17 support track. - -Current release claim boundary: -- Mandatory gates are validated through committed evidence under `docs/qa/evidence/`. -- Optional DB pooler/PostgreSQL 17 tracks are documented only and not executed in this MVP release. -- Conversion lifecycle transitions now use an explicit in-repo state-store - boundary and append a process-local lifecycle event trail for submission, - dedupe hits, processing start, retry scheduling, success, failure, and - operator retry acceptance. The worker can also recover due submitted jobs and - stale processing leases from the available repository state at startup. The - default runtime remains process-local until a SQL profile is introduced. - -## Delivery schedule - -- One-day customer delivery plan (including security verification): - - `docs/plans/2026-02-20-24h-customer-delivery-plan.md` - -## Documentation references - -- `docs/architecture.md` -- `docs/prd-integrated-document-viewer-platform.md` -- `docs/trd-integrated-document-viewer-platform.md` -- `docs/diagrams/submit-flow.md` -- `docs/diagrams/status-flow.md` -- `docs/diagrams/preview-flow.md` -- `docs/diagrams/retry-deadletter-flow.md` -- `docs/business/2026-07-02-krw2b-valuation-kpi-model.md` -- `docs/design/2026-07-02-buyer-demo-kpi-figjam-handoff.md` -- `docs/deployment/2026-07-02-buyer-deployment-integration-playbook.md` -- `docs/deployment/clearfolio-buyer-connector.openapi.yaml` -- `docs/persistence/2026-07-02-durable-conversion-job-repository-plan.md` -- `docs/diligence/2026-07-02-buyer-diligence-index.md` -- `docs/security/2026-07-02-threat-model-data-handling.md` -- `docs/security/2026-07-02-signed-artifact-link-design.md` -- `docs/security/2026-07-02-auth-tenant-model.md` -- `docs/security/2026-07-02-license-allowlist-review.md` -- `docs/security/2026-07-02-license-policy.json` -- `docs/analytics/2026-07-02-durable-metrics-event-model.md` -- `docs/qa/evidence/2026-07-02-krw2b-sale-readiness/seeded-demo-story-verification.md` - -## Transfer metadata - -- Target owner repo: to be set during transfer. -- Tech stack: Java 21 / Spring Boot / Maven. -- Primary package: `com.clearfolio.viewer`. -- Current branch default assumption: `main`. +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/clearfolio) + +Clearfolio is an independently deployable secure document-conversion and viewing service. It can also compose with naruon and other ContextualWisdomLab hosts through explicit versioned APIs without shared application-database ownership. + +The current runtime is Java 21 / Spring Boot WebFlux. Upload submission is non-blocking, conversion lifecycle work is delegated to bounded background workers, viewer bootstrap and artifact access are tenant-scoped, and the canonical artifact route uses short-lived signed token, revocation, integrity, Range, and read-audit controls. + +## Product truth + +- Validated PDF passthrough and PDF.js viewing are implemented on protected main. +- Non-PDF `PdfBoxArtifactGenerator` output is a **development/demo one-page placeholder**, not a faithful DOCX/HWP/Office conversion claim. +- The default conversion-job repository remains process-local. Filesystem artifacts and selected append-only ledgers provide only their documented local restart continuity, not a durable distributed job system. +- Production identity-provider integration, real Office fidelity, durable distributed jobs/backpressure/cancellation, full OpenTelemetry/SLO evidence, and release-grade remote persistence remain product gaps or active work as classified in the canonical docs. +- Active pull-request behavior is never treated as protected-main functionality until integrated and revalidated. + +## Quick start + +Run repository acceptance first: + +```bash +mvn -B --no-transfer-progress verify +python3 scripts/verify_maven_test_reports.py +python3 -m unittest discover -s scripts +``` + +Then run the local application smoke in two terminals so the foreground Spring process remains alive: + +```bash +# Terminal 1 +mvn spring-boot:run +``` + +```bash +# Terminal 2, after Terminal 1 reports the application started +curl -fsS http://localhost:8080/healthz +``` + +Focused tests may be used during RED/GREEN development, but final local acceptance uses `mvn verify` plus the test-report verifier. Merge/release acceptance requires current exact-head GitHub evidence according to repository policy. + +## Primary API surfaces + +- `POST /api/v1/convert/jobs` — bounded asynchronous upload/submit. +- `GET /api/v1/convert/jobs/{jobId}` — tenant-scoped lifecycle status. +- `POST /api/v1/convert/jobs/{jobId}/retry` — controlled dead-letter retry. +- `GET /viewer/{docId}` — canonical HTML/PDF.js viewer entry. +- `GET /api/v1/viewer/{docId}` and compatibility alias — protected viewer bootstrap. +- `POST /api/v1/viewer/{docId}/artifact-links` — create a short-lived tenant-bound signed artifact link. +- `POST /api/v1/viewer/artifact-links/{tokenId}/revoke` — revoke an issued artifact token when authorized. +- `GET /api/v1/viewer/{docId}/artifact-read-events` — tenant-scoped artifact-read audit evidence. +- `GET /artifacts/{docId}.pdf` — canonical signed PDF byte delivery with zero-or-one HTTP Range support. +- `GET /api/v1/convert/jobs/{jobId}/download` — direct-download convenience path; its final accepted security contract is the canonical signed artifact-delivery authority, not permission-only byte access. +- analytics/admin/health surfaces are documented in `docs/API_CONTRACT.md` and live source; some stronger admin/readiness contracts remain active-PR work and are explicitly labelled there. + +Protected JSON/document APIs use Clearfolio tenant/subject/permission authority. The current buyer-demo scaffold can carry header-based claims and configured signatures; it is not a substitute for production IdP/OIDC issuer, audience, expiry, revocation, and enterprise role mapping. + +## Lifecycle + +Public conversion lifecycle values are: + +```text +SUBMITTED → PROCESSING → SUCCEEDED + ↘ retry/recovery → SUBMITTED + ↘ terminal/exhausted → FAILED +FAILED + dead-letter evidence → authorized retry → SUBMITTED +``` + +Retry exhaustion remains `FAILED` with dead-letter evidence rather than a separate public terminal enum. + +## Security highlights + +- uploaded documents and document-derived strings are untrusted; +- blocked/oversized/malformed requests fail closed; +- cross-tenant resources are concealed according to the API contract; +- document-byte permission and signed artifact-token authority are separate controls; +- canonical signed reads bind tenant, subject, document, scope, purpose, checksum, issuance, expiry and revocation and record controlled read evidence; +- raw tokens, signatures and unnecessary sensitive identifiers must not be logged; +- active-content/macro execution and implicit external network access are not accepted deterministic-conversion authority paths; +- cryptographic purposes use separated reviewed secrets when required by the integrated revision. + +See `SECURITY.md`, `docs/THREAT_MODEL.md`, and the ADR index for the complete boundary. + +## Standalone and CWL composition + +Clearfolio owns its conversion/viewer behavior, tenant enforcement, artifact-delivery rules, state interfaces and product evidence. A host such as naruon may own higher-level user workflow, upstream identity/federation and deployment composition, but communicates over explicit contracts and does not mutate Clearfolio application persistence directly. + +`contextual-orchestrator` is optional for genuinely model-backed features and is not authority for deterministic conversion, authorization, fidelity measurement, merge, or release. + +## Canonical documentation + +Start here rather than reconstructing the product from dated plans or PR bodies: + +- [`docs/PRD.md`](docs/PRD.md) — product requirements, users, maturity and fidelity/release outcomes. +- [`docs/TRD.md`](docs/TRD.md) — technical requirements and implemented/active/planned boundaries. +- [`ARCHITECTURE.md`](ARCHITECTURE.md) — system/trust/ownership architecture. +- [`docs/adr/README.md`](docs/adr/README.md) — architecture decisions and implementation maturity. +- [`docs/DATA_MODEL.md`](docs/DATA_MODEL.md) — logical ERD and persistence ownership. +- [`docs/UML.md`](docs/UML.md) — component/sequence/state/recovery/deployment diagrams. +- [`docs/API_CONTRACT.md`](docs/API_CONTRACT.md) — public API and MSA contract authority. +- [`docs/THREAT_MODEL.md`](docs/THREAT_MODEL.md) — product threat model. +- [`docs/TEST_STRATEGY.md`](docs/TEST_STRATEGY.md) — realistic TDD/security/fidelity/recovery strategy. +- [`docs/OPERABILITY.md`](docs/OPERABILITY.md) — startup, degraded mode, incident, backup/restore and release operations. +- [`docs/TRACEABILITY.md`](docs/TRACEABILITY.md) — requirement/ADR to implementation/test/PR/evidence mapping. +- [`docs/DOCUMENTATION_ASSESSMENT.md`](docs/DOCUMENTATION_ASSESSMENT.md) — documentation completeness and drift assessment. +- [`docs/engineering/acceptance-criteria.md`](docs/engineering/acceptance-criteria.md) — executable acceptance policy. + +Legacy `docs/prd-integrated-document-viewer-platform.md`, `docs/trd-integrated-document-viewer-platform.md`, dated delivery plans, feature design notes and `docs/qa/evidence/` remain useful history/evidence. They do not override the canonical spine or live protected code. + +## Development automation + +Clearfolio's dedicated commercial loop is execution-first and work-conserving: an approval wait, queued check, one RCA, one commit, one documentation update or one merge blocks only the affected action while other safe work continues. Autonomous product development uses an immutably pinned OpenCode Agent with GitHub Secret `NVIDIA_NIM_API_KEY`, never `COPILOT_GITHUB_TOKEN` as a development-model credential, and preserves independent review/merge authority. + +Central ContextualWisdomLab `.github` owns privileged PR-maintenance/review/merge control-plane behavior. Clearfolio must not copy that privileged implementation into the product repository. + +## Release boundary + +Do not release from a feature PR merely because its tests are green. Release only from an exact integrated protected head after CI, security, exact coverage/docstrings, realistic document-fidelity acceptance, accessibility, packaging, SBOM/provenance, reproducibility, API/schema compatibility, migrations/rollback/recovery where applicable, independent review, and protected-main operational acceptance all pass. + +## License + +Clearfolio source and documentation are available under the [Apache License 2.0](LICENSE). Third-party components retain their own licenses and attribution obligations; release artifacts must verify those obligations against the exact dependency inventory, SBOM, and generated attribution evidence required by [release acceptance](docs/RELEASE_ACCEPTANCE.md). The repository grant does not replace third-party terms. diff --git a/SECURITY.md b/SECURITY.md index e4dbc7385..7f83cd247 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,27 +1,120 @@ # Security Policy -## Supported Version +## Supported Version and Evidence Boundary -Clearfolio is currently maintained on the `main` branch. Security fixes are -accepted for the active application and are released through reviewed pull -requests. +Clearfolio is maintained on the protected `main` branch. Security fixes are +accepted through reviewed pull requests and are not treated as shipped until +they integrate through repository protection. Active PRs may document proposed +or implemented-on-branch controls, but protected `main` remains the authority +for deployed product behavior. + +Clearfolio is a document-conversion and viewing service. Uploaded documents, +document-derived strings, browser data, filenames, PDF actions, remote-resource +references, model output, and pull-request text are untrusted input. The +non-PDF development/demo placeholder on protected main is not production Office +conversion or fidelity evidence. Production transformed-format support requires +the qualification gates in `docs/FIDELITY_ACCEPTANCE.md` and issue #5. ## Reporting a Vulnerability Please report suspected vulnerabilities privately through GitHub Security Advisories for this repository: https://github.com/ContextualWisdomLab/clearfolio/security/advisories/new. -Include the affected endpoint, dependency, input, observed impact, and a -minimal reproduction when possible. -For automation failures, include the failing check name, run URL, and the -package/CVE or SARIF rule that triggered the alert. Reports that involve -document parsing, artifact links, tenant boundaries, or dependency resolution -are treated as high-sensitivity until triage proves otherwise. +Include the affected endpoint or component, the input and preconditions, the +observed impact, and a minimal reproduction when possible. Do not place customer +document content, raw credentials, signed tokens, private tenant identifiers, or +other sensitive evidence in a public issue. + +For automation failures, include the failing check name, run URL, exact source +revision when known, and the package/CVE or SARIF rule that triggered the alert. +Reports involving document parsing, artifact delivery, tenant boundaries, +conversion runtimes, or dependency resolution are treated as high-sensitivity +until triage proves otherwise. + +## Authorization and Artifact Delivery + +Tenant authorization and signed artifact delivery are separate authorities. +Protected document-byte flows must preserve same-tenant concealment and the +least privilege required by the route. Where the signed artifact contract +applies, a request must also pass the canonical signature, expiry, scope, +document/tenant/checksum binding, issued-ledger, revocation, single-range, and +controlled read-audit checks defined by the current implementation and +`docs/API_CONTRACT.md`. + +A generic tenant permission does not replace the signed artifact boundary. The +signed artifact-link route exists on protected main. Direct conversion-job +download consistency is being hardened in active PR #270; until that work is +integrated, its stronger direct-download behavior is `ACTIVE_PR`, not shipped +truth. + +Cross-tenant and missing-resource cases must remain non-enumerating where the +API contract requires concealment. Public error payloads and logs must not expose +local filesystem paths, document content, raw signed claims, raw artifact tokens, +or exception-controlled sensitive values. + +## Audit Pseudonymization and Key Separation + +Privacy-safe audit evidence uses purpose-bound, domain-separated HMAC +pseudonymization where that control is implemented. Active PR #270 strengthens +this boundary with separate policy-signing and audit-pseudonym purposes and +fail-closed key configuration. Do not reuse one key across those purposes. + +Pseudonymized identifiers remain personal data when they can still single out or +be related back to a person or tenant. They therefore remain subject to +purpose limitation, least privilege, retention, key governance, and access +auditing; pseudonymization is not anonymization and is not permission for +unbounded reuse. + +Key material must never be committed to source control or copied into logs, +error responses, documentation examples, model prompts, or build artifacts. +Current detailed key requirements and maturity are maintained in +`docs/adr/0003-audit-pseudonymization-key-separation.md` and the current reviewed +security implementation. + +## Document Conversion Trust Boundary + +Clearfolio must not execute document macros or active content as application +authority, and production conversion must not depend on following arbitrary +external document links. A future Office runtime must remain outside the +Clearfolio API-container trust boundary behind the provider-neutral conversion +contract, with bounded resources, isolated temporary storage, no inherited +application/cloud credentials, deny-by-default outbound network access, and +explicit source/output policy checks. + +PDF output acceptance distinguishes inert fidelity-preserving navigation from +executable behavior. Supported URI/internal-link fidelity may be retained only +under the explicit versioned output policy; script, launch, submission/import, +embedded-file, chained active-action, or equivalent executable behavior fails +closed according to that policy. + +## Dependency, CI, and Supply-Chain Expectations + +Medium, high, and critical dependency advisories are remediated by updating the +affected package or its managing BOM/parent and re-running the exact applicable +security and test evidence. Workflow or repository-governance findings are not +dismissed silently; logs and review evidence must preserve the reason when a +gate cannot complete. + +Release evidence must remain bound to the exact protected source and artifact +identity and include the applicable CI/security gates, exact owned production +coverage, public Javadocs, realistic fidelity and recovery evidence, SBOM and +third-party attribution, provenance/reproducibility, migration/rollback where +applicable, and required independent review. See +`docs/RELEASE_ACCEPTANCE.md` for the canonical integrated release gate. -## Remediation Expectations +## Security Documentation Map -Medium, high, and critical dependency advisories are fixed by bumping the -affected package or its managing BOM/parent. Workflow or repository-governance -findings are not dismissed silently; the relevant workflow logs must show the -reason when a check cannot complete. +- `docs/THREAT_MODEL.md` — trust boundaries, abuse cases, mitigations, and + residual risks. +- `docs/API_CONTRACT.md` — route-level authority and controlled failure + semantics. +- `docs/adr/0002-tenant-artifact-authorization.md` — tenant permission versus + artifact-delivery authority. +- `docs/adr/0003-audit-pseudonymization-key-separation.md` — HMAC purpose/key + separation and privacy treatment. +- `docs/adr/0005-deterministic-conversion-fidelity.md` — converter isolation, + active-content policy, and fidelity truth. +- `docs/FIDELITY_ACCEPTANCE.md` — realistic transformed-format qualification. +- `docs/MIGRATION_ROLLBACK.md` — security-preserving migration and recovery. +- `docs/RELEASE_ACCEPTANCE.md` — exact integrated release evidence. diff --git a/docs/ACQUISITION_DILIGENCE.md b/docs/ACQUISITION_DILIGENCE.md new file mode 100644 index 000000000..a11f1b4cc --- /dev/null +++ b/docs/ACQUISITION_DILIGENCE.md @@ -0,0 +1,225 @@ +# Clearfolio Acquisition Diligence + +Status: Canonical acquisition-readiness index +Assessment date: 2026-09-20 +Protected-main baseline assessed: `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +This document is the current buyer/acquirer diligence authority for Clearfolio. It separates protected-main evidence, `ACTIVE_PR` evidence, historical evidence, external evidence, and unresolved acquisition risk. It does **not** convert a green pull request, demo artifact, valuation narrative, model review, or historical snapshot into production or acquisition readiness. + +The dated `docs/diligence/2026-07-02-buyer-diligence-index.md`, `docs/diligence/2026-07-03-buyer-readiness-scorecard.md`, buyer-demo screenshots, FigJam/Slides handoffs, and `docs/qa/evidence/2026-07-02-krw2b-sale-readiness/` remain **historical evidence**. They explain what was proven at that time but do not override current protected main, this document, live pull requests, current checks/reviews, or the canonical architecture graph. + +## Evidence vocabulary + +- `PROTECTED_MAIN`: observable in the current protected default branch. +- `ACTIVE_PR`: implemented or documented only in an open PR; not shipped. +- `PARTIAL`: material evidence exists but an acquirer would still inherit an unresolved risk or missing control. +- `PLANNED`: accepted backlog/architecture without complete implementation evidence. +- `EXTERNAL`: requires evidence outside this repository, such as legal ownership records, customer contracts, certification reports, or infrastructure-specific attestations. +- `HISTORICAL`: dated evidence that must not be reused as current proof without revalidation. + +Every acquisition claim must identify its evidence class. `ACTIVE_PR`, `HISTORICAL`, model output, status checks, and PR-body prose are never silently promoted to `PROTECTED_MAIN`. + +## Current acquisition verdict + +Clearfolio has a substantial technical diligence base but is **not yet acquisition-ready as a finished commercial document-conversion product**. The repository can explain its current architecture, security boundaries, tenant/artifact authority, release controls, historical quality evidence, and major product gaps. However, an acquirer would still inherit material product and operational completion risk: + +- non-PDF conversion on protected main still includes a development/demo placeholder rather than production Office fidelity; +- production Office qualification remains issue #5 with provider-neutral contract/security work in `ACTIVE_PR` #306; +- durable asynchronous job persistence, distributed idempotency/backpressure/cancellation and production-grade restart recovery remain issue #312 and are not shipped; +- production identity/federation remains issue #314; the current HMAC-signed gateway-claim mode is not a complete provider-neutral OIDC/JWT enterprise federation contract; +- OpenTelemetry and measured operational acceptance remain incomplete and are part of issue #312's execution evidence; +- the tenant-safe end-to-end deletion/download user journey remains issue #263 after active lifecycle/accessibility/security substrate integrates; +- several high-value security/accessibility/availability branches are stacked behind the current parent and require refreshed exact-head/live-base evidence after integration; +- qualifying independent approval and repository protection still govern integration where required. + +This repository does not claim certification, legal title assurance, customer traction, valuation, or transferability merely because engineering evidence exists. + +## Product diligence + +| Question | Evidence status | Current authority | Acquisition implication | +| --- | --- | --- | --- | +| Can Clearfolio accept and view validated PDFs? | `PROTECTED_MAIN` | `ARCHITECTURE.md`, current source, `docs/PRD.md` | Core viewing/passthrough path exists. | +| Does protected main provide faithful production Office conversion? | `PARTIAL` | issue #5, `docs/FIDELITY_ACCEPTANCE.md`, `ACTIVE_PR` #306 | No. The non-PDF placeholder is not production Office fidelity. | +| Is the Office engine isolated from the API trust boundary? | `ACTIVE_PR` / `PLANNED` | ADR-0005, `docs/UML.md`, issue #5, #306 | Provider-neutral publication boundary is being developed; sandboxed sidecar or independently operated remote runtime still requires qualification. | +| Are accepted conversion jobs durably recoverable with explicit backpressure/cancellation? | `PLANNED` | issue #312, protected-main worker/repository/executor code | No. Protected main is safely bounded but process-local; production durable acceptance/outbox, lease fencing, cancellation and admission semantics remain to implement. | +| Does Clearfolio have provider-neutral production identity federation? | `PLANNED` | issue #314, current tenant-claim verifier/config code, Draft #313 key-strength hardening | No. Current shared-HMAC gateway claims are a deployment adapter; issuer/audience/algorithm/key-rotation/tenant-role federation still requires a versioned verifier contract. | +| Is the complete tenant-safe deletion/download journey shipped? | `PARTIAL` | issue #263; protected #270; #268 `SUPERSEDED`; bounded lifecycle/accessibility successors | Signed download is `IMPLEMENTED_ON_MAIN`, while buyer-facing deletion completion and integrated recovery evidence remain unintegrated. | +| Are APIs versioned and host/MSA boundaries explicit? | `PARTIAL` | `docs/API_CONTRACT.md`, `ARCHITECTURE.md`, ADR-0001 | Current boundaries are documented; full generated/public schema completeness remains partial. | + +## Architecture and engineering diligence + +The current canonical architecture set is: + +- `docs/PRD.md` +- `docs/TRD.md` +- root `ARCHITECTURE.md` +- `docs/adr/README.md` and detailed ADRs +- `docs/UML.md` +- `docs/DATA_MODEL.md` +- `docs/API_CONTRACT.md` +- `SECURITY.md` +- `docs/THREAT_MODEL.md` +- `docs/TEST_STRATEGY.md` +- `docs/OPERABILITY.md` +- `docs/TRACEABILITY.md` +- `docs/FIDELITY_ACCEPTANCE.md` +- `docs/MIGRATION_ROLLBACK.md` +- `docs/RELEASE_ACCEPTANCE.md` +- `docs/RESEARCH_TRACEABILITY.md` +- `docs/DOCUMENTATION_ASSESSMENT.md` +- `docs/ACQUISITION_DILIGENCE.md` + +The set is `DESIGN_SUFFICIENT` on the active documentation line when its current review findings and exact-head checks are clean. It is not `PROTECTED_MAIN_SUFFICIENT` until that exact documentation head integrates through repository policy. + +### Data/persistence truth + +`docs/DATA_MODEL.md` is a conceptual/logical ERD and ownership map. It explicitly distinguishes process memory, file artifacts, file ledgers, host-owned identity, `ACTIVE_PR` lifecycle evidence, and `PLANNED` entities. A diagrammed entity does not imply a database table. Protected-main job state remains process-local by default, so distributed durability must not be inferred from filesystem artifact persistence. + +Issue #312 is the bounded authority for future general durable conversion-job acceptance, transactional outbox/idempotency, lease-fenced execution, explicit backpressure/cancellation, restart/redelivery recovery and related OpenTelemetry evidence. It must reuse the accepted generation/fencing requirements preserved from `SUPERSEDED` #268 and current bounded successors rather than duplicating them. + +Issue #314 is the bounded authority for provider-neutral production identity verification/federation. It must preserve the internal `tenant_context` authorization contract and artifact-token separation instead of copying object authorization into each identity provider adapter. + +## Security and privacy diligence + +### Current evidence + +- root `SECURITY.md` is the current product security entrypoint; +- server-side tenant authorization and same-tenant concealment are explicit design/runtime concerns; +- signed artifact delivery is distinct from metadata permission; +- signed-token issuance, checksum binding, revocation and controlled read audit exist on the canonical artifact path; +- HMAC pseudonymization and purpose/key separation are documented, with active hardening in the security stack; +- uploaded documents and derived content are untrusted; +- Office-process execution inside the API-container trust boundary is rejected by the accepted architecture; +- release acceptance requires security/review/SBOM/provenance evidence rather than one green status. + +### Remaining risk + +- production issuer/audience/key-rotation/tenant-role identity federation remains issue #314; Draft #313 hardens shared-HMAC production key length but does not close the broader federation gap; +- distributed tenant-safe persistence and centralized audit/retention controls remain incomplete; +- real Office sandbox/malware/active-content/fidelity qualification is not complete; +- privacy/data-retention behavior must be validated against the actual deployment/customer legal basis before production use. + +Pseudonymized identifiers remain personal data; Clearfolio does not treat pseudonymization as anonymization. + +## Reliability and operations diligence + +`docs/OPERABILITY.md`, `docs/MIGRATION_ROLLBACK.md`, and issue #312 are the current operator/recovery authorities for the transition from process-local execution to durable asynchronous operation. They deliberately do not invent SLO/RPO/RTO values. + +Material remaining risks include: + +- durable asynchronous job state and crash/restart recovery (issue #312); +- idempotent distributed execution and backpressure/cancellation (issue #312); +- OpenTelemetry traces/metrics and measured capacity evidence (issue #312 plus OPERABILITY); +- real backup/restore for future durable state; +- production Office sidecar/remote-service process recycling, cleanup, no-network enforcement and resource-pressure behavior; +- integrated protected-main release acceptance after the current stacked PRs converge. + +Until measured deployment-profile evidence exists, availability, recovery-time and throughput claims remain `PARTIAL` or `PLANNED`. + +## Supply-chain, license and provenance diligence + +### Repository-owned evidence + +- root `LICENSE` grants Apache License 2.0 terms for the repository work as published; +- `docs/security/2026-07-02-license-allowlist-review.md` records the repository dependency-license review baseline; +- `docs/legal/2026-07-03-third-party-attribution.md` provides generated third-party attribution evidence; +- `scripts/render_third_party_attribution.py` and related contract tests reduce attribution drift; +- CycloneDX SBOM evidence exists under dated QA evidence and current release policy requires SBOM/provenance tied to the exact release identity; +- issue #5 separately requires the exact Office runtime image, fonts, native libraries, codecs/dictionaries and transitive license obligations to be reviewed before production qualification. + +### Acquisition/IP limits + +The public repository license is not the same thing as an acquisition-chain **intellectual property** title report. This repository does not independently prove employment invention assignment, contractor assignment, contributor authority, trademark ownership, customer-data rights, patent freedom-to-operate, or all historical contribution provenance. Those matters require **external legal review** and organization records. + +Contributor identity in Git history can support diligence, but contributor identity alone does not prove assignment or authority. Consequently, complete transfer-chain evidence is **not independently proven** by the repository and must not be represented as closed acquisition risk. + +A production Office distribution also requires legal review of the exact selected runtime and redistribution package. Library-level license compatibility is not sufficient approval for an entire container/runtime image. + +## NOTICE and third-party attribution + +Apache-2.0 redistribution requires applicable attribution/NOTICE obligations to be preserved. Clearfolio currently generates third-party attribution from reviewed dependency evidence; release acceptance must verify the generated attribution against the exact SBOM/dependency set. If a dependency or Office runtime introduces a NOTICE or source-offer obligation, the release package must carry it explicitly. + +Absence of a repository-root `NOTICE` file is not by itself proof that no NOTICE obligation exists. The exact release dependency/runtime inventory is authoritative for that determination. + +## Compliance and certification boundary + +Clearfolio may design controls toward enterprise assurance and evidence readiness, but this repository **does not claim certification** for SOC 2, CSAP, ISO/IEC 27001/42001, accessibility conformance, or another external scheme merely from code or documentation. Any certification/conformance claim requires the relevant independent assessment and scoped evidence. + +Privacy, security and accessibility documentation supports engineering diligence; it does not replace applicable customer, regulator, accessibility or legal review. + +## Review, CI and merge diligence + +An acquirer should expect exact evidence, not screenshots of old green runs: + +- source head and independently resolved live base are separate identities; +- required checks must be terminal-success for the applicable exact candidate; +- queued, pending, cancelled, skipped-required, absent, stale, predecessor, synthetic-only, model-only and status-only evidence is non-passing; +- CodeRabbit/OpenCode/Noema output is advisory/review evidence according to its actual formal state, not automatic approval; +- **independent approval** is counted only when it is a qualifying formal non-author approval under the applicable repository/governance rule; +- a merge is not a release; protected-main operational, fidelity, recovery, SBOM/provenance and publication verification remain distinct gates. + +`docs/RELEASE_ACCEPTANCE.md` is the release authority. + +## Current open work that materially affects diligence + +Live PR and issue numbers and heads must be refetched before a transaction or release decision. At this assessment date, the material work families include: + +- #270 — security/privacy/test-evidence and signed direct-download parent; +- #264 — nested-safe accessible async viewer controls, stacked behind #270; +- #268 — `SUPERSEDED`; signed admin tenant/lifecycle/deletion requirements are preserved in bounded successors and are not protected-main behavior by inheritance; +- #271 — repository-local OpenCode development-loop controls, stacked behind #270 and central `.github` dependency; +- #276 — malformed artifact-token structure hardening, stacked behind #270; +- #295 — liveness/readiness separation, stacked behind #270; +- #305 — canonical documentation and acquisition-truth line; +- #306 / issue #5 — provider-neutral Office adapter and fail-closed qualification work; +- #311 — Junrar dependency-management update; exact-head checks are useful source evidence but it overlaps the parent dependency/SBOM surface and must be reconciled after the stable security parent before integration; +- #313 — production signed-tenant HMAC key-length readiness hardening; this is not equivalent to full identity federation; +- issue #312 — durable job acceptance/outbox/idempotency/backpressure/cancellation/restart/observability work, dependency-aware behind the security/lifecycle baseline; +- issue #314 — provider-neutral production OIDC/JWT identity federation and tenant-role mapping, dependency-aware behind overlapping authentication/security work. + +These numbers are navigation hints, not permanent evidence. Any head/base movement invalidates predecessor-head merge conclusions. + +## Historical evidence policy + +The following remain useful only as dated **historical evidence** unless explicitly refreshed against current code: + +- `docs/diligence/2026-07-02-buyer-diligence-index.md`; +- `docs/diligence/2026-07-03-buyer-readiness-scorecard.md`; +- `docs/diligence/2026-07-03-buyer-data-room-manifest.json`; +- `docs/qa/evidence/2026-07-02-krw2b-sale-readiness/`; +- dated buyer-demo/FigJam/Slides artifacts and screenshots; +- stale PR numbers, SHAs, local coverage logs, or queued-check narratives embedded in those artifacts. + +They must not be used to assert that PR #74 is the current baseline or that PR #82 is the current active product line. Current GitHub refs, protected source, canonical docs and exact evidence supersede them. + +## External diligence still required before an acquisition claim + +Repository engineering cannot autonomously close every transaction risk. The buyer/seller must separately obtain or verify, as applicable: + +- corporate authority and cap-table/transaction authority; +- employment/contractor/contributor intellectual property assignment chain; +- trademark/domain ownership and naming conflicts; +- patent/freedom-to-operate analysis where material; +- third-party commercial/runtime distribution rights; +- customer contracts, DPAs, processor/subprocessor obligations and data-transfer rights; +- production infrastructure/cloud/secret/KMS/data-residency evidence; +- incident/legal/regulatory history; +- independent security/compliance/accessibility assessments if claimed; +- customer/pilot/usage/revenue evidence and valuation assumptions. + +These are `EXTERNAL` evidence categories. Their absence from source control is not a software bug, but the repository must not imply they are complete. + +## Acquisition readiness exit criteria + +Engineering may describe the software package as acquisition-ready only when, at minimum: + +1. canonical docs are `PROTECTED_MAIN_SUFFICIENT`; +2. issue #5's production Office qualification and realistic fidelity/security evidence are integrated for every advertised transformed format, or those formats are explicitly unsupported; +3. issue #312's durable asynchronous lifecycle/backpressure/cancellation/restart recovery and tenant-safe persistence have release-grade evidence; +4. issue #314's production identity/federation boundary is integrated for the deployment profile, with privacy, retention, audit and authorization evidence; +5. issue #263's user-facing tenant-safe lifecycle is complete where included in product scope; +6. exact integrated CI/security/coverage/Javadocs/fidelity/accessibility/package/SBOM/provenance/migration/recovery/review gates pass; +7. the released artifact is verified after publication; and +8. the seller/buyer completes the required external legal/IP/commercial diligence without the repository claiming what it cannot prove. + +Until those conditions are met, the correct status is substantial engineering evidence with explicit remaining risk—not a finished acquisition-ready product. diff --git a/docs/API_CONTRACT.md b/docs/API_CONTRACT.md new file mode 100644 index 000000000..7e1c2dddd --- /dev/null +++ b/docs/API_CONTRACT.md @@ -0,0 +1,120 @@ +# Clearfolio API and Integration Contract + +Status: Canonical API authority index +Baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` + +This document is an index and compatibility contract. Source controllers and versioned OpenAPI artifacts remain executable schema authority. Active-PR behavior is labelled explicitly. + +## Contract principles + +- public URLs and payloads change only through reviewed compatibility analysis; +- tenant/permission authority is enforced server-side; +- cross-tenant resource existence is concealed where required; +- document-byte delivery requires the artifact authority defined in ADR-0002; +- asynchronous job APIs do not wait for conversion completion; +- errors use controlled response shapes and do not expose internal exception details; +- hosts such as naruon integrate over versioned interfaces rather than application-database access. + +## Current endpoint matrix + +| Endpoint | Authority | Maturity / notes | +| --- | --- | --- | +| `POST /api/v1/convert/jobs` | tenant `job:create` + validation/policy boundary | `IMPLEMENTED_ON_MAIN`; returns async job contract | +| `GET /api/v1/convert/jobs/{jobId}` | tenant job read + same-tenant ownership | `IMPLEMENTED_ON_MAIN` | +| `POST /api/v1/convert/jobs/{jobId}/retry` | retry permission + same-tenant/dead-letter eligibility | `IMPLEMENTED_ON_MAIN`; stricter signed-claim/admin contracts evolve in active stack | +| `DELETE /api/v1/convert/jobs/{jobId}` | delete permission + same tenant | present on current code path; durable deletion semantics `PARTIAL`; #268 `SUPERSEDED`, bounded successors unintegrated | +| `GET /api/v1/viewer/{docId}` | viewer permission + same tenant | `IMPLEMENTED_ON_MAIN`; returns bootstrap only for succeeded job | +| `GET /api/v1/convert/viewer/{docId}` | same as canonical viewer bootstrap | compatibility alias | +| `POST /api/v1/viewer/{docId}/artifact-links` | artifact-link-create permission + same tenant | `IMPLEMENTED_ON_MAIN`; returns signed short-lived URL/token metadata | +| `POST /api/v1/viewer/artifact-links/{tokenId}/revoke` | artifact-link-revoke permission + tenant-bound token record | `IMPLEMENTED_ON_MAIN` | +| `GET /api/v1/viewer/{docId}/artifact-read-events` | audit-read permission + tenant scope | `IMPLEMENTED_ON_MAIN` | +| `GET /artifacts/{docId}.pdf` | signed artifact token; token scope/doc/tenant/checksum/expiry/ledger/revocation | `IMPLEMENTED_ON_MAIN`; zero or one Range; controlled read audit | +| `GET /api/v1/convert/jobs/{jobId}/download` | tenant `artifact:read` + same tenant + signed artifact delivery | `IMPLEMENTED_ON_MAIN`; protected-main #270 requires signed artifact token verification, zero-or-one Range handling, checksum binding and verified-read audit | +| admin list/retry/delete surfaces | signed tenant claims + least-privilege admin permission + tenant-scoped service contract | protected #270 controls `IMPLEMENTED_ON_MAIN`; #268 `SUPERSEDED`, remaining lifecycle hardening `PARTIAL` in bounded successors | +| `GET /healthz` | orchestration probe | protected-main health surface; liveness semantic target `ACTIVE_PR` #295 | +| `GET /readyz` | orchestration traffic-readiness probe | `ACTIVE_PR` #295 only; not protected-main behavior | + +## Artifact delivery contract + +An authorized document-byte read must satisfy all applicable layers: + +1. tenant/actor permission for the endpoint; +2. same-tenant resource ownership/concealment; +3. succeeded conversion/artifact existence; +4. signed artifact token presence and signature validity; +5. non-expired `artifact:read` scope; +6. route document identity match; +7. issued-token ledger presence and not revoked; +8. tenant/job binding; +9. current artifact checksum match; +10. zero or one valid HTTP byte range; +11. controlled read-audit recording for verified-token reads. + +A valid tenant permission does not waive token/revocation/audit checks. A token does not waive endpoint tenant permission where the endpoint requires it. + +## Asynchronous lifecycle contract + +Public job state is currently: + +- `SUBMITTED` +- `PROCESSING` +- `SUCCEEDED` +- `FAILED` + +Retry exhaustion remains `FAILED` plus dead-letter evidence. Clients must not invent a separate terminal enum unless a versioned API change introduces one. + +A valid submit returns an accepted response and a status URL. Heavy conversion remains outside the request completion path. + +## Error semantics + +Errors use the repository's stable API error response fields (`errorCode`, compatibility `code` where present, `message`, `traceId`, `details`). Internal paths, secrets, token content, document bytes, raw tenant identifiers beyond required contract fields, and uncontrolled exceptions are not valid error details. + +Authorization status can intentionally distinguish: + +- authentication absent/invalid → controlled unauthorized; +- permission denied → controlled forbidden when disclosure is safe; +- cross-tenant or concealed resource → not found; +- valid resource not ready → conflict or other endpoint-specific controlled state; +- invalid range → range-not-satisfiable. + +## Versioning and compatibility + +Protected `main` currently identifies its shipped HTTP contract through the `/api/v1/**` path namespace. It does **not** yet implement a separate request/response API-version negotiation header. + +PR #379 is `ACTIVE_PR` evidence for a bounded negotiation contract and must not be represented as shipped until protected merge. Its proposed behavior keeps existing `/api/v1/**` callers backward compatible when `X-Clearfolio-Api-Version` is absent or blank, accepts an explicit exact `v1`, declares the current `v1` version on API responses, and rejects any explicitly unsupported version with a controlled error before controller dispatch. The exact active-PR head and its checks/reviews remain non-transferable evidence and must be revalidated before maturity changes. + +Breaking changes include removing/renaming public endpoints or fields, changing tenant/permission semantics in a way that makes previously authorized data inaccessible without a security reason/migration, changing lifecycle state meanings, changing artifact-token claim interpretation, or changing error semantics relied on by clients. + +Security tightening that closes an unsafe bypass may intentionally reject previously accepted requests. Such changes require migration notes and updated client/contract tests rather than preserving an insecure compatibility path. + +## naruon / MSA composition + +A host may: + +- submit documents under an explicitly trusted tenant/actor mapping; +- poll status or receive future versioned callbacks; +- request viewer bootstrap/signed artifact links; +- surface Clearfolio failure/recovery state to its own UI. + +A host may not: + +- read/write Clearfolio application persistence directly; +- mint Clearfolio artifact claims without delegated reviewed authority; +- bypass Clearfolio format/security validation; +- reinterpret placeholder output as supported-format fidelity; +- merge/release Clearfolio because its own model or status says the product is ready. + +## OpenAPI synchronization + +`docs/deployment/clearfolio-buyer-connector.openapi.yaml` is an existing integration artifact. It must be compared with current controllers and this contract whenever endpoints/permissions/payloads change. Until a single generated/openapi-tested schema covers every current endpoint, the OpenAPI surface is `PARTIAL`, not a complete product specification. + +## Contract tests + +Required changes to public API authority should include: + +- controller/serialization tests; +- tenant/permission and cross-tenant tests; +- malformed/edge input tests; +- artifact token/range/revocation/audit tests for byte delivery; +- OpenAPI/schema compatibility tests where the path is represented; +- naruon/integration contract tests when a versioned host interface is published. diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md new file mode 100644 index 000000000..cb7362175 --- /dev/null +++ b/docs/DATA_MODEL.md @@ -0,0 +1,302 @@ +# Clearfolio Data and Domain Model + +Status: Canonical logical model +Baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` + +This document is a logical/ownership model, **not a claim that every entity is stored in PostgreSQL**. Clearfolio currently uses in-memory repository state plus selected filesystem and append-only ledger persistence. Future SQL/object-store work must preserve these domain identities and authority boundaries rather than inventing new semantic IDs during migration. + +## Persistence legend + +- `MEMORY`: current default runtime process memory. +- `FILE_ARTIFACT`: current filesystem artifact persistence when configured. +- `FILE_LEDGER`: current append-only/file-backed evidence when configured. +- `ACTIVE_PR`: persistence/lifecycle behavior exists only in an open PR. +- `CONCEPTUAL`: required logical entity for target architecture; no protected-main persistence claim. +- `HOST_OWNED`: external host concern; not Clearfolio application persistence. +- `EXTERNAL_CONTROL_PLANE`: evidence whose durable storage belongs to the executing scheduler/platform rather than the Clearfolio application. + +## Logical entities + +| Entity | Ownership / persistence | Purpose | +| --- | --- | --- | +| `tenant_context` | request-scoped; `HOST_OWNED` identity source, Clearfolio-enforced claims | tenant, subject and permission authority for a request | +| `conversion_job` | `MEMORY` on protected main | immutable job identity, tenant ownership, source metadata and lifecycle | +| `content_identity` | `MEMORY` index on protected main | content-hash dedupe identity scoped by current repository rules | +| `conversion_artifact` | `FILE_ARTIFACT` or memory | converted/passthrough PDF bytes and artifact metadata | +| `artifact_token` | runtime + optional `FILE_LEDGER` metadata | short-lived signed artifact-read authority tied to tenant, subject, document, checksum, scope and expiry | +| `artifact_read_event` | runtime + optional `FILE_LEDGER` | read audit following token verification | +| `artifact_revocation` | runtime + optional `FILE_LEDGER` | revocation evidence for an issued artifact token | +| `policy_override` | audit event / request evidence | explicit blocked-format exception intent; raw approval token is not durable audit output | +| `audit_event` | mixed current event/log/ledger surfaces | purpose-bound audit evidence; identifiers may be pseudonymized but remain personal data | +| `analytics_snapshot` | runtime + optional `FILE_LEDGER` | buyer/operations snapshot evidence; not a complete durable telemetry model | +| `deletion_receipt` | `CONCEPTUAL` / `PARTIAL` target; issue #263; #268 `SUPERSEDED`; bounded successors #350/#351/#353 | generation-bound durable evidence for artifact deletion and cleanup recovery; no protected-main durable receipt store | +| `lifecycle_generation` | `CONCEPTUAL` / `PARTIAL` target; issues #263/#312; #268 `SUPERSEDED`; bounded successors #345/#350/#351/#353/#361 | prevents stale retry/delete work from acting on a replacement lifecycle; not protected-main durable fencing | +| `fidelity_evidence` | `CONCEPTUAL` / `PLANNED` | fixture, expected outcome, renderer version, source/artifact hashes and acceptance result for a supported-format claim | +| `conversion_engine` | `CONCEPTUAL` / `PLANNED` issue #5 | qualified Office-conversion provider/runtime identity, version, isolation profile, license/SBOM/provenance and support status | +| `conversion_attempt` | `CONCEPTUAL` / `PLANNED` issue #5 | one immutable job/generation/source-digest/engine/policy-bound conversion execution and typed outcome | +| `conversion_quarantine` | `CONCEPTUAL` / `PLANNED` issue #5 | controlled disposition of hostile, policy-denied, crashed or otherwise quarantined source/result evidence | +| `adapter_health_snapshot` | `CONCEPTUAL` / `PLANNED` issue #5 | bounded capacity/readiness/resource/process-health evidence for a qualified adapter runtime | +| `format_support_record` | `CONCEPTUAL` / `PLANNED` issue #5 | versioned format support tier, limitations and fidelity/security acceptance bound to an engine/policy version | +| `conversion_audit_event` | `CONCEPTUAL` / `PLANNED` issue #5 | privacy-safe Office attempt lifecycle evidence for accepted, rejected, timed-out, cancelled, crashed, retried, quarantined and completed outcomes | +| `automation_run` | GitHub/workflow/scheduler evidence; `CONCEPTUAL` locally | autonomous run identity, control version, and exact historical source/base evidence | +| `automation_checkpoint` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | last safe execution phase and exact historical evidence identity | +| `queue_snapshot` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | bounded fresh-state inventory used to select the next atomic action | +| `deferred_lane` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | branch-local wait keyed by exact PR/head/live-base/run/review identity | +| `action_receipt` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | proof that one atomic repository action completed at an exact evidence boundary | +| `failure_envelope` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | controlled observed failure phase/category without invented hidden cause or sensitive exception text | +| `continuation_handoff` | `CONCEPTUAL` / `EXTERNAL_CONTROL_PLANE`; issue #331 | privacy-safe next-run context that is historical and requires a full live refetch | +| `review_evidence` | GitHub-owned; `CONCEPTUAL` locally | independent review identity/state distinct from checks/status/model output | +| `naruon_contract` | versioned interface; `HOST_OWNED` orchestration | explicit host/service composition contract without cross-database access | + +The Office qualification entities above are a logical vocabulary for issue #5 and active Draft #306. Their presence **does not imply a database table**, physical foreign key, durable SQL schema, deployed sidecar, approved remote service, or production Office-format support. PR #306 supplies `ACTIVE_PR` provider-neutral contract/publication-validation evidence; the production runtime qualification remains `PLANNED` until issue #5 acceptance is integrated. + +The scheduler execution-receipt entities are likewise conceptual/external. Their presence **does not imply a database table** in Clearfolio. The executing external scheduler, central control plane, or a future repository-local workflow owns durable receipt storage when supported. Clearfolio owns only the semantic contract, privacy boundary, traceability, and tests unless a later accepted ADR explicitly changes persistence ownership. + +## Core relationships + +```mermaid +erDiagram + tenant_context ||--o{ conversion_job : authorizes + content_identity ||--o{ conversion_job : deduplicates + conversion_job ||--o| conversion_artifact : produces + conversion_artifact ||--o{ artifact_token : authorizes_read_of + artifact_token ||--o{ artifact_read_event : records + artifact_token ||--o| artifact_revocation : may_have + conversion_job ||--o{ audit_event : emits + policy_override ||--o{ audit_event : is_audited_by + conversion_job ||--o{ analytics_snapshot : contributes_to + conversion_job ||--o{ deletion_receipt : "PARTIAL target delete evidence" + conversion_artifact ||--o{ fidelity_evidence : "PLANNED validates" + conversion_job ||--o{ conversion_attempt : "PLANNED attempts" + conversion_engine ||--o{ conversion_attempt : "PLANNED executes" + conversion_attempt ||--o| conversion_quarantine : "PLANNED may quarantine" + conversion_engine ||--o{ adapter_health_snapshot : "PLANNED reports" + conversion_engine ||--o{ format_support_record : "PLANNED qualifies" + fidelity_evidence ||--o{ format_support_record : "PLANNED supports" + conversion_attempt ||--o{ conversion_audit_event : "PLANNED emits" +``` + +The diagram is conceptual. It deliberately does not imply foreign keys or a physical relational schema for entities that are currently in memory, filesystem ledgers, GitHub, host-owned systems, planned Office-adapter qualification evidence, or an external scheduler control plane. + +## `conversion_job` + +Required semantics: + +- opaque UUID job identity; +- tenant ownership; +- original filename/content type/size metadata; +- content identity/dedupe linkage; +- lifecycle status (`SUBMITTED`, `PROCESSING`, `SUCCEEDED`, `FAILED`); +- retry/dead-letter evidence; +- created/started/completed/retry timing; +- immutable generation identity when durable mutation fencing is integrated. + +A job UUID must never be rebound to another tenant or lifecycle generation merely because earlier metadata was deleted. + +## `content_identity` + +Protected main uses cryptographic content hashing for dedupe. A future durable implementation must define whether identity is tenant-scoped or otherwise privacy-safe before global dedupe is introduced. The hash must not be treated as anonymous data or exposed as an uncontrolled metric label. + +## `conversion_artifact` + +Minimum semantics: + +- owner job identity; +- artifact media type; +- byte length; +- checksum/digest; +- origin (`passthrough`, `transformed`, `degraded`, or development-only placeholder); +- renderer/converter version when transformed; +- publication and deletion state; +- storage generation/version for remote object stores. + +`FILE_ARTIFACT` on protected main means local disk durability only. It does not imply distributed object-store atomicity or durable job-state convergence. + +## `artifact_token` + +Signed-token claims include at least: + +- token identifier; +- tenant identifier; +- subject identifier; +- document/job identifier; +- read scope; +- purpose; +- artifact checksum; +- issue time; +- expiry time; +- signature/version metadata. + +Validation authority combines signature, expiry, scope, document binding, ledger presence, revocation, tenant ownership and current artifact checksum. Dedicated tenant permission and a valid artifact token are separate controls where both are required. + +## `artifact_read_event` + +A read event is emitted only after token verification and captures controlled audit fields such as token/document identity, range request, response status, trace identifier and read time. It must not record raw authorization headers or token secrets. + +## `deletion_receipt` (`PARTIAL` target; issue #263; #268 `SUPERSEDED`) + +The target receipt binds: + +- deletion request identity; +- tenant/job/lifecycle generation; +- pending or verified artifact checksum state; +- durable state transition; +- attempt count and last attempt time; +- controlled failure code; +- cleanup completion evidence. + +Historical PR #268 is superseded. Current bounded successors preserve individual lifecycle primitives, but protected `main` does not yet own a durable deletion-receipt store or the complete restart-recovery sequence. A failed initial artifact read must not be misrepresented as confirmed absence. Recovery order must be reconstructable from durable evidence rather than process-local cursors before this model can become `IMPLEMENTED_ON_MAIN`. + +## `fidelity_evidence` (`PLANNED`) + +Recommended logical fields use two-or-more-word `snake_case` names: + +- `fidelity_evidence_id` +- `source_fixture_id` +- `source_format_code` +- `source_content_hash` +- `converter_version` +- `artifact_content_hash` +- `expected_page_count` +- `structural_result` +- `visual_result` +- `extraction_result` +- `accessibility_result` +- `security_result` +- `accepted_difference` +- `verification_time` +- `source_commit_sha` + +This entity becomes authoritative only after a persistence/evidence contract is implemented; until then it is a release-domain model. + +## Office qualification model (`ACTIVE_PR` #306 / `PLANNED` issue #5) + +The provider-neutral Java contract in Draft #306 is implementation evidence for the request/result boundary and post-provider PDF publication checks. The entities below describe the larger qualification domain that must still be proven before production Office support can be claimed. + +### `conversion_engine` + +Logical fields should bind at least: + +- `conversion_engine_id` +- `adapter_contract_version` +- `engine_provider_code` +- `engine_runtime_version` +- `runtime_image_digest` +- `runtime_license_manifest` +- `runtime_sbom_digest` +- `runtime_provenance_digest` +- `isolation_policy_version` +- `qualification_status` + +A `conversion_engine` describes a qualified execution boundary. It is not permission to start LibreOffice/JODConverter inside the API container. + +### `conversion_attempt` + +An attempt binds the exact immutable execution inputs and result identity, including: + +- tenant and `conversion_job` identity; +- lifecycle generation; +- source format and source digest; +- `conversion_engine` and adapter version; +- conversion policy version; +- correlation/request identity; +- queue/start/finish timing; +- typed result/failure classification; +- output digest when publication validation succeeds. + +Retry creates another bounded attempt; it must not rewrite the provenance of the earlier attempt. + +### `conversion_quarantine` + +Quarantine is a controlled state/evidence object for source or result material that must not continue through normal publication. It records a non-sensitive reason code, policy/engine identity, bounded retention/review state and cleanup outcome without exposing document content or local paths. + +### `adapter_health_snapshot` + +This low-cardinality evidence records whether a qualified runtime has bounded capacity to accept work and may include process count/age, queue depth, saturation class, crash/timeout counters and policy/configuration identity. It must not expose tenant or document identifiers as metric labels. + +### `format_support_record` + +A support record binds one source-format/version family to exact engine/policy/fidelity/security evidence and a support tier such as qualified, limited, unsupported or disabled. A successful process exit alone cannot create this record as qualified. + +### `conversion_audit_event` + +This logical Office-specific audit event narrows the generic `audit_event` purpose to one conversion attempt and records controlled lifecycle outcomes such as accepted, rejected, timed out, cancelled, crashed, retried, quarantined and completed. If future persistence unifies it with the generic audit ledger, that must be a reviewed schema decision rather than an implicit alias. + +None of these conceptual names imply a database table. Future persistence may choose tables, events, object metadata or another durable representation, but it must preserve their semantic identity, tenant boundary and evidence provenance. + +## Scheduler execution receipt model (`PLANNED`; issue #331) + +The receipt domain makes external autonomous execution diagnosable without granting runtime authority to repository prose. + +### `automation_checkpoint` + +Minimum conceptual fields include: + +- `automation_checkpoint_id` +- `automation_run_id` +- `control_version_digest` +- `execution_phase_code` +- `protected_main_sha` +- `source_head_sha` +- `live_base_tip_sha` +- `target_ref_name` +- `target_blob_sha` +- `last_action_code` +- `checkpoint_time` + +All Git identities are historical after movement and must be refetched before another write. + +### `action_receipt` + +An action receipt binds one atomic action to its exact observable result. It may reference a commit, issue, PR state transition, workflow/run/check identity, thread resolution, or controlled no-mutation proof. It never means that another gate, approval, merge, release, or product requirement passed. + +### `failure_envelope` + +A failure envelope stores a controlled phase/category such as admission, queue construction, connector, permission, repository policy, provider, publication CAS, verification, or telemetry unavailable. It must not contain raw secrets, document content, tenant/subject identifiers, uncontrolled provider exception text, or an invented root cause. + +### `continuation_handoff` + +A continuation handoff identifies the last safe checkpoint, any branch-local deferred lanes, and the next permitted class of work. It cannot authorize a mutation. The next run must rebuild a fresh queue from live GitHub state. + +### Receipt lifecycle + +```mermaid +erDiagram + automation_run ||--o{ automation_checkpoint : records + automation_run ||--o{ queue_snapshot : builds + queue_snapshot ||--o{ deferred_lane : defers + automation_checkpoint ||--o| action_receipt : proves + automation_checkpoint ||--o| failure_envelope : classifies + automation_checkpoint ||--o| continuation_handoff : resumes_with +``` + +Storage remains `EXTERNAL_CONTROL_PLANE` unless a later accepted ADR assigns a repository-local durable implementation. No Clearfolio application database is introduced by ADR-0012. + +## Automation/evidence model + +GitHub evidence must keep authorities separate: + +```mermaid +erDiagram + automation_run ||--|| source_revision : checks_out + automation_run ||--|| base_revision : evaluates_against + automation_run ||--o{ check_evidence : produces + automation_run ||--o{ workflow_evidence : produces + source_revision ||--o{ review_evidence : reviewed_at + source_revision ||--o{ status_evidence : reports_at +``` + +A green status does not become an independent approval; a synthetic merge does not become exact-head product proof; a PR-body SHA does not override live source revision; an action receipt does not become release or merge authority. + +## Future physical persistence rules + +When SQL persistence is added: + +1. every Clearfolio-owned object name has at least two descriptive `snake_case` words; +2. tenant ownership is represented in keys/constraints, not reconstructed only in application filters; +3. lifecycle generation and idempotency keys participate in mutation predicates; +4. state transitions are transactional and append audit/outbox evidence atomically where required; +5. secrets and raw approval/token values are not persisted as ordinary business data; +6. migration and rollback tests prove tenant isolation and identity preservation; +7. no other CWL service may write Clearfolio tables directly; +8. external scheduler receipt entities remain external unless a new accepted ADR changes their ownership and supplies migration, retention, privacy, recovery, and rollback evidence. diff --git a/docs/DOCUMENTATION_ASSESSMENT.md b/docs/DOCUMENTATION_ASSESSMENT.md new file mode 100644 index 000000000..2795b64b9 --- /dev/null +++ b/docs/DOCUMENTATION_ASSESSMENT.md @@ -0,0 +1,209 @@ +# Clearfolio Documentation Completeness Assessment + +Date: 2026-08-10 +Scope baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` +Assessment rule: active pull requests and issues are work-in-progress evidence, never shipped capability. + +## Executive finding + +Protected `main` is **PROTECTED_MAIN_INSUFFICIENT** as a complete acquisition-grade documentation source of truth because the current canonical graph remains on PR #305 and has not passed the repository's counted independent-human approval gate and integrated. + +The current PR #305 documentation line is **DESIGN_SUFFICIENT** when its exact-head semantic, Maven, and security contract suite is green. It covers product requirements, technical requirements, architecture, status-bearing ADRs, component/sequence/state/deployment/authority/recovery UML, conceptual/logical ERD and persistence ownership, API/schema authority, security/threat model, test strategy, operability/recovery, fidelity acceptance, migration/rollback, release/provenance, acquisition diligence, research/standards, and requirement-to-evidence traceability. + +This verdict does not mean product, release, certification, transaction, valuation, or operational readiness. It means the reviewed branch can reconstruct what is `IMPLEMENTED_ON_MAIN`, `ACTIVE_PR`, `PARTIAL`, `ACCEPTED_ARCHITECTURE`, `PLANNED`, `RESEARCH_ONLY`, `SUPERSEDED`, or `OUT_OF_SCOPE` without relying on this conversation. + +## Protected-main advancement since the first assessment + +PR #270 is now integrated on protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11`. Its privacy-safe failure logging, signed direct artifact delivery, exact-head CI, zero-missed owned production coverage, strict public Javadocs, SBOM/attribution, and related security evidence are therefore `IMPLEMENTED_ON_MAIN`, not `ACTIVE_PR`. + +Former descendants are not automatically current because their old ancestry and checks were bound to predecessor identities. The repository has begun clean reconciliation on the protected baseline: + +- PR #271: recurring OpenCode product loop and early-stop execution contract, clean `ACTIVE_PR`; +- PR #276: malformed artifact-token structural rejection, clean `ACTIVE_PR`; +- PR #313: production HMAC key-readiness guard, clean `ACTIVE_PR`; +- PR #318: bounded external-company branding removal, clean `ACTIVE_PR`; +- current #264: nested-safe accessible asynchronous controls, clean `ACTIVE_PR` with dependency-free Node integration evidence; +- current #295: liveness/readiness separation, clean `ACTIVE_PR`; +- current #340: deterministic single logging runtime binding, clean `ACTIVE_PR` with `spring-jcl` retained and standalone `commons-logging` excluded. + +PR #264 and PR #295 have been rebuilt on the protected baseline and are current-base active work. Historical #268 is `SUPERSEDED`; its valuable requirements are mapped to bounded current successors #341/#342/#345/#350/#351/#353/#361/#363/#380, none of which is protected-main behavior merely by inheritance. No predecessor review or check transfers automatically. + +## Conversation decision coverage + +Conversation verdict: **`CONVERSATION_COVERAGE_SUFFICIENT`**. +Control-plane enforcement verdict: **`CONTROL_ENFORCEMENT_INCOMPLETE`**. + +The assessment covers **repository-scoped conversation decisions** for Clearfolio: do not stop after one useful result; **prompt repair earns zero completion credit**; rebuild the live queue and perform a **same-invocation substantive action** when a safe action exists; use **multi-lane rotation** when more than one independent safe lane is executable; and terminate normally only after **two consecutive fresh exit sweeps** discover no safe work. A final response, status summary, documentation assessment, execution receipt, green check, merge, or blocker remains intermediate while another safe action exists. + +Those decisions are represented across ADR-0009, ADR-0011, ADR-0012, `docs/OPERABILITY.md`, `docs/TRACEABILITY.md`, this assessment, and the active PR #271 scheduler/operator contract. The canonical graph also captures the product, security, privacy, testing, fidelity, recovery, release, provenance, acquisition, API, data-ownership, standalone, and naruon/MSA decisions that apply to Clearfolio. + +The scope is intentionally repository-bounded. **Other ContextualWisdomLab projects are not imported** into Clearfolio's PRD, TRD, Architecture, ADR, UML, or ERD merely because their reports appeared in the same conversation or attached source set. Cross-repository dependencies are referenced only where they define an actual Clearfolio boundary, such as organization-owned PR maintenance or optional naruon composition. + +`CONTROL_ENFORCEMENT_INCOMPLETE` remains accurate because: + +- PR #271 is `ACTIVE_PR` and its stronger prompt contract is not protected-main behavior; +- the external scheduler still lacks durable phase/action/checkpoint receipts and issue #331 remains `PLANNED`; +- the canonical documentation graph itself remains on PR #305; +- issue #321's counted independent-human reviewer route is unresolved; and +- branch protection, exact-head checks, writer leases, and review independence still control integration. + +Therefore conversation coverage is sufficient at the design/source-of-truth level, while operational enforcement and protected-main integration remain incomplete. + +## Evidence identity + +A source document cannot embed the SHA of the commit that contains itself. Dated SHAs and run IDs in documents are historical evidence only. Live GitHub APIs are the merge authority for current source head, independently resolved live base, required checks, formal reviews, unresolved threads, and protected policy. + +The following evidence classes are distinct: + +- scheduler enabled/last-run evidence; +- scheduler admission and queue-construction evidence; +- action receipt and last safe checkpoint evidence; +- exact-head source checks; +- synthetic-merge evidence; +- status/model/advisory review evidence; +- formal counted human approval; +- protected-main operational/release evidence. + +A generic scheduled-task error is not completion and does not identify its hidden internal cause. Issue #331 and ADR-0012 define the planned scheduler execution receipt, controlled failure envelope, continuation handoff, and budget continuation contract without inventing platform telemetry. + +## Fitness scale + +Document-family fitness: + +- `PRESENT_CURRENT` +- `PRESENT_STALE` +- `PARTIAL` +- `MISSING` +- `NOT_APPLICABLE` +- `OWNED_BY_OTHER_REPO` +- `OWNED_BY_ACTIVE_PR` +- `SUPERSEDED` + +Capability maturity: + +- `IMPLEMENTED_ON_MAIN` +- `ACTIVE_PR` +- `PARTIAL` +- `ACCEPTED_ARCHITECTURE` +- `PLANNED` +- `RESEARCH_ONLY` +- `SUPERSEDED` +- `OUT_OF_SCOPE` + +Integration fitness: + +- `DESIGN_SUFFICIENT`: one reviewed branch explains current/active/planned product truth and passes semantic documentation contracts. +- `PROTECTED_MAIN_SUFFICIENT`: the same code-current graph is integrated on protected main after all live checks and review policy. +- `PROTECTED_MAIN_INSUFFICIENT`: protected main does not yet contain the current complete canonical graph or its current acceptance evidence. + +## Artifact assessment and branch remediation + +| Artifact family | Protected-main assessment | PR #305 remediation / current authority | +| --- | --- | --- | +| Product requirements | `PRESENT_STALE` | `docs/PRD.md` is the current product spine and marks active/planned behavior truthfully. | +| Technical requirements | `PRESENT_STALE` | `docs/TRD.md` replaces deferred-MVP assumptions with current technical and evidence contracts. | +| Architecture | `PRESENT_STALE` | root `ARCHITECTURE.md` is the canonical system, trust, deployment, and document index. | +| ADRs | `MISSING` | `docs/adr/README.md` plus ADR-0001…0012 records ownership, authorization, lifecycle, fidelity, availability, evidence, automation, release, thin scheduler, and scheduler execution receipt decisions. | +| UML / behavior / deployment | `PARTIAL` | `docs/UML.md` covers components, submit/view/auth, deletion recovery, liveness/readiness, Office isolation, automation authority, scheduler receipt/continuation, standalone/MSA, and degraded modes. | +| ERD / data-domain model | `MISSING` | `docs/DATA_MODEL.md` distinguishes memory, file-ledger, file-artifact, host-owned, conceptual, and external-control-plane entities and explicitly avoids inventing physical tables. | +| API/schema contracts | `PARTIAL` | `docs/API_CONTRACT.md` indexes current authority; issue #315 owns the complete versioned API and standalone+naruon compatibility. PR #316 remains bounded integrity work. | +| Security policy | `PARTIAL` | root `SECURITY.md` is the current product security entrypoint; canonical architecture, API, threat, and ADR documents remain detailed control authorities. | +| Threat model | `PARTIAL` | `docs/THREAT_MODEL.md` reconciles protected-main threats and active hardening without claiming unmerged controls are shipped. | +| Test strategy | `PARTIAL` | `docs/TEST_STRATEGY.md` pairs exact coverage/Javadocs with realistic security, fidelity, recovery, accessibility, and data-integrity tests. | +| Operability / incident / recovery | `PARTIAL` | `docs/OPERABILITY.md` covers startup/degraded/recovery, activation versus execution, fail-closed prewrite refetch/freeze, issue #331 receipts, last safe checkpoint, controlled failure, budget continuation, and fresh GitHub state. | +| Fidelity acceptance | `MISSING` | `docs/FIDELITY_ACCEPTANCE.md` prevents placeholder, extension, or HTTP success from becoming Office-support claims. | +| Migration / rollback | `PARTIAL` | `docs/MIGRATION_ROLLBACK.md` records current process-local/file truth and future durable migration invariants without fake SQL. | +| Release / provenance | `PARTIAL` | `docs/RELEASE_ACCEPTANCE.md` binds one exact integrated source/artifact identity across CI, security, fidelity, recovery, review, SBOM, and provenance. | +| Requirement-to-evidence traceability | `MISSING` | `docs/TRACEABILITY.md` includes current issues and bounded implementation PRs with explicit maturity. | +| Research / standards | `MISSING` | `docs/RESEARCH_TRACEABILITY.md` separates normative, primary, and research evidence from implementation claims. | +| Acquisition / IP / legal diligence | `PRESENT_STALE` | `docs/ACQUISITION_DILIGENCE.md` is the current technical diligence authority; dated buyer artifacts are historical. Repository evidence remains separate from external legal, IP, and commercial evidence. | +| Automation authority | `PARTIAL` | ADR-0008, ADR-0009, ADR-0011, ADR-0012, UML, DATA_MODEL, OPERABILITY, and TRACEABILITY separate central PR maintenance, local OpenCode proposal work, external scheduling, receipts, reviewer authority, and writer leases. | + +## Current executable product, security, reliability, and governance gaps + +Documentation is sufficient only when it exposes rather than hides open work: + +- issue #5 / Draft #306: real production Office conversion, sandbox/runtime, hostile source, license/SBOM/provenance, and Korean/English fidelity; +- issue #263: complete tenant-safe download, deletion, and recovery user journey; +- issue #312: durable asynchronous acceptance, outbox/idempotency, leases, backpressure, cancellation, restart/redelivery, and privacy-safe OpenTelemetry; +- issue #314: production OIDC/JWT federation; +- issue #315 / PR #316: one complete **versioned API** and standalone+naruon compatibility; bounded OpenAPI integrity is not completeness; +- issue #317 / PR #318: truthful **production workspace** and session bootstrap instead of buyer-demo authority; branding cleanup is not a workspace; +- issue #319 / PR #313: runtime **credential registry** rather than environment-backed runtime secret authority; key readiness is not source migration; +- issue #320 / current #340: one intentional logging runtime bridge and deterministic warning-free startup/SBOM evidence; +- issue #321: observable CODEOWNERS and independent human reviewer route; advisory bots are not counted approval; +- issue #322 / PR #323: viewer generation safety; stale DOM publication is partially addressed, while active PDF.js cancellation/destruction and broader parity remain; +- issue #324 / current #334: robust light/dark focus appearance with executable contrast and geometry evidence; +- issue #326: push analytics tenant isolation into repository queries after the accepted scoped-API contract preserved from `SUPERSEDED` #268; +- issue #327 / current #338: terminal-outcome conversion success rate; +- issue #329 / current #339: fail-closed finite and domain-valid KPI ledger evidence; +- issue #331: scheduler **execution receipt**, failure envelope, resumable checkpoint, and budget continuation evidence. + +These gaps demonstrate why `DESIGN_SUFFICIENT` is not product, release, acquisition, or commercial completion. + +## Material source-of-truth drift found and repaired on the branch + +1. Exact-head drift: PR prose can retain predecessor SHAs; live GitHub state is authoritative. +2. Security maturity drift: PR #270 is now implemented on main; its former descendants require deliberate current-base reconciliation rather than inherited evidence. +3. Conversion drift: non-PDF placeholder output is not production Office fidelity. +4. Durability drift: file-backed artifacts and ledgers do not make conversion-job state a distributed durable queue. +5. Availability drift: separate readiness is current #295 on the protected baseline and remains `ACTIVE_PR` until integrated. +6. Automation drift: central maintenance, local OpenCode work, external scheduling, counted human review, and model/advisory evidence are separate authorities. +7. Scheduler evidence drift: activation or a generic scheduled-task error is not an execution receipt, root-cause verdict, or repository completion. Issue #331 and ADR-0012 model the missing evidence as conceptual/external rather than fake application persistence. +8. Release drift: green individual PRs do not prove one integrated protected release artifact. +9. Acquisition drift: dated diligence material is historical and cannot prove present transaction readiness. +10. API drift: issue #315 is the complete versioned contract; PR #316 is bounded integrity work. +11. Workspace drift: issue #317 is the production workspace; PR #318 is bounded branding work. +12. Credential drift: issue #319 is runtime authority migration; PR #313 is key readiness only. +13. Viewer drift: issue #322 requires operation-generation ownership and active cancellation; PR #323 is only the first stale-publication slice. +14. Accessibility drift: one mixed focus color is not robust light/dark focus evidence; issue #324 and current #334 own the current slice. +15. Analytics isolation drift: global materialization plus application filtering is weaker than a tenant-scoped repository query; issue #326 owns adoption after the scoped-API contract preserved from `SUPERSEDED` #268. +16. KPI semantic drift: in-flight work must not count as failed conversion outcomes; issue #327 and current #338 own the denominator correction. +17. KPI integrity drift: parseable NaN, infinity, or out-of-range values are invalid buyer evidence; issue #329 and current #339 own fail-closed replay validation. +18. Governance drift: protected merge requires a counted write-authorized human approval; central `.github#772`, not broader bot authority, owns provisioning. + +## Canonical documentation graph on PR #305 + +Core spine: + +- `docs/PRD.md` +- `docs/TRD.md` +- root `ARCHITECTURE.md` +- `SECURITY.md` +- `docs/adr/README.md` and ADR-0001…0012 +- `docs/DATA_MODEL.md` +- `docs/UML.md` +- `docs/API_CONTRACT.md` +- `docs/THREAT_MODEL.md` +- `docs/TEST_STRATEGY.md` +- `docs/OPERABILITY.md` +- `docs/TRACEABILITY.md` +- `docs/RESEARCH_TRACEABILITY.md` +- `docs/ACQUISITION_DILIGENCE.md` + +Release and operational authorities: + +- `docs/FIDELITY_ACCEPTANCE.md` +- `docs/MIGRATION_ROLLBACK.md` +- `docs/RELEASE_ACCEPTANCE.md` +- `docs/engineering/acceptance-criteria.md` + +Dated evidence under QA, deployment, and buyer paths remains provenance only and cannot override this graph or live protected code. + +## Machine-checkable documentation gate + +The branch contract suite includes: + +- `scripts/test_documentation_spine_contract.py`; +- `scripts/test_release_loop_adr_contract.py`; +- `scripts/test_documentation_assessment_security_contract.py`; +- `scripts/test_office_architecture_documentation_contract.py`; +- `scripts/test_acquisition_diligence_contract.py`; +- `scripts/test_live_product_gap_traceability_contract.py`; +- `scripts/test_scheduler_execution_receipt_documentation_contract.py`. + +Together these protect required and indexed documents and ADRs; maturity vocabulary; root security authority; exact-head, live-base, and review separation; OpenCode plus `NVIDIA_NIM_API_KEY` and no `COPILOT_GITHUB_TOKEN`; placeholder versus fidelity truth; Office isolation and conceptual data entities; migration, release, and acquisition boundaries; current issue/PR traceability; scheduler execution receipt, budget continuation, and failure-recovery semantics; balanced Mermaid/code fences; and the rule that documentation work is intermediate. + +## Completion rule + +Documentation completion is not repository completion. PR #305 must pass exact-head semantic, Maven, and security checks; receive the counted independent human approval required by live protection; integrate on protected main; and then be reconciled after later product merges. Even then product, release, transaction, and valuation readiness require the open implementation, operational, legal/IP, customer-data, certification, and commercial evidence described above. diff --git a/docs/FIDELITY_ACCEPTANCE.md b/docs/FIDELITY_ACCEPTANCE.md new file mode 100644 index 000000000..b5753a419 --- /dev/null +++ b/docs/FIDELITY_ACCEPTANCE.md @@ -0,0 +1,126 @@ +# Clearfolio Document Fidelity Acceptance + +Status: Canonical product-quality gate +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +This document defines what Clearfolio may call a supported document conversion. It is deliberately stricter than “the endpoint returned a PDF.” Conversion support is a buyer-facing claim and therefore requires deterministic, reproducible evidence. + +## Maturity vocabulary + +- `IMPLEMENTED_ON_MAIN`: protected-main behavior exists and may be described as current, subject to the evidence limits below. +- `ACTIVE_PR`: implementation exists only on an open pull request and is not shipped behavior. +- `PARTIAL`: a usable boundary exists but does not yet satisfy the full production-fidelity contract. +- `ACCEPTED_ARCHITECTURE`: the design is accepted but implementation/evidence is incomplete. +- `PLANNED`: backlog only. + +## Current support truth + +| Source class | Current maturity | Allowed product claim | +| --- | --- | --- | +| PDF passthrough with `%PDF-` validation | `IMPLEMENTED_ON_MAIN` | Native PDF bytes can be stored and served through the artifact path subject to validation/security gates. | +| Non-PDF placeholder generation | `IMPLEMENTED_ON_MAIN` as development/demo behavior | A one-page development/demo PDF may be generated. This is **not** Office rendering or fidelity evidence. | +| DOCX/XLSX/PPTX production conversion | `PLANNED` / issue #5 | No production support claim until a sandboxed adapter and realistic fixture evidence pass this contract. | +| HWP/HWPX | `PLANNED` / currently blocked by policy | Unsupported unless a separately reviewed format/security/fidelity decision changes the policy. | + +The existence of a file extension parser, MIME acceptance path, placeholder PDF, or successful HTTP response is never sufficient evidence for a production-format support claim. + +## Production conversion acceptance + +A format may move to production-supported status only when one exact integrated protected head proves all applicable gates. + +### Deterministic rendering + +- identical source bytes, converter version/configuration, fonts, locale, and declared platform profile produce semantically equivalent output; +- network access is not required by default and is denied at the converter boundary; conversion must not dereference external hyperlinks or fetch remote resources; +- network isolation does not by itself authorize stripping inert URI/internal hyperlinks from output; fidelity fixtures must prove legitimate hyperlinks remain usable unless a versioned product policy explicitly disallows them; +- converter/version/configuration and source/output digests are recorded in provenance evidence; +- nondeterministic metadata is normalized or explicitly excluded from the equality model without hiding semantic differences. + +### Realistic fixtures and rights + +The benchmark corpus must contain authorized or redistributable fixtures representative of actual buyer documents. Synthetic-only examples may supplement but not replace realistic fixtures. Fixture provenance and redistribution rights must be documented. + +At minimum, fixtures must exercise where the format supports them: + +- text, fonts, Unicode and Korean text; +- tables, pagination and page breaks; +- images and transparency; +- headers/footers, sections and orientation; +- hyperlinks and annotations; +- charts/drawings or other embedded visual objects; +- formulas, merged cells and print areas for spreadsheets; +- slide/page geometry for presentations; +- malformed and adversarial inputs. + +### Fidelity assertions + +Evidence must be machine-comparable where practical and manually inspectable where semantics cannot be reduced safely to one metric. The suite should include: + +- page count and page geometry; +- normalized extracted text and ordering; +- rendered-page image comparison with documented tolerances; +- object/table/cell/slide assertions relevant to the source format; +- no silent page/object loss; +- controlled degraded-mode classification when exact fidelity is impossible. + +One global similarity score is not sufficient to authorize a format claim when a critical semantic object can disappear while the score remains high. + +### Security and resource bounds + +- conversion runs in a bounded sandbox/process boundary appropriate to the selected adapter; +- active content/macros/embedded executables are rejected, neutralized, or handled by an explicit reviewed policy; +- external-link dereferencing and remote-resource loading are disabled by default; inert output URI/internal-link actions may be preserved for fidelity, but executable action classes such as script, launch, submit/import, or remote-execution behavior require explicit fail-closed policy and regression evidence; +- CPU, memory, file size, page/object count, recursion/archive expansion and wall-clock limits are finite; +- converter failures produce controlled error states and never substitute placeholder success; +- temporary files are app-owned, bounded and cleaned after success/failure/cancellation; +- artifacts remain subject to tenant authorization and signed-delivery controls. + +### Failure taxonomy + +Every conversion attempt is classifiable as one of: + +- `native` — no semantic transform, such as accepted PDF passthrough; +- `transformed` — production adapter completed and fidelity/security gates passed; +- `degraded` — a documented non-critical fidelity limitation occurred and the API/UI surfaces it explicitly; +- `unsupported` — no approved adapter/format policy exists; +- `failed` — approved processing started but could not produce an accepted artifact; +- `development_placeholder` — non-production/demo output only. + +A `development_placeholder` result must never be emitted as `transformed` or used in production-fidelity KPI evidence. + +## Release evidence + +For every supported transformed format, a release candidate must preserve: + +- fixture manifest with source licenses/rights metadata and integrity digests; +- converter binary/package/version/configuration provenance; +- deterministic focused and full-suite results from the exact release head; +- security/resource-bound regressions; +- representative rendered outputs or privacy-safe derived evidence; +- rollback criteria when the converter/runtime version changes. + +Historical evidence from another converter version, head SHA, platform profile, or fixture corpus does not transfer automatically. + +## Adapter qualification workflow + +Issue #5 remains the product gap for the first sandboxed Office adapter. Qualification order is: + +1. establish a legal/redistributable fixture corpus and expected semantics; +2. pin and sandbox the converter/runtime; +3. add RED tests that currently demonstrate unsupported/placeholder behavior; +4. implement the smallest adapter boundary; +5. prove deterministic rendering, security limits, failure taxonomy and provenance; +6. run exact-head product/security/package evidence; +7. update PRD/TRD/API/support matrix only after the implementation is integrated. + +The selected adapter must remain replaceable behind a narrow Clearfolio-owned contract so standalone operation and MSA composition do not depend on one vendor-specific storage or orchestration design. + +## Traceability + +- Product support claims: `docs/PRD.md` +- Technical conversion boundary: `docs/TRD.md` +- Architecture and component ownership: `ARCHITECTURE.md` +- Decision record: `docs/adr/0005-deterministic-conversion-fidelity.md` +- Test strategy: `docs/TEST_STRATEGY.md` +- Standards/research index: `docs/RESEARCH_TRACEABILITY.md` +- Requirement/evidence mapping: `docs/TRACEABILITY.md` diff --git a/docs/MIGRATION_ROLLBACK.md b/docs/MIGRATION_ROLLBACK.md new file mode 100644 index 000000000..abe51bde1 --- /dev/null +++ b/docs/MIGRATION_ROLLBACK.md @@ -0,0 +1,148 @@ +# Clearfolio Migration, Rollback, and Recovery Contract + +Status: Canonical operational-change authority +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +This document defines how Clearfolio changes runtime state, persistence formats, security contracts, converters, and deployment behavior without turning rollback into an improvised incident procedure. + +## Current persistence truth + +Protected main is not a database-backed production architecture yet. + +- Conversion-job state is process-local/in-memory and is therefore lost on process restart. +- Converted artifacts can use the file-backed artifact store and survive restart on the same durable volume. +- Selected audit/snapshot evidence uses append-only file-ledger mechanisms where configured. +- Database/object-store distributed durability remains `PLANNED` or `ACTIVE_PR` depending on the bounded feature. + +A conceptual ERD in `docs/DATA_MODEL.md` does not imply that every entity is persisted on protected main. + +## Change classes + +Every release must identify which classes it changes: + +1. **stateless application change** — code/API behavior without durable-format changes; +2. **configuration/security change** — secrets, permissions, validation, feature flags or policy; +3. **file-format/ledger change** — artifact metadata or append-only ledger encoding/replay semantics; +4. **converter/runtime change** — document renderer, fonts, PDF/runtime dependency or sandbox profile; +5. **persistence-schema change** — future SQL/object-store metadata schema; +6. **public-contract change** — API payload, status, token, integration, failure or compatibility semantics. + +The rollback strategy must match the class instead of assuming that restoring an old application binary restores durable state safely. + +## Pre-deployment gate + +Before a state-affecting release reaches production, capture: + +- exact release/source/artifact digest and dependency/SBOM provenance; +- changed durable formats and compatibility direction; +- migration command or deterministic startup migration, if any; +- backup/snapshot requirements and restore rehearsal evidence; +- downgrade compatibility statement; +- rollback trigger and maximum safe rollback point; +- affected API/client versions; +- tenant/security/data-retention impact; +- operational signals that distinguish bad rollout from external dependency failure. + +A migration without a tested restore/rollback or explicit irreversible-change acceptance is not release-ready. + +## File-ledger compatibility + +Append-only ledger readers must fail closed on malformed or semantically impossible records rather than silently discarding valid history. A format evolution must provide one of: + +- backward-compatible reader support; +- an explicit offline/atomic rewrite with integrity verification and backup; +- a versioned new ledger path with controlled cutover. + +Recovery evidence must cover clean replay, truncated/torn tail handling where the format supports it, invalid UTF-8/record rejection, duplicate/replayed records, and restart after an interrupted write. + +Old binaries must not be reintroduced after they can no longer interpret the current ledger safely. + +## Artifact-store rollback + +Artifact bytes and metadata are security-sensitive durable state. + +- Never roll back to code that interprets a current artifact as belonging to another tenant/generation. +- Preserve checksum/integrity validation across rollback. +- Deletion/retention operations that have crossed an irreversible boundary require recovery/compensation, not resurrection from stale metadata. +- If a release changes on-disk layout, migrate by copy/verify/atomic switch where practical rather than in-place destructive mutation. +- Restore exercises must prove that signed-delivery and tenant-ownership checks still protect recovered artifacts. + +#268 is `SUPERSEDED`. Bounded successors preserve stronger lifecycle/deletion fencing as `PARTIAL` target evidence, not protected-main recovery behavior. + +## Converter/runtime rollback + +A converter or PDF/runtime upgrade can change rendered semantics even if APIs remain compatible. Rollback decisions therefore require the fidelity evidence in `docs/FIDELITY_ACCEPTANCE.md`. + +For each converter/runtime change: + +- preserve the previous pinned version and its evidence until the new release is accepted; +- compare realistic fixtures before rollout; +- record converter/font/configuration provenance; +- treat output-semantic regression as a release rollback trigger; +- do not preserve an insecure old converter solely for byte-for-byte compatibility. + +Security fixes can intentionally break backward compatibility when continuing old behavior is unsafe; the migration note must explain the client/operational impact. + +## Public API and token migration + +Breaking public changes require a versioned migration or an explicit security-tightening exception with client updates. + +Artifact-token interpretation changes must define: + +- version recognition; +- issue/accept overlap period where safe; +- expiration/revocation behavior for old tokens; +- rollback behavior when old code cannot safely validate newly issued tokens. + +Rollback must not re-enable a known authorization bypass simply to accept legacy requests. + +## Future SQL persistence + +When SQL job persistence is implemented, migrations must additionally prove: + +- descriptive two-or-more-word `snake_case` object naming except externally mandated names; +- forward and rollback/recovery paths for every schema change; +- transactional boundaries and lock/runtime bounds; +- concurrent old/new application compatibility for rolling deployment where claimed; +- uniqueness/tenant/generation invariants in the database, not only application memory; +- backup/restore and point-in-time recovery evidence appropriate to the deployment profile; +- migration failure does not produce a partially authorized or partially upgraded serving state. + +The database must remain service-owned; naruon and other hosts integrate through versioned Clearfolio interfaces rather than cross-service table access. + +## Deployment rollback sequence + +A normal rollback sequence is: + +1. stop or drain new traffic where the failure can corrupt/complicate state; +2. capture failure evidence and exact deployed revision; +3. determine whether durable state crossed a non-backward-compatible boundary; +4. if safe, deploy the last proven compatible artifact/configuration; +5. if not safe, execute the tested data/ledger restore or forward-fix path; +6. revalidate liveness/readiness, tenant isolation, signed artifact delivery, conversion lifecycle, and representative buyer flow; +7. preserve incident evidence and update the migration/ADR/release record. + +Rollback completion is an operational proof, not merely “the old pod started.” + +## Recovery acceptance + +A recovery-capable release has deterministic tests or rehearsals for the states it claims to survive, including as applicable: + +- process crash/restart; +- torn or invalid file-ledger tail; +- missing/corrupt artifact metadata; +- cleanup/deletion retry; +- stale in-flight work; +- failed configuration/secret rotation; +- previous/new converter version transition; +- future SQL migration interruption; +- restore to a clean environment followed by buyer-critical smoke tests. + +## Traceability + +- Domain/persistence ownership: `docs/DATA_MODEL.md` +- Operational degraded/recovery behavior: `docs/OPERABILITY.md` +- Fidelity rollback gate: `docs/FIDELITY_ACCEPTANCE.md` +- Release acceptance: `docs/RELEASE_ACCEPTANCE.md` +- Security threats: `docs/THREAT_MODEL.md` +- Requirements/evidence: `docs/TRACEABILITY.md` diff --git a/docs/OPERABILITY.md b/docs/OPERABILITY.md new file mode 100644 index 000000000..3e809cd2a --- /dev/null +++ b/docs/OPERABILITY.md @@ -0,0 +1,225 @@ +# Clearfolio Operability and Recovery Guide + +Status: Canonical operability index +Baseline: protected `main` at `83ec6f7fe2b04bdcd28bf98ec350e41e55730a18` + +This guide states current operational truth and links active target changes without inventing SLO/RPO/RTO values that have not been measured. + +## Deployment modes + +### Standalone + +Clearfolio can run as its own Spring Boot/WebFlux service with configured artifact storage and current default job repository. Local filesystem artifact/ledger settings can provide selected restart continuity, but protected-main conversion jobs remain process-local by default. + +### Composed MSA + +A host such as naruon may call Clearfolio through versioned APIs and provide trusted scoped identity claims. The host does not receive implicit write access to Clearfolio application persistence. Central `.github` workflows are development governance, not runtime infrastructure. + +## Startup acceptance + +Before routing production-like traffic: + +1. application starts with required security configuration; +2. configured cryptographic keys satisfy strength/separation requirements of the integrated revision; +3. artifact store root is app-owned and writable only by intended service identity; +4. local ledgers can be opened/replayed without integrity failure when configured; +5. required converter dependencies for advertised formats are present; +6. health/readiness semantics match the deployed revision; +7. startup does not silently substitute production-required secrets with demo behavior. + +## Liveness and readiness + +Protected main currently has `/healthz`. `ACTIVE_PR` #295 proposes explicit liveness `/healthz` and readiness `/readyz`. Until that PR integrates, deployment documents must not claim separate probes are shipped. + +After integration: + +- liveness answers whether the process is restart-eligible; +- readiness answers whether this instance accepts traffic; +- shared dependency failure does not automatically make the process dead; +- responses remain low-information and `no-store`. + +## Queue and worker operations + +Protected-main worker behavior is bounded and asynchronous. Operators should monitor: + +- executor/queue saturation when exposed; +- processing age and stale lease recovery; +- retry count and dead-letter frequency; +- conversion failure class; +- artifact publication success; +- process restart effects on in-memory job state. + +`PLANNED`: durable queue depth/lag, cancellation, explicit backpressure and distributed worker lease metrics. + +## Artifact operations + +- PDF artifacts may live in memory or `FileSystemArtifactStore` depending on configuration. +- Signed artifact links are short-lived and revocable; canonical artifact reads are audited. +- Local filesystem durability is not equivalent to remote object-store transactional guarantees. +- #268 is `SUPERSEDED`; bounded successors preserve deletion-receipt and restart-safe cleanup requirements as `PARTIAL` evidence, so operators must not rely on those semantics before integration. + +## Failure and recovery matrix + +| Failure | Current/target behavior | Operator action | +| --- | --- | --- | +| invalid/unsupported/oversized upload | controlled fail-closed rejection | correct source/config; do not bypass validation | +| missing tenant permission | 401/403 or concealed 404 according to boundary | fix upstream authorization, not repository data | +| cross-tenant resource attempt | concealed not-found behavior | investigate caller/tenant routing; no data disclosure | +| conversion transient failure | bounded retry when eligible | observe retry/dead-letter state | +| retry exhaustion | `FAILED` + dead-letter evidence | authorized operator retry only after cause review | +| process restart | startup recovery only for state still represented by current repository | recognize process-local job durability limitation | +| artifact missing/unreadable | controlled artifact failure | verify store ownership/integrity; never fabricate success | +| signed token expired/revoked | fail closed | issue a new authorized link if business policy permits | +| local ledger torn tail/corruption | behavior depends on ledger contract; active deletion work validates crash tails | preserve evidence; recover per ledger-specific runbook | +| readiness false (`ACTIVE_PR`) | remove instance from routing | repair local readiness contributor; avoid restart cascade | +| central CI/reviewer outage | blocks affected integration evidence only | continue safe repository work; do not weaken gates | +| scheduled-task execution failure | blocks only that automation execution until classified | distinguish scheduling/activation from execution failure, preserve writer leases, capture the last safe checkpoint when supported, then continue safe repository work | + +## Privacy-safe observability + +Current logs/audit/analytics surfaces are incomplete compared with the target OpenTelemetry model. Future telemetry must avoid uncontrolled high-cardinality or sensitive labels such as raw tenant/subject IDs, filenames, paths, approval tokens, artifact tokens, document bytes/digests, and exception-selected values unless an explicitly reviewed evidence purpose requires them. + +Recommended low-cardinality metrics include: + +- conversion outcome by controlled reason code; +- queue saturation/rejection; +- retry/dead-letter counts; +- readiness state; +- artifact cleanup outcome; +- converter/fidelity profile version where bounded. + +No SLO is considered established until measurements from the actual deployment profile are retained and reviewed. + +## Scheduled automation incident handling + +A **scheduled-task execution failure** is not the same failure boundary as scheduling/activation failure. Operators and autonomous maintainers must classify the boundary before changing code or credentials. + +Use this order: + +1. Read the live automation definition and determine whether it is enabled, has a valid recurrence, and records a recent invocation. Healthy scheduling/activation evidence proves only that the task was accepted and invoked; it does not prove successful repository execution. +2. If activation is healthy but the returned task outcome is a **generic scheduled-task error**, classify it as an **execution failure** until a more precise failing boundary is observed. +3. Refetch repository state before assuming the failed run was side-effect free. **Before every write, refetch the exact source head and exact target head, the independently resolved live base, the target blob/ref, and the relevant review state: formal reviews, unresolved threads, exact-head checks, and security gates. If the target identity changes or another writer moves the target after that snapshot, reject the stale write, freeze that branch/path, and rotate to another safe item.** Apply the same fail-closed rule before a retry or other recovery mutation. +4. Build separate hypotheses for connector/tool failure, authentication or permission failure, provider/rate/runtime state, repository policy, and prompt-size/duplicated-state amplification. Do not label any hypothesis as root cause without evidence. +5. Treat reviewed **repository authority** as the detailed product/security/architecture source of truth. The recurring scheduler should carry execution control semantics rather than a second copy of the full product specification. Simplifying duplicate scheduler detail must not delete requirements from GitHub or weaken review, test, security, fidelity, writer-lease, or release gates. +6. Apply the smallest reversible feasible control change and verify the automation remains enabled with the intended cadence when the control-plane tool exposes that state. +7. Continue the safe Clearfolio executable queue in the same invocation. Scheduler debt is local and must not become a global excuse to stop product, security, documentation, or PR work. +8. On later runs, re-evaluate whether the failure recurs. Success after a simplification supports the remediation but does not retroactively prove the original internal failure mechanism when detailed telemetry was unavailable. + +ADR-0011 owns the thin-scheduler decision. ADR-0008 continues to own central-vs-local automation credentials/authority, while ADR-0009 owns work-conserving RCA and the double-exit termination semantics. + +## Scheduler execution receipts and budget continuation (`PLANNED`; issue #331) + +Issue #331 and ADR-0012 define the missing diagnosability contract. This is planned architecture, not proof that the external scheduler currently persists structured receipts. + +### Required evidence boundaries + +A diagnosable run distinguishes: + +1. `schedule`: the recurrence became due; +2. `admission`: the scheduler accepted the invocation; +3. `fresh queue`: the run independently resolved current GitHub identities; +4. `atomic action`: one bounded mutation or acceptance operation began; +5. `action receipt`: the operation reached a stable exact evidence boundary; +6. `budget continuation`: the run intentionally stopped before a hard boundary after preserving a verified safe checkpoint; +7. `controlled failure`: an observed failing phase/category was recorded without inventing hidden platform detail; +8. `continuation handoff`: the next run was told what historical evidence exists and that all live state must be refetched. + +Activation without execution is not success. A run-start signal without `fresh queue` construction is not repository execution. A generic scheduled-task error is a symptom, not a source defect and not a root-cause verdict. + +### Clean budget continuation + +Use `budget_continuation` only when actual available tool/runtime signals show that a new RED test, broad refactor, new branch, or multi-step mutation cannot reasonably reach a verified or safely test-only boundary in the current invocation. + +Before continuing later: + +- finish or safely abandon the current atomic mutation; +- record the **last safe checkpoint** and exact observable action result; +- refetch the affected head/base/ref/blob after any mutation; +- leave no temporary, one-shot, self-modifying, encoded-patch, or competing writer workflow; +- leave no knowingly broken non-test-only source head; +- treat every checkpoint SHA as historical; +- start the next invocation from **fresh GitHub state**. + +Budget continuation is not repository completion and must not be used merely because one branch is waiting or because a bounded safe action still fits in the observable budget. + +### Controlled failure envelope + +Where the platform permits, a privacy-safe failure envelope records: + +- opaque run and action identity; +- control/prompt version or digest; +- observed failing phase; +- low-cardinality category such as admission, queue construction, connector/tool, authentication/permission, provider/rate/runtime, repository policy, CAS/publication, verification, or telemetry unavailable; +- last completed action receipt; +- last safe checkpoint; +- branch-local deferred lanes and recovery eligibility. + +It excludes raw credentials, authorization values, cookies, approval/artifact tokens, document bytes/content, filenames/paths, tenant/subject/approver identifiers, uncontrolled exception messages, and user-visible stack traces. If detailed telemetry is unavailable, retain `telemetry_unavailable`; **do not invent** a more specific cause. + +### Recovery procedure + +1. Read the receipt only as historical evidence. +2. Revalidate the external scheduler/control identity where observable. +3. Rebuild the entire executable queue from fresh GitHub state. +4. Recheck writer ownership immediately before each write. +5. Resume at the highest-value safe lane; do not blindly replay the previous action. +6. If a source/ref moved, reject stale publication and freeze only that lane. +7. If Draft PR #271 later implements repository-local receipts, keep its evidence separate from the external scheduler and reacquire all exact-head/base-sensitive gates after its parent dependencies stabilize. + +## Incident handling + +1. bind incident evidence to exact deployed source/build identity; +2. preserve privacy-safe logs and relevant append-only evidence; +3. identify the first failing component boundary; +4. distinguish product, infrastructure, central CI/governance and dependency-provider ownership; +5. select the smallest reversible repair after feasibility proof; +6. validate in a non-destructive environment where possible; +7. roll forward or back according to data/artifact compatibility; +8. require protected-main/runtime evidence before closing an automation or deployment incident. + +## Backup and restore + +`PARTIAL`: filesystem artifacts/ledgers can be backed up as files when configured, but this is not yet a complete business-continuity design because default job state remains process-local. + +`PLANNED` durable state requires: + +- consistent job/state/outbox backup semantics; +- artifact/object-store version alignment; +- tenant-scoped restore; +- generation/idempotency preservation; +- tested restore into an isolated environment; +- measured recovery evidence before publishing RPO/RTO claims. + +External scheduler execution receipts remain owned by the executing control plane under ADR-0012. They are not included in Clearfolio application backup scope unless a later accepted ADR explicitly changes ownership. + +## Rollback + +Code rollback is permitted only when persistence/artifact format and security semantics remain compatible. Never roll back a security boundary merely because a prior release is operationally convenient. Database or durable-ledger migrations require explicit rollback/forward-recovery instructions before release. + +Scheduler-receipt rollback must return to fail-closed generic classification and fresh-state reconstruction. It must not restore stale-checkpoint writes, guessed root causes, weaker writer leases, or self-modifying workflows. + +## Release operational acceptance + +Before release: + +- exact integrated protected source is known; +- all required CI/security/coverage/fidelity/accessibility/provenance checks pass; +- deployment starts with release-like security configuration; +- supported-format smoke/fidelity tests run using release artifacts; +- health/readiness behavior matches documentation; +- retry/recovery and artifact access smoke tests pass; +- no production claim depends on demo fixtures or placeholder conversion; +- rollback/recovery constraints are documented; +- published artifact and SBOM/provenance are verified after publication. + +## Related runbooks and evidence + +- `docs/deployment/2026-07-02-buyer-deployment-integration-playbook.md` +- `docs/operations/artifact-deletion-cleanup.md` when the corresponding active PR integrates +- `docs/persistence/2026-07-02-durable-conversion-job-repository-plan.md` +- `docs/analytics/2026-07-02-durable-metrics-event-model.md` +- `docs/engineering/acceptance-criteria.md` +- `docs/qa/evidence/` dated snapshots +- issue #331 and ADR-0012 for scheduler receipt/continuation acceptance + +Dated evidence is provenance, not timeless operational truth; this document and protected source define the current contract. diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 000000000..afd80b0ed --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,156 @@ +# Clearfolio Product Requirements Document + +Status: Canonical product requirements spine +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +## Product purpose + +Clearfolio is an independently deployable secure document-conversion and viewing service that can also compose as a bounded service inside naruon and other ContextualWisdomLab systems. It accepts untrusted document uploads, performs bounded asynchronous processing, exposes state and controlled artifacts, and preserves tenant, evidence, and release boundaries. Browsers, LLMs, and downstream hosts do not become authority for conversion correctness, authorization, or release acceptance. + +## Maturity vocabulary + +Requirements use explicit implementation labels: + +- `IMPLEMENTED_ON_MAIN`: evidenced on protected default branch. +- `ACTIVE_PR`: implemented on an open PR only; not shipped. +- `PARTIAL`: useful subset exists but the requirement is incomplete. +- `ACCEPTED_ARCHITECTURE`: accepted target boundary without protected-main implementation evidence. +- `PLANNED`: backlog work. +- `OUT_OF_SCOPE`: intentionally excluded. + +## Primary users and buyer outcomes + +### Document user + +A document user needs to submit a document without blocking on conversion, understand progress and failures, open accessible converted output, and retrieve only artifacts they are authorized to read. + +### Tenant operator + +A tenant operator needs least-privilege administrative listing/retry/delete workflows, concealed cross-tenant resources, restart-safe recovery evidence, privacy-safe audit trails, and accurate liveness/readiness signals. + +### Platform integrator + +A platform integrator needs stable versioned APIs and explicit service-ownership boundaries so Clearfolio works alone or behind naruon/gateway/orchestration infrastructure without hidden database coupling. + +### Security, procurement, and acquisition reviewer + +A reviewer needs exact evidence for tenant isolation, active-content and external-resource boundaries, dependency and supply-chain integrity, document fidelity, accessibility, recovery, observability, and release provenance. + +## Core product journeys + +### Submit and process + +`IMPLEMENTED_ON_MAIN`: `POST /api/v1/convert/jobs` performs bounded upload intake and delegates conversion work away from the WebFlux request path. Jobs expose status, retries, dead-letter state, content identity, and current process-local recovery behavior. + +`PLANNED`: durable distributed job intake, cancellation, backpressure, persistent queue semantics, and database-backed recovery. + +### View and retrieve output + +`IMPLEMENTED_ON_MAIN`: PDF passthrough, PDF.js viewer bootstrap, signed viewer artifact links, token ledger/revocation/read audit for the canonical artifact route, and tenant-scoped JSON APIs exist. + +`PARTIAL`: direct conversion-job download is under security remediation to align with the signed artifact-delivery contract. + +`PARTIAL`: protected main does not provide production-fidelity Office conversion. `PdfBoxArtifactGenerator` generates a placeholder one-page PDF for non-PDF sources; that behavior is not a supported-format fidelity claim. + +### Recover from failure + +`IMPLEMENTED_ON_MAIN`: bounded worker retry/dead-letter behavior, operator retry, process-local startup recovery, and file-backed artifact options exist. + +`PARTIAL`: #268 is `SUPERSEDED`; bounded successor lanes preserve individual lifecycle primitives, but durable deletion receipts, generation fencing, retry fairness, crash-tail validation, and restart recovery are not protected-main behavior. + +`ACTIVE_PR`: #295 separates process liveness from traffic readiness. + +### Enforce tenant and privacy boundaries + +`IMPLEMENTED_ON_MAIN`: tenant contexts, permissions, same-tenant checks, cross-tenant concealment, signed viewer artifact links, revocation and audit evidence exist in the current scaffold. + +`IMPLEMENTED_ON_MAIN`: protected #270 strengthened signed tenant claims, dedicated artifact/admin least privilege, purpose-separated cryptographic keys, and domain-separated HMAC audit pseudonyms. #268 is `SUPERSEDED`; remaining immutable lifecycle and durable mutation work is `PARTIAL` in unintegrated bounded successors. + +`PLANNED`: production IdP/OIDC issuer/audience/expiry/revocation/role mapping, enterprise tenant lifecycle, data-retention policy, and distributed policy enforcement. + +### Operate an accessible viewer + +`IMPLEMENTED_ON_MAIN`: a responsive viewer shell and PDF.js rendering path exist. + +`ACTIVE_PR`: #264 adds contextual accessible names and nested/reentrant busy-state handling with exact DOM/ARIA/disabled restoration. + +`PLANNED`: broader keyboard, screen-reader, print/export and assistive-technology acceptance evidence. + +## Functional requirements + +| ID | Requirement | Maturity | +| --- | --- | --- | +| FR-01 | Accept bounded document uploads and return an asynchronous job contract without inline conversion | `IMPLEMENTED_ON_MAIN` | +| FR-02 | Expose tenant-scoped job status and stable lifecycle metadata | `IMPLEMENTED_ON_MAIN` | +| FR-03 | Expose state-gated viewer bootstrap and controlled artifact delivery | `IMPLEMENTED_ON_MAIN` through protected #270 | +| FR-04 | Preserve content identity and idempotent dedupe behavior at the current repository boundary | `IMPLEMENTED_ON_MAIN` | +| FR-05 | Fail closed for blocked formats, malformed input and bounded upload limits | `IMPLEMENTED_ON_MAIN` | +| FR-06 | Require least-privilege authorization before tenant repository/artifact/admin access | `IMPLEMENTED_ON_MAIN`, strengthened in `ACTIVE_PR` | +| FR-07 | Preserve immutable lifecycle identity and restart-safe artifact deletion evidence | `PARTIAL`; #268 `SUPERSEDED`; bounded successors are not integrated | +| FR-08 | Separate liveness from traffic readiness | `ACTIVE_PR` #295 | +| FR-09 | Make repeated asynchronous viewer actions contextually named and nested-safe | `ACTIVE_PR` #264 | +| FR-10 | Convert each claimed Office format using deterministic real-fixture fidelity acceptance instead of placeholder output | `PLANNED`, issue #5 | +| FR-11 | Provide durable async jobs with cancellation, backpressure, restart recovery and distributed idempotency | `PLANNED` | +| FR-12 | Publish OpenTelemetry traces and low-cardinality privacy-safe metrics | `PLANNED` | +| FR-13 | Maintain versioned naruon/MSA contracts without cross-service application-database access | `ACCEPTED_ARCHITECTURE` | +| FR-14 | Produce SBOM, attribution, reproducibility and release-provenance evidence tied to exact source | `PARTIAL`, strengthened in #270 | +| FR-15 | Support truthful tenant-safe deletion/download UX on the durable lifecycle substrate without duplicate cleanup or signed-link bypass | `PLANNED` after protected #270 and bounded lifecycle/accessibility successor integration, issue #263 | + +## Non-functional requirements + +- **Fail closed:** malformed, unauthorized, cross-tenant, stale-generation, unsupported, oversized and unverifiable requests fail with controlled semantics. +- **Exact evidence:** predecessor-head, stale-base, queued, pending, skipped-required, cancelled, absent, synthetic-only, model-only and status-only evidence is not exact-head release evidence. +- **Coverage:** owned production statement and branch coverage is exactly 100%; public production APIs have beginner-readable Javadocs/docstrings. +- **Naming:** owned database/domain persistence objects use at least two descriptive `snake_case` words unless an external protocol requires another style. +- **Privacy:** pseudonymized identifiers remain personal data; use purpose-bound access, minimization, encryption, retention controls and auditable privileged access rather than blanket masking. +- **Cryptography:** cryptographic purposes use separated keys and versioned/domain-separated constructions where applicable. +- **Deterministic authority:** conversion, authorization, security validation, fidelity measurement and release decisions remain deterministic; LLM output cannot override them. +- **Workflow security:** use least privilege and immutable workflow/action/source pinning where practical. +- **Compatibility:** preserve standalone deployment and modular MSA integration through explicit APIs and replaceable adapters. + +## Document fidelity contract + +A format may be advertised as supported only when the current implementation has all of the following: + +1. authorized or redistributable realistic source fixtures; +2. expected structural, visual, extraction, accessibility, security and failure outcomes; +3. deterministic source/artifact identity where meaningful; +4. documented tolerated differences and unsupported constructs; +5. active-content, macro, external-resource, malformed-container and resource-boundary tests; +6. exact-head regression evidence and release acceptance. + +Until those conditions are met, non-PDF placeholder generation is development/demo behavior and must not be described as Office conversion fidelity. `docs/FIDELITY_ACCEPTANCE.md` is the detailed qualification authority and issue #5 is the current executable product gap. + +## Automation and development contract + +`ACTIVE_PR`: #271 defines a repository-local OpenCode product-development loop. It uses `NVIDIA_NIM_API_KEY` only through GitHub Secrets, never `COPILOT_GITHUB_TOKEN` as a development-model credential, and separates credential-bearing model proposal from credential-free verification/publication. Its decision loop is RCA → materially distinct remedies → real-world feasibility → same-run action → proof. Review/check latency blocks only the affected action; path-disjoint safe work continues. + +`ACCEPTED_ARCHITECTURE`: pull-request maintenance and merge governance belong to the central ContextualWisdomLab `.github` control plane. Clearfolio must not duplicate that privileged control-plane implementation. + +Autonomous development may propose bounded Draft PRs but may not self-approve, weaken tests/protection, merge, release or deploy its own work. + +## Product gaps prioritized by acquisition impact + +1. Real deterministic Office conversion plus realistic fidelity benchmark suite (issue #5). +2. Durable async job state, idempotency, backpressure, cancellation and distributed recovery. +3. Production identity/tenant lifecycle and least-privilege enterprise authorization. +4. Privacy-safe OpenTelemetry traces/metrics and operator SLO evidence. +5. Object-store/database durability with rollback/recovery and tenant isolation. +6. Accessible, truthful tenant-safe download/deletion UX after the active lifecycle/security substrate integrates (issue #263). +7. Accessibility/print/export acceptance across other supported viewer workflows. +8. Versioned naruon integration contracts and compatibility tests. +9. Reproducible packaging, SBOM/provenance, signed release and operational acceptance. + +## Release acceptance + +A release may be published only from an exact integrated protected head that passes CI, security, exact coverage/docstrings, realistic document-fidelity acceptance, accessibility, packaging, SBOM/provenance, reproducibility, API/schema compatibility, migration/rollback/recovery where applicable, qualifying independent review, and protected-main operational acceptance. Version bumps and release notes are consequences of this evidence, not substitutes for it. + +Detailed release authorities are `docs/FIDELITY_ACCEPTANCE.md`, `docs/MIGRATION_ROLLBACK.md`, and `docs/RELEASE_ACCEPTANCE.md`. These documents define evidence/compatibility/recovery rules; they do not turn an `ACTIVE_PR` implementation into `IMPLEMENTED_ON_MAIN` behavior. + +## Non-goals + +- LLM-based decisions for deterministic conversion or access control. +- Hidden cross-service database access to naruon or other CWL services. +- Claiming Office fidelity from placeholder output. +- Self-approval or protection bypass by autonomous development. +- Claiming certifications, SLOs, durability, distributed transactions, or enterprise IdP behavior without measured or integrated evidence. diff --git a/docs/RELEASE_ACCEPTANCE.md b/docs/RELEASE_ACCEPTANCE.md new file mode 100644 index 000000000..db1a589b6 --- /dev/null +++ b/docs/RELEASE_ACCEPTANCE.md @@ -0,0 +1,150 @@ +# Clearfolio Integrated Release Acceptance + +Status: Canonical release gate +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +Clearfolio is releasable only from one exact integrated protected head. A green feature PR, documentation PR, local build, predecessor SHA, synthetic merge, or model/reviewer status is not a release candidate by itself. + +## Release identity + +Every candidate must bind evidence to: + +- exact protected source commit SHA; +- exact version/tag to be created; +- immutable dependency lock/resolution state; +- build artifact digests; +- SBOM/provenance material generated from that source; +- converter/PDF/runtime versions and configuration affecting document output; +- supported deployment profile and configuration assumptions. + +If the source head changes, source-sensitive acceptance evidence is stale and must be reacquired. + +## Mandatory release gates + +### Build and tests + +- `mvn -B --no-transfer-progress verify` succeeds from the exact release head; +- Surefire evidence contains positive executed tests and zero skipped, failed, or errored tests; +- Failsafe evidence, when present, satisfies the same contract; +- owned production line and branch coverage gates remain at zero missed statements/branches as configured; +- public Javadocs/docstrings are generated without accepted warning debt; +- changed documentation contract tests pass. + +### Product behavior + +The release proves representative buyer workflows rather than only primitive units: + +- submit → asynchronous status → terminal state; +- succeeded document → viewer bootstrap → signed artifact read; +- rejected/unsupported/failed document → controlled user/operator state; +- retry/dead-letter behavior where applicable; +- tenant-isolation and permission boundaries; +- viewer accessibility/keyboard behavior where changed; +- restart/recovery behavior for every durability claim in the release. + +A product route that returns success with development/demo placeholder output cannot satisfy a production Office-fidelity release claim. + +### Document fidelity + +Every format advertised as transformed/supported must pass `docs/FIDELITY_ACCEPTANCE.md` on the exact release converter/runtime profile. PDF passthrough and development placeholder behavior must remain separately labelled. + +If no Office adapter has qualified, the release notes and product documentation must say so plainly. + +### Security and privacy + +The exact release head must pass applicable repository security gates and preserve: + +- signed tenant/claim validation where required; +- least-privilege permissions and cross-tenant concealment; +- signed artifact token integrity, expiry, scope, document/tenant/checksum binding, issued-ledger membership and revocation; +- canonical zero/single-range artifact delivery and controlled read audit; +- privacy-safe audit pseudonymization/key separation where present; +- blocked/unsupported active-content behavior and bounded conversion resources; +- no secrets or document contents in committed evidence/logging paths; +- immutable/pinned supply-chain automation according to repository policy. + +A cancelled, skipped-required, absent-required, queued, stale-head, or predecessor security result is not passing evidence. + +### Operability and recovery + +- health/readiness behavior matches the implemented release rather than an active-PR target; +- configuration/secret requirements are documented and fail closed where required; +- rollback/recovery follows `docs/MIGRATION_ROLLBACK.md`; +- any durable-format or storage change has tested compatibility/restore evidence; +- representative startup, shutdown, failure, and recovery smoke tests pass for the declared deployment profile; +- monitoring/diagnostic behavior avoids secret/tenant/document leakage. + +### Packaging and provenance + +The release evidence must include or verify as applicable: + +- reproducible or otherwise provenance-bound build inputs; +- dependency inventory/SBOM; +- third-party attribution/license policy evidence; +- artifact checksums; +- source/tag relationship; +- published package/container/artifact verification after publication. + +Generated evidence is not automatically trustworthy because it exists; generation and verification must be tied to the exact release source and reviewed policy. + +### Review and governance + +- zero valid unresolved current-head human, CodeRabbit, GHAS, Dependabot, OpenCode, Noema, Strix, or equivalent findings that repository policy treats as blockers; +- all branch/ruleset required checks pass on the exact protected candidate; +- qualifying independent non-author formal approval is present when required by live repository/CWL governance; +- advisory comments, statuses, reactions, author reviews, dismissed reviews, predecessor reviews, and model-only verdicts are not counted as independent approval. + +## Evidence authorities + +Release evidence intentionally distinguishes: + +- `source_head_sha` — exact protected source being released; +- `live_base_tip_sha` — protected branch tip used for integration decisions before release; +- workflow/check identity — execution and outcome on the source head; +- review identity — reviewer, review state and reviewed commit; +- security evidence — scanner/gate outcome on the exact source; +- product/fidelity evidence — realistic behavior on the release runtime profile; +- artifact evidence — hashes/SBOM/provenance for the built/published output. + +No single green status collapses these authorities. + +## Release sequence + +1. drain/merge the intended PR stack through normal protection; +2. refetch the exact protected head and freeze the candidate identity; +3. run integrated CI, security, coverage/docstring, product, fidelity, packaging and recovery gates; +4. obtain required current-head independent review/governance evidence; +5. update version and `CHANGELOG.md` according to repository policy without changing accepted behavior unexpectedly; +6. rebuild/reverify from the final exact tagged head if versioning changed the source; +7. publish/create the release and its SBOM/provenance/checksums; +8. verify the published artifact against the expected source/digest; +9. run protected-main/released-artifact operational smoke/acceptance where the deployment profile supports it; +10. preserve dated release evidence without turning volatile SHA/run IDs into timeless architecture claims. + +If any step changes source, dependencies, converter runtime, packaging or security-sensitive configuration, reacquire the affected evidence rather than reusing an older pass. + +## Rollback triggers + +Examples of release rollback/forward-fix triggers include: + +- cross-tenant or artifact authorization regression; +- document corruption/data loss; +- deterministic conversion/fidelity regression beyond accepted tolerances; +- inability to replay required durable state; +- readiness/traffic-routing failure that makes safe serving impossible; +- package/provenance mismatch; +- newly confirmed high-impact vulnerability not safely mitigated by configuration. + +Rollback must not reintroduce a known security bypass merely to restore compatibility. + +## Traceability + +- Product acceptance: `docs/PRD.md` +- Technical gates: `docs/TRD.md` +- Fidelity: `docs/FIDELITY_ACCEPTANCE.md` +- Migration/rollback/recovery: `docs/MIGRATION_ROLLBACK.md` +- Test strategy: `docs/TEST_STRATEGY.md` +- Operability: `docs/OPERABILITY.md` +- Threats: `docs/THREAT_MODEL.md` +- Release decision: `docs/adr/0010-release-provenance-fidelity-gate.md` +- Requirement/evidence mapping: `docs/TRACEABILITY.md` diff --git a/docs/RESEARCH_TRACEABILITY.md b/docs/RESEARCH_TRACEABILITY.md new file mode 100644 index 000000000..e2b7e1a0d --- /dev/null +++ b/docs/RESEARCH_TRACEABILITY.md @@ -0,0 +1,100 @@ +# Clearfolio Standards and Research Traceability + +Status: Canonical standards/research index +Reviewed: 2026-08-10 + +This document records the primary standards, official technical documentation, and peer-reviewed research that materially constrain Clearfolio architecture and acceptance. It does not convert a standard into an implementation claim; `docs/TRACEABILITY.md` remains the requirement→source/test/PR evidence map. + +## Standards and technical-source mapping + +| Area | Primary source | Clearfolio decision / acceptance consequence | +| --- | --- | --- | +| HTTP byte ranges | Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110, STD 97). RFC Editor. https://www.rfc-editor.org/rfc/rfc9110 | Signed artifact delivery implements a deliberately narrower zero-or-one `bytes` Range profile; valid partial responses use correct `Content-Range`/`Accept-Ranges`, and malformed/unsatisfiable/multiple ranges fail under the reviewed endpoint contract instead of silently broadening behavior. | +| Web accessibility | World Wide Web Consortium. (2024, December 12). *Web Content Accessibility Guidelines (WCAG) 2.2* (W3C Recommendation). https://www.w3.org/TR/2024/REC-WCAG22-20241212/ | Repeated viewer controls require distinguishable accessible names; async/busy/focus/keyboard states need explicit tests. Viewer accessibility claims require measured acceptance rather than visual inspection alone. Clearfolio pins the December 2024 Recommendation publication for reproducible implementation evidence rather than a moving latest-version URI. | +| Kubernetes probes | Kubernetes Authors. (2026, April 17). *Liveness, readiness, and startup probes*. Kubernetes Documentation. https://kubernetes.io/docs/concepts/workloads/pods/probes/ | Liveness answers restart eligibility; readiness answers traffic eligibility. Shared/transient dependency failures must not casually turn liveness into a restart cascade. This grounds ADR-0006 and active PR #295. | +| Spring availability model | Broadcom. (2026). *Spring Boot 3.5 reference: Actuator endpoints—Kubernetes probes*. https://docs.spring.io/spring-boot/3.5/reference/actuator/endpoints.html#actuator.endpoints.kubernetes-probes | Spring `ApplicationAvailability` and the dedicated liveness/readiness model support Clearfolio's explicit availability semantics. Application construction should not invent availability state when the required Spring authority is absent. The 3.5 documentation URI is version-pinned; adopting another Spring line requires a fresh compatibility check. | +| Telemetry architecture | OpenTelemetry Authors. (2025). *OpenTelemetry Specification 1.59.0*. https://opentelemetry.io/docs/specs/otel/ | Planned traces/metrics use OTel-compatible concepts while preserving privacy-safe low-cardinality attributes. No SLO is claimed merely because telemetry instrumentation exists. The official page identifies the current specification version; implementation must pin the dependency/specification combination it actually qualifies. | +| CycloneDX SBOM | Ecma International. (2025). *CycloneDX Bill of materials specification* (ECMA-424, 2nd ed.; CycloneDX v1.7). https://ecma-international.org/publications-and-standards/standards/ecma-424/ | The authoritative current ECMA-424 edition defines CycloneDX v1.7. Clearfolio's committed/active supply-chain evidence remains a historical CycloneDX 1.6 baseline until a separately reviewed implementation migration changes it. Documentation of the current standard does not silently upgrade generated SBOM schema versions. | +| SPDX | SPDX Workgroup. (2024). *SPDX Specification 3.0.1*. Linux Foundation. https://spdx.github.io/spdx-spec/v3.0.1/ | SPDX is an accepted interoperable software/supply-chain metadata model where required by release tooling. Adoption of a specific serialization/version is an implementation/release decision, not a generic documentation claim. | +| Software provenance | SLSA Community. (2025). *SLSA specification, version 1.2*. https://slsa.dev/spec/v1.2/ | Release provenance should make source/build/material identity verifiable. Clearfolio uses SLSA concepts as a target quality model; do not claim a SLSA level without conformance evidence. | +| Secure development | Souppaya, M., Scarfone, K., & Dodson, D. (2022). *Secure Software Development Framework (SSDF) Version 1.1: Recommendations for mitigating the risk of software vulnerabilities* (NIST SP 800-218). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218 | Root-cause remediation, dependency/supply-chain controls, secure release practices, and documented security gates align with SSDF principles. The repository does not claim NIST certification. | +| Office conversion process boundary | JODConverter. (2025). *Office Managers*; JODConverter. (2025). *Configuration overview*; JODConverter. (2025). *Release 4.4.11*; JODConverter. (2025). *Migration 4.4.11*. | Current JODConverter documentation distinguishes local, externally managed local, and remote office managers and documents task/process lifecycle controls. Clearfolio deliberately rejects starting the office runtime inside the API-container trust boundary: issue #5 requires a sandboxed sidecar/external process or separately operated remote service behind a provider-neutral adapter. The existence of `LocalOfficeManager` support is not a Clearfolio architecture approval. | +| Office runtime licensing | The Document Foundation. (n.d.). *Licenses*. https://www.libreoffice.org/licenses/ | LibreOffice is made available under MPL-2.0 and includes components under other licenses that can vary by distribution/version. Qualification therefore reviews the exact image/runtime/fonts/codecs/dictionaries and generated SBOM/attribution rather than inferring redistribution approval from the Java adapter license. | + +## Current Office-adapter source refresh + +The product-gap issue originally cited JODConverter 4.4.10 configuration as if that represented the current documented line. Official project documentation now identifies release 4.4.11 and its migration guide, while the current `Office Managers` and `Configuration overview` pages are published under the project's rolling `/latest/` documentation tree. A search of the official published site on 2026-08-09 found a fixed 4.4.10 documentation tree but did **not** establish equivalent fixed `4.4.11/getting-started/office-managers/` or `4.4.11/configuration/` pages. Those two rolling pages are therefore cited with a retrieval date instead of inventing a versioned URL that was not verified. + +The version qualification anchor is the explicit 4.4.11 release/migration content, not the word `latest`. Clearfolio must pin the exact reviewed adapter and office-runtime versions in dependency/image configuration, exercise the realistic fidelity/security corpus, and record them in SBOM/provenance evidence. A future documentation refresh should replace a rolling source with a fixed project URL if the project publishes and verifies one. + +APA 7 technical-source entries used for the Office-adapter decision: + +- JODConverter. (2025). *Office Managers*. Retrieved August 9, 2026, from https://jodconverter.github.io/jodconverter/latest/getting-started/office-managers/ +- JODConverter. (2025). *Configuration overview*. Retrieved August 9, 2026, from https://jodconverter.github.io/jodconverter/latest/configuration/ +- JODConverter. (2025, August 21). *Release 4.4.11*. https://jodconverter.github.io/jodconverter/latest/release-notes/release-notes-4.4.11/ +- JODConverter. (2025). *Migration 4.4.11*. https://jodconverter.github.io/jodconverter/latest/migration-guides/migration-guide-4.4.11/ +- JODConverter. (2025, August 21). *Apache License, Version 2.0* [License file, v4.4.11]. GitHub. https://raw.githubusercontent.com/jodconverter/jodconverter/v4.4.11/LICENSE +- The Document Foundation. (n.d.). *Licenses*. https://www.libreoffice.org/licenses/ + +The fixed JODConverter 4.4.10 documentation root remains useful provenance for the previously reviewed line but is not substituted for 4.4.11 behavior: https://jodconverter.github.io/jodconverter/4.4.10/ + +## Stable APA 7 reference entries + +- Broadcom. (2026). *Spring Boot 3.5 reference: Actuator endpoints*. https://docs.spring.io/spring-boot/3.5/reference/actuator/endpoints.html +- Ecma International. (2025). *CycloneDX Bill of materials specification* (ECMA-424, 2nd ed.; CycloneDX v1.7). https://ecma-international.org/publications-and-standards/standards/ecma-424/ +- Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110, STD 97). RFC Editor. https://www.rfc-editor.org/rfc/rfc9110 +- Kubernetes Authors. (2026, April 17). *Liveness, readiness, and startup probes*. https://kubernetes.io/docs/concepts/workloads/pods/probes/ +- OpenTelemetry Authors. (2025). *OpenTelemetry Specification 1.59.0*. https://opentelemetry.io/docs/specs/otel/ +- OWASP Foundation & Ecma International. (2024). *CycloneDX Bill of Materials Standard, version 1.6 (ECMA-424, 1st ed.)* [Historical Clearfolio SBOM baseline]. https://cyclonedx.org/news/cyclonedx-v1.6-now-an-ecma-international-standard/ +- SLSA Community. (2025). *SLSA specification, version 1.2*. https://slsa.dev/spec/v1.2/ +- Souppaya, M., Scarfone, K., & Dodson, D. (2022). *Secure Software Development Framework (SSDF) Version 1.1: Recommendations for mitigating the risk of software vulnerabilities* (NIST SP 800-218). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218 +- SPDX Workgroup. (2024). *SPDX Specification 3.0.1*. Linux Foundation. https://spdx.github.io/spdx-spec/v3.0.1/ +- World Wide Web Consortium. (2024, December 12). *Web Content Accessibility Guidelines (WCAG) 2.2*. https://www.w3.org/TR/2024/REC-WCAG22-20241212/ + +## Peer-reviewed testing evidence + +### Coverage is a structural gate, not a product oracle + +Inozemtseva and Holmes empirically found that coverage is not strongly correlated with test-suite effectiveness once suite size is controlled. Clearfolio therefore retains exact 100% owned production statement/branch coverage as a structural contract **and separately requires realistic security, lifecycle, fidelity, accessibility, concurrency, crash/restart, migration/rollback, and release assertions**. + +APA 7: + +Inozemtseva, L., & Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. In *Proceedings of the 36th International Conference on Software Engineering* (pp. 435–445). Association for Computing Machinery. https://doi.org/10.1145/2568225.2568271 + +### Test results require trustworthy oracles/evidence + +The software-testing oracle problem explains why “the command exited zero” is not a complete correctness claim. Clearfolio's Maven evidence verifier therefore checks that tests actually executed, were not skipped, and report zero failures/errors, while domain-specific tests assert security/fidelity/recovery outcomes. + +APA 7: + +Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The oracle problem in software testing: A survey. *IEEE Transactions on Software Engineering, 41*(5), 507–525. https://doi.org/10.1109/TSE.2014.2372785 + +## Decision notes + +### RFC 9110 does not require Clearfolio to implement every multi-range feature + +The standard permits one or more ranges; Clearfolio intentionally supports a smaller single-range product contract. That restriction must be explicit and consistently rejected/tested, not accidentally implemented differently across two byte-delivery endpoints. + +### Accessibility conformance is more than ARIA strings + +Contextual accessible names in active PR #264 are necessary for repeated actions, but full viewer accessibility also depends on keyboard/focus/state/error/print behavior and must remain `PARTIAL` until those paths have acceptance evidence. + +### Liveness is intentionally narrower than readiness + +Kubernetes documentation warns that poorly designed liveness probes can amplify failures. Clearfolio therefore treats restart-worthiness as a process property while routing readiness may include temporary instance conditions. External shared-service health is not automatically a reason to restart the process. + +### SBOM and provenance formats are versioned products + +The current standard line is CycloneDX v1.7 / ECMA-424 2nd edition, while Clearfolio's repository evidence is still a CycloneDX 1.6 historical baseline. The existence of either a CycloneDX/SPDX document is not supply-chain assurance by itself. Release acceptance binds generated metadata to exact source/dependency/build identity and verifies deterministic regeneration/provenance as applicable; a schema upgrade requires its own implementation and compatibility evidence. + +### Office conversion support requires two independent evidence classes + +JODConverter's API/process-management features can support a conversion adapter, but they do not prove document fidelity or sandbox safety. Clearfolio therefore requires both (1) a provider-neutral isolated execution boundary with bounded resources/network/secrets and (2) realistic authorized fixture evidence under `docs/FIDELITY_ACCEPTANCE.md`. A successful LibreOffice/JODConverter process exit is not a product-support oracle. + +## Source lifecycle rule + +For stable standards, cite the normative/stable publication. For fast-moving project specifications such as OpenTelemetry, CycloneDX, SLSA, Spring Boot, Kubernetes, JODConverter, and LibreOffice packaging, re-check the official current source before making a new version-specific claim. Do not encode a transient “latest version” statement in timeless architecture unless the project deliberately pins and tests that version. + +## PDF / copyright policy + +A paper PDF is committed only when redistribution is explicitly permitted. Otherwise retain the full citation, DOI/stable source link, and a concise relevance summary. Documentation completeness does not justify copying copyrighted papers into the repository without redistribution rights. diff --git a/docs/TEST_STRATEGY.md b/docs/TEST_STRATEGY.md new file mode 100644 index 000000000..838323090 --- /dev/null +++ b/docs/TEST_STRATEGY.md @@ -0,0 +1,122 @@ +# Clearfolio Test Strategy + +Status: Canonical quality strategy +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +## Goal + +Tests must prove product behavior and release claims, not merely inflate coverage. Exact 100% owned production statement/branch coverage remains a floor; realistic security, lifecycle, fidelity, accessibility and recovery evidence supplies the behavioral meaning that coverage alone cannot provide. + +## Test layers + +### Unit and contract tests + +Cover deterministic validation, lifecycle transitions, token parsing/signing, tenant permissions, audit pseudonymization, range parsing, filename/content boundaries, configuration and public API contracts. + +### Integration tests + +Exercise controller → service → repository/artifact boundaries with realistic request/response semantics, including tenant concealment, signed artifact delivery, worker state transitions and configured filesystem/ledger behavior. + +### Security tests + +Required classes include: + +- missing/malformed/expired/revoked/wrong-scope/wrong-document artifact tokens; +- cross-tenant lookup/mutation concealment; +- weak/missing/shared cryptographic keys; +- hostile filenames, NUL/control characters, malformed Base64/UUID/epoch fields; +- oversized upload and parser resource limits; +- active-content/external-resource/macro boundaries when real Office conversion is introduced; +- privacy-safe logs and audit output; +- dependency/SAST/fuzz supply-chain gates. + +### Concurrency and recovery tests + +Cover duplicate submission races, bounded executor saturation, retry/dead-letter transitions, lifecycle serialization, stale processing recovery, crash/restart replay and—because #268 is `SUPERSEDED`, when its bounded lifecycle successors integrate—deletion receipt fairness, torn tails, generation mismatch and cleanup recovery. + +### Document-fidelity tests + +`PLANNED` and mandatory before advertising non-PDF source formats as supported. Use authorized realistic Office/PDF fixtures with expected: + +- text and structure; +- page count/layout/pagination where deterministic enough to assert; +- fonts/styles/tables/images/links; +- extracted metadata where claimed; +- rendered artifact integrity; +- unsupported construct/degraded warning behavior; +- active-content/security outcome; +- print/accessibility outcome where claimed. + +Fixture and expectation provenance must be reviewable. A synthetic-only mixture of trivial documents is insufficient for a commercial fidelity claim. + +### Accessibility tests + +Exercise accessible names for repeated actions, keyboard operation, busy-state semantics, focus restoration, ARIA restoration, no markup injection through document labels, error/status announcements, and print/export state as those surfaces mature. + +### Packaging / provenance / release tests + +Verify reproducible dependency resolution, deterministic SBOM/attribution regeneration where claimed, immutable workflow/action sourcing, package contents, startup smoke, supported runtime compatibility and exact release source identity. + +## TDD rule + +For a source/product defect: + +1. write the smallest realistic regression that reaches the intended production boundary; +2. observe RED for the diagnosed reason, not a fixture/setup/import failure; +3. implement the narrowest root-cause fix; +4. observe GREEN on the focused regression; +5. run the relevant full suite and exact coverage/docstring gates; +6. publish and require exact-head GitHub evidence; +7. resolve only the review thread actually addressed. + +If the first repair does not change the failing boundary, return to RCA rather than stack unrelated fixes. + +## Coverage and test-report gate + +Canonical Maven and script acceptance: + +```bash +mvn -B --no-transfer-progress verify +python3 scripts/verify_maven_test_reports.py +python3 scripts/test_documentation_spine_contract.py +python3 -m unittest discover -s scripts +``` + +The direct documentation-contract command gives a focused failure surface when the canonical graph changes; standard-library discovery is the repository-wide canonical acceptance for `scripts/*.py` checks. Neither replaces Maven verification or Surefire/Failsafe report validation. + +Required test-report properties: + +- at least one executed Surefire test; +- zero skipped tests; +- zero failures and errors; +- explicit non-negative outcome counts; +- the same contract for Failsafe reports when present; +- bounded strict UTF-8 XML with DTD/entity/NUL rejection before parsing. + +JaCoCo must report zero missed owned production lines and branches. Java public production APIs must pass warning-free Javadoc/doclint. JS production viewer utilities follow the repository's JS coverage gate when included in the production path. + +## Exact-head evidence taxonomy + +A local test or predecessor-head workflow result is diagnostic only. Merge evidence must bind the unchanged candidate head. Base-sensitive integration evidence is separate. Queued, pending, skipped-required, cancelled, absent, stale-head, predecessor-head, synthetic-only, model-only or failed evidence cannot be promoted to passing. + +## Test data policy + +- never commit confidential customer documents; +- prefer redistributable standards/examples or fixtures authored specifically for the repository; +- preserve fixture source/license/provenance; +- include hostile and edge-case fixtures without embedding live secrets or PII; +- size large fidelity assets deliberately and keep CI/runtime budgets bounded. + +## Scientific / mathematical components + +Clearfolio currently has no material psychometric numerical kernel. If one is introduced through future integration, production arithmetic follows the repository/CWL Rust-first CPU-multithreaded and parity-verified GPU policy where computationally material, with appropriate multilevel/multiple-membership and temporal structure rather than naive aggregation. + +## References + +Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The oracle problem in software testing: A survey. *IEEE Transactions on Software Engineering, 41*(5), 507–525. https://doi.org/10.1109/TSE.2014.2372785 + +Relevance: test execution is only useful when the oracle/evidence meaning is explicit; Clearfolio therefore validates report outcomes and domain assertions instead of treating process exit alone as acceptance. + +Inozemtseva, L., & Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. In *Proceedings of the 36th International Conference on Software Engineering* (pp. 435–445). ACM. https://doi.org/10.1145/2568225.2568271 + +Relevance: exact coverage is retained as a structural gate, but realistic security/fidelity/recovery assertions are separately required because coverage percentage alone is not evidence of fault-detection effectiveness. diff --git a/docs/THREAT_MODEL.md b/docs/THREAT_MODEL.md new file mode 100644 index 000000000..08c6434aa --- /dev/null +++ b/docs/THREAT_MODEL.md @@ -0,0 +1,130 @@ +# Clearfolio Threat Model + +Status: Canonical threat-model index +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +Root `SECURITY.md` remains the vulnerability-reporting policy. This document models product trust boundaries and links active hardening work without presenting it as shipped. + +## Protected assets + +- customer document bytes and converted artifacts; +- tenant, subject and authorization context; +- signed artifact tokens and signing keys; +- audit pseudonymization secrets and policy-signing secrets; +- conversion-job identity and lifecycle generation; +- artifact/analytics/deletion ledgers; +- source/build/SBOM/provenance evidence; +- reviewer, CI and automation credentials; +- fidelity fixtures and expected results. + +## Trust boundaries + +```mermaid +flowchart LR + Untrusted[Untrusted upload / browser input] + Gateway[Trusted gateway / identity source] + HTTP[Clearfolio HTTP boundary] + Auth[Authorization boundary] + Worker[Conversion worker] + Artifact[Artifact storage] + Ledger[Audit/evidence ledgers] + GitHub[GitHub CI/review control plane] + Model[OpenCode/NVIDIA model path - development only] + + Untrusted --> HTTP + Gateway --> Auth --> HTTP + HTTP --> Worker --> Artifact + HTTP --> Ledger + Worker --> Ledger + GitHub -. develops/reviews .-> HTTP + Model -. proposes bounded patches only .-> GitHub +``` + +Uploaded documents and all document-derived strings are untrusted data. Model output is untrusted development proposal data. Neither can change authorization or release policy. + +## Threat register + +| Threat | Boundary / impact | Current/target mitigation | +| --- | --- | --- | +| cross-tenant IDOR | job/artifact/admin APIs disclose or mutate another tenant | same-tenant concealment and protected #270 signed/admin controls `IMPLEMENTED_ON_MAIN`; #268 `SUPERSEDED`, remaining lifecycle successors `PARTIAL` | +| artifact-token bypass | permission-only route returns document bytes without revocation/checksum/audit | canonical `/artifacts` signed delivery and direct-download alignment `IMPLEMENTED_ON_MAIN` through protected #270 | +| stale/replayed artifact token | old token reads changed/revoked artifact | expiry, ledger presence, revocation, doc/scope/tenant/checksum binding | +| malformed token/parser ambiguity | surplus delimiters, empty claims, malformed Base64/UUID/epoch alter verification semantics | strict parser hardening `ACTIVE_PR` #276; fail closed before authorization/ledger continuation | +| weak/shared cryptographic keys | one compromise crosses signing/audit purposes | protected #270 key strength/separation and domain-separated HMAC pseudonyms `IMPLEMENTED_ON_MAIN`; #268 `SUPERSEDED` for remaining lifecycle work | +| raw PII/secret audit leakage | approver, tenant, token or exception-selected data leaves protected boundary | purpose-bound pseudonymization and controlled failure codes; privacy review; no raw tokens | +| job UUID rebinding / stale mutation | old retry/delete work acts on a replacement lifecycle | permanent identity + generation fences `PARTIAL`; #268 `SUPERSEDED`, bounded successors unintegrated | +| deletion evidence confusion | failed read interpreted as artifact absence, causing metadata loss | persist pending receipt first; confirmed digest/absence distinction `PARTIAL`; #268 `SUPERSEDED`, bounded successors unintegrated | +| recovery starvation after restart | repeatedly failing oldest receipt blocks later cleanup | durable attempt transitions and replay-derived fairness `PARTIAL`; #268 `SUPERSEDED`, bounded successors unintegrated | +| malicious/oversized document | CPU/memory/disk exhaustion, parser/converter exploit | upload bounds/fuzz/blocklist current; real converter must add sandbox/resource/active-content/no-network controls | +| macro/script/external-resource execution | document content gains code/network authority | forbidden in deterministic conversion architecture; real converter acceptance must prove boundary | +| placeholder fidelity overclaim | buyer believes non-PDF documents were faithfully converted | ADR-0005; release/support matrix; placeholder labelled development-only | +| path traversal / unsafe publication | attacker-controlled filename/path escapes artifact root | app-owned artifact paths, sanitization; future converter temp/publication tests | +| denial of service through queue | uncontrolled jobs exhaust workers | bounded executor current; durable backpressure/cancellation planned | +| liveness/readiness coupling | dependency outage causes restart storm or traffic to unready instance | separation `ACTIVE_PR` #295 | +| prompt injection in autonomous development | repository/document text changes agent authority or exfiltrates secret | model step is proposal-only, protected paths denied, credential-free verification, no self-merge `ACTIVE_PR` #271 | +| development credential leakage | model or logs expose `NVIDIA_NIM_API_KEY`/App tokens | GitHub Secrets only, model-step-only secret, short-lived scoped publisher, no raw secret echo | +| reviewer/approval spoofing | comment/status/model output treated as independent approval | evidence-class separation, exact-head formal review governance | +| stale-head merge evidence | old checks/reviews authorize moved code | ADR-0007 exact-head/live-base refetch and current-head gates | +| dependency/supply-chain compromise | malicious dependency/action/build artifact | immutable action/source pins where practical, SAST/security scan, SBOM/attribution/provenance | + +## Data privacy model + +Pseudonymization does not make audit data anonymous. Tenant/subject/artifact metadata remains purpose-limited personal/business data. Controls should prefer least privilege, selective disclosure, encryption, bounded retention and auditable privileged access rather than masking that prevents legitimate operations. + +Never publish or use as low-cardinality metric labels without an explicit reviewed need: + +- raw tenant/subject IDs; +- filenames or document paths; +- approval/artifact tokens or signatures; +- document content; +- uncontrolled document digests; +- exception-selected text/class names; +- local filesystem paths. + +## Converter threat requirements (`PLANNED` real Office conversion) + +A production converter must be treated as an untrusted-document execution boundary: + +- macro and active-content execution disabled; +- external resource/network access denied by default; +- bounded CPU, memory, process count, output size and wall time; +- app-owned temporary directories with cleanup; +- no user-controlled executable path/argument injection; +- MIME/magic/container validation before processing; +- malformed archive/XML/relationship/bomb fixtures; +- least-privilege OS/container identity; +- converter/version provenance in fidelity evidence; +- deterministic controlled failure when safety cannot be established. + +## Automation threat requirements + +The product-development agent is not a trusted merge authority. It may read untrusted repository content to propose a bounded patch, but: + +- permissions are fail-closed; +- credentials are not exposed to arbitrary verification code; +- model output cannot modify workflows, protected paths, binaries, symlinks or excessive file scope unless an explicitly reviewed contract says otherwise; +- a fresh credential-free runner verifies the immutable patch; +- publication uses short-lived minimum GitHub App authority; +- central review/merge authority remains independent. + +## Incident reopening conditions + +Reopen the threat model and relevant ADR when any of these changes: + +- a new document parser/converter is added; +- persistence moves to SQL/object storage; +- identity moves to a new issuer/federation model; +- token claims or cryptographic purposes change; +- a new cross-service integration receives state or secret authority; +- an autonomous agent receives broader write/tool/network permissions; +- support/fidelity claims expand to a new format; +- release/provenance authority changes. + +## Related security documents + +- `SECURITY.md` +- `docs/security/2026-07-02-threat-model-data-handling.md` +- `docs/security/2026-07-02-auth-tenant-model.md` +- `docs/security/2026-07-02-signed-artifact-link-design.md` +- `docs/security/2026-08-04-audit-pseudonymization.md` when integrated with its source PR +- ADR-0002 through ADR-0004 and ADR-0007 through ADR-0010 diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md new file mode 100644 index 000000000..2f515bc74 --- /dev/null +++ b/docs/TRACEABILITY.md @@ -0,0 +1,171 @@ +# Clearfolio Requirements and Evidence Traceability + +Status: Canonical traceability index +Baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` +Assessment date: 2026-09-20 + +This document maps product and technical requirements to current implementation, +verification, decisions, and open work. It prevents five recurring errors: + +1. treating an active pull request as shipped behavior; +2. treating a green check, status, or model comment as independent approval; +3. treating a development placeholder as production Office fidelity; +4. leaving a buyer-visible gap only in issue or PR prose; and +5. trusting predecessor SHAs, reviews, or checks after the source or base moved. + +## Maturity and evidence rules + +Canonical maturity values are: + +- `IMPLEMENTED_ON_MAIN` +- `ACTIVE_PR` +- `PARTIAL` +- `ACCEPTED_ARCHITECTURE` +- `PLANNED` +- `RESEARCH_ONLY` +- `SUPERSEDED` +- `OUT_OF_SCOPE` + +A PR-body SHA or run number is narrative. Live GitHub state is authoritative for +the exact source head, independently resolved base, current checks, formal +reviews, unresolved threads, and protected merge policy. `ACTIVE_PR` evidence +never becomes `IMPLEMENTED_ON_MAIN` merely because a branch is mergeable or all +technical checks pass. + +## Product requirement traceability + +| Requirement | Maturity | Primary implementation or evidence | Verification | Governing authority | +| --- | --- | --- | --- | --- | +| Bounded asynchronous submit | `IMPLEMENTED_ON_MAIN` | `ConversionController`, `DefaultDocumentConversionService`, `DefaultConversionWorker` | controller, service, retry, concurrency, Maven verify | PRD FR-01; TRD §§2, 4, 5 | +| Tenant-scoped job status | `IMPLEMENTED_ON_MAIN` | `TenantAccessService`, status controller, repository ownership checks | authorization, concealment, controller, repository tests | PRD FR-02/FR-06; ADR-0002 | +| Signed viewer artifact delivery | `IMPLEMENTED_ON_MAIN` | `ArtifactLinkService`, `ArtifactController`, issued-token ledger, revocation and range handling | signature, expiry, scope, tenant, document, checksum, range and audit tests | PRD FR-03; ADR-0002; API contract | +| Direct-download signed delivery | `IMPLEMENTED_ON_MAIN` | protected-main #270 direct job download uses tenant permission plus canonical signed artifact-token authority | exact-head CI, authorization, token, range, revocation and read-audit regressions | ADR-0002; API contract; release acceptance | +| Content identity and deduplication | `IMPLEMENTED_ON_MAIN` | conversion service and repository digest identity | service and concurrent repository tests | PRD FR-04; DATA_MODEL `content_identity` | +| Blocked-format and upload bounds | `IMPLEMENTED_ON_MAIN` | validation service and multipart limits | validation, fuzz and multipart tests | PRD FR-05; SECURITY; THREAT_MODEL | +| Privacy-safe HMAC audit pseudonymization | `IMPLEMENTED_ON_MAIN` | protected-main #270 domain-separated, purpose-specific pseudonymous audit evidence | key separation, privacy, no-raw-identifier and no-token regressions | ADR-0003; SECURITY; THREAT_MODEL | +| Privacy-safe public and logged failure envelopes | `IMPLEMENTED_ON_MAIN` | protected-main #270 class-only controlled failure categories and trace identity | provider-controlled message, path, filename, email, stack-trace and raw-job-id regressions | SECURITY; TEST_STRATEGY; OPERABILITY | +| Exact-head CI/test evidence | `IMPLEMENTED_ON_MAIN` | protected-main #270 exact source-head checkout, Maven `verify`, report verification, synthetic-merge separation, coverage and public-Javadoc gates | CI, Maven reports, zero missed owned lines/branches, warning-free public Javadocs | ADR-0007; RELEASE_ACCEPTANCE | +| Strict artifact-token structural parsing | `ACTIVE_PR`; issue #263 security dependency; current #276 | clean current-base token claim boundary preserves trailing empty fields and rejects malformed claims before ledger or authorization access | exact-head CI, Security Scan, SAST, fuzz and focused token tests | ADR-0002; API_CONTRACT; TEST_STRATEGY | +| Production HMAC key readiness | `ACTIVE_PR`; issue #319 dependency; current #313 | clean current-base `ProductionAuthReadinessConfig` rejects missing/null, undersized, normalized, unstable or purpose-reused signing material | exact-head CI, Security Scan, SAST, fuzz and 100% coverage | SECURITY; THREAT_MODEL; MIGRATION_ROLLBACK | +| Runtime credential registry | `PLANNED`; issue #319 | protected main still receives tenant-claim and artifact-token HMAC values from environment-backed Spring configuration; #313 is readiness, not authority migration | registry lookup, rotation, restart, replica, least-privilege and no-secret-log tests required | issue #319; AGENTS; SECURITY; THREAT_MODEL | +| Provider-neutral production identity federation | `PLANNED`; issue #314 | current HMAC-signed gateway claim adapter remains bounded; no complete OIDC/JWT federation is shipped | issuer, audience, algorithm, JWK rotation/outage, tenant mapping and migration tests required | issue #314; PRD; API_CONTRACT; THREAT_MODEL | +| Immutable job identity and durable deletion receipts | `PARTIAL`; issue #263; #268 `SUPERSEDED`; current #345/#350/#351/#353 | the stale broad descendant was closed after its valuable semantics were decomposed into current bounded lanes for permanent job-id reservation, same-job lifecycle serialization, deletion state vocabulary and immutable deletion-receipt identity; durable receipt persistence and recovery are still not shipped | current-base identity/state/locking tests plus future durable replay, crash-tail, fairness, privacy and lifecycle acceptance | ADR-0004; DATA_MODEL; MIGRATION_ROLLBACK | +| User-facing tenant-safe deletion journey | `PLANNED`; issue #263 | lower-layer lifecycle work is not yet an integrated protected-main upload/view/download/delete/recovery workflow | idempotent delete, link invalidation, pending/failure/completion UI, restart/recovery and accessibility tests required | issue #263; PRD; API_CONTRACT; RELEASE_ACCEPTANCE | +| Nested-safe accessible asynchronous controls | `ACTIVE_PR`; current #264 | clean current-base reusable `dom-utils.js` busy-state behavior and a dependency-free Node integration suite preserve nested state, inert repeated activation, focus and restoration semantics | exact-head CI, Security Scan, SAST, fuzz, JavaScript unit/integration and branch-coverage evidence | PRD FR-09; UML viewer flows; issue #263 | +| Viewer render generation safety | `PARTIAL`; issue #322; current `ACTIVE_PR` #323 | clean current-base first slice suppresses stale canvas, metadata, preview-link and ready-state publication after supersession | behavioral Node late-render regression plus exact-head CI/security/SAST/fuzz | issue #322; UML viewer lifecycle; TEST_STRATEGY | +| Active PDF.js task cancellation and destruction | `PLANNED`; issue #322 | #323 discards late results but does not yet complete `RenderTask.cancel()`, loading-task destruction, signed-token parity or multi-generation ownership | cancellation invocation, three-generation, old-success/new-failure, focus/status ownership and no-unhandled-rejection tests required | issue #322; OPERABILITY; TEST_STRATEGY | +| Robust keyboard focus appearance | `PARTIAL`; issue #324; current `ACTIVE_PR` #334 | two independent 3px black/white `:focus-visible` bands reconstructed on protected main | executable CSS geometry and WCAG contrast regression plus exact-head CI/security/SAST/fuzz | issue #324; TEST_STRATEGY | +| Liveness and readiness separation | `ACTIVE_PR`; current #295 | clean current-base `/healthz` process liveness and `/readyz` traffic readiness with controlled no-store payloads | state, failure, recovery, no-cache and missing-authority tests | ADR-0006; OPERABILITY | +| Production workspace and session bootstrap | `PLANNED`; issue #317 | protected main still exposes buyer-demo shell and browser/service demo authority; a real server-controlled bootstrap is absent | production-profile, session authority, real upload/poll/view, tenant isolation and browser-storage tests required | issue #317; PRD; ARCHITECTURE; SECURITY | +| Bounded external-company branding repair | `PARTIAL`; issue #317; current `ACTIVE_PR` #318 | two footer claims change from unrelated HYOSUNG ownership to Clearfolio Apache-2.0 branding | focused root/viewer HTML contract and exact-head CI/security/SAST/fuzz | issue #317; ACQUISITION_DILIGENCE | +| Real Office conversion fidelity | `PLANNED`; issue #5 | protected main has validated PDF passthrough and a development/demo placeholder, not production transformed-format support | authorized realistic Korean/English corpus, security, resource, deterministic rendering, license/SBOM/provenance and recovery gates required | issue #5; ADR-0005; FIDELITY_ACCEPTANCE | +| Provider-neutral Office adapter boundary | `ACTIVE_PR`; issue #5; #306 | Draft adapter and package/output policies remain non-shipped; `docs/UML.md` and `docs/DATA_MODEL.md` model the isolation and conceptual qualification entities | exact-head contract, hostile-package, output-policy, CPU/resource and later runtime-fidelity evidence required | issue #5; #306; ADR-0005; `docs/UML.md`; `docs/DATA_MODEL.md` | +| Durable distributed jobs, backpressure and cancellation | `PLANNED`; issue #312 | protected main still uses process-local job repository, executor and delayed retry timers | atomic accept/outbox, idempotency, lease fencing, redelivery, saturation, cancellation-race, restart and rollback tests required | issue #312; ADR-0004; OPERABILITY; MIGRATION_ROLLBACK | +| Privacy-safe OpenTelemetry | `PLANNED`; issue #312 | selected logs and analytics exist, but no complete trace/metric cardinality and privacy contract | trace correlation, low-cardinality labels, queue/lease/cancellation/recovery metrics and SLO evidence required | issue #312; TRD; OPERABILITY; ACQUISITION_DILIGENCE | +| Complete versioned public API/schema and naruon compatibility | `PARTIAL`; issue #315 | current controllers, records, buyer OpenAPI seed and prose contract do not form one complete released machine-readable authority | route/DTO/error/lifecycle/example/version/breaking-change, standalone and naruon consumer tests required | issue #315; API_CONTRACT; ADR-0001; RELEASE_ACCEPTANCE | +| Buyer OpenAPI license, neutral example and delete-route integrity | `PARTIAL`; issue #315; current `ACTIVE_PR` #316 | bounded integrity slice aligns `info.license` to Apache-2.0, removes demo examples and verifies the required delete path parameter | structured YAML and route contract tests plus exact-head CI/security/SAST/fuzz | issue #315; #316; API_CONTRACT; ACQUISITION_DILIGENCE | +| Stable unique OpenAPI operation identities | `PARTIAL`; issue #315; current `ACTIVE_PR` #337 | offline standard-library checker requires one non-empty unique `operationId` per HTTP operation | positive current-schema test, missing/duplicate/path-metadata regressions and exact-head CI/security/SAST | issue #315; current `ACTIVE_PR` #337; API_CONTRACT | +| Analytics storage-level tenant isolation | `PARTIAL`; issue #326; current `ACTIVE_PR` #342 with stacked #361/#380 | current-base #342 pushes analytics list isolation into a tenant-scoped repository query; #361/#380 extend fail-closed tenant-scoped identifier lookup through repository/service boundaries, while none of these ACTIVE_PR slices is protected-main behavior yet | hostile global-query seam, scoped repository/service interaction and two-tenant concealment/KPI isolation tests | issue #326; ADR-0002; THREAT_MODEL | +| Terminal-outcome conversion success rate | `PARTIAL`; issue #327; current `ACTIVE_PR` #338 | clean current-base KPI response and evidence Javadocs use succeeded divided by succeeded plus failed, excluding in-flight work | mixed terminal/in-flight, zero-terminal, controller and exact-head CI/security/SAST/fuzz tests | issue #327; current `ACTIVE_PR` #338; API_CONTRACT; TEST_STRATEGY | +| Finite and domain-valid persisted KPI evidence | `PARTIAL`; issue #329; current `ACTIVE_PR` #389 | current-base KPI ledger validation rejects NaN, infinities, rates outside `[0,1]` and negative p95 latency without clamping; former #339 is closed historical evidence | invalid replay and valid historical round-trip tests plus exact-head CI/security/SAST/fuzz | issue #329; current `ACTIVE_PR` #389; DATA_MODEL; TEST_STRATEGY | +| Deterministic single logging runtime binding | `ACTIVE_PR`; issue #320; current #340 | clean current-base exclusion keeps Spring `spring-jcl` as the sole Commons Logging bridge and removes standalone `commons-logging` from the PDFBox path | exact-head CI, Security Scan, SAST, fuzz, runtime provider enumeration and dependency-intent contract | issue #320; OPERABILITY; RELEASE_ACCEPTANCE | +| Hourly RCA/feasibility product development | `ACTIVE_PR`; current #271 | clean current-base bounded OpenCode workflow, immutable patch, credential-free verification and Draft-only publication | prompt/workflow contract tests, exact-head CI/security/SAST and protected-main operational run after integration | ADR-0008; ADR-0009; ADR-0011 | +| User-redirection and no-early-stop execution contract | `ACTIVE_PR`; current #271 | actual recurring prompt classifies premature-stop reports, gives prompt repair zero completion credit, rebuilds the live queue, performs same-run substantive work and resets exit sweeps | direct prompt regression and exact-head Buyer-readiness/Maven/merge acceptance | ADR-0009; ADR-0011; hourly operations guide | +| Scheduler execution receipt and resumable continuation | `PLANNED`; issue #331 | external scheduler exposes no durable phase/action/checkpoint evidence; #271 carries only a bounded repository-local semantic slice | run/admission/queue/action/CAS/privacy/failure/budget-continuation simulations and platform evidence required | issue #331; ADR-0012; DATA_MODEL; UML; OPERABILITY | +| Qualifying independent formal review route | `PLANNED`; issue #321; organization `.github#772` ownership | protected merge requires at least one approval from a write-authorized independent reviewer; advisory bots are not counted | eligible human-team provisioning, CODEOWNERS assignment, stale-review and permission-loss acceptance required | issue #321; ADR-0007; ADR-0008; repository metadata | +| Reproducible release, SBOM and provenance | `PARTIAL`; current `ACTIVE_PR` #391 and Draft #392 | #270 integrated deterministic SBOM/attribution and strict source tests; #391 binds buyer OpenAPI bytes to a source revision and #392 proposes a read-only tagged acceptance evidence bundle, but neither is protected-main release truth and no complete protected release has passed integrated fidelity/recovery/review acceptance | release artifact/schema digests, exact tag/source binding, SBOM/license/provenance, rollback and post-publication verification required | ADR-0010; RELEASE_ACCEPTANCE; issue #315 | +| Acquisition, IP and legal diligence | `PARTIAL` | repository license/SBOM/attribution and canonical diligence are inspectable; contributor/IP assignment, FTO, contracts and certification remain external | repository contract tests plus external legal, customer, infrastructure and commercial evidence | ACQUISITION_DILIGENCE; DOCUMENTATION_ASSESSMENT | + +## Current clean replacements and superseded predecessors + +The following predecessor PRs were closed only after a current-base replacement +was compared by exact blob or semantic patch and independently reverified: + +| Requirement lane | Current evidence | Superseded predecessor | Reason | +| --- | --- | --- | --- | +| Focus appearance | issue #324; current `ACTIVE_PR` #334 | #325 `SUPERSEDED` | identical two-file behavior rebuilt on protected main and exact-head checks rerun | +| Terminal KPI denominator | issue #327; current `ACTIVE_PR` #338 | #328 `SUPERSEDED` | all five output blobs are byte-identical on a clean protected-main base | +| KPI ledger numeric validity | issue #329; current `ACTIVE_PR` #389 | #330 `SUPERSEDED`; #339 closed historical evidence | current #389 carries the live bounded invalid-evidence contract; older unmerged heads are not current proof | +| OpenAPI operation identity | issue #315; current `ACTIVE_PR` #337 | #332 `SUPERSEDED` | identical checker/test blobs; predecessor stale ancestry lacked current report verifier | +| HMAC readiness | issue #319; current `ACTIVE_PR` #313 | #333 `SUPERSEDED` | duplicate production blob and equivalent focused tests | +| Availability probes | ADR-0006; current `ACTIVE_PR` #295 | #335 `SUPERSEDED` | duplicate production/test blobs; #295 also retains a scoped operations runbook | +| Branding boundary | issue #317; current `ACTIVE_PR` #318 | #336 `SUPERSEDED` | both changed blobs were byte-identical | +| Broad admin/deletion descendant | issues #263/#312 and bounded current successors #341/#342/#345/#350/#351/#353/#361/#363/#380 | #268 `SUPERSEDED` | 139-file conflict-bearing descendant was closed only after its valuable tenant, identity, lifecycle and audit semantics were mapped to current-base or intentionally stacked bounded lanes; old checks/reviews are non-transferable | + +A superseded closure is queue hygiene, not proof that the replacement is shipped. +Each replacement still requires exact-head policy, counted independent approval, +and protected integration. + +## Evidence-authority matrix + +| Evidence | What it proves | What it does not prove | +| --- | --- | --- | +| Exact-head CI | Deterministic tests and configured quality gates passed for one source identity | Independent approval, future-base compatibility, protected runtime acceptance | +| Synthetic-merge job | Candidate was compatible with the tested base composition | Future live-base compatibility or source-head behavior by itself | +| Security Scan, SAST or fuzz | The configured scanner or target completed successfully on the named identity | Absence of every vulnerability or human approval | +| CodeRabbit, OpenCode, Noema or model comment | Advisory review or model judgment for a named identity | GitHub-counted write-authorized independent approval | +| Formal `APPROVED` review | An eligible reviewer approved a specific review identity subject to policy | Current checks, current base, release or deployment readiness after movement | +| PR-body SHA/run list | Historical narrative and investigation provenance | Live head, base, review, thread or check state | +| Local test output | Developer diagnostic confidence | Protected integration or release evidence | +| Protected-main commit | The merged source is now repository implementation authority | Production deployment, SLO, fidelity, certification or transaction readiness | +| Scheduler activation | A recurrence was enabled or admitted | Queue construction, repository action or successful completion | +| Scheduler execution receipt | The executing control plane reached a named phase and evidence boundary | Current GitHub state, merge authority or an unrecorded hidden root cause | +| `budget continuation` | A run stopped at an observed safe checkpoint before a practical hard boundary | Repository completion or permission to trust checkpoint SHAs as current | +| Fidelity corpus result | Exact runtime satisfied declared fixture assertions | Every customer document, security review or another runtime/corpus | +| SBOM/provenance and artifact digest | Artifact relationship to declared source and materials | Functional correctness, independent approval or legal FTO | +| Repository license and attribution | Source/dependency terms are inspectable | Contributor assignment, patent/trademark FTO, customer data rights or certification | + +Any source, base, required-check, review-policy, or reviewer-permission movement +invalidates assumptions not bound to the new exact identity. + +## Conversation decision capture + +The Clearfolio-scoped decisions repeatedly established in this conversation are +mapped as follows: + +- review/check latency never terminates disjoint safe work → ADR-0009; +- blocker → RCA → distinct remedies → feasibility → same-run action → proof → ADR-0009; +- prompt repair earns zero completion credit and user redirection requires same-run substantive execution → ADR-0009, ADR-0011 and current #271; +- two consecutive fresh exit sweeps are required after the last action → ADR-0009 and OPERABILITY; +- central `.github` owns privileged PR maintenance while Clearfolio owns bounded product proposals → ADR-0008; +- OpenCode uses GitHub Secret `NVIDIA_NIM_API_KEY`, not `COPILOT_GITHUB_TOKEN` as a model credential → ADR-0008; +- scheduler prompts are a thin control plane; repository documents are detailed authority → ADR-0011; +- scheduler activation, execution, receipt, controlled failure and continuation are separate evidence classes → ADR-0012 and issue #331; +- signed tenant permission and signed artifact-token byte authority are separate controls → ADR-0002; +- pseudonymized audit identity remains personal data and uses purpose-separated keys → ADR-0003; +- lifecycle identity requires durable generation fencing and truthful cleanup evidence → ADR-0004; +- Office process ownership remains outside the API-container trust boundary → ADR-0005 and issue #5; +- liveness and readiness answer different operational questions → ADR-0006 and current #295; +- PR-body and predecessor checks are not live exact-head/live-base evidence → ADR-0007; +- one integrated protected source/artifact identity owns release truth → ADR-0010; +- documentation sufficiency is an intermediate state, never a stop condition → DOCUMENTATION_ASSESSMENT and ADR-0009; +- acquisition diligence separates repository engineering evidence from external legal, customer and commercial evidence → ACQUISITION_DILIGENCE. + +Reports from ThreadWeave, fast-mlsirm, BandScope, TEPP, OriginWeave, LifeOS, +EmbedRelay, MHTML ETL Gateway, or another ContextualWisdomLab repository are not +silently imported into Clearfolio requirements. Only actual Clearfolio runtime, +composition, or organization-control dependencies are referenced here. + +## Documentation dependency graph + +```mermaid +flowchart TD + PRD[docs/PRD.md] --> TRD[docs/TRD.md] + PRD --> TRACE[docs/TRACEABILITY.md] + TRD --> ARCH[ARCHITECTURE.md] + TRD --> DATA[docs/DATA_MODEL.md] + TRD --> UML[docs/UML.md] + ARCH --> ADR[docs/adr/README.md] + ADR --> TRACE + DATA --> TRACE + UML --> TRACE + SEC[SECURITY.md + THREAT_MODEL] --> ADR + TEST[docs/TEST_STRATEGY.md] --> TRACE + OPS[docs/OPERABILITY.md] --> TRACE + FID[docs/FIDELITY_ACCEPTANCE.md] --> REL[docs/RELEASE_ACCEPTANCE.md] + MIG[docs/MIGRATION_ROLLBACK.md] --> REL + REL --> TRACE + RES[docs/RESEARCH_TRACEABILITY.md] --> TRACE + ACQ[docs/ACQUISITION_DILIGENCE.md] --> TRACE + TRACE --> ASSESS[docs/DOCUMENTATION_ASSESSMENT.md] +``` diff --git a/docs/TRD.md b/docs/TRD.md new file mode 100644 index 000000000..c3bf17c1c --- /dev/null +++ b/docs/TRD.md @@ -0,0 +1,237 @@ +# Clearfolio Technical Requirements Document + +Status: Canonical technical requirements spine +Baseline: protected `main` at `f3cc09a9838f0f88c81a2ceae22138fab80a2edb` + +## 1. Technical objective + +Clearfolio provides a bounded Spring WebFlux service for untrusted document intake, asynchronous conversion lifecycle management, tenant-scoped viewer/artifact access, and operational evidence. The technical design must preserve standalone operation while exposing replaceable interfaces for naruon and other CWL hosts. + +This document separates protected-main implementation from active pull requests and target architecture. An `ACTIVE_PR` statement is never release evidence. + +## 2. Runtime boundaries + +### HTTP/control plane + +`IMPLEMENTED_ON_MAIN`: + +- `ConversionController` accepts uploads, exposes status/retry/viewer/direct-download APIs, and delegates heavy work away from request processing. +- `ArtifactController` owns signed artifact-link creation, revocation, read-audit access, signed PDF delivery, and single-range behavior for the canonical artifact route. +- `ViewerUiController` serves the HTML/PDF.js viewer shell. +- `ApiExceptionHandler` provides controlled API failures. + +### Conversion plane + +`IMPLEMENTED_ON_MAIN`: + +- `DocumentConversionService` / `DefaultDocumentConversionService` orchestrates validation, content identity, dedupe, state-store interaction, artifact seeding and worker enqueue. +- `DefaultConversionWorker` runs bounded background work with retry/dead-letter behavior and startup recovery against available repository state. +- PDF uploads with validated PDF magic can be passed through to the artifact store. + +`PARTIAL`: + +- `PdfBoxArtifactGenerator` produces placeholder one-page PDFs for non-PDF sources. It is not an Office-fidelity implementation. + +`PLANNED`: + +- production Office conversion adapter(s) with no unreviewed active-content execution or external network dependency; +- fixture-based fidelity evidence for each claimed source format; +- explicit unsupported/degraded behavior. + +The detailed transformed-format qualification authority is `docs/FIDELITY_ACCEPTANCE.md`; issue #5 is the active product gap. A successful converter process exit, accepted extension, or generated placeholder cannot independently authorize a support claim. + +### Persistence/evidence plane + +`IMPLEMENTED_ON_MAIN`: + +- `InMemoryConversionJobRepository` is the default job repository. +- `FileSystemArtifactStore` can persist artifact bytes and metadata locally; in-memory mode remains available for tests/development. +- selected evidence uses append-only/file-backed ledgers such as artifact-link and analytics snapshot ledgers when configured. + +`PARTIAL` target; #268 is `SUPERSEDED` and bounded successors remain unintegrated: + +- durable artifact deletion receipts, generation fencing, append-only recovery evidence and fairness/restart hardening. + +`PLANNED`: + +- SQL-backed durable conversion jobs/state transitions; +- distributed lease/idempotency/cancellation/backpressure contracts; +- remote object-store atomicity and lifecycle coordination. + +Current/future persistence compatibility, rollback and restore semantics are governed by `docs/MIGRATION_ROLLBACK.md`; the conceptual ERD in `docs/DATA_MODEL.md` never implies that planned SQL entities already exist. + +## 3. Security and authorization boundaries + +### Tenant authority + +`IMPLEMENTED_ON_MAIN`: `TenantContext`, `TenantAccessService`, and `TenantPermissions` provide tenant/subject/permission boundaries and same-tenant concealment semantics. Signed gateway claims can be configured; unsigned demo scaffolding is not a production internet trust boundary. + +`IMPLEMENTED_ON_MAIN` through protected #270 for signed claims and audit controls; #268 is `SUPERSEDED` and the remaining lifecycle bullets are `PARTIAL` target behavior: + +- signed tenant claim requirements; +- dedicated least-privilege admin/artifact permissions; +- purpose-separated signing/audit keys; +- domain-separated HMAC audit pseudonyms; +- cross-tenant concealment before artifact/repository mutation; +- immutable lifecycle identity/generation fences. + +### Runtime credential authority + +`PRESENT-DEVIATION-ON-MAIN`: the repository-level `AGENTS.md` records that the artifact-token HMAC secret and tenant-claim HMAC secret are currently transported directly into Spring runtime configuration through environment-backed placeholders. This is an acknowledged governance deviation, not the target credential architecture. + +`ACCEPTED_ARCHITECTURE`: + +- runtime code reads secrets from a credential/KV registry rather than treating raw process environment as the runtime source of truth; +- deployment/CI may use an environment value only as bounded bootstrap transport into that registry; +- the credential record is purpose-scoped, tenant/deployment scoped where applicable, auditable, rotation-capable, and never exposed through logs, metrics, public errors, health payloads, or document-derived data; +- artifact-token signing, tenant-claim signing, audit pseudonymization, model credentials, and repository automation credentials remain separate purposes and authorities; +- standalone mode may use a replaceable local credential-store adapter, but production must not silently fall back to unsigned/demo behavior when the configured registry is unavailable. + +Migrating this deviation is a product/security task. Documentation must not claim that the current protected-main secret source is already KV-backed. + +### Artifact authority + +`IMPLEMENTED_ON_MAIN`: `ArtifactLinkService` issues HMAC-bound, tenant/document/checksum/scoped tokens with expiry, ledger presence, revocation, and read-audit evidence. `ArtifactController#getPdf` validates signed tokens and single-range requests before serving bytes. + +`IMPLEMENTED_ON_MAIN` through protected #270: direct `ConversionController` download uses the same signed-delivery semantics. Dedicated `artifact:read` permission remains necessary but cannot bypass signed token, revocation, or audit requirements. + +### Privacy-safe audit + +Raw secrets and uncontrolled exception-selected details must not become durable/shared audit evidence. Audit correlation identifiers must be purpose-bound. Pseudonymization reduces direct identifiability but does not remove personal-data status. + +## 4. API requirements + +### Asynchronous conversion + +- `POST /api/v1/convert/jobs`: bounded multipart intake; return accepted job contract; no conversion completion wait. +- `GET /api/v1/convert/jobs/{jobId}`: tenant-scoped lifecycle snapshot. +- `POST /api/v1/convert/jobs/{jobId}/retry`: controlled dead-letter retry. +- `DELETE /api/v1/convert/jobs/{jobId}` and admin mutation surfaces must enforce tenant/generation-safe semantics appropriate to the integrated branch. + +### Viewer and artifacts + +- `/viewer/{docId}`: HTML viewer entry. +- `/api/v1/viewer/{docId}` and alias: protected bootstrap data and signed artifact-link metadata for succeeded jobs. +- `/api/v1/viewer/{docId}/artifact-links`: tenant-authorized signed artifact-link issuance. +- `/artifacts/{docId}.pdf`: signed token + revocation + checksum binding + single-range + read audit. +- `/api/v1/convert/jobs/{jobId}/download`: direct-download convenience endpoint; protected #270 aligned it to the same signed artifact-delivery authority while retaining dedicated tenant permission and attachment semantics (`IMPLEMENTED_ON_MAIN`). + +All new public API/schema changes require version/compatibility analysis and regression coverage. `docs/API_CONTRACT.md` is the version/authority index; implementation-specific examples must not silently redefine its tenant, token, lifecycle, or error semantics. + +## 5. Job lifecycle requirements + +Protected-main lifecycle uses `SUBMITTED`, `PROCESSING`, `SUCCEEDED`, and `FAILED`, with retry exhaustion represented through dead-letter evidence rather than a separate public terminal enum. + +Required invariants: + +1. one immutable job identity must not be rebound to a different tenant/content generation; +2. only valid transitions may be persisted; +3. worker retry is bounded and observable; +4. stale processing can be recovered only under an explicit lease/age contract; +5. duplicate submissions do not create inconsistent content ownership; +6. delete/retry operations must not cross tenant or generation fences; +7. future distributed storage must preserve idempotency and recovery under worker crash/restart. + +Historical #268 is `SUPERSEDED`. Current bounded successors preserve individual immutable-generation, deletion-receipt, cleanup-recovery, and administrative-lifecycle primitives, but they remain unintegrated `PARTIAL` evidence. Umbrella issue #263 owns the remaining integrated user-facing deletion/download API and accessible UX after those successors and the accessibility lane integrate. New work must reuse the accepted contracts rather than duplicate them on another branch. + +## 6. Conversion and fidelity requirements + +### Format classification + +- `passthrough`: source bytes are the claimed view artifact after deterministic validation (for example, accepted PDF passthrough). +- `transformed`: actual conversion is executed and validated against a fidelity profile. +- `degraded`: controlled lower-fidelity output with explicit user/operator warning and release-approved scope. +- `unsupported`: controlled failure; no placeholder may masquerade as successful conversion. +- `development_placeholder`: test/demo-only behavior not eligible for supported-format claims. + +### Fidelity evidence + +Each transformed format requires: + +- authorized or redistributable realistic fixtures covering text, fonts, tables, images, pagination and representative edge cases; +- expected extraction/render structure and artifact properties; +- comparison rules that tolerate only documented nondeterminism; +- macro/active-content/external-resource/malformed-container tests; +- size/time/resource bounds and cleanup tests; +- accessibility/print/export behavior where user-visible; +- exact-head benchmark evidence retained separately from timeless requirements. + +`docs/FIDELITY_ACCEPTANCE.md` defines the detailed support taxonomy, deterministic/provenance requirements and release evidence for issue #5. + +## 7. Availability and recovery + +`IMPLEMENTED_ON_MAIN`: health endpoint and process-local startup recovery exist. + +`ACTIVE_PR` #295: `/healthz` becomes process liveness and `/readyz` traffic readiness, with controlled low-information responses, `no-store`, and deterministic startup/failure/recovery tests. + +Future readiness contributors must be instance-local routing conditions. Shared-service health must not turn liveness into a dependency restart cascade. + +Durable-format, artifact-store, converter/runtime, API/token and future SQL migration/rollback/recovery are governed by `docs/MIGRATION_ROLLBACK.md`. Recovery is complete only after restored state passes the tenant/security and representative buyer-flow checks appropriate to the release. + +## 8. Observability + +`PARTIAL`: structured API errors and selected audit/analytics evidence exist. + +`PLANNED`: + +- OpenTelemetry traces for intake, validation, enqueue, conversion, artifact publication and recovery; +- low-cardinality metrics for queue depth/lag, conversion outcome, retry/dead-letter, readiness and artifact cleanup; +- privacy-safe correlation that does not publish tenant, filename, raw token, document digest or uncontrolled exception values as metric labels; +- measured SLO evidence before any availability claim. + +## 9. Standalone and MSA interoperability + +Clearfolio owns document conversion/viewing behavior, local authorization enforcement, artifact-delivery rules, its own state interfaces and evidence contracts. A host such as naruon may own user-facing orchestration, identity exchange, transport composition and deployment policy, but must not reach directly into Clearfolio application persistence. + +Integration requirements: + +- stable versioned HTTP/schema contracts; +- replaceable repository/artifact/conversion/credential adapters; +- explicit tenant/actor authority handoff; +- no hidden dependency on a central CWL repository at runtime; +- central `.github` reusable automation remains control-plane tooling, not a product runtime dependency. + +## 10. Automation architecture + +### PR maintenance + +`ACCEPTED_ARCHITECTURE`: central `ContextualWisdomLab/.github` owns privileged review → repair → exact-head revalidation → protected merge automation. Leaf repositories should use thin callers/contracts rather than copy privileged implementation. + +### Product development + +`ACTIVE_PR` #271 proposes a Clearfolio-local, bounded, path-disjoint OpenCode development workflow. It uses `NVIDIA_NIM_API_KEY` only at the model step, performs credential-free verification, publishes only a Draft through short-lived scoped authority, and never approves/merges/releases/deploys its own proposal. + +Decision contract: exact evidence → RCA → distinct remedies → empirical feasibility → test-first action → exact proof → next executable queue item. A blocker report or one completed artifact is not a successful run while safe work remains. + +## 11. Quality gates + +- Java 21 and repository-defined Maven toolchain. +- `mvn -B --no-transfer-progress verify` is the canonical full Maven acceptance command. +- Surefire evidence: positive executed count; zero skip/failure/error; strict bounded safe XML parsing. +- optional Failsafe evidence follows the same rule when present. +- JaCoCo: zero missed owned production lines and branches. +- warning/deprecation failures remain enabled. +- warning-free public Javadocs for production public API. +- JS viewer logic touched by production flows is covered by its repository coverage gate. +- security/SAST/fuzz and other required exact-head checks must pass according to live repository policy. +- queued/pending/skipped-required/cancelled/absent/stale/predecessor/synthetic-only evidence is never promoted to success. + +## 12. Supply-chain and release requirements + +- dependencies and actions are explicitly governed and immutably pinned where practical; +- third-party attribution and CycloneDX SBOM are reproducible from source when claimed; +- release provenance must bind source revision, dependency lock/BOM state, build workflow and produced artifact; +- release only after realistic document-fidelity, security, coverage, accessibility, compatibility, recovery, independent-review and protected-main operational evidence is complete. + +`docs/RELEASE_ACCEPTANCE.md` is the detailed integrated gate. It requires one exact protected source/artifact identity and keeps source-head, live-base, workflow, review, security, fidelity/recovery and SBOM/provenance evidence separate. A successful PR check, local build, predecessor head or model verdict cannot substitute for integrated release acceptance. + +## 13. Technical non-goals + +- arbitrary JavaScript/macro execution from uploaded documents; +- uncontrolled external fetches during deterministic conversion; +- LLM-based authorization, format validation or fidelity pass/fail; +- raw process environment as the production runtime credential source of truth; +- assuming in-memory state is durable distributed persistence; +- treating a PR-body SHA or status-only bot result as current integration evidence; +- duplicating central control-plane write authority inside the product repository; +- duplicating the accepted #268 lifecycle/deletion requirements, now `SUPERSEDED` into bounded successors, to bypass their dependency/review gates. diff --git a/docs/UML.md b/docs/UML.md new file mode 100644 index 000000000..9b98c6a5d --- /dev/null +++ b/docs/UML.md @@ -0,0 +1,336 @@ +# Clearfolio UML and Architecture Diagrams + +Status: Canonical diagram index +Baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` + +Diagrams are architecture views, not deployment evidence. `ACTIVE_PR` labels identify behavior that is not yet protected-main functionality. + +## 1. Bounded-context / component view + +```mermaid +flowchart LR + Client[Document user / host client] + Operator[Tenant operator] + Gateway[Trusted identity / gateway boundary] + + subgraph Clearfolio[Clearfolio service] + HTTP[WebFlux HTTP controllers] + Auth[Tenant access boundary] + Convert[Conversion orchestration] + Worker[Bounded conversion worker] + Repo[Job repository / state store] + Artifact[Artifact store] + Link[Signed artifact link service] + Ledger[Audit / artifact link ledgers] + Viewer[Viewer shell / PDF.js] + Availability[Health / availability] + end + + Client --> HTTP + Operator --> HTTP + Gateway --> Auth + HTTP --> Auth + HTTP --> Convert + Convert --> Repo + Convert --> Worker + Worker --> Repo + Worker --> Artifact + HTTP --> Link + Link --> Artifact + Link --> Ledger + HTTP --> Viewer + HTTP --> Availability + + Naruon[naruon / other CWL host] -. versioned API contract .-> HTTP + Central[ContextualWisdomLab/.github] -. development control plane only .-> Clearfolio +``` + +## 2. Submit → convert → view sequence + +```mermaid +sequenceDiagram + actor User + participant API as ConversionController + participant Auth as TenantAccessService + participant Service as DocumentConversionService + participant Repo as ConversionJobRepository + participant Worker as DefaultConversionWorker + participant Store as ArtifactStore + participant Links as ArtifactLinkService + participant Viewer as Viewer/PDF.js + + User->>API: POST /api/v1/convert/jobs + tenant claims + file + API->>Auth: require job:create + Auth-->>API: tenant_context + API->>Service: submit(file, policy, tenant_context) + Service->>Service: validate + content identity + dedupe + Service->>Repo: save conversion_job + Service->>Worker: enqueue(job) + API-->>User: 202 + jobId + statusUrl + Worker->>Store: publish PDF artifact + Worker->>Repo: mark SUCCEEDED + User->>API: GET viewer bootstrap + API->>Auth: require viewer:read + same tenant + API->>Links: createLink(job, tenant_context) + Links->>Store: checksum current artifact + Links-->>API: signed artifact URL + API-->>Viewer: bootstrap metadata +``` + +For non-PDF inputs on protected main, artifact generation may be development placeholder behavior; this sequence does not imply Office-fidelity acceptance. + +## 3. Tenant-authorized signed artifact read + +```mermaid +sequenceDiagram + actor Caller + participant API as Artifact/download endpoint + participant Auth as TenantAccessService + participant Repo as Job repository + participant Store as ArtifactStore + participant Token as ArtifactLinkService + participant Ledger as ArtifactLinkLedger + + Caller->>API: request bytes + tenant claims + signed artifact token + API->>Auth: require artifact:read where endpoint policy requires + Auth-->>API: tenant_context + API->>Repo: get tenant-owned conversion_job + API->>Store: get artifact bytes + API->>Token: verifyReadToken(job, bytes, token) + Token->>Ledger: verify issuance + not revoked + Token->>Token: expiry/scope/doc/tenant/checksum checks + Token-->>API: verified claims + API->>API: resolve zero or one Range + API->>Ledger: record controlled read evidence + API-->>Caller: 200 / 206 / 416 or controlled auth failure +``` + +Protected-main `ArtifactController` already follows this authority pattern. Direct conversion-job download signed-delivery alignment is `IMPLEMENTED_ON_MAIN` on protected `main`; later active work may harden adjacent token/parser or lifecycle boundaries but does not downgrade this shipped maturity. + +## 4. Conversion job state machine + +```mermaid +stateDiagram-v2 + [*] --> SUBMITTED + SUBMITTED --> PROCESSING: worker claim + PROCESSING --> SUCCEEDED: artifact published + PROCESSING --> SUBMITTED: bounded retry scheduled / recovery + PROCESSING --> FAILED: retry exhausted or terminal failure + FAILED --> SUBMITTED: authorized dead-letter retry + SUCCEEDED --> [*] + FAILED --> [*]: terminal until authorized retry/delete +``` + +Dead-letter exhaustion is represented by `FAILED` plus dead-letter evidence rather than a separate public lifecycle enum. + +## 5. Durable deletion recovery (`PARTIAL` target; issue #263; #268 `SUPERSEDED`) + +```mermaid +sequenceDiagram + actor Operator + participant API as Admin/delete boundary + participant Auth as TenantAccessService + participant Receipt as Deletion receipt store + participant Artifact as Lifecycle-fenced artifact store + participant Repo as Job repository + participant Recovery as Recovery coordinator + + Operator->>API: delete job + API->>Auth: require signed tenant + admin/write authority + API->>Receipt: persist pending deletion receipt first + API->>Artifact: read/bind exact generation + checksum + alt artifact readable + Artifact-->>API: bytes/checksum or confirmed absence + API->>Receipt: bind verified digest state + API->>Repo: tombstone exact tenant/job/generation + API->>Artifact: cleanup exact generation + API->>Receipt: mark cleanup completion + else artifact read fails + API->>Receipt: append controlled failed attempt + API-->>Operator: controlled retryable failure + Recovery->>Receipt: rebuild next eligible order after restart + end +``` + +This is `PARTIAL` target behavior, not protected-main durable deletion persistence. Historical PR #268 is `SUPERSEDED`; current bounded successor slices such as #341, #342, #345, #350, #351, #353, #361, and #363 preserve individual authorization, isolation, identity, lifecycle, and audit semantics without making the end-to-end receipt/recovery flow `IMPLEMENTED_ON_MAIN`. + +## 6. Liveness/readiness flow (`ACTIVE_PR` #295) + +```mermaid +flowchart TD + K[Kubernetes / orchestrator] + L[/healthz liveness/] + R[/readyz readiness/] + Process[Process health] + Traffic[Instance accepts traffic] + + K --> L --> Process + K --> R --> Traffic + Process -->|broken| Restart[restart candidate] + Traffic -->|not ready| Remove[remove from routing] + Traffic -->|ready| Route[route traffic] +``` + +Shared dependency health must not create restart cascades by contaminating process liveness. + +## 7. Office conversion adapter isolation (`ACTIVE_PR` #306 / `PLANNED` issue #5) + +PR #306 defines the provider-neutral `OfficeConversionAdapter` contract and deterministic publication-validation boundary as `ACTIVE_PR` evidence. The production Office runtime remains `PLANNED`: a `sandboxed_office_sidecar` or independently operated `remote_office_service` must own Office-process execution **outside the API container**. The `deterministic_fixture_adapter` is an `ACTIVE_PR` offline contract oracle, not a production Office engine. In-process `LocalOfficeManager` inside the Clearfolio API container remains rejected. + +```mermaid +sequenceDiagram + actor Caller + participant API as Clearfolio API container + participant Preflight as Source preflight / policy + participant Adapter as OfficeConversionAdapter - ACTIVE_PR #306 + participant Fixture as deterministic_fixture_adapter - ACTIVE_PR oracle + participant Sidecar as sandboxed_office_sidecar - PLANNED + participant Remote as remote_office_service - PLANNED + participant Publish as PDF publication validator + participant Store as ArtifactStore + + Caller->>API: non-PDF document + tenant/job/generation contract + API->>Preflight: validate source format/container/size/active-content policy + Preflight-->>API: accepted qualified request or fail-closed rejection + API->>Adapter: immutable conversion request + alt deterministic contract test only + Adapter->>Fixture: convert bounded fixture + Fixture-->>Adapter: deterministic PDF result + provenance + else sandboxed production candidate after qualification + Adapter->>Sidecar: authenticated bounded conversion request + Sidecar-->>Adapter: PDF result + engine/runtime evidence + else remote production candidate after qualification + Adapter->>Remote: authenticated bounded conversion request + Remote-->>Adapter: PDF result + engine/runtime evidence + end + Adapter-->>API: provider-neutral result + API->>Publish: validate request/result binding, PDF structure, size/pages, active-action policy + Publish-->>API: accepted artifact or typed failure + API->>Store: publish only accepted output +``` + +The sidecar and remote providers are alternatives, not simultaneous requirements. Their qualification must prove isolation, no inherited application secrets, deny-by-default outbound networking, bounded CPU/RAM/disk/process lifetime, cancellation/restart/cleanup, exact runtime/license/SBOM/provenance, hostile-source handling, and realistic Office fidelity. Neither this diagram nor a successful adapter invocation establishes production Office support; support requires issue #5 release evidence and protected-main integration. + +## 8. Automation authority flow + +```mermaid +flowchart LR + Central[Central .github PR maintenance] + Review[Review / repair / exact-head revalidation] + Merge[Protected merge authority] + + Local[Clearfolio product-development loop - ACTIVE_PR] + RCA[RCA + remedies + feasibility] + Proposal[Bounded path-disjoint Draft proposal] + Verify[Credential-free full verification] + + Central --> Review --> Merge + Local --> RCA --> Proposal --> Verify + Verify -. never self-approve/merge .-> Review +``` + +The central control plane and leaf product-development agent are intentionally separate. A local scheduler must not copy central reviewer/merge credentials. + +## 9. Standalone and MSA deployment topology + +```mermaid +flowchart TB + subgraph Standalone[Standalone deployment] + Browser1[Browser / API client] --> CF1[Clearfolio] + CF1 --> LocalState[Configured local state/artifacts] + end + + subgraph Composed[MSA composition] + Browser2[User client] --> Host[naruon / host service] + Host -->|versioned Clearfolio API| CF2[Clearfolio] + Identity[Host identity / gateway] -->|signed/scoped claims| CF2 + CF2 --> State[Clearfolio-owned state/artifacts] + Orchestrator[contextual-orchestrator] -. optional model routing outside deterministic conversion authority .-> Host + end + + CentralCI[ContextualWisdomLab/.github] -. CI/review control plane .-> CF1 + CentralCI -. CI/review control plane .-> CF2 +``` + +No composition arrow authorizes direct host writes to Clearfolio application persistence. + +## 10. Degraded/failure modes + +```mermaid +flowchart TD + Input[Request / background work] --> Gate{Boundary result} + Gate -->|unauthorized / cross-tenant| Conceal[controlled 401/403/404] + Gate -->|unsupported / malformed / oversized| Reject[controlled fail-closed rejection] + Gate -->|queue saturated| Backpressure[bounded refusal / planned durable backpressure] + Gate -->|conversion terminal failure| Failed[FAILED + retry/dead-letter evidence] + Gate -->|artifact unavailable| NoArtifact[controlled not-found/not-ready] + Gate -->|valid| Continue[continue workflow] + Failed --> Recovery[authorized retry / restart recovery] +``` + +## 11. Scheduler execution receipt and resumable continuation (`PLANNED`; issue #331) + +This diagram models the contract from ADR-0012. It does not claim that the external scheduler platform already stores every receipt or that Draft PR #271 is protected-main behavior. + +```mermaid +sequenceDiagram + participant Clock as Schedule + participant Control as External scheduler / admission + participant GitHub as Live GitHub authority + participant Queue as Fresh queue + participant Writer as Single Clearfolio writer + participant Receipt as External action receipt store + + Clock->>Control: schedule recurrence + Control->>Control: admission + run identity + Control->>GitHub: refetch main, PRs/issues, reviews/checks, blobs/refs, writers + GitHub-->>Control: exact live evidence + Control->>Queue: construct fresh queue + Queue-->>Control: next bounded atomic action + Control->>Writer: execute exact-head/blob/ref-bound action + alt action succeeds at stable boundary + Writer-->>Receipt: action receipt + exact proof + Receipt-->>Control: next action permitted + Control->>GitHub: refetch affected state + else practical budget nearly exhausted + Writer-->>Receipt: last safe checkpoint + Receipt-->>Control: budget continuation + Note over Control,GitHub: Next run must rebuild a fresh queue; checkpoint SHAs are historical + else observed execution failure + Writer-->>Receipt: controlled failure envelope + last safe checkpoint + Receipt-->>Control: continuation handoff or branch-local defer + Note over Control,Receipt: Generic scheduled-task error remains a symptom; do not invent hidden cause + end +``` + +### Receipt state machine + +```mermaid +stateDiagram-v2 + [*] --> SCHEDULED: schedule + SCHEDULED --> ADMITTED: admission + ADMITTED --> QUEUE_BUILT: fresh queue + QUEUE_BUILT --> ACTION_RUNNING: atomic action + ACTION_RUNNING --> ACTION_PROVEN: action receipt + ACTION_PROVEN --> QUEUE_BUILT: next safe item + ACTION_PROVEN --> COMPLETED: two fresh exit sweeps + ACTION_RUNNING --> BUDGET_CONTINUATION: verified safe checkpoint + ACTION_RUNNING --> FAILURE_ENVELOPE: observed controlled failure + BUDGET_CONTINUATION --> [*]: continuation handoff + FAILURE_ENVELOPE --> [*]: continuation handoff / local defer + COMPLETED --> [*] +``` + +`BUDGET_CONTINUATION` is not product completion. `FAILURE_ENVELOPE` is not a source-code finding or a guessed root cause. The next invocation independently resolves every live GitHub identity before another mutation. + +## Related detailed diagrams + +Existing dated/feature diagrams remain useful detailed views: + +- `docs/diagrams/submit-flow.md` +- `docs/diagrams/submit-policy-adapter-flow.md` +- `docs/diagrams/status-flow.md` +- `docs/diagrams/preview-flow.md` +- `docs/diagrams/retry-deadletter-flow.md` + +When a detailed diagram conflicts with this canonical index or protected code, it must be updated or explicitly marked historical. diff --git a/docs/adr/0001-standalone-msa-ownership.md b/docs/adr/0001-standalone-msa-ownership.md new file mode 100644 index 000000000..b78f117a7 --- /dev/null +++ b/docs/adr/0001-standalone-msa-ownership.md @@ -0,0 +1,48 @@ +# ADR-0001: Preserve standalone Clearfolio and explicit MSA ownership boundaries + +Status: Accepted +Implementation maturity: `PARTIAL` / `ACCEPTED_ARCHITECTURE` + +## Context and drivers + +Clearfolio must work as an independently deployable document-conversion/viewing service while also composing with naruon and other CWL hosts. Hidden runtime coupling or cross-service database access would make deployment, security review, rollback and acquisition diligence harder. + +## Alternatives + +1. Make Clearfolio a naruon-internal module with shared persistence. +2. Keep Clearfolio standalone but allow hosts to read/write its application database. +3. Keep Clearfolio independently operable and compose only through explicit versioned contracts. + +## Decision + +Choose alternative 3. Clearfolio owns document intake, conversion lifecycle, artifact-delivery rules, local authorization enforcement, its state interfaces and its evidence. Hosts may own user-facing orchestration, upstream identity exchange, deployment composition and product workflow, but communicate through explicit APIs/contracts. No host obtains implied authority to mutate Clearfolio persistence. + +## Consequences + +Clearfolio requires stable public/service interfaces and replaceable adapters. Some data may be duplicated at service boundaries, but ownership and rollback remain understandable. Central `.github` automation is a development control plane, not a runtime dependency. + +## Failure and recovery + +If a host integration is unavailable, standalone Clearfolio behavior remains testable and deployable. Host-specific failures must not corrupt Clearfolio state. Integration retries and idempotency are contract concerns rather than cross-database repair. + +## Security and privacy + +Tenant/actor authority must cross the service boundary through explicit trusted claims or tokens with least privilege. Hosts do not bypass Clearfolio tenant checks. Data minimization applies to integration payloads. + +## Compatibility and migration + +Existing HTTP flows remain valid **except where an accepted security contract explicitly requires a versioned tightening**. The direct conversion-job download route is such an exception: permission-only byte access is not the final contract and must converge on the signed artifact-delivery authority defined by `docs/API_CONTRACT.md` and ADR-0002. Existing direct-download clients therefore require an explicit migration rather than an assertion that their current request shape remains indefinitely valid. + +Future naruon contracts are versioned and introduced additively or with explicit migration. A move to shared persistence would require a superseding ADR and a tenant/data-ownership migration plan. + +## Tests and acceptance + +- standalone application startup and API tests; +- versioned integration contract tests when naruon composition stabilizes; +- tenant authorization tests at the service boundary; +- direct-download consumer tests that prove the signed-delivery migration and controlled failure semantics; +- no cross-service application-DB dependency in deployment manifests. + +## Rollback / supersession + +Individual adapters may be rolled back without changing service ownership. Security-contract migration must not roll document-byte delivery back to permission-only access. Supersede this ADR only if product ownership itself changes and the new architecture documents data authority, migration and rollback. diff --git a/docs/adr/0002-tenant-artifact-authorization.md b/docs/adr/0002-tenant-artifact-authorization.md new file mode 100644 index 000000000..430916628 --- /dev/null +++ b/docs/adr/0002-tenant-artifact-authorization.md @@ -0,0 +1,59 @@ +# ADR-0002: Separate tenant authorization from signed artifact-delivery authority + +Status: Accepted +Implementation maturity: `IMPLEMENTED_ON_MAIN` for canonical artifact route; direct-download alignment is `ACTIVE_PR` + +## Context and drivers + +A caller that may read job metadata should not automatically receive document bytes. Tenant permission and a signed artifact token protect different threats: the former authorizes the actor within a tenant, while the latter binds a short-lived read to a specific document, artifact checksum, purpose, scope, issuance record and revocation state. + +## Alternatives + +1. Authorize artifact bytes with `job:read` only. +2. Authorize bytes with `artifact:read` only. +3. Require endpoint-appropriate tenant permission plus the signed artifact-delivery contract, with cross-tenant concealment. + +## Decision + +Choose alternative 3. `ArtifactLinkService` is the signed-delivery authority. Artifact token verification covers signature, expiry, read scope, document binding, issued-token ledger state, revocation, tenant ownership and artifact checksum. Canonical delivery supports only a single HTTP byte range and records controlled read-audit evidence. Direct-download convenience routes must not bypass these controls. + +## Consequences + +Clients first obtain or receive a valid short-lived artifact link/token. Additional requests and token lifecycle complexity are accepted in exchange for revocation, purpose binding, checksum binding and auditability. + +## Failure and recovery + +Missing/expired/unknown/revoked/mismatched tokens fail closed. Cross-tenant resources remain concealed. Invalid or unsatisfiable range requests return controlled range failure without silently serving whole bytes. A new link can be issued after legitimate expiry/revocation according to authorization policy. + +## Security and privacy + +Raw tokens are secrets and are never logged or persisted as ordinary audit content. Read events contain controlled identifiers/trace/range/status only. Dedicated artifact permission is least privilege but is not itself a substitute for signed token verification. + +## Compatibility and migration + +Viewer links already use the signed route. Direct-download callers that previously relied only on tenant permission must migrate through an explicit versioned client/contract change when the active remediation integrates; permission-only behavior is not preserved as a compatibility fallback. + +The rollout order is: + +1. make signed artifact-link issuance and canonical token verification available to every supported client; +2. update client/consumer contract tests to send a valid signed token and to handle the documented `401`/`403`/`404`/`409`/`416` failure semantics without probing resource existence; +3. deploy the direct-download implementation that applies the same signature, expiry, scope, document, tenant, checksum, issued-ledger, revocation, single-Range and read-audit authority; +4. remove any legacy permission-only expectation only after the migrated clients are proven against the integrated contract. + +Rollback may revert a client rollout while the old server behavior is still intentionally available during a bounded migration window, but once the signed-delivery enforcement is the protected contract, rollback must not re-enable permission-only byte access. Any incompatible response-shape or endpoint-semantic change requires an explicit API version/migration note and consumer tests. + +## Tests and acceptance + +- missing permission and cross-tenant concealment; +- missing/malformed/expired/scope/doc/checksum/ledger token rejection; +- revocation rejection; +- full and single-range reads; +- multi-range/invalid/unsatisfiable controlled failure; +- versioned client/contract tests for migrated direct-download success and failure responses; +- rollout compatibility tests proving both canonical signed delivery and migrated direct-download semantics before legacy assumptions are retired; +- read-audit evidence; +- exact-head security/fuzz/coverage gates. + +## Rollback / supersession + +Do not roll back to permission-only document-byte access. A successor mechanism must preserve equivalent or stronger short-lived binding, revocation, integrity, tenant and audit properties. diff --git a/docs/adr/0003-audit-pseudonymization-key-separation.md b/docs/adr/0003-audit-pseudonymization-key-separation.md new file mode 100644 index 000000000..d25413106 --- /dev/null +++ b/docs/adr/0003-audit-pseudonymization-key-separation.md @@ -0,0 +1,66 @@ +# ADR-0003: Use purpose-separated, domain-separated audit pseudonymization + +Status: Proposed +Implementation maturity: `IMPLEMENTED_ON_MAIN` through protected #270; #268 is `SUPERSEDED` for remaining lifecycle work + +## Context and drivers + +Audit evidence needs stable correlation without writing raw approver/operator identifiers or approval tokens. Reusing a policy-signing key for audit pseudonymization couples unrelated purposes and increases blast radius. Plain unsalted hashing of low-entropy identities is not an adequate pseudonymization boundary. + +## Alternatives + +1. Store raw identifiers in audit events. +2. Store plain SHA-256 identifiers. +3. Use a versioned domain-separated HMAC with an audit-only key and retain raw identity only at its authorized source. + +## Decision + +Choose alternative 3. Audit pseudonyms use **HMAC-SHA-256** with an audit-only key and an explicit, versioned domain/context prefix. Clearfolio policy requires at least **32 uniformly random bytes (256 bits of generated key material)** for a production audit-pseudonymization key; this is an application policy chosen to match the SHA-256 security target rather than a claim that NIST mandates that exact byte count. A human password, low-entropy phrase, reused signing secret, or truncated identifier is not acceptable key material. + +Production key material must be generated by an approved cryptographically secure generator and held by an authenticated secret manager/KMS or equivalent credential registry that enforces access control, auditability and purpose separation. Environment variables may be bounded bootstrap transport where deployment policy permits, but raw process environment is not the accepted long-term production key authority. Audit-pseudonym, artifact-token, tenant-claim, policy-signing, model and repository-automation credentials are distinct purposes and must not share key material. + +Length is necessary but not sufficient evidence of acceptable key generation. Clearfolio must not infer unpredictability from a 32-byte value alone: production configuration also requires an approved provenance classification bound to the loaded credential, such as CSPRNG-generated material registered by the credential authority or a KMS-generated/held key. Known placeholders, repeated-byte values and deterministic low-entropy fixtures are explicit negative cases, and a credential whose approved provenance cannot be established fails closed. This provenance rule avoids pretending that arbitrary key bytes can prove their own randomness. + +Pseudonym records carry a key/version identifier such as `v1` plus the domain context. New writes use only the currently active version. During a bounded rotation/retention window, authorized readers may recognize explicitly configured predecessor versions to interpret retained evidence; they must not silently downgrade new writes, accept an unconfigured version, or rewrite historical pseudonyms without a privacy-reviewed migration. Once retention/rollback requirements for a predecessor expire, that key version is disabled and destroyed according to credential policy. Pseudonymized values remain personal data and remain subject to access/retention controls. + +## Consequences + +Cross-event correlation is possible within the intended domain, while direct identifiers are omitted from ordinary audit output. Key rotation/versioning, authenticated credential storage and purpose-specific configuration become operational requirements. + +## Failure and recovery + +Missing, shorter-than-policy, non-random/placeholder, provenance-unverified, improperly shared, unavailable or unrecognized-version required keys fail closed in production-configured paths. Rotation preserves version-aware interpretation without requiring raw historic identifiers to be rewritten. Recovery restores an authorized key version only from the governed credential backup/recovery mechanism; application logs or audit payloads are never a recovery source. + +## Security and privacy + +Do not log raw approval tokens, raw subject/tenant identifiers where pseudonymized evidence suffices, signatures, uncontrolled exception text, filenames/paths/digests in low-cardinality telemetry, or secret material. HMAC pseudonyms are not anonymization. + +## Compatibility and migration + +Existing raw/plain-hash evidence must not be silently relabeled. New evidence records the pseudonymization version/domain. Any migration of retained legacy evidence requires a privacy review. Key rotation is additive first: introduce and verify the new version, switch new writes, retain the prior version only for the bounded read/rollback window, then retire it. + +## Tests and acceptance + +- production key-strength/provenance acceptance rejects fewer than 32 bytes, known placeholder/default material, repeated-byte 32-byte values, other deterministic low-entropy fixtures, and any key lacking approved CSPRNG/KMS/credential-registry provenance; a positive fixture proves an approved provenance-bound 32-byte-or-stronger key is accepted; +- purpose-separation test rejects reuse of audit pseudonymization key material by artifact-token, tenant-claim or policy-signing authority; +- deterministic same-domain correlation and different-domain non-equality; +- version test accepts only explicitly configured active/predecessor versions and proves new writes never downgrade; +- rotation regression proves active-version switch, bounded predecessor reads and final predecessor retirement; +- credential configuration test proves production fail-closed behavior when the authenticated key authority is unavailable; +- no raw identifier/token/key logs; +- standalone/Spring fail-closed construction; +- exact-head owned production statement/branch coverage and security review. + +## Rollback / supersession + +Rollback may disable a new audit consumer but must not restore raw identifier logging or an already-retired weak/shared key. Supersede only with a privacy-reviewed construction having equal or stronger separation, key-management and rotation semantics. + +## References + +National Institute of Standards and Technology. (2008). *The keyed-hash message authentication code (HMAC) (FIPS PUB 198-1).* https://doi.org/10.6028/NIST.FIPS.198-1 + +Barker, E. (2020). *Recommendation for key management: Part 1—General (NIST Special Publication 800-57 Part 1 Revision 5).* National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-57pt1r5 + +Sönmez Turan, M., & Brandão, L. T. A. N. (2024). *Keyed-hash message authentication code (HMAC): Specification of HMAC and recommendations for message authentication* (NIST Special Publication 800-224, Initial Public Draft). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-224.ipd + +SP 800-224 remains a draft at this ADR revision; it is tracked for transition planning and does not supersede the current final NIST HMAC/key-management authorities until NIST publishes the final replacement. diff --git a/docs/adr/0004-durable-lifecycle-generation-fencing.md b/docs/adr/0004-durable-lifecycle-generation-fencing.md new file mode 100644 index 000000000..ab62894f2 --- /dev/null +++ b/docs/adr/0004-durable-lifecycle-generation-fencing.md @@ -0,0 +1,44 @@ +# ADR-0004: Fence durable job/artifact deletion by immutable lifecycle generation + +Status: Proposed +Implementation maturity: `PARTIAL` target; #268 is `SUPERSEDED`; bounded successors remain unintegrated + +## Context and drivers + +Deletion and retry span repository metadata, artifact bytes and process restart. A job UUID alone is unsafe if stale work can act on a later replacement or if a failed artifact read is misclassified as confirmed absence. Process-local recovery cursors cannot provide durable fairness after restart. + +## Alternatives + +1. Best-effort delete artifact then metadata with process-memory retries. +2. Use the UUID alone as the deletion identity. +3. Permanently reserve identity, bind operations to tenant + immutable lifecycle generation, and persist a deletion receipt before artifact inspection. + +## Decision + +Choose alternative 3. Durable mutation uses tenant/job/generation fences. A deletion receipt is persisted before the first artifact read, advances only through validated transitions, binds the exact artifact digest or explicit confirmed-absence state before metadata tombstoning, and records retryable controlled failure evidence. Recovery ordering is reconstructed from durable transition times/evidence rather than a process-local cursor. + +## Consequences + +Lifecycle logic and append-only validation become more complex, but stale work, UUID rebinding, restart starvation and ambiguous deletion outcomes become detectable and fail closed. + +## Failure and recovery + +Unreadable artifact bytes are not confirmed absence. Failed attempts retain metadata and append controlled retry evidence. Torn ledger tails, invalid UTF-8, identity conflicts, fabricated initial records, illegal successor states and inconsistent attempt evidence are rejected or handled under documented crash-tail rules. + +## Security and privacy + +Deletion evidence avoids raw sensitive content and exception-selected log data. Generation fences protect cross-tenant/cross-lifecycle integrity as well as availability. + +## Compatibility and migration + +Standalone local ledgers remain valid deployment choices. Distributed SQL/object-store adapters must implement equivalent atomicity/fencing or explicitly reject unsupported operations. Existing identifiers are not recycled. + +## Tests and acceptance + +Crash/restart replay, permanently failing oldest receipt, bounded recovery fairness, generation mismatch, duplicate identity, initial-read failure, cleanup failure, torn-tail validation, privacy logging, concurrency serialization, and **exact owned production statement/branch coverage with JaCoCo line-missed = 0 and branch-missed = 0** for the lifecycle code introduced by the implementing change. + +Coverage is structural evidence only; the lifecycle acceptance tests above must prove the state, concurrency, recovery and tenant-generation invariants independently. + +## Rollback / supersession + +Do not roll back to UUID reuse or process-only fairness. A successor distributed transaction/outbox design may supersede this ADR only after equivalent generation, idempotency and recovery guarantees are proven. diff --git a/docs/adr/0005-deterministic-conversion-fidelity.md b/docs/adr/0005-deterministic-conversion-fidelity.md new file mode 100644 index 000000000..bef0bcd8a --- /dev/null +++ b/docs/adr/0005-deterministic-conversion-fidelity.md @@ -0,0 +1,91 @@ +# ADR-0005: Separate deterministic supported conversion from demo placeholder output + +Status: Accepted +Implementation maturity: protected-main PDF passthrough `IMPLEMENTED_ON_MAIN`; non-PDF production conversion `PLANNED` + +## Context and drivers + +Protected main can pass through validated PDFs and can generate a placeholder one-page PDF for non-PDF sources. Calling that placeholder a DOCX/HWP/Office conversion would create a false fidelity claim and an unsafe release boundary. Enterprise buyers need reproducible evidence for what survives conversion. + +Office conversion engines also process highly complex attacker-controlled formats through native and parser-heavy runtimes. A converter crash, memory leak, parser exploit, macro/active-content behavior, or external-resource fetch must not inherit the Clearfolio API process/container's filesystem, credentials, network namespace, or tenant authority merely because an adapter library can start an office process locally. + +## Alternatives + +1. Treat any generated PDF as a successful conversion. +2. Use an LLM or visual opinion as the primary fidelity oracle. +3. Classify output as passthrough/transformed/degraded/unsupported/development-placeholder and allow supported-format claims only after deterministic real-fixture acceptance, while running production Office conversion behind an isolated provider-neutral adapter boundary. +4. Embed/start the Office runtime inside the Clearfolio API process/container for operational simplicity. + +## Decision + +Choose alternative 3 and reject alternative 4 for production Office transformation. + +Deterministic converter adapters are the production authority. No uploaded macro/active content is executed, external resource fetching is disabled by default, and unsupported or unverifiable inputs fail closed. LLMs may assist documentation or investigation but do not decide conversion correctness. + +The Clearfolio application depends on a versioned provider-neutral `office_conversion_adapter` contract. A production Office engine is owned by a separately isolated execution boundary, such as a sandboxed sidecar/external process service or separately operated authenticated remote converter. The Office runtime does **not** run inside the Clearfolio API-container trust boundary and does not inherit application secrets or unrestricted network/filesystem authority. + +JODConverter, LibreOffice, Collabora, or another engine may implement that adapter after qualification; none is architectural authority by itself. JODConverter's support for a local manager is a library capability, not approval to start LibreOffice inside the API container. Exact adapter/runtime versions, images, fonts, codecs, dictionaries and licenses are pinned and reviewed as release materials rather than floated to a moving “latest” version. + +## Consequences + +The supported-format list grows more slowly and deployment gains an explicit converter boundary, but product claims and blast-radius controls become defensible. Realistic Office/PDF fixtures, expected outcomes, converter/version/configuration provenance and sandbox policy become release artifacts. + +The service interface remains replaceable for standalone and naruon/MSA composition. Hosts integrate through Clearfolio's versioned API/service boundary rather than gaining direct access to converter temporary storage or Clearfolio persistence. + +## Failure and recovery + +A converter crash, timeout, malformed container, unsupported construct, missing required renderer dependency, sandbox exhaustion, network-policy violation or fidelity-gate failure produces controlled failure evidence; it never silently returns a placeholder as a supported conversion. + +The current public lifecycle remains `SUBMITTED`, `PROCESSING`, `SUCCEEDED`, and `FAILED`. **Degraded conversion evidence is a reason/profile attached to the public `FAILED` terminal state, not a fifth terminal state.** A degraded result cannot create a supported-format claim and cannot be published as a successful converted artifact. Introducing a separate public degraded state would require an explicit versioned API change, migration note, client/consumer contract tests, and corresponding state-machine/release updates. + +Cancellation/timeout must terminate or quarantine the exact converter process/job generation. A later job must not inherit its temporary profile or files. Retry is bounded and limited to explicit transient categories; unsupported, policy-denied, password-protected or structurally rejected inputs are not retried as transient failures. + +Converter/runtime rollback follows the realistic fixture and provenance contract in `docs/FIDELITY_ACCEPTANCE.md` and `docs/MIGRATION_ROLLBACK.md`; an insecure old converter is not restored solely for output compatibility. + +## Security and privacy + +Production converter execution uses bounded CPU/memory/time/process count/temp storage, a non-root identity, least filesystem permissions, no inherited API/cloud credentials, no Docker/host control socket, and deny-by-default outbound network access. Seccomp/AppArmor/SELinux or equivalent platform controls are applied where supported. + +Inputs are validated before conversion for extension/MIME/container policy, active content, encryption/password state, malformed structures, archive depth/decompression expansion and configured size/object limits. Outputs are published only after PDF/media-type/size/page/integrity validation and applicable content/security checks. + +No unreviewed macros/scripts execute. No implicit external links/resources are followed. Temporary files are app-owned to the converter boundary, bounded, and cleaned after success, failure, cancellation or crash. Fixtures used in public CI must be redistributable and free of confidential data. + +## Supply-chain and license boundary + +Adapter and runtime qualification are separate: + +- Java adapter license compatibility does not prove the complete Office runtime/container is redistributable or secure; +- generate and verify SBOM/provenance/attribution for the exact converter runtime and the Clearfolio client independently; +- pin images by digest and dependencies by reviewed version/BOM; +- record the exact office runtime, fonts, codecs, dictionaries and native packages that affect output; +- document vulnerability response, rebuild/rollback and emergency converter-disable procedures. + +Issue #5 is the executable product qualification record. `docs/RESEARCH_TRACEABILITY.md` records the current official JODConverter manager/configuration sources and LibreOffice licensing source without turning a transient current version into a timeless pin. + +## Compatibility and migration + +Existing PDF passthrough remains compatible. Non-PDF placeholder behavior remains labelled development/demo until replaced. New converter adapters are introduced through a versioned support matrix and fixture suite. + +Moving from placeholder behavior to transformed Office support is a product-contract expansion, not a transparent implementation swap: API/UI/support documentation must expose unsupported/degraded/failure semantics accurately during migration. + +## Tests and acceptance + +For each claimed format require: + +- authorized/redistributable realistic fixtures; +- structure/text/images/tables/fonts/page expectations and controlled visual comparison; +- macro/active-content/external-link/password/malformed/container-expansion security fixtures; +- deterministic or explicitly bounded output/provenance comparison; +- sandbox no-secret/no-network/no-host-control assertions; +- CPU/memory/time/process/temp-storage pressure tests; +- cancellation, crash, cleanup, process-recycle, retry-category and stale-generation tests; +- degraded-output tests proving `FAILED`, no fifth public state and no supported-format claim; +- license/SBOM/provenance evidence for the exact adapter/runtime image; +- accessibility/print/export checks where user-visible; +- exact-head CI/security/fuzz/review and integrated release acceptance. + +A successful converter process exit alone is never a production support oracle. + +## Rollback / supersession + +A converter version may be rolled back only to a release that still satisfies current security, compatibility and fixture requirements. Rollback must not reintroduce a known sandbox or authorization weakness. Supersede this ADR only if a new authority can provide equal or stronger deterministic fidelity, isolation, privacy, supply-chain and recovery evidence. diff --git a/docs/adr/0006-liveness-readiness-separation.md b/docs/adr/0006-liveness-readiness-separation.md new file mode 100644 index 000000000..2bed2fa90 --- /dev/null +++ b/docs/adr/0006-liveness-readiness-separation.md @@ -0,0 +1,73 @@ +# ADR-0006: Separate process liveness from traffic readiness + +Status: Proposed +Implementation maturity: `ACTIVE_PR` #295 + +## Context and drivers + +An orchestrator needs two different answers: whether the Clearfolio process should be restarted and whether this instance can safely receive user traffic. Combining these signals can send traffic to an unready process or create restart cascades when a shared dependency is unhealthy. + +## Alternatives + +1. Keep one generic health endpoint for all conditions. +2. Make liveness depend on all shared dependencies. +3. Expose controlled liveness and readiness separately; keep readiness contributors instance-local and deterministic. + +## Decision + +Choose alternative 3. `/healthz` represents process liveness and `/readyz` represents traffic readiness. Responses expose only controlled labels, use `Cache-Control: no-store`, and do not disclose tenant, topology, credentials, queue details, dependency names, build identifiers or exception text. + +### Probe access boundary + +The endpoints are unauthenticated **only on the orchestration probe path**: + +- HTTP method: `GET` only; mutation methods are not probe contracts; +- allowed network sources: same-instance/loopback health tooling and the deployment platform's explicitly configured node/orchestrator probe source set; +- Kubernetes/container deployments must route those sources over the internal workload/control-plane path and apply NetworkPolicy, security-group, service-mesh or equivalent controls where the platform supports source restriction; +- the public/user application ingress must not publish `/healthz` or `/readyz`; ingress/gateway route tests must prove they are absent from the public route table; +- if a deployment cannot keep a probe endpoint off a broader network path, that broader path requires the deployment's normal service authentication/authorization rather than inheriting the unauthenticated probe exception; +- no query parameter or request header enables diagnostic expansion. + +A deployment manifest is non-compliant if an Internet/public ingress can reach either unauthenticated probe merely because the application controller exposes it. + +## Consequences + +Deployment manifests and runbooks must configure two probes plus their network boundary. Operational semantics become clearer and dependency incidents are less likely to amplify through unnecessary restarts. + +## Failure and recovery + +Liveness failure makes restart eligibility visible. Readiness failure removes an instance from routing without asserting the process is dead. Recovery returns readiness after the deterministic contributor recovers. + +## Security and privacy + +Probe endpoints remain low-information and never become diagnostic-detail endpoints. The unauthenticated exception is limited to the controlled probe sources above; it is not a general anonymous HTTP API policy. + +## Compatibility and migration + +Existing `/healthz` clients may currently interpret it as a generic/readiness signal, so the meaning must not be switched in place before consumers move. + +Rollout order is mandatory: + +1. deploy `/readyz` while preserving the pre-migration `/healthz` behavior expected by existing clients; +2. deploy internal-only probe routing/network policy for both endpoints and verify public ingress cannot reach them anonymously; +3. migrate orchestrator readiness probes and every known readiness consumer to `/readyz` while compatibility tests exercise both endpoints; +4. confirm no supported readiness consumer still depends on old `/healthz` readiness semantics; +5. only then switch `/healthz` to the process-liveness contract or retire any obsolete readiness behavior under an explicit compatibility note. + +Before step 5, rollback restores the previous application/manifests and keeps the old `/healthz` semantics so migrated and non-migrated clients can be recovered safely. After step 5 is accepted on protected main, rollback must restore a release whose endpoint semantics and client manifests are a compatible pair; independently reverting only the server or only the probes is prohibited. + +## Tests and acceptance + +- startup, broken/recovered liveness and not-ready/ready transitions; +- `GET` probe behavior and rejection/absence of mutation routes; +- cache headers and no diagnostic leakage; +- deployment-contract tests proving only loopback/platform probe sources have anonymous network reachability; +- ingress/gateway tests proving neither probe is anonymously exposed on the public application ingress; +- compatibility tests covering old `/healthz` behavior plus new `/readyz` during migration; +- migration tests proving readiness consumers move before `/healthz` semantics change; +- rollback tests that restore a mutually compatible server/probe pair; +- exact-head CI/security gates. + +## Rollback / supersession + +Rollback may restore the previous compatible deployment while investigating an availability regression, but a successor design must preserve the semantic distinction and the probe-network boundary rather than call dependency readiness process liveness or expose diagnostics anonymously. diff --git a/docs/adr/0007-exact-head-live-base-evidence.md b/docs/adr/0007-exact-head-live-base-evidence.md new file mode 100644 index 000000000..3d85840f0 --- /dev/null +++ b/docs/adr/0007-exact-head-live-base-evidence.md @@ -0,0 +1,46 @@ +# ADR-0007: Bind merge evidence to exact source head and current live base + +Status: Proposed +Implementation maturity: policy accepted; CI enforcement strengthened in `ACTIVE_PR` #270 + +## Context and drivers + +PR bodies, remembered SHAs, earlier workflow runs and GitHub PR base snapshots can become stale while branches move. Stacked PRs amplify this risk. A predecessor-head green check cannot establish current-head correctness, and a synthetic merge result does not replace source-head evidence. + +## Alternatives + +1. Trust the latest PR body/check narrative. +2. Trust any successful run on the same PR number. +3. Refetch the exact source head and independently resolve the current base-ref tip; classify every check/review/status by the revision it actually covers. + +## Decision + +Choose alternative 3. Every review/fix/merge cycle begins from live source-head and live-base identity. Required source behavior is verified on exact head; base-sensitive compatibility is verified separately. Any relevant head/base movement invalidates predecessor assumptions and triggers revalidation. + +A protected merge has two atomic identity preconditions at merge time: the **expected source-head SHA** and the **supported base-tip SHA**. A merge path is compliant only when the merge operation atomically binds or compares both exact values, for example through a compare-and-swap-equivalent integration primitive. If either exact SHA differs from the values used for the final acceptance decision, the operation must reject the stale attempt and require fresh-state revalidation from newly fetched head/base/review/check state. A platform or tool path that can bind only the source head, or that relies on live-base refetch plus ancestry/ruleset checks without exact base-tip equality at the merge boundary, is not an authorized merge path under this ADR and must be rejected rather than treated as sufficiently atomic. + +This requirement deliberately separates read-side preparation from write authority. A final refetch can reduce uncertainty, but it cannot close the refetch-to-merge TOCTOU window by itself. If the available merge interface cannot enforce both exact identity preconditions, automation may continue read-only review, validation, and preparation while another governed integration mechanism is selected; it must not infer atomicity from timing proximity. + +## Consequences + +More reruns may be necessary after branch movement, but stale evidence cannot silently authorize integration. PR prose becomes narrative rather than authority. Some otherwise convenient merge APIs may be unusable until the repository has an exact-head/exact-base compare-and-swap-equivalent integration path. + +## Failure and recovery + +Queued, pending, skipped-required, cancelled, absent, stale-head, predecessor-head, synthetic-only, status-only and failed evidence is non-passing. If a branch moves during a write or refetch-to-merge operation, the writer freezes that attempt and reconciles from fresh state instead of racing. Lack of a two-SHA atomic merge primitive is a merge-path capability gap, not permission to weaken the precondition. + +## Security and governance + +Formal GitHub review, commit status, model verdict, check run and workflow evidence remain separate authority classes. COMMENTED/model/status evidence never becomes independent human approval by wording alone. + +## Compatibility and migration + +Stacked branches must be reconciled after their parent integrates or moves materially. No review/check evidence transfers automatically to a replacement head. + +## Tests and acceptance + +Workflow contract tests assert job-scoped exact-head checkout and synthetic-merge behavior; pre-merge logic verifies current head/base; exact-head CI/security/fuzz and live review state are refetched immediately before integration. Merge-control acceptance must also cover the refetch-to-merge race: after the candidate snapshot, move either the exact source-head SHA or the exact supported base-tip SHA and prove the merge operation rejects the stale attempt because exact SHA equality no longer holds. An ancestry-compatible but different base tip must still be rejected. The next attempt begins only after fresh-state revalidation and reacquisition of evidence affected by the changed identity. + +## Rollback / supersession + +Do not roll back to PR-number-based evidence reuse or refetch-plus-ancestry pseudo-atomicity. A successor may optimize evidence caching only if cryptographically/revision-bound invalidation preserves the same correctness properties and the final merge operation still binds both exact identities atomically. diff --git a/docs/adr/0008-central-vs-local-automation-authority.md b/docs/adr/0008-central-vs-local-automation-authority.md new file mode 100644 index 000000000..d355939ab --- /dev/null +++ b/docs/adr/0008-central-vs-local-automation-authority.md @@ -0,0 +1,48 @@ +# ADR-0008: Separate central PR-maintenance authority from local product-development authority + +Status: Proposed +Implementation maturity: central dependency plus `ACTIVE_PR` #271 + +## Context and drivers + +PR review/repair/merge automation needs privileged organization governance, while buyer-gap development benefits from repository-specific context. Copying privileged central workflows into each product creates duplicated policy, competing writers, excess credentials and inconsistent review semantics. + +## Alternatives + +1. Give one local scheduler authority to develop, review, approve and merge. +2. Copy the central maintenance implementation into Clearfolio. +3. Keep privileged PR maintenance in `ContextualWisdomLab/.github` and allow a bounded Clearfolio-local product agent to propose only path-disjoint Draft work. + +## Decision + +Choose alternative 3. Central PR-maintenance automation owns organization-governed review dispatch, gate evaluation and protected merge semantics; it does not grant a general fleet product-development writer a second Clearfolio source lease. + +The dedicated Clearfolio-local OpenCode development loop is the **only actor permitted to process `ACTIVE_PR` product-development source mutations** while it holds the Clearfolio writer lease. General **central fleet loops keep Clearfolio disabled** for product-development writes and must not race that lease. Central automation may perform read-only review and gate evaluation, and may merge commits produced by the leased local writer when the merge path itself satisfies repository governance. A central repair that mutates Clearfolio source is permitted only after the local source writer is released or fenced and the central action successfully acquires the **same Clearfolio writer lease** before mutation. Without that lease transfer, central automation is not a source-repair writer. + +Lease transfer is a state transition, not a timing convention: the central action must refetch current head/base, prove the prior writer is fenced or released, acquire the same lease identity, and revalidate the target blob/ref before its first mutation. If any of those preconditions cannot be proven, it remains read-only for Clearfolio. This aligns central repair authority with ADR-0009's branch-local active-writer freeze instead of creating a repair exception that can overlap the local writer. + +The local OpenCode execution path is supply-chain constrained: its agent/action source is **immutable and pinned** to a reviewed full commit/digest rather than a floating branch or tag; dependency/bootstrap artifacts are integrity-verified and fail closed when provenance cannot be established. The local agent may diagnose, test and propose bounded changes, but does not self-approve, weaken protection, merge, release or deploy. Independent reviewer credentials and authorities stay separate from proposal credentials. + +## Consequences + +Central defects are fixed centrally rather than hidden by leaf workarounds. Local product development can continue during unrelated approval/check latency without receiving merge authority. A single product-development writer lease prevents duplicate mutation while retaining organization-level review/merge governance. Central source repair has an explicit lease-handoff cost rather than an implicit privileged bypass. + +## Failure and recovery + +If central maintenance is unavailable, local product work may continue only within its safe proposal/verification boundary. If the local agent lacks model/publication credentials, cannot verify its pinned agent/dependency provenance, or detects path/ref ownership conflict, it fails closed for that action and rotates to another safe task rather than broadening credentials. If a competing writer is detected, the affected branch is frozen for the run and refetched later. A failed central lease handoff leaves central maintenance read-only; it does not authorize a best-effort competing repair. + +## Security and privacy + +Model-backed local development uses GitHub Secret `NVIDIA_NIM_API_KEY` only at the model step, never `COPILOT_GITHUB_TOKEN` as a development-model credential. Credential-bearing proposal execution is separated from credential-free verification and short-lived scoped publication authority. Floating agent/action references and blanket secret inheritance are not accepted substitutes for reviewed immutable sourcing and explicit secret contracts. + +## Compatibility and migration + +Leaf callers/contracts may evolve with the central reusable workflow, but central implementation remains the PR-maintenance authority. Reviewer credential names/scopes are not casually changed by product feature work. Enabling another Clearfolio product-development writer requires an explicit successor ADR that replaces, rather than silently overlaps, this single-writer lease. + +## Tests and acceptance + +Scheduler contract tests; Clearfolio-disabled assertions for general fleet product-development loops; path-overlap and protected-path refusal; immutable pinned agent/action and patch/source checks; credential-free full verification; short-lived scoped publication; live writer-lease refetch before mutation; negative test proving central source mutation is rejected while the local writer lease remains active; handoff test proving central mutation is accepted only after release/fencing plus acquisition of the same lease and a fresh head/base/blob revalidation; central protected-main operational acceptance before claiming central repair closure. + +## Rollback / supersession + +Disable the local proposal loop without affecting ordinary product runtime. Supersede only if a reviewed control-plane architecture preserves writer separation, reviewer independence, immutable supply-chain sourcing and least privilege. diff --git a/docs/adr/0009-work-conserving-rca-loop.md b/docs/adr/0009-work-conserving-rca-loop.md new file mode 100644 index 000000000..d98d169a2 --- /dev/null +++ b/docs/adr/0009-work-conserving-rca-loop.md @@ -0,0 +1,70 @@ +# ADR-0009: Make autonomous development work-conserving and action-oriented + +Status: Proposed +Implementation maturity: scheduler/prompt policy in progress; `ACTIVE_PR` #271 contains repository workflow contract + +## Context and drivers + +A recurring automation failure mode is to diagnose one blocker, report it, and terminate while unrelated safe work remains. Review latency, queued checks, missing approval, a failed first remedy, a documentation update, or a single completed PR is not a repository outcome that justifies idling the rest of the invocation. A blocker report or plan alone is not a successful run when another safe executable item remains. + +## Alternatives + +1. Stop after the first blocker or notable result and rely on the next hourly recurrence. +2. Retry the same blocked action repeatedly until it changes. +3. Maintain a live executable queue and use RCA → distinct remedies → empirical feasibility → same-run action → proof, then immediately rotate to the next safe item. + +## Decision + +Choose alternative 3. The hourly recurrence is continuation after real invocation-budget exhaustion, not a reason to stop voluntarily. A blocked action is deferred by exact identity while the loop works elsewhere. Every successful mutation, proof, merge, closure, documentation repair, or defer decision is followed by another queue selection. Two fresh exit sweeps are required before a normal run may terminate for lack of work. + +## Feasibility contract + +For a non-passing state the loop must: + +1. identify the first failing boundary and correction owner; +2. separate symptom, immediate cause, root cause and systemic/control cause where material; +3. enumerate materially distinct bounded remedies; +4. verify actual tool/API support, permissions, credentials, reviewer eligibility, branch/ruleset semantics, dependency ownership, writer lease, runtime/rate-limit state, blast radius, rollback and an exact acceptance test; +5. reject invented authority/secrets/reviewers, weakened gates, stale evidence and duplicate writers; +6. execute the smallest safe root-cause-changing remedy test-first; +7. treat a failed/no-op attempt as new evidence and either try another distinct remedy or rotate. + +## Executable exit-sweep protocol + +Each fresh exit sweep is a new evidence pass, not a replay of cached blocker state. It must: + +1. refetch live head and live base for every still-relevant PR or branch; +2. refetch current reviews, commit statuses, model verdicts, check runs, workflow evidence, required checks, issue/PR state and protected-main state needed to rebuild the executable queue; +3. bind evidence to the current exact PR/head/base/run/review identity, carry forward deferred identities only as exact PR/head/base/run/review references, invalidate evidence not bound to the current exact identity, and then revalidate whether the reason for deferral still exists; +4. revalidate the writer lease before any newly executable mutation; +5. execute any safe item discovered by the sweep and restart the exit-sweep count after that action. + +If a safe item appears between sweep one and sweep two, the loop must not terminate. It must execute or safely defer that item using fresh evidence, then begin the two-sweep exit proof again. Only two consecutive fresh sweeps with no safe executable work establish the normal no-work exit condition. + +## Consequences + +Runs use more of their practical execution budget and produce fewer status-only outcomes. The scheduler needs stronger branch/path ownership checks and a deferred-work set to avoid races and polling loops. + +## Failure and recovery + +When one branch has another active writer, freeze only that branch for the run. When no model/publication credential exists, mark that specific mutation infeasible rather than inventing a credential. When all remaining work is genuinely external-only or unsafe, the exit sweep records that boundary and the next hourly run refetches from scratch. + +## Security and governance + +Work conservation never means bypassing protection, self-approval, force-push, weakening tests, racing writers, or expanding credential scope. Safety and repository policy dominate throughput. + +## Tests and acceptance + +- scheduler contract tests for RCA, feasibility and same-run action; +- path-disjoint ownership and protected-path refusal; +- queue-capacity and credential fail-closed tests; +- exact base/head/patch rechecks before and after publication; +- stale-evidence regressions for every ADR-0007 authority class: current reviews, commit statuses, model verdicts, check runs, workflow evidence and required-check interpretations must be rejected when their exact PR/head/base/run/review identity no longer matches; +- regression coverage that discovering a new source head or live base resets the affected evidence and queue decision rather than inheriting predecessor evidence; +- regression test that a blocker report/plan alone is not an accepted product outcome when a safe item remains; +- final double-exit-sweep policy in the active scheduler prompt; +- contract coverage for the executable fresh-sweep protocol and its reset-on-new-work rule. + +## Rollback / supersession + +The loop can be disabled without changing product runtime. Supersede only with an automation policy that preserves work conservation, writer safety, feasibility proof and no-report-as-completion semantics. diff --git a/docs/adr/0010-release-provenance-fidelity-gate.md b/docs/adr/0010-release-provenance-fidelity-gate.md new file mode 100644 index 000000000..a0cf1b3c3 --- /dev/null +++ b/docs/adr/0010-release-provenance-fidelity-gate.md @@ -0,0 +1,57 @@ +# ADR-0010: Gate release on integrated provenance, fidelity, accessibility and operability evidence + +Status: Accepted +Implementation maturity: `PARTIAL` + +## Context and drivers + +A green unit-test run or a generated SBOM does not establish that a document-conversion product is releasable. Clearfolio handles untrusted customer documents, tenant boundaries, generated artifacts and asynchronous recovery. Release therefore needs evidence across correctness, security, fidelity, accessibility, supply chain and operations on one integrated protected revision. + +## Alternatives + +1. Release whenever the feature PR's CI is green. +2. Release from a manually selected commit after local verification. +3. Release only from an exact integrated protected head after the full acceptance graph passes. + +## Decision + +Choose alternative 3. `docs/RELEASE_ACCEPTANCE.md` is the canonical complete release checklist; this ADR defines why that checklist is authoritative rather than maintaining a weaker parallel list. Release acceptance includes required CI/security/SAST/fuzz, exact owned coverage/docstrings, realistic document-fidelity acceptance for every claimed format, accessibility, package/runtime compatibility, SBOM/third-party attribution/provenance, reproducibility, API/schema compatibility, migration/rollback/recovery where applicable, independent review according to governance, and protected-main operational evidence. + +At minimum the exact integrated candidate must demonstrate zero skips, failures, and errors; zero warnings and deprecations; warning-free public Javadocs/doclint; applicable JavaScript coverage for owned executable browser code; Markdown lint; and live required review and security gates. These are necessary release-evidence classes, not substitutes for realistic fidelity, security, recovery, accessibility or operational assertions. + +## Consequences + +Release cadence may be slower than feature cadence. Evidence is easier for customers and acquirers to audit, and unsupported format claims cannot slip into release notes merely because placeholder rendering exists. + +## Failure and recovery + +Any required failed, pending, cancelled, skipped-required, absent or stale evidence blocks release. A release failure triggers correction on a new candidate revision; predecessor evidence does not silently transfer. Published artifacts with a discovered defect are superseded using the repository's release/incident process rather than rewriting history. + +## Security and privacy + +Provenance and evidence artifacts must not leak source documents, raw tenant identifiers, secrets, approval tokens, internal paths or uncontrolled exception data. Supply-chain metadata is public/shareable only after disclosure review appropriate to the artifact. + +## Compatibility and migration + +Version bumps and schema changes require explicit compatibility notes. Stateful migrations require tested forward/rollback or recovery paths before release. Standalone and supported MSA integration profiles must be validated independently when both are claimed. + +## Tests and acceptance + +The executable acceptance source remains `docs/RELEASE_ACCEPTANCE.md`. This ADR requires that source to preserve, when applicable: + +- exact integrated source identity; +- zero skips, failures, and errors in required test evidence; +- zero warnings and deprecations in owned production compilation; +- warning-free public Javadocs/doclint; +- exact owned Java line/branch coverage and applicable JavaScript coverage; +- Markdown lint and documentation-contract validation; +- realistic format fixtures and fidelity thresholds; +- accessibility acceptance; +- SBOM/attribution deterministic regeneration; +- provenance/reproducible package checks; +- migration/rollback/restart acceptance when state changes; +- live required review and security gates, including independent review and protected-main operational verification. + +## Rollback / supersession + +Rollback uses a previously accepted artifact/version only when data/schema compatibility permits. Supersede this ADR only with an equal or stronger evidence model documented with migration and customer-impact handling. diff --git a/docs/adr/0011-thin-scheduler-control-plane.md b/docs/adr/0011-thin-scheduler-control-plane.md new file mode 100644 index 000000000..f7166593e --- /dev/null +++ b/docs/adr/0011-thin-scheduler-control-plane.md @@ -0,0 +1,124 @@ +# ADR-0011: Keep the recurring scheduler as a thin control plane + +- Status: Accepted +- Implementation maturity: `PARTIAL` — the external hourly Clearfolio scheduler has adopted the decision; repository-local automation remains `ACTIVE_PR` #271 until protected integration. +- Decision date: 2026-08-10 + +## Context + +The Clearfolio hourly development task was active and recording executions, yet repeated invocations surfaced only a generic scheduled-task failure. The available automation control surface proved that scheduling/activation was healthy, but it did not expose a sufficiently detailed execution log to prove one internal root cause. + +The previous scheduler prompt had also grown into a second copy of detailed product, security, documentation, release, and architecture policy. That duplication was a supported failure-amplifier hypothesis rather than a proven single root cause: it increased per-run input size, made stale product detail easier to preserve outside GitHub, and coupled scheduler execution to a large amount of information already maintained in reviewed repository documents. + +A scheduler execution failure must not silently convert an execution-first development loop into a one-line status reporter. The scheduler needs enough control policy to recover work, but GitHub must remain the reconstructable source of detailed product truth. + +## Decision drivers + +- scheduled development must continue after local check/review/provider/scheduler failures when another safe action exists; +- exact product and architecture detail must be reviewable, versioned, testable, and attributable in GitHub; +- repeating that detail in an external scheduler creates two mutable sources of truth; +- scheduler activation state and scheduler execution failure are different diagnostic boundaries; +- loss of detailed scheduler-run telemetry must not justify inventing a root cause; +- the existing writer-lease, exact-head, TDD, governance, and double-exit requirements must survive prompt simplification. + +## Considered alternatives + +### A. Keep a monolithic, self-contained scheduler prompt + +Rejected. It is convenient when repository documents are incomplete, but duplicates reviewed policy, increases stale-state risk, and makes every scheduled invocation carry product detail unrelated to control-plane execution. + +### B. Disable the scheduler and rely on manual development + +Rejected. It avoids scheduler-specific failures but abandons the required hourly continuation and does not address execution reliability. + +### C. Move all control behavior into a repository GitHub Actions writer + +Rejected for the current authority model. Central `.github` and repository-local automation have separate writer/reviewer credentials and protection boundaries; moving the external orchestration authority into Actions would require a separate security/governance decision. + +### D. Thin control plane with repository-owned detailed authority + +Accepted. + +## Decision + +The recurring Clearfolio scheduler is a **thin control plane**. Repository documents are the detailed authority for product scope, architecture, security, fidelity, data model, testing, operability, research traceability, release acceptance, and current implementation maturity. The scheduler **must not duplicate the full product specification**. + +The scheduler retains only execution semantics that cannot safely be delegated away: + +1. refetch exact live repository/PR/check/review/writer state at run start; +2. respect the single Clearfolio writer lease and read-only dependency boundaries; +3. maintain a live executable queue and treat waiting as local; +4. apply RCA → distinct remedies → real-world feasibility → action → proof; +5. enforce exact-head review/merge evidence and never manufacture approval; +6. **read current canonical repository documents before selecting or changing product work**; +7. if a prior scheduler execution failure is observed, distinguish scheduling/activation failure from execution failure using the evidence actually available; +8. do not claim an unobservable internal cause as proven; +9. simplify duplicated/stale control state when it is a plausible failure amplifier without weakening safety or acceptance requirements; +10. continue safe repository work in the **same invocation** after scheduler RCA rather than treating scheduler debt as global blocking work; and +11. require the **double exit sweep** before normal termination. + +Every source, documentation, ref, PR-state, review-trigger, or merge-adjacent write has an additional fail-closed prewrite contract. **Before every write, refetch the exact source head and exact target head, the independently resolved live base, the target blob/ref, and the relevant review state: formal reviews, unresolved threads, exact-head checks, and security gates. If the target identity changes or another writer moves the target after that snapshot, reject the stale write, freeze that branch/path, and rotate to another safe item.** The write is prepared only against those exact identities. + +## Consequences and trade-offs + +### Benefits + +- detailed product truth remains versioned and reviewed in GitHub; +- each scheduled invocation carries a smaller, more stable control contract; +- prompt simplification can happen without deleting product requirements; +- documentation tests can detect missing authority rather than relying on conversational memory; +- scheduler failures become an operability problem with a bounded recovery procedure instead of a routine status message. + +### Costs + +- the scheduler must read the relevant repository authority before acting; +- a missing/stale canonical document becomes an actionable repository defect; +- external automation state and repository docs are two evidence systems and must be linked explicitly in traceability rather than conflated. + +## Failure and recovery behavior + +When a scheduled task fails: + +1. read live automation state and last-run evidence available to the actor; +2. classify whether the task failed to schedule/activate or failed during execution; +3. inspect repository/GitHub state to determine whether the prior run mutated any target before failing; +4. before any recovery write, refetch current source head, independently resolved current base, formal reviews, unresolved threads, exact-head checks, security gates, target blob/ref, and writer lease; if any target identity moved, reject the stale operation, freeze that lane, and rotate; +5. test separate hypotheses such as prompt-size and duplicated-state amplification, connector/tool failure, authentication/authorization, GitHub provider/rate state, or repository policy; +6. execute the smallest reversible supported remedy; +7. continue the repository executable queue in the same invocation when safe; +8. record durable new operating decisions in canonical docs; and +9. require fresh exact-head evidence after any repository mutation. + +If detailed scheduler execution telemetry is unavailable, record the internal cause as unresolved rather than guessing. Activation evidence alone cannot prove execution success. + +## Security, privacy, and governance impact + +- no new repository token or model secret is introduced by this decision; +- scheduler prompt reduction must not weaken branch protection, independent review, exact-head checks, tenant boundaries, model credential isolation, or writer leases; +- repository authority must not include secrets; secrets remain GitHub Secret/runtime configuration material; +- central `.github`, naruon, contextual-orchestrator, and repositories with dedicated writer loops remain read-only to this scheduler. + +## Compatibility and migration + +Existing detailed scheduler clauses migrate into the corresponding canonical GitHub documents instead of being deleted as requirements. The scheduler may reference those files by path and maturity. If an applicable canonical document has not yet reached protected `main`, an active-PR version may be read only as `ACTIVE_PR` evidence and must not be represented as shipped behavior. + +Repository-local OpenCode automation in #271 is compatible with this decision but does not become protected-main behavior until its own dependencies, reviews, checks, and central `.github` integration satisfy governance. + +## Tests and acceptance evidence + +Acceptance requires: + +- `scripts/test_release_loop_adr_contract.py` to require this ADR and the scheduled-task runbook terms in `docs/OPERABILITY.md`; +- the same contract to prove the scheduler reads current repository authority before product selection, and to verify the complete prewrite refetch/freeze behavior plus Markdown trailing-newline discipline; +- the ADR index and `docs/TRACEABILITY.md` to link this decision; +- scheduler control state to remain enabled with the thin prompt after update; +- a future scheduled execution to distinguish scheduler activation from execution failure instead of emitting only a generic failure message where the control surface allows that distinction; +- no regression in the work-conserving double-exit, exact-head, writer-lease, TDD, approval, or release gates. + +A successful prompt update is not proof that all future scheduler failures are solved; it proves only that the duplicated-control-state failure amplifier has been removed and the recovery contract is explicit. + +## Rollback and supersession + +Rollback to a monolithic prompt is allowed only if evidence shows repository-authority lookup itself is the material reliability failure and a bounded alternative cannot preserve execution requirements. Such a change requires a new ADR because it would reintroduce duplicated detailed authority. + +Supersede this ADR if the product later adopts a repository-native orchestration control plane with equivalent writer isolation, scheduler-failure telemetry, exact-head/live-base evidence, and work-conserving termination semantics. diff --git a/docs/adr/0012-scheduler-execution-receipts.md b/docs/adr/0012-scheduler-execution-receipts.md new file mode 100644 index 000000000..1d60f7387 --- /dev/null +++ b/docs/adr/0012-scheduler-execution-receipts.md @@ -0,0 +1,160 @@ +# ADR-0012: Scheduler execution receipts and resumable continuation + +Status: Proposed +Implementation maturity: `PLANNED`; issue #331 + +## Context + +Clearfolio uses an external recurring scheduler as a thin execution control plane and is also developing a repository-local OpenCode product-development workflow in Draft PR #271. These are separate execution authorities. The external scheduler has repeatedly returned only the generic scheduled-task error `There was a problem with your scheduled task.` while the repository still contained executable work. + +The available control-plane interface does not expose an exception type, failing phase, action receipt, stack trace, or durable checkpoint. Therefore maintainers must **do not invent** a hidden root cause such as prompt length, provider failure, connector failure, permissions, or hard runtime exhaustion without evidence. Scheduler activation, repository execution, exact GitHub mutation, clean budget continuation, and failure are different evidence classes. + +Issue #331 records the operational gap: a scheduled invocation needs privacy-safe evidence showing how far it progressed, what atomic action was last proven, whether a continuation is safe, and which state must be refetched next. + +## Decision drivers + +- Diagnose generic scheduler failures without fabricating platform internals. +- Resume work after practical tool or runtime exhaustion without trusting stale SHAs. +- Prevent half-published changes, temporary writer machinery, and knowingly broken non-test-only heads. +- Keep the scheduler prompt thin while GitHub documents remain detailed product authority. +- Preserve the single Clearfolio writer lease and branch-local freeze-and-rotate behavior. +- Avoid persisting secrets, document content, tenant/subject identifiers, tokens, or uncontrolled exception text. +- Distinguish the external scheduler from Draft PR #271 and from central `.github` PR maintenance. + +## Considered alternatives + +### Alternative A — Keep only the generic task error + +Rejected. A generic sentence does not distinguish schedule admission, execution start, queue construction, repository mutation, controlled deferment, budget continuation, or failure. It also encourages repeated speculative prompt edits instead of evidence-backed RCA. + +### Alternative B — Store all scheduler state in the Clearfolio application database + +Rejected. The external platform owns the scheduler runtime. Clearfolio must not invent a physical database or pretend it owns platform execution persistence. A future repository-local workflow may persist GitHub-owned evidence, but that is a separate reviewed implementation decision. + +### Alternative C — Versioned execution receipts owned by the executing control plane + +Accepted. The executing scheduler or workflow owns durable receipt storage where supported. Clearfolio owns the semantic contract, privacy constraints, traceability, and tests. The repository models these records as conceptual/external evidence rather than shipped application tables. + +## Decision + +Every autonomous development invocation should expose or persist, where the executing platform permits, a versioned execution receipt composed of conceptual records. An **automation checkpoint** is the human-readable concept represented by `automation_checkpoint`: the last safe execution phase plus exact historical evidence identity. + +- `automation_run`: immutable run identity and scheduler/control version or digest; +- `automation_checkpoint`: the last safe phase and exact historical evidence identity; +- `queue_snapshot`: protected-main, PR, issue, dependency, review, check, and writer-lease identities observed when the queue was built; +- `deferred_lane`: one locally blocked lane keyed by exact PR/head/live-base/run/review identity and defer reason; +- `action_receipt`: the last completed atomic repository action and its observable proof; +- `failure_envelope`: controlled failing phase/category and recovery eligibility; +- `continuation_handoff`: next-run context that is explicitly historical and requires a fresh GitHub refetch. + +The lifecycle distinguishes at least: + +```text +schedule +→ admission +→ fresh queue +→ atomic action +→ action receipt +→ next action +→ completed | budget continuation | failure envelope +``` + +### Phase semantics + +1. `schedule` proves only that recurrence was due. +2. `admission` proves only that the platform accepted the invocation. +3. `fresh queue` records the GitHub identities used for selection; those identities become historical after any movement. +4. `atomic action` identifies one mutation or acceptance operation that can reach a stable proof boundary. +5. `action receipt` binds the operation to an exact source/ref/blob/PR/issue/check identity where observable. +6. `budget continuation` is a clean non-completion state used only when actual runtime/tool signals indicate that starting another multi-step mutation cannot reasonably reach a verified boundary. +7. `failure envelope` records only an observed controlled category and phase. It does not substitute speculation for missing telemetry. + +### Budget continuation + +Before a hard invocation boundary, the runner should finish or safely abandon the current atomic action, refetch the resulting exact state, remove temporary writer machinery, and avoid leaving a knowingly broken non-test-only source head. The next invocation starts from a full fresh queue. A budget continuation is not repository completion and does not waive normal double-exit-sweep semantics when sufficient budget remains. + +### Fresh-state rule + +A continuation handoff never authorizes writes by itself. The next run must independently refetch protected main, every open PR/issue, exact heads/live bases, reviews/threads, checks/security evidence, target blobs/refs, dependencies, and active-writer state before any decision or mutation. + +### Privacy and security + +Receipts must not contain: + +- raw credentials, API keys, authorization headers, cookies, approval tokens, or artifact tokens; +- document bytes, extracted document content, filenames/paths, or uncontrolled digests without an approved evidence purpose; +- raw tenant, subject, approver, or operator identifiers; +- provider-controlled exception messages or stack traces in user-visible receipts; +- model prompt content that can contain untrusted repository/document text. + +Use controlled phase codes, low-cardinality reason codes, opaque run/action identifiers, and exact Git object identities only where needed for repository evidence. Pseudonymized values remain personal data and retain purpose/retention controls. + +### Authority separation + +- The external scheduler remains a thin control plane. +- Reviewed repository documents, current source, and live GitHub policy remain detailed authority. +- Draft PR #271 is future repository-local automation and must not be presented as proof that the external scheduler executed successfully. +- Central `.github` retains its separately reviewed PR-maintenance authority. +- No receipt authorizes self-approval, gate bypass, release, deployment, or a competing writer. + +## Consequences + +### Positive + +- Operators can distinguish activation from execution and clean continuation from failure. +- A recurring run can resume without trusting stale source/base/check/review evidence. +- Generic task errors become an observable control gap rather than a guessed root cause. +- Privacy-safe action lineage improves incident review, acquisition diligence, and scheduler reliability evidence. +- The design preserves the thin prompt and one canonical GitHub documentation graph. + +### Negative + +- The executing platform must expose or store additional structured evidence. +- Receipt schemas and retention require versioning and compatibility management. +- External platforms may not expose enough telemetry for full implementation; in that case the receipt must explicitly record `unavailable` evidence rather than inventing detail. +- More exact identities increase traceability volume and require bounded retention. + +## Failure and recovery + +- If no run identity is available, classify the observation as schedule/control-plane evidence only and continue repository work through another safe lane when tools remain. +- If admission succeeds but queue construction does not, record the failing phase without claiming repository execution. +- If a repository action fails before mutation, record the exact attempted boundary and no-mutation evidence where observable. +- If contents/ref publication races another writer, reject stale CAS, freeze only that branch, and rotate. +- If a test-only RED head is published, the same run should reach GREEN when feasible; otherwise the handoff must explicitly identify a safe test-only continuation and must not represent it as mergeable product code. +- If a platform cannot expose detailed failure telemetry, preserve the generic scheduled-task error as the symptom and **do not invent** the underlying cause. +- Recovery always starts from fresh GitHub state, not from the checkpoint SHA as current authority. + +## Security, privacy, and governance impact + +The receipt model strengthens least privilege and evidence separation. It does not expand scheduler credentials, review authority, branch protection, release authority, model access, or cross-repository write scope. Retention and access to receipts must be purpose-bound and auditable. Repository-local autonomous model use continues to require OpenCode and GitHub Secret `NVIDIA_NIM_API_KEY`; `COPILOT_GITHUB_TOKEN` is not a development-model credential. + +## Compatibility and migration + +- Existing external runs without receipts remain historical `telemetry_unavailable` evidence. +- New receipt versions must remain parseable or explicitly migrated; unknown fields must not grant authority. +- Draft PR #271 may adopt the semantic model after #270 and central prerequisites stabilize, but predecessor checks/reviews do not transfer. +- No Clearfolio application database migration is implied by this ADR. + +## Tests and acceptance + +Issue #331 owns implementation acceptance. Machine-checkable repository documentation must verify: + +- activation without execution is distinguishable; +- run start without queue construction is not success; +- the generic scheduled-task error maps to a controlled failing phase without invented cause; +- hard-budget simulation produces `budget continuation`, not completion; +- partial publication is bound to exact parent/blob/ref identity; +- writer-lease conflict is branch-local; +- the next run refetches live GitHub identities; +- receipts exclude raw secrets, tokens, document content, tenant/subject identifiers, and uncontrolled exception text; +- external scheduler receipts and Draft PR #271 are distinct authorities. + +Repository documents must keep ADR, data model, UML, operability, traceability, documentation assessment, and changelog aligned. + +## Rollback + +A receipt implementation may be disabled if it leaks sensitive data, blocks safe execution, or cannot preserve exact evidence identity. Rollback returns to fail-closed generic classification and fresh-state reconstruction; it must not restore invented root causes, stale checkpoint writes, self-modifying workflows, or weaker writer leases. + +## Supersession + +Supersession requires a later ADR that identifies a demonstrably stronger execution-evidence mechanism, preserves activation/execution/evidence-authority separation, defines migration and rollback, and keeps privacy-safe resumable continuation. A platform-native durable run ledger may supersede this conceptual contract only after equivalent acceptance evidence exists. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 000000000..64ca19c60 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,40 @@ +# Clearfolio Architecture Decision Records + +Status: Canonical ADR index +Baseline: protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11` + +ADR status and implementation maturity are intentionally separate. `Accepted` means the architecture decision is authoritative; it does not imply that every consequence is implemented on protected main. Each ADR therefore records implementation status explicitly. + +| ADR | Decision | Status | Implementation maturity | +| --- | --- | --- | --- | +| [0001](0001-standalone-msa-ownership.md) | Clearfolio remains independently deployable with explicit host/MSA authority boundaries | Accepted | `PARTIAL` / `ACCEPTED_ARCHITECTURE` | +| [0002](0002-tenant-artifact-authorization.md) | Tenant authorization and signed artifact delivery are separate required controls | Accepted | `IMPLEMENTED_ON_MAIN` through protected #270, including direct-download alignment | +| [0003](0003-audit-pseudonymization-key-separation.md) | Audit pseudonyms use purpose-separated, domain-separated cryptographic material | Proposed | `IMPLEMENTED_ON_MAIN` through protected #270; #268 `SUPERSEDED` for remaining lifecycle work | +| [0004](0004-durable-lifecycle-generation-fencing.md) | Durable job/artifact deletion uses immutable identity, generation fencing and restart-safe evidence | Proposed | `PARTIAL` target; #268 `SUPERSEDED`; bounded successors unintegrated | +| [0005](0005-deterministic-conversion-fidelity.md) | Supported-format claims require deterministic real conversion and fidelity evidence; placeholders are non-production | Accepted | `PARTIAL`; real Office conversion `PLANNED` | +| [0006](0006-liveness-readiness-separation.md) | Process liveness and traffic readiness are independent signals | Proposed | `ACTIVE_PR` #295 | +| [0007](0007-exact-head-live-base-evidence.md) | Merge/release evidence binds exact source head and independently resolved live base | Proposed | CI semantics `ACTIVE_PR` #270; operational policy accepted | +| [0008](0008-central-vs-local-automation-authority.md) | Central PR maintenance and leaf product development have separate authority and credentials | Proposed | `.github` dependency + `ACTIVE_PR` #271 | +| [0009](0009-work-conserving-rca-loop.md) | Automation uses work-conserving RCA→remedies→feasibility→action→proof and cannot stop after one item | Proposed | scheduler/prompt policy in progress | +| [0010](0010-release-provenance-fidelity-gate.md) | Release requires integrated security, fidelity, accessibility, provenance and operability evidence | Accepted | `PARTIAL` | +| [0011](0011-thin-scheduler-control-plane.md) | Recurring scheduler stays a thin execution control plane while reviewed repository documents own detailed product authority | Accepted | external scheduler applied; repository-local automation remains `ACTIVE_PR` #271 | +| [0012](0012-scheduler-execution-receipts.md) | Scheduler execution receipt, failure envelope, and budget continuation semantics make autonomous runs diagnosable and resumable without inventing hidden causes | Proposed | `PLANNED`; issue #331 | + +## Required ADR structure + +Every material ADR contains: + +- context and decision drivers; +- considered alternatives; +- decision; +- consequences and trade-offs; +- failure and recovery behavior; +- security/privacy/governance impact; +- compatibility/migration implications; +- tests/acceptance evidence; +- rollback or supersession criteria; +- implementation maturity. + +## Supersession rule + +Do not edit a historical decision to pretend a different decision was always in force. Materially changed decisions receive a new ADR that marks the old record superseded and explains migration and rollback. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..efc808682 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,35 @@ +# Clearfolio + +Clearfolio is ContextualWisdomLab's secure document conversion and viewing service. It accepts bounded document jobs, exposes tenant-scoped conversion state, and delivers verified PDF artifacts through a browser viewer and controlled artifact-link flows. + +> This landing reflects protected `main` product truth. Active pull requests, planned Office-format fidelity, production identity integrations, and unpublished deployment/release evidence are not presented as shipped capabilities. + +## Start here + +- [Repository overview and local run guide](https://github.com/ContextualWisdomLab/clearfolio#readme) +- [Architecture](architecture.md) +- [Product requirements](prd-integrated-document-viewer-platform.md) +- [Technical requirements](trd-integrated-document-viewer-platform.md) +- [Buyer deployment integration](deployment/2026-07-02-buyer-deployment-integration-playbook.md) +- [Threat model and data handling](security/2026-07-02-threat-model-data-handling.md) +- [Signed artifact-link design](security/2026-07-02-signed-artifact-link-design.md) +- [Authentication and tenant model](security/2026-07-02-auth-tenant-model.md) +- [Durable job repository plan](persistence/2026-07-02-durable-conversion-job-repository-plan.md) +- [Durable metrics event model](analytics/2026-07-02-durable-metrics-event-model.md) +- [Buyer diligence index](diligence/2026-07-02-buyer-diligence-index.md) +- [Repository releases](https://github.com/ContextualWisdomLab/clearfolio/releases) +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/clearfolio) + +## Product responsibility + +Clearfolio owns the document-conversion/viewing boundary, tenant-scoped job lifecycle, artifact-delivery controls, viewer bootstrap, retry/recovery evidence available in the current runtime, and the public API contracts that expose those capabilities. Higher-level hosts may compose Clearfolio through explicit contracts, but they do not acquire direct ownership of Clearfolio application state. + +The protected default branch currently proves validated PDF passthrough and viewer behavior plus bounded asynchronous conversion/job primitives. Development placeholders and planned provider adapters are not treated as faithful production conversion for unsupported formats. Header-based buyer-demo identity scaffolding is likewise not a claim of complete production OIDC/JWT federation. + +## Security and evidence + +Uploaded documents and document-derived values are untrusted input. Cross-tenant access, artifact tokens, permission checks, expiry, revocation, integrity, range delivery, and audit evidence are separate controls. Source and tests are evidence of implemented contracts; deployment, certification, durability, and fidelity claims require their own runtime or release proof. + +## Publication boundary + +This file is only a GitHub Pages source candidate. Pages is live only after protected integration, repository Pages configuration/deployment through the organization-owned control path, and verification of the resulting HTTPS content. diff --git a/docs/repository-metadata.md b/docs/repository-metadata.md index 0f771cdd5..1ecd95e71 100644 --- a/docs/repository-metadata.md +++ b/docs/repository-metadata.md @@ -1,28 +1,93 @@ # GitHub Transfer Metadata -This file captures repository handoff details required before GitHub ownership transfer. +Last updated: 2026-08-09 + +This file captures repository handoff and acquisition-readiness details that must +be reproducible from GitHub without conversation history. It describes the +current protected-main repository plus explicitly identified active work; it +does not promote an active pull request to shipped behavior. ## Identity -- Repository intent: `clearfolio-viewer` (MVP backend for integrated document viewer) -- Language/runtime: Java 21, Spring Boot -- Primary package: `com.clearfolio.viewer` -- Initial artifact: `clearfolio-viewer` (Maven artifact from `pom.xml`) +- Repository: `ContextualWisdomLab/clearfolio`. +- Product intent: secure integrated document conversion and viewing. +- Language/runtime: Java 21 / Spring Boot / Maven. +- Primary package: `com.clearfolio.viewer`. +- Maven artifact: `clearfolio-viewer`. +- License: Apache License 2.0 in root `LICENSE`. +- Protected default branch: `main`. + +## Authoritative documentation graph + +PR #305 introduces the canonical product/architecture spine. Until that PR +integrates, protected `main` still contains historical documentation that may be +stale. The review target is: -## Transfer readiness checklist +- `docs/PRD.md`; +- `docs/TRD.md`; +- root `ARCHITECTURE.md`; +- `docs/DATA_MODEL.md`; +- `docs/UML.md`; +- `docs/API_CONTRACT.md`; +- `docs/THREAT_MODEL.md`; +- `docs/TEST_STRATEGY.md`; +- `docs/OPERABILITY.md`; +- `docs/TRACEABILITY.md`; +- `docs/RESEARCH_TRACEABILITY.md`; +- `docs/FIDELITY_ACCEPTANCE.md`; +- `docs/MIGRATION_ROLLBACK.md`; +- `docs/RELEASE_ACCEPTANCE.md`; +- `docs/adr/README.md` and detailed ADRs. -- [x] Root README exists (`README.md`) -- [x] Architecture baseline exists (`docs/architecture.md`) -- [x] TRD + PRD alignment exists (`docs/trd-integrated-document-viewer-platform.md`) -- [x] UML flow documentation exists for submit/status/preview (`docs/diagrams/*-flow.md`) -- [x] Acceptance mapping updated for AC-1 ~ AC-15 in architecture/TRD docs -- [x] PR review bot workflow validated (`@coderabbitai` review/resolve flow on PR) -- [ ] License file added -- [ ] Explicit `CODEOWNERS` set to transfer team -- [ ] CI/security workflow files added for long-term governance +## Transfer and acquisition-readiness checklist + +- [x] Root README exists and points to current product boundaries. +- [x] Apache-2.0 root `LICENSE` exists. +- [x] CI workflow exists and enforces exact-head tests, zero-missed owned + coverage, Javadocs, no-skipped-test evidence, and merge compatibility. +- [x] Security Scan, SAST Semgrep, and fuzz workflows exist. +- [x] CodeRabbit review/status integration is active. +- [x] SBOM, third-party attribution, security, fidelity, migration/rollback, and + release-acceptance authorities are represented in repository evidence or the + canonical documentation PR. +- [x] PRD, TRD, Architecture, ADR, conceptual ERD/data model, UML, API contract, + threat model, test strategy, operability, and traceability are consolidated by + PR #305 and guarded by `scripts/test_documentation_spine_contract.py`. +- [ ] Explicit `CODEOWNERS` for the eventual transfer team. No transfer-team + identity is invented here; configure this only when an eligible team/user is + actually known and accepted by repository governance. +- [ ] Qualifying independent non-author review route demonstrated for the final + transfer/release head. Automated review/status evidence does not substitute + for a counted independent approval when policy requires one. +- [ ] Production Office transformed-format conversion qualified. Protected + `main` currently has PDF passthrough plus non-PDF development/demo placeholder + generation; issue #5 owns the sandboxed provider-neutral Office adapter and + fidelity qualification gap. +- [ ] Production identity and distributed durable-state integrations completed + or contractually assigned to the receiving platform; current limitations must + remain explicit in PRD/TRD/Architecture/release evidence. +- [ ] One exact protected release head has complete CI, security, independent + review, fidelity, recovery, SBOM/provenance, packaging, rollback and + operational-acceptance evidence before transfer is described as release-ready. ## Handover pointers -- Core implementation entry points: `src/main/java/com/clearfolio/viewer` -- Public API entry points: `src/main/java/com/clearfolio/viewer/controller` -- Current test surface: `src/test/java/com/clearfolio/viewer` +- Core implementation: `src/main/java/com/clearfolio/viewer`. +- Public HTTP/API entry points: `src/main/java/com/clearfolio/viewer/controller`. +- Tests: `src/test/java/com/clearfolio/viewer` and repository contract tests + under `scripts/`. +- Security authority: root `SECURITY.md` plus `docs/security/` and + `docs/THREAT_MODEL.md` in the canonical spine. +- Operations: `docs/OPERABILITY.md` plus `docs/operations/`. +- Release/fidelity authority: `docs/RELEASE_ACCEPTANCE.md` and + `docs/FIDELITY_ACCEPTANCE.md` in the canonical spine. +- Product gaps: issue #5 for Office conversion and issue #263 for the remaining + tenant-safe user lifecycle after active substrate PRs integrate. + +## Evidence rule + +A checked repository capability means the repository contains the named source +or evidence boundary; it does not mean the software is commercially complete. +Active-PR behavior remains `ACTIVE_PR`, planned architecture remains `PLANNED` or +`ACCEPTED_ARCHITECTURE`, and only protected-main code may be described as +`IMPLEMENTED_ON_MAIN`. diff --git a/scripts/test_acquisition_diligence_contract.py b/scripts/test_acquisition_diligence_contract.py new file mode 100644 index 000000000..4588aa9ab --- /dev/null +++ b/scripts/test_acquisition_diligence_contract.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Require acquisition diligence to use current canonical evidence, not stale snapshots.""" + +from __future__ import annotations + +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one repository text file using strict UTF-8 decoding.""" + + return (REPOSITORY_ROOT / relative_path).read_text(encoding="utf-8") + + +class AcquisitionDiligenceContractTest(unittest.TestCase): + """Keep acquisition readiness reconstructable from current repository authority.""" + + def test_current_acquisition_diligence_document_exists_and_is_discoverable(self) -> None: + """Require one non-dated canonical diligence authority linked from architecture.""" + + diligence_path = REPOSITORY_ROOT / "docs/ACQUISITION_DILIGENCE.md" + self.assertTrue(diligence_path.is_file()) + architecture = read_text("ARCHITECTURE.md") + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + self.assertIn("`docs/ACQUISITION_DILIGENCE.md`", architecture) + self.assertIn("`docs/ACQUISITION_DILIGENCE.md`", assessment) + + def test_diligence_separates_current_evidence_from_legacy_snapshots(self) -> None: + """Prevent dated sale-readiness material from masquerading as current proof.""" + + diligence = read_text("docs/ACQUISITION_DILIGENCE.md").lower() + for required in ( + "protected main", + "active_pr", + "historical evidence", + "license", + "third-party attribution", + "sbom", + "intellectual property", + "contributor", + "legal", + "issue #5", + "issue #263", + "production office", + "durable asynchronous", + "opentelemetry", + "independent approval", + ): + with self.subTest(required=required): + self.assertIn(required, diligence) + + self.assertIn("2026-07-02-buyer-diligence-index.md", diligence) + self.assertIn("must not be used to assert", diligence) + + current_work = diligence.split( + "## current open work that materially affects diligence", + maxsplit=1, + )[1].split("## historical evidence policy", maxsplit=1)[0] + self.assertNotIn("#74", current_work) + self.assertNotIn("#82", current_work) + + def test_diligence_does_not_claim_unproven_transfer_or_certification(self) -> None: + """Require rights/compliance gaps to remain explicit until independently proven.""" + + diligence = read_text("docs/ACQUISITION_DILIGENCE.md").lower() + self.assertIn("not independently proven", diligence) + self.assertIn("does not claim certification", diligence) + self.assertIn("external legal review", diligence) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_documentation_assessment_security_contract.py b/scripts/test_documentation_assessment_security_contract.py new file mode 100644 index 000000000..a3b1d304f --- /dev/null +++ b/scripts/test_documentation_assessment_security_contract.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Prevent the documentation fitness matrix from regressing security authority.""" + +from __future__ import annotations + +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one repository text file with strict UTF-8 decoding.""" + + return (REPOSITORY_ROOT / relative_path).read_text(encoding="utf-8") + + +class DocumentationAssessmentSecurityContractTest(unittest.TestCase): + """Keep the documentation assessment aligned with the canonical security entrypoint.""" + + def test_security_fitness_row_matches_current_root_security_authority(self) -> None: + """Reject the superseded reporting-policy-only description of SECURITY.md.""" + + security = " ".join(read_text("SECURITY.md").lower().split()) + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + + for required in ( + "signed artifact", + "same-tenant", + "hmac", + "document conversion trust boundary", + "release evidence", + ): + with self.subTest(required=required): + self.assertIn(required, security) + + self.assertNotIn("root `SECURITY.md` remains the reporting policy", assessment) + self.assertIn( + "root `SECURITY.md` is the current product security entrypoint", + assessment, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_documentation_spine_contract.py b/scripts/test_documentation_spine_contract.py new file mode 100644 index 000000000..e6b7fc64d --- /dev/null +++ b/scripts/test_documentation_spine_contract.py @@ -0,0 +1,346 @@ +#!/usr/bin/env python3 +"""Verify that Clearfolio's canonical architecture documentation stays coherent.""" + +from __future__ import annotations + +import re +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] +CANONICAL_DOCUMENTS = ( + "docs/PRD.md", + "docs/TRD.md", + "ARCHITECTURE.md", + "SECURITY.md", + "docs/DATA_MODEL.md", + "docs/UML.md", + "docs/API_CONTRACT.md", + "docs/THREAT_MODEL.md", + "docs/TEST_STRATEGY.md", + "docs/OPERABILITY.md", + "docs/TRACEABILITY.md", + "docs/RESEARCH_TRACEABILITY.md", + "docs/DOCUMENTATION_ASSESSMENT.md", + "docs/engineering/acceptance-criteria.md", + "docs/adr/README.md", +) +SUPPLEMENTAL_RELEASE_DOCUMENTS = ( + "docs/FIDELITY_ACCEPTANCE.md", + "docs/MIGRATION_ROLLBACK.md", + "docs/RELEASE_ACCEPTANCE.md", +) +ADR_FILES = tuple( + f"docs/adr/{number:04d}-{slug}.md" + for number, slug in ( + (1, "standalone-msa-ownership"), + (2, "tenant-artifact-authorization"), + (3, "audit-pseudonymization-key-separation"), + (4, "durable-lifecycle-generation-fencing"), + (5, "deterministic-conversion-fidelity"), + (6, "liveness-readiness-separation"), + (7, "exact-head-live-base-evidence"), + (8, "central-vs-local-automation-authority"), + (9, "work-conserving-rca-loop"), + (10, "release-provenance-fidelity-gate"), + (11, "thin-scheduler-control-plane"), + (12, "scheduler-execution-receipts"), + ) +) +GUIDE_FILES = ("README.md", "AGENTS.md", "CLAUDE.md") +FINAL_VERIFY_COMMAND = "mvn -B --no-transfer-progress verify" + + +def read_text(relative_path: str) -> str: + """Return one repository text file using strict UTF-8 decoding.""" + + return (REPOSITORY_ROOT / relative_path).read_text(encoding="utf-8") + + +class DocumentationSpineContractTest(unittest.TestCase): + """Protect the canonical PRD/TRD/architecture/ADR/UML/ERD document graph.""" + + def test_canonical_documents_exist(self) -> None: + """Require every canonical document and detailed ADR to remain present.""" + + for relative_path in (*CANONICAL_DOCUMENTS, *SUPPLEMENTAL_RELEASE_DOCUMENTS, *ADR_FILES): + with self.subTest(path=relative_path): + self.assertTrue((REPOSITORY_ROOT / relative_path).is_file()) + + def test_architecture_links_the_canonical_spine(self) -> None: + """Require root architecture to remain the discoverable documentation index.""" + + architecture = read_text("ARCHITECTURE.md") + for relative_path in (*CANONICAL_DOCUMENTS, *SUPPLEMENTAL_RELEASE_DOCUMENTS): + if relative_path == "ARCHITECTURE.md": + continue + with self.subTest(path=relative_path): + self.assertIn(f"`{relative_path}`", architecture) + + def test_public_guides_link_product_requirements_and_decisions(self) -> None: + """Keep README and agent guides anchored to the same canonical spine.""" + + required_links = ("docs/PRD.md", "docs/TRD.md", "docs/adr/README.md") + for relative_path in GUIDE_FILES: + text = read_text(relative_path) + for canonical_path in required_links: + with self.subTest(path=relative_path, canonical=canonical_path): + self.assertIn(canonical_path, text) + + def test_claude_authority_classification_matches_repository_guidance(self) -> None: + """Keep canonical and supplemental documentation authority unambiguous for agents.""" + + claude = read_text("CLAUDE.md") + self.assertIn("docs/DOCUMENTATION_ASSESSMENT.md", claude) + self.assertIn("Supplemental release and research authorities", claude) + for supplemental in ( + "docs/FIDELITY_ACCEPTANCE.md", + "docs/MIGRATION_ROLLBACK.md", + "docs/RELEASE_ACCEPTANCE.md", + "docs/RESEARCH_TRACEABILITY.md", + ): + with self.subTest(supplemental=supplemental): + self.assertIn(supplemental, claude) + + def test_claude_premerge_contract_requires_protection_and_independent_approval(self) -> None: + """Prevent advisory/model evidence from silently replacing protected merge governance.""" + + claude = read_text("CLAUDE.md").lower() + for required in ( + "protected `main` branch rules", + "all required checks", + "required reviewer count", + "qualifying independent approval", + "automated comments, check results, and model output do not count as approval", + ): + with self.subTest(required=required): + self.assertIn(required, claude) + + def test_public_guides_use_final_maven_verify_command(self) -> None: + """Prevent agent/user guidance from regressing to a partial final test command.""" + + for relative_path in GUIDE_FILES: + with self.subTest(path=relative_path): + self.assertIn(FINAL_VERIFY_COMMAND, read_text(relative_path)) + + def test_readme_does_not_restore_stale_mvp_runtime_claims(self) -> None: + """Reject the historical MVP/in-memory-artifact summary as current product truth.""" + + readme = read_text("README.md").lower() + self.assertNotIn("contains the mvp backend", readme) + self.assertNotIn("asynchronous conversion that produces an in-memory pdf artifact", readme) + self.assertIn("development/demo one-page placeholder", readme) + + def test_readme_exposes_repository_license_boundary(self) -> None: + """Keep the public entrypoint explicit about source and third-party terms.""" + + readme = read_text("README.md") + self.assertIn("## License", readme) + self.assertIn("[Apache License 2.0](LICENSE)", readme) + self.assertIn("Third-party components retain", readme) + + def test_security_policy_matches_current_trust_boundaries(self) -> None: + """Require root security guidance to expose current authorization and privacy boundaries.""" + + security = " ".join(read_text("SECURITY.md").lower().split()) + for required in ( + "signed artifact", + "same-tenant", + "hmac", + "pseudonym", + "purpose", + "security advisories", + ): + with self.subTest(required=required): + self.assertIn(required, security) + self.assertIn("pseudonymized", security) + self.assertIn("personal data", security) + self.assertNotIn("permission alone authorizes document bytes", security) + + def test_changelog_has_one_unreleased_section(self) -> None: + """Keep current unreleased changes in one reviewable section.""" + + changelog = read_text("CHANGELOG.md") + self.assertEqual(1, len(re.findall(r"^## \[Unreleased\]$", changelog, flags=re.MULTILINE))) + + def test_adr_index_links_every_detailed_decision(self) -> None: + """Require the ADR index to link exactly the canonical detailed ADR files.""" + + index = read_text("docs/adr/README.md") + linked = set(re.findall(r"\]\((\d{4}[-a-z0-9]+\.md)\)", index)) + expected = {Path(path).name for path in ADR_FILES} + self.assertEqual(expected, linked) + + def test_documents_distinguish_main_active_and_planned_work(self) -> None: + """Prevent target architecture or active PR work from masquerading as shipped.""" + + required_labels = ( + "IMPLEMENTED_ON_MAIN", + "ACTIVE_PR", + "PARTIAL", + "ACCEPTED_ARCHITECTURE", + "PLANNED", + ) + for relative_path in ("docs/PRD.md", "docs/TRD.md", "ARCHITECTURE.md"): + text = read_text(relative_path) + for label in required_labels: + with self.subTest(path=relative_path, label=label): + self.assertIn(label, text) + + def test_documentation_assessment_uses_canonical_status_labels(self) -> None: + """Keep machine-readable documentation fitness labels canonical.""" + + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + self.assertIn("DESIGN_SUFFICIENT", assessment) + self.assertIn("PROTECTED_MAIN_SUFFICIENT", assessment) + self.assertNotIn("DESIGN-SUFFICIENT", assessment) + self.assertNotIn("PROTECTED-MAIN-SUFFICIENT", assessment) + + def test_test_strategy_runs_canonical_script_acceptance(self) -> None: + """Require documented acceptance to execute the documentation contract and all script tests.""" + + strategy = read_text("docs/TEST_STRATEGY.md") + self.assertIn("python3 scripts/test_documentation_spine_contract.py", strategy) + self.assertIn("python3 -m unittest discover -s scripts", strategy) + + def test_exact_head_adr_defines_atomic_merge_preconditions(self) -> None: + """Require merge-time head/base movement to fail closed and trigger revalidation.""" + + adr = read_text("docs/adr/0007-exact-head-live-base-evidence.md").lower() + for required in ( + "expected source-head sha", + "supported base-tip sha", + "merge time", + "reject", + "fresh-state revalidation", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + def test_automation_adr_names_single_active_pr_writer_and_pinning(self) -> None: + """Prevent duplicate ACTIVE_PR writers and unpinned local agent execution.""" + + adr = read_text("docs/adr/0008-central-vs-local-automation-authority.md").lower() + for required in ( + "only actor permitted to process `active_pr`", + "central fleet loops", + "clearfolio disabled", + "immutable", + "pinned", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + def test_placeholder_conversion_is_not_documented_as_office_fidelity(self) -> None: + """Keep placeholder PDF generation outside supported Office conversion claims.""" + + combined = "\n".join( + read_text(path) + for path in ( + "docs/PRD.md", + "docs/TRD.md", + "ARCHITECTURE.md", + "docs/adr/0005-deterministic-conversion-fidelity.md", + "docs/FIDELITY_ACCEPTANCE.md", + ) + ).lower() + self.assertIn("placeholder", combined) + self.assertIn("not", combined) + self.assertIn("office", combined) + self.assertIn("fidelity", combined) + self.assertIn("development_placeholder", combined) + + def test_fidelity_contract_requires_realistic_deterministic_evidence(self) -> None: + """Prevent extension/HTTP success from becoming a production format claim.""" + + fidelity = read_text("docs/FIDELITY_ACCEPTANCE.md").lower() + for required in ( + "realistic fixtures", + "authorized or redistributable", + "deterministic rendering", + "active content", + "network access is not required by default", + "output digests", + "unsupported", + "failed", + "development_placeholder", + ): + with self.subTest(required=required): + self.assertIn(required, fidelity) + + def test_office_converter_isolation_is_a_durable_decision(self) -> None: + """Prevent an adapter library capability from collapsing the API trust boundary.""" + + adr = read_text("docs/adr/0005-deterministic-conversion-fidelity.md").lower() + for required in ( + "provider-neutral `office_conversion_adapter`", + "does **not** run inside the clearfolio api-container trust boundary", + "no inherited api/cloud credentials", + "deny-by-default outbound network access", + "a successful converter process exit alone is never a production support oracle", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + def test_release_contract_binds_exact_source_and_artifact_evidence(self) -> None: + """Require one integrated exact-head release gate rather than green-PR inference.""" + + release = read_text("docs/RELEASE_ACCEPTANCE.md") + for required in ( + "exact protected source commit SHA", + "source_head_sha", + "live_base_tip_sha", + "docs/FIDELITY_ACCEPTANCE.md", + "docs/MIGRATION_ROLLBACK.md", + "SBOM", + "independent non-author formal approval", + ): + with self.subTest(required=required): + self.assertIn(required, release) + + def test_migration_contract_does_not_invent_current_database_persistence(self) -> None: + """Keep rollback guidance honest about process-local job state and future SQL.""" + + migration = read_text("docs/MIGRATION_ROLLBACK.md") + self.assertIn("Conversion-job state is process-local/in-memory", migration) + self.assertIn("Future SQL persistence", migration) + self.assertIn("must not be reintroduced", migration) + self.assertIn("must not re-enable a known authorization bypass", migration) + + def test_automation_contract_names_required_agent_secret_boundary(self) -> None: + """Keep autonomous development on OpenCode/NVIDIA rather than Copilot credentials.""" + + combined = "\n".join( + read_text(path) + for path in ( + "docs/PRD.md", + "docs/TRD.md", + "ARCHITECTURE.md", + "docs/adr/0008-central-vs-local-automation-authority.md", + ) + ) + self.assertIn("NVIDIA_NIM_API_KEY", combined) + self.assertIn("COPILOT_GITHUB_TOKEN", combined) + self.assertIn("OpenCode", combined) + + def test_work_conserving_contract_forbids_report_as_completion(self) -> None: + """Keep RCA and documentation outcomes as intermediate work while safe work exists.""" + + adr = read_text("docs/adr/0009-work-conserving-rca-loop.md").lower() + self.assertIn("work-conserving", adr) + self.assertIn("two fresh exit sweeps", adr) + self.assertIn("blocker report", adr) + self.assertIn("not a successful run", adr) + + def test_diagram_fences_are_balanced(self) -> None: + """Reject malformed Markdown diagram fences in canonical diagram-bearing docs.""" + + for relative_path in ("ARCHITECTURE.md", "docs/UML.md", "docs/DATA_MODEL.md", "docs/THREAT_MODEL.md"): + text = read_text(relative_path) + with self.subTest(path=relative_path): + self.assertEqual(text.count("```") % 2, 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_live_product_gap_traceability_contract.py b/scripts/test_live_product_gap_traceability_contract.py new file mode 100644 index 000000000..3ce23e80a --- /dev/null +++ b/scripts/test_live_product_gap_traceability_contract.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Keep canonical documentation aligned with newly accepted Clearfolio product gaps.""" + +from __future__ import annotations + +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one repository document as normalized lowercase text.""" + + return " ".join( + (REPOSITORY_ROOT / relative_path) + .read_text(encoding="utf-8") + .lower() + .split() + ) + + +class LiveProductGapTraceabilityContractTest(unittest.TestCase): + """Require current buyer-visible gaps and bounded remediation slices to be traceable.""" + + def test_traceability_tracks_production_workspace_demo_authority_gap(self) -> None: + """Require issue #317 and its bounded branding slice to remain explicit.""" + + traceability = read_text("docs/TRACEABILITY.md") + for required in ( + "issue #317", + "production workspace", + "buyer-demo", + "#318", + "branding", + ): + with self.subTest(required=required): + self.assertIn(required, traceability) + + def test_traceability_separates_openapi_license_integrity_from_schema_completeness(self) -> None: + """Require #316 to stay a bounded Apache-2.0 integrity slice beneath issue #315.""" + + traceability = read_text("docs/TRACEABILITY.md") + for required in ( + "issue #315", + "#316", + "apache-2.0", + "bounded integrity", + "partial", + ): + with self.subTest(required=required): + self.assertIn(required, traceability) + + def test_traceability_tracks_runtime_credential_registry_gap(self) -> None: + """Require issue #319 to stay separate from the active key-readiness slice.""" + + traceability = read_text("docs/TRACEABILITY.md") + for required in ( + "issue #319", + "credential registry", + "environment", + "#313", + ): + with self.subTest(required=required): + self.assertIn(required, traceability) + + def test_traceability_tracks_current_clean_replacements_and_main_advancement(self) -> None: + """Prevent closed predecessor PR identities from remaining the canonical live mapping.""" + + traceability = read_text("docs/TRACEABILITY.md") + for required in ( + "protected `main` at `55d7ae8647208e301f282350f076eeddaba61d11`", + "direct-download signed delivery | `implemented_on_main`", + "privacy-safe hmac audit pseudonymization | `implemented_on_main`", + "exact-head ci/test evidence | `implemented_on_main`", + "issue #324; current `active_pr` #334", + "issue #327; current `active_pr` #338", + "issue #329; current `active_pr` #389", + "issue #315; current `active_pr` #337", + "#325", + "superseded", + "#328", + "#330", + ): + with self.subTest(required=required): + self.assertIn(required, traceability) + + def test_traceability_tracks_current_accessibility_and_logging_reconciliations(self) -> None: + """Require the newly verified current-base slices to replace historical/deferred prose.""" + + traceability = read_text("docs/TRACEABILITY.md") + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + + for required in ( + "nested-safe accessible asynchronous controls | `active_pr`; current #264", + "deterministic single logging runtime binding | `active_pr`; issue #320; current #340", + "dependency-free node", + "spring-jcl", + "commons-logging", + ): + with self.subTest(document="traceability", required=required): + self.assertIn(required, traceability) + + for required in ( + "current #264", + "current #340", + "#268", + "superseded", + ): + with self.subTest(document="assessment", required=required): + self.assertIn(required, assessment) + self.assertNotIn("only #268 remains unreconciled", assessment) + + def test_canonical_docs_reject_stale_268_and_direct_download_maturity(self) -> None: + """Require shipped direct download and superseded deletion ancestry to be classified truthfully.""" + + uml = read_text("docs/UML.md") + data_model = read_text("docs/DATA_MODEL.md") + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + + self.assertIn("direct conversion-job download", uml) + self.assertIn("`implemented_on_main`", uml) + self.assertNotIn("direct conversion-job download alignment is an `active_pr`", uml) + + for document_name, document in (("uml", uml), ("data_model", data_model)): + with self.subTest(document=document_name): + self.assertIn("#268", document) + self.assertIn("superseded", document) + self.assertIn("partial", document) + self.assertNotIn("`active_pr` #268", document) + + self.assertIn("#268", assessment) + self.assertIn("superseded", assessment) + self.assertNotIn("only #268 remains unreconciled", assessment) + + def test_all_canonical_docs_reject_superseded_268_and_stale_download_maturity(self) -> None: + """Keep lifecycle and signed-download maturity consistent across canonical docs.""" + + supersession_documents = ( + "docs/PRD.md", + "docs/TRD.md", + "ARCHITECTURE.md", + "docs/OPERABILITY.md", + "docs/API_CONTRACT.md", + "docs/THREAT_MODEL.md", + "docs/MIGRATION_ROLLBACK.md", + "docs/TEST_STRATEGY.md", + "docs/adr/README.md", + "docs/adr/0003-audit-pseudonymization-key-separation.md", + "docs/adr/0004-durable-lifecycle-generation-fencing.md", + "docs/DOCUMENTATION_ASSESSMENT.md", + "docs/ACQUISITION_DILIGENCE.md", + ) + for relative_path in supersession_documents: + document = read_text(relative_path) + with self.subTest(document=relative_path): + self.assertIn("#268", document) + self.assertIn("superseded", document) + self.assertNotIn("`active_pr` #268", document) + + signed_download_documents = ( + "docs/PRD.md", + "docs/TRD.md", + "ARCHITECTURE.md", + "docs/THREAT_MODEL.md", + "docs/adr/README.md", + ) + for relative_path in signed_download_documents: + document = read_text(relative_path) + with self.subTest(document=relative_path): + self.assertIn("implemented_on_main", document) + self.assertNotIn("direct-download hardening `active_pr`", document) + self.assertNotIn("direct-download alignment `active_pr`", document) + self.assertNotIn( + "direct conversion-job download endpoint is currently under `active_pr`", + document, + ) + + traceability = read_text("docs/TRACEABILITY.md") + self.assertIn("assessment date: 2026-09-20", traceability) + + def test_assessment_exposes_current_api_workspace_and_credential_gaps(self) -> None: + """Require the fitness assessment to expose current executable product/security gaps.""" + + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + for required in ( + "issue #315", + "issue #317", + "issue #319", + "production workspace", + "versioned api", + "credential registry", + ): + with self.subTest(required=required): + self.assertIn(required, assessment) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_office_architecture_documentation_contract.py b/scripts/test_office_architecture_documentation_contract.py new file mode 100644 index 000000000..7705f132b --- /dev/null +++ b/scripts/test_office_architecture_documentation_contract.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Require truthful Office-adapter architecture and logical data-model coverage.""" + +from __future__ import annotations + +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one canonical repository document as strict UTF-8 text.""" + + return (REPOSITORY_ROOT / relative_path).read_text(encoding="utf-8") + + +class OfficeArchitectureDocumentationContractTest(unittest.TestCase): + """Keep Office conversion isolation visible without claiming it is shipped.""" + + def test_uml_distinguishes_active_adapter_from_planned_runtime(self) -> None: + """Require an explicit Office adapter isolation flow with truthful maturity.""" + + uml = read_text("docs/UML.md").lower() + for required in ( + "officeconversionadapter", + "active_pr", + "sandboxed_office_sidecar", + "remote_office_service", + "deterministic_fixture_adapter", + "planned", + "api container", + "outside", + ): + with self.subTest(required=required): + self.assertIn(required, uml) + + def test_data_model_labels_office_qualification_entities_conceptual(self) -> None: + """Require logical qualification entities without inventing persistence.""" + + data_model = read_text("docs/DATA_MODEL.md").lower() + for required in ( + "conversion_engine", + "conversion_attempt", + "conversion_quarantine", + "adapter_health_snapshot", + "format_support_record", + "conversion_audit_event", + "conceptual", + "planned", + ): + with self.subTest(required=required): + self.assertIn(required, data_model) + + self.assertIn("does not imply a database table", data_model) + + def test_traceability_keeps_issue_5_and_pr_306_non_shipped(self) -> None: + """Keep Office qualification linked to its issue and active Draft implementation.""" + + traceability = read_text("docs/TRACEABILITY.md").lower() + for required in ( + "issue #5", + "#306", + "active_pr", + "docs/uml.md", + "docs/data_model.md", + ): + with self.subTest(required=required): + self.assertIn(required, traceability) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_release_loop_adr_contract.py b/scripts/test_release_loop_adr_contract.py new file mode 100644 index 000000000..584fbdab7 --- /dev/null +++ b/scripts/test_release_loop_adr_contract.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Protect high-value release and work-conserving automation ADR details.""" + +from __future__ import annotations + +import re +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one repository text file as original UTF-8 text.""" + + return (REPOSITORY_ROOT / relative_path).read_text(encoding="utf-8") + + +class ReleaseLoopAdrContractTest(unittest.TestCase): + """Keep review-driven release and loop semantics explicit and testable.""" + + def test_fresh_exit_sweep_is_an_executable_revalidation_procedure(self) -> None: + """Require each sweep to rebuild authority and queue state from live evidence.""" + + adr = read_text("docs/adr/0009-work-conserving-rca-loop.md").lower() + for required in ( + "each fresh exit sweep", + "refetch live head and live base", + "rebuild the executable queue", + "deferred identities", + "writer lease", + "commit statuses", + "model verdicts", + "check runs", + "workflow evidence", + "invalidate evidence not bound to the current exact identity", + "safe item appears between sweep one and sweep two", + "must not terminate", + "execute any safe item", + "restart the exit-sweep count after that action", + "two consecutive fresh sweeps with no safe executable work", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + def test_release_adr_delegates_to_canonical_complete_acceptance(self) -> None: + """Prevent a green but incomplete evidence set from satisfying release policy.""" + + raw_adr = read_text("docs/adr/0010-release-provenance-fidelity-gate.md") + self.assertIn("docs/RELEASE_ACCEPTANCE.md", raw_adr) + + adr = raw_adr.lower() + for required in ( + "zero skips, failures, and errors", + "zero warnings and deprecations", + "warning-free public javadocs/doclint", + "applicable javascript coverage", + "markdown lint", + "live required review and security gates", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + canonical_release = read_text("docs/RELEASE_ACCEPTANCE.md").lower() + for required in ( + "exact version/tag", + "immutable dependency lock/resolution state", + "build artifact digests", + "sbom/provenance material generated from that source", + "third-party attribution/license policy evidence", + "artifact checksums", + "rollback/recovery follows `docs/migration_rollback.md`", + "restart/recovery behavior", + ): + with self.subTest(contract_owner=required): + self.assertIn(required, canonical_release) + + def test_scheduler_control_plane_delegates_detail_to_repository_authority(self) -> None: + """Keep recurring scheduler control compact while preserving repository authority.""" + + adr = read_text("docs/adr/0011-thin-scheduler-control-plane.md").lower() + for required in ( + "thin control plane", + "repository documents are the detailed authority", + "must not duplicate the full product specification", + "read current canonical repository documents before selecting or changing product work", + "scheduler execution failure", + "same invocation", + "double exit sweep", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + operability = read_text("docs/OPERABILITY.md").lower() + for required in ( + "scheduled-task execution failure", + "scheduling/activation", + "execution failure", + "prompt-size", + "duplicated-state", + "repository authority", + ): + with self.subTest(required=required): + self.assertIn(required, operability) + + def test_scheduler_prewrite_refetch_is_complete_and_fail_closed(self) -> None: + """Require each write to bind an ordered fresh-state contract before mutation.""" + + ordered_contract = re.compile( + r"before every write.*" + r"exact (?:source|target) head.*" + r"live base.*" + r"target blob/ref.*" + r"relevant review state.*" + r"(?:target identity|another writer).*(?:freeze|stop|reject).*(?:rotate|abort)", + flags=re.DOTALL, + ) + for relative_path in ( + "docs/adr/0011-thin-scheduler-control-plane.md", + "docs/OPERABILITY.md", + ): + text = " ".join(read_text(relative_path).lower().split()) + with self.subTest(path=relative_path): + self.assertRegex(text, ordered_contract) + for required in ( + "formal reviews", + "unresolved threads", + "exact-head checks", + "security gates", + ): + with self.subTest(path=relative_path, required=required): + self.assertIn(required, text) + + def test_scheduler_adr_ends_with_one_newline(self) -> None: + """Keep ADR-0011 compliant with the repository Markdown trailing-newline gate.""" + + adr = read_text("docs/adr/0011-thin-scheduler-control-plane.md") + self.assertTrue(adr.endswith("\n")) + self.assertFalse(adr.endswith("\n\n")) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_scheduler_execution_receipt_documentation_contract.py b/scripts/test_scheduler_execution_receipt_documentation_contract.py new file mode 100644 index 000000000..5a0b9bccb --- /dev/null +++ b/scripts/test_scheduler_execution_receipt_documentation_contract.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +"""Require canonical documentation for diagnosable scheduler continuation.""" + +from __future__ import annotations + +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def read_text(relative_path: str) -> str: + """Return one repository document as normalized lowercase text.""" + + document_path = REPOSITORY_ROOT / relative_path + if not document_path.exists(): + return "" + return " ".join(document_path.read_text(encoding="utf-8").lower().split()) + + +class SchedulerExecutionReceiptDocumentationContractTest(unittest.TestCase): + """Keep scheduler failures diagnosable without inventing hidden causes.""" + + def test_adr_index_links_execution_receipt_decision(self) -> None: + """Require a dedicated decision beyond the thin-control-plane ADR.""" + + adr_index = read_text("docs/adr/README.md") + for required in ( + "0012-scheduler-execution-receipts.md", + "execution receipt", + "budget continuation", + ): + with self.subTest(required=required): + self.assertIn(required, adr_index) + + def test_execution_receipt_adr_covers_failure_and_recovery(self) -> None: + """Require exact failure-envelope, checkpoint, and supersession semantics.""" + + adr = read_text("docs/adr/0012-scheduler-execution-receipts.md") + for required in ( + "status: proposed", + "automation checkpoint", + "action receipt", + "failure envelope", + "continuation handoff", + "budget continuation", + "generic scheduled-task error", + "do not invent", + "rollback", + "supersession", + ): + with self.subTest(required=required): + self.assertIn(required, adr) + + def test_data_model_marks_scheduler_receipts_conceptual_or_external(self) -> None: + """Prevent the ERD from inventing Clearfolio-owned scheduler persistence.""" + + data_model = read_text("docs/DATA_MODEL.md") + for required in ( + "automation_checkpoint", + "action_receipt", + "failure_envelope", + "continuation_handoff", + "conceptual", + "external", + ): + with self.subTest(required=required): + self.assertIn(required, data_model) + + def test_uml_and_operability_show_resumable_execution_flow(self) -> None: + """Require a visible flow from admission through receipt or continuation.""" + + uml = read_text("docs/UML.md") + operability = read_text("docs/OPERABILITY.md") + for required in ( + "schedule", + "admission", + "fresh queue", + "atomic action", + "action receipt", + "budget continuation", + "failure envelope", + ): + with self.subTest(document="uml", required=required): + self.assertIn(required, uml) + for required in ( + "issue #331", + "generic scheduled-task error", + "last safe checkpoint", + "budget continuation", + "controlled failure", + "fresh github state", + ): + with self.subTest(document="operability", required=required): + self.assertIn(required, operability) + + def test_traceability_and_assessment_expose_live_scheduler_gap(self) -> None: + """Require issue #331 and its non-shipped maturity to stay explicit.""" + + traceability = read_text("docs/TRACEABILITY.md") + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + for required in ( + "issue #331", + "scheduler execution receipt", + "planned", + ): + with self.subTest(document="traceability", required=required): + self.assertIn(required, traceability) + for required in ( + "issue #331", + "execution receipt", + "protected_main_insufficient", + ): + with self.subTest(document="assessment", required=required): + self.assertIn(required, assessment) + + def test_assessment_states_conversation_coverage_and_enforcement_limits(self) -> None: + """Require an explicit verdict for the repeated early-stop and documentation request.""" + + assessment = read_text("docs/DOCUMENTATION_ASSESSMENT.md") + for required in ( + "conversation decision coverage", + "conversation_coverage_sufficient", + "control_enforcement_incomplete", + "prompt repair earns zero completion credit", + "same-invocation substantive action", + "multi-lane rotation", + "two consecutive fresh exit sweeps", + "repository-scoped conversation decisions", + "other contextualwisdomlab projects are not imported", + ): + with self.subTest(required=required): + self.assertIn(required, assessment) + + +if __name__ == "__main__": + unittest.main()