[Product Gap] Make conversion jobs durable with backpressure and cancellation #312

Buyer-visible outcome

Make Clearfolio conversion jobs survive process restart and scale beyond one process-local executor without duplicating work, losing accepted jobs, or reporting false success. Buyers must receive explicit backpressure and cancellation semantics instead of a process-local queue that can disappear on restart.

Current protected-main evidence

Protected main remains the implementation authority. At f3cc09a9838f0f88c81a2ceae22138fab80a2edb:

• DefaultDocumentConversionService.submit(...) persists the job through the configured repository and then calls conversionWorker.enqueue(...); there is no atomic durable job+outbox acceptance boundary.
• ConversionExecutorConfig uses a bounded process-local ThreadPoolTaskExecutor with configured worker threads and queue capacity.
• DefaultConversionWorker.enqueue(...) dispatches through CompletableFuture.runAsync; queue rejection marks the job dead-lettered rather than exposing a capacity/backpressure contract to the submitter.
• startup recovery can re-enqueue jobs represented by the configured repository, but the default conversion-job repository remains process-local, so process restart loses the authoritative default job inventory.
• retry timers use process-local delayed executors.
• no versioned public cancellation lifecycle is shipped.

This is not a criticism of the current bounded executor: it is an explicit transition from a safe local asynchronous implementation to a production durable execution model.

Dependency / non-duplication boundary

Do not duplicate work already owned by the active security/lifecycle stack:

• fix(security): harden audit pseudonymization and refresh Netty evidence #270 owns the current security/privacy/test-evidence parent and signed direct-download alignment;
• fix(security): require signed tenant claims on admin endpoints #268 owns immutable job identity/generation fencing plus deletion receipts/restart-safe artifact cleanup;
• [Product Gap] Complete tenant-safe document lifecycle and artifact deletion #263 owns the end-to-end tenant-safe deletion/download product journey;
• feat(operations): separate liveness and readiness probes #295 owns liveness/readiness separation;
• issue [Product Gap] Qualify a sandboxed Office conversion adapter #5/feat(conversion): establish provider-neutral Office adapter contract #306 own Office-runtime qualification and converter-specific cancellation/process cleanup.

This issue owns general conversion-job acceptance, durable dispatch, backpressure, worker lease, cancellation, and crash/restart execution semantics. Reuse lifecycle generation/fencing from #268 after it integrates; do not fork it.

Required architecture

Introduce versioned replaceable boundaries rather than coupling controllers to one broker/database:

• durable conversion_job state repository;
• transactional conversion_outbox or equivalent atomic accepted-job dispatch record;
• idempotency identity bound to tenant, source digest, conversion policy version and requested output contract;
• lease-fenced conversion_attempt / worker claim with immutable generation;
• durable retry scheduling rather than process-local delayed executors;
• explicit queue capacity/admission and backpressure result;
• explicit cancellation request/state with generation-bound worker/process termination semantics;
• inbox/receipt or equivalent duplicate-delivery suppression where the selected broker needs it;
• privacy-safe append-only execution/audit evidence;
• adapter-neutral execution so standalone/local and MSA deployments share the same public contract.

Database objects, when introduced, must use descriptive two-or-more-word snake_case names such as:

• conversion_job
• conversion_outbox
• conversion_attempt
• worker_lease
• cancellation_request
• idempotency_record
• execution_audit_event

The logical model does not force PostgreSQL for every deployment. A standalone profile may use another durable adapter if it preserves the same semantics and recovery tests.

API contract

Define versioned public behavior for at least:

• accepted job response and idempotent duplicate response;
• capacity/backpressure response, including deterministic retry guidance where appropriate;
• status polling with monotonic lifecycle/generation semantics;
• cancellation request, repeated cancellation, cancellation-vs-completion race and terminal result;
• retryable versus terminal failure;
• restart/recovery behavior;
• tenant-scoped authorization and object-existence concealment.

Do not overload FAILED with capacity admission rejection if the job was never durably accepted. Do not report 202 Accepted until the job is durably recoverable.

Reliability invariants

• A successful acceptance response implies the exact job can be recovered after process/worker restart.
• One idempotency identity cannot publish multiple successful artifacts for the same output contract.
• A worker that loses its lease/generation cannot publish or mutate the current job.
• Redelivery/retry is safe under duplicate messages and concurrent workers.
• Cancellation is monotonic and cannot cancel a later generation or another tenant's job.
• A cancelled/failed attempt cannot leak its temporary files into a later attempt.
• Queue saturation is observable and produces explicit backpressure rather than silent loss.
• Recovery after crash between state transition, outbox write, broker delivery, worker claim, artifact publication and completion is deterministic and tested.

Security and privacy

• tenant identity/permission is server authoritative;
• broker/database credentials use least privilege and are never surfaced to clients/model prompts/logs;
• no raw tenant/subject/file/token/document content in high-cardinality telemetry;
• durable audit evidence remains purpose-bound and pseudonymized where appropriate;
• cross-tenant IDs remain non-enumerating;
• cancellation/retry/admin operations require separate least-privilege permissions;
• no hidden cross-service database writes from naruon or other CWL hosts.

Observability

Expose privacy-safe OpenTelemetry-compatible traces/metrics for:

• acceptance/admission result;
• durable queue depth/age and oldest-job age;
• active/expired worker leases;
• attempt latency and retry delay;
• duplicate/idempotency hits;
• cancellation latency/outcome;
• recovery count;
• dead-letter/quarantine count;
• capacity saturation.

Do not claim an SLO until the target deployment profile has measured evidence.

Test-first acceptance

Before implementation, reproduce the current limits with deterministic tests, then require:

• crash after durable acceptance but before broker dispatch;
• broker redelivery and duplicate messages;
• two workers racing one job;
• lease expiry followed by stale worker completion;
• restart with pending/retryable jobs;
• bounded queue saturation/backpressure;
• idempotent duplicate submit under concurrency;
• cancellation before claim, during execution, after completion and repeated cancellation;
• cancellation/retry generation races;
• database/broker temporary outage and recovery;
• tenant-isolation and permission tests;
• migration and rollback tests from the current process-local profile;
• realistic integration tests using an ephemeral supported durable backend/profile;
• exact 100% owned production line/statement and branch coverage, complete public Javadocs/docstrings, zero required skips/failures/errors;
• exact-head CI/security/SAST/fuzz/review and independent approval under repository policy.

Documentation / release updates

Update the canonical docs/PRD.md, docs/TRD.md, ARCHITECTURE.md, ADR index/decisions, docs/DATA_MODEL.md, docs/UML.md, docs/API_CONTRACT.md, docs/THREAT_MODEL.md, docs/TEST_STRATEGY.md, docs/OPERABILITY.md, docs/MIGRATION_ROLLBACK.md, docs/TRACEABILITY.md, docs/ACQUISITION_DILIGENCE.md, and CHANGELOG.md with truthful maturity.

Release only after one exact integrated protected head proves durability, cancellation, backpressure, tenant isolation, migration/rollback/recovery, observability, packaging/SBOM/provenance, review and operational acceptance.

Suggested implementation order

1. Persisted state + atomic outbox/idempotent acceptance.
2. Lease-fenced worker claim and durable retry schedule.
3. Explicit backpressure/admission response.
4. Versioned cancellation contract and worker fencing.
5. Restart/redelivery/concurrency/migration/rollback acceptance.
6. OpenTelemetry operational evidence and protected-main release acceptance.

Do not start this source slice on a branch that conflicts with the current #270/#268 stack. Re-evaluate the integrated protected baseline first.