[Commercialization Gap] Productionize tenancy, durability, observability, release, and support #174

Buyer problem

TEPP has strong repository-level contracts and partial PostgreSQL/API/security slices, but it is not yet an operable multi-tenant product. Buyers cannot install a supported release, authenticate users/services, run durable jobs, monitor SLOs, recover from failures, upgrade safely, or obtain a complete procurement/assurance evidence pack.

Product outcome

Turn the scientifically validated product vertical into a production service that can run standalone and as a modular CWL service. The service must preserve TEPP's purpose-bound PII utility without blanket masking and must remain honest about CSAP/SOC 2 readiness versus certification.

Required capabilities

Deployment and runtime

• versioned public API/CLI and generated clients;
• durable analysis queue, status lifecycle, cancellation, retry, dead-letter and idempotency;
• PostgreSQL reference store plus object storage for immutable source/model/export artifacts;
• Compose deployment first, with Kubernetes-ready probes/configuration and no hidden monolithic assumptions;
• production TLS and authenticated service boundaries;
• bounded resource/capacity controls and backpressure.

Identity, tenancy, and privacy

• Keyverse-compatible OIDC/OAuth/JWT and service identity adapter;
• tenant isolation through PostgreSQL RLS or equivalent mandatory enforcement;
• purpose-, role-, lifetime-, and field-bound access;
• protected re-identification/identity-mapping path without disabling legitimate PII-dependent work;
• retention, legal hold, export, deletion, backup, and restore controls with evidence.

Destructive-operation authorization

• loopback binding, tepp-consumer, tenant/workspace IDs, idempotency keys, resource IDs, or other caller-supplied routing/context fields are not authentication or authorization grants;
• cancellation, deletion, revocation, replacement, export withdrawal, and other state-destroying mutations require an authenticated human/service principal plus a purpose- and resource-bound authorization decision issued through the production identity/policy boundary;
• mutation authorization must bind at least principal, tenant/workspace, consumer/application identity, operation, target resource, purpose, issued/expiry time, and replay identity; stale, cross-tenant, cross-resource, cross-operation, or replayed grants fail closed;
• credentials/grants must not be passed through ordinary CLI arguments, echoed into receipts, logs, traces, process listings, persisted project-history/export records, or scientific artifacts;
• mutation handlers validate and serialize the complete success receipt before committing irreversible state changes; any fallible validation/serialization failure leaves the original resource intact;
• every externally consumable mutation receipt has an explicit semantic contract version and preserves the semantic invariants of the resource from which it was derived;
• accepted resources must not become uncancellable because creation and cancellation disagree on identifier encoding/length contracts;
• authorization decisions and successful destructive mutations emit immutable, redacted audit/provenance evidence without becoming scientific evidence or architecture authority.

The unauthenticated loopback cancellation experiments carried by PRs #361/#373/#378, #440/#442, #445/#446, and #447/#448 are research/delivery lineage only until this boundary exists. Their useful lifecycle/versioning, parsing, atomicity, receipt, path-limit, consumer-isolation, and fail-closed tests should be folded into the authenticated Analysis Run/operations landing vehicle rather than reviving header-trust mutation endpoints. Descendant read/retry/operator stacks that were based on these branches must be reconstructed so closure of the cancellation experiments does not silently reintroduce their mutation code.

Operability and assurance

• OpenTelemetry traces/metrics/logs without source-text or credential leakage;
• SLOs for submission, queue latency, run completion, artifact retrieval, error rate, and recovery;
• runbooks for provider outage, database/object-store outage, corrupted artifact, model regression, GPU loss, queue backlog, security incident, and rollback;
• capacity/load/soak/failover tests and measured hot-partition mitigation;
• immutable audit/provenance, SBOM, dependency/license evidence, signed build provenance, reproducible release artifacts, and vulnerability-response process;
• backup/restore, point-in-time recovery, migration upgrade/downgrade rehearsal, and disaster-recovery evidence.

Acceptance criteria

• A clean environment can install and run a versioned release from documented Compose commands without repository-internal credentials.
• Liveness, readiness, startup, and dependency health are separate and fail with content-redacting responses.
• Durable jobs survive worker/API restarts; duplicate submission and retry produce no duplicate chargeable/scientific artifacts.
• Cross-tenant reads/writes, forged tenant headers, invalid JWT audience/signature, stale grants, purpose escalation, operation/resource grant reuse, and unauthenticated destructive loopback mutations fail closed.
• Every destructive operation proves authenticated principal + tenant + operation + resource + purpose authorization before mutation, and a failed receipt build/serialization leaves state unchanged.
• External consumers use versioned APIs/events only; no CWL product reads TEPP application tables directly.
• OpenTelemetry spans connect request → queue → compute → validation → artifact → result without copying source text, identity mapping, provider keys, or unrestricted model output.
• Capacity tests cover realistic corpus sizes and concurrent tenants; queue depth, CPU/GPU/RAM/VRAM, database/object-store, and backpressure limits are documented.
• Backup/restore and corrupted-manifest rehearsals prove restored artifacts are unusable until integrity, tenant, cutoff, and provenance checks pass.
• Every release has version/CHANGELOG consistency, signed tag/artifacts, SPDX or CycloneDX SBOM, build provenance, checksums, migration manifest, rollback steps, and exact protected-head evidence.
• Security/threat-model tests cover SSRF, request smuggling/framing, hostile Unicode/JSON/archive, prompt injection, dependency compromise, log/export leakage, privilege escalation, destructive-operation spoofing/replay, and denial-of-service bounds.
• CSAP/SOC 2/ISO/NIST mappings identify control owner, implementation, runtime evidence, test, known limitation, and deployment/customer responsibility; no certification or attestation is claimed without independent evidence.
• Support policy defines compatibility, upgrade window, deprecation, incident severity, response/communication, data portability, and vulnerability disclosure.
• A release candidate executes the end-to-end product issue with exact-head CI/security/scientific evidence and no unresolved release blocker.

Commercial proof

Prepare a buyer-facing but source-backed package containing architecture, data flow, threat model, deployment sizing, SLOs, security/privacy controls, validation evidence, limitations, pricing/meter hooks, migration/rollback, and an anonymized reproducible demo. Every claim must link to protected-main or released evidence.

Scope boundary

This issue does not ask the repository to self-certify CSAP/SOC 2, store payment-card data, silently mask all PII, or weaken scientific gates for availability. A failed scientific-integrity gate is an operational failure, not a warning to hide.

Authority

Use OPERABILITY, COMPLIANCE_READINESS, SECURITY, THREAT_MODEL, ADR 0011/0013/0014, and the latest authoritative standards recorded in the repository doctoring register.