diff --git a/.github/workflows/terraform-modules.yml b/.github/workflows/terraform-modules.yml index 5c2c2a280c1..d8c6ddcdcbe 100644 --- a/.github/workflows/terraform-modules.yml +++ b/.github/workflows/terraform-modules.yml @@ -38,7 +38,7 @@ jobs: run: | python3 - <<'PY' import json, os, subprocess - modules = ["openwork-aws-ecs", "openwork-gcp-cloud-run", "openwork-azure-container-apps"] + modules = ["openwork-aws-ecs", "openwork-gcp-cloud-run", "openwork-azure-container-apps", "openwork-self-host"] base = os.environ["BASE_SHA"] try: if base == "MANUAL" or not base.strip("0"): @@ -101,11 +101,16 @@ jobs: - name: Format, initialize, validate and mocked tests with both tools working-directory: infra/terraform/modules/${{ matrix.module }} run: | + example=examples/complete + if [[ "$MODULE" == openwork-self-host ]]; then example=examples/basic; fi for tool in terraform tofu; do "$tool" fmt -check -recursive # A lock belongs to one registry. Regenerate explicitly when switching tools. if [[ "$tool" == tofu ]]; then - rm -f .terraform.lock.hcl examples/complete/.terraform.lock.hcl + rm -f .terraform.lock.hcl "$example/.terraform.lock.hcl" + if [[ -f "$example/opentofu.lock.hcl" ]]; then + cp "$example/opentofu.lock.hcl" "$example/.terraform.lock.hcl" + fi fi # Reusable modules ship no locks; preserve a future committed lock if present. if [[ ! -f .terraform.lock.hcl ]]; then @@ -118,7 +123,7 @@ jobs: fi "$tool" test ( - cd examples/complete + cd "$example" if [[ ! -f .terraform.lock.hcl ]]; then "$tool" init -backend=false -input=false fi @@ -126,6 +131,13 @@ jobs: "$tool" validate ) done + - uses: actions/setup-node@v6 + if: matrix.module == 'openwork-gcp-cloud-run' + with: + node-version: '24.22.1' + - name: Cloud SQL Gateway entrypoint compatibility (offline) + if: matrix.module == 'openwork-gcp-cloud-run' + run: node --test infra/terraform/modules/openwork-gcp-cloud-run/scripts/cloud-sql-entrypoint.test.mjs - name: Install verified Trivy id: trivy-install if: ${{ !cancelled() }} diff --git a/infra/managed-deployments/README.md b/infra/managed-deployments/README.md index 1b61d2ea068..83790583615 100644 --- a/infra/managed-deployments/README.md +++ b/infra/managed-deployments/README.md @@ -27,6 +27,18 @@ tools/ Release build and publication (provider argument) tests/ Installer, runner and health agent tests ``` +## Optional AI Gateway module support + +AWS and Google Cloud roots consume the native Terraform modules **directly**, not +Helm. The modules' new `ai_gateway_*` inputs are documented in +[the AI Gateway guide](../terraform/AI_GATEWAY.md); these managed roots do not set +them, so the AI Gateway remains **off**. This does not change installer payload +publication, bootstrap permissions, managed defaults or rollout state. Enabling a +managed install needs a separately reviewed root/image/DNS/certificate/permission +change, required re-approval and fresh live install/upgrade proof. Mock module +plans are not a managed-readiness receipt. The AI Gateway is distinct from +OpenWork Web's den-gateway and Den API's MCP gateway. + ## AWS shared-module adoption The AWS root consumes `infra/terraform/modules/openwork-aws-ecs`. The former diff --git a/infra/terraform/AI_GATEWAY.md b/infra/terraform/AI_GATEWAY.md new file mode 100644 index 00000000000..c7c89129c37 --- /dev/null +++ b/infra/terraform/AI_GATEWAY.md @@ -0,0 +1,155 @@ +# Optional AI Gateway in Terraform + +The native AWS ECS, Google Cloud Run and Azure Container Apps modules can deploy +**the AI Gateway** (`ee/apps/gateway`, image `openwork-inference`, port **8791**). +Select a reviewed source revision containing these inputs; older module versions +do not support them. This guide is not a live qualification or release receipt. +It manages organization inference providers, member keys and limits. This is not +**den-gateway**, the AWS OpenWork Web/sandbox proxy on port **8788**, nor Den API's +**MCP gateway**. Enabling one does not enable the others. + +## Common native-module interface + +Add these inputs to an existing module call (alongside its required inputs): + +```hcl +ai_gateway_enabled = true +ai_gateway_domain_name = "models.example.com" +ai_gateway_image = var.ai_gateway_image # image@sha256: +``` + +| Input/output | Default | Meaning | +| --- | --- | --- | +| `ai_gateway_enabled` | `false` | Installation configuration, like Helm `gateway.enabled`, not a product rollout flag | +| `ai_gateway_domain_name` | `""` | Required when enabled; dedicated qualified non-local hostname, without scheme, port or path | +| `ai_gateway_image` | `""` | Required when enabled; digest-pinned `openwork-inference` image from the **same release as Den** (or its mirror) | +| `ai_gateway_url` (output) | `null` | When enabled, `https://` for desktops | + +No digest is invented or selected from a mutable tag. Use **Gateway-capable Den +API/web and Gateway images from the same release**; the historical built-in Den +0.18.57/0.18.58 conformance pins are not proof of Gateway capability. For another +release, supply the Den image overrides required by that module as well. Verify +the release's image manifests before deployment. Cloud Run cannot pull directly from GHCR: mirror +`ghcr.io/different-ai/openwork-inference` to Artifact Registry, or use its existing +pull-through repository and the corresponding digest-pinned repository path. +`ai_gateway_image` is explicit even if `allow_mutable_image_tags` is enabled for +legacy Den images. All native Gateway services reserve 1 vCPU / 2 GiB; ECS/Azure +run one replica, Cloud Run keeps one warm and permits up to three. These are +unbenchmarked starting sizes, not production capacity recommendations. + +When disabled, the module creates no AI Gateway service, route, DNS/certificate +addition or new `GATEWAY_*` environment names. Missing `GATEWAY_ENABLED` is false +at runtime. Existing explicit environment overrides remain unchanged on AWS/GCP; +they still do **not** deploy a service. Remove manual Gateway/Inference overrides +before opting in. Enabled modules reject `GATEWAY_*`, legacy `INFERENCE_*` and +shared database/runtime overrides in `extra_environment` **and** `extra_secrets`. +Azure's narrower plaintext allowlist continues to apply even when disabled. + +## Wiring and networking + +The chart's `_gateway.tpl` / `inference.yaml` are the reference. Both den-api and +the AI Gateway get `GATEWAY_ENABLED=true`, `DB_MODE=mysql`, production mode, +`GATEWAY_PROXY_BASE_URL`, `GATEWAY_PUBLIC_BASE_URL` and the **existing** database +URL and `DEN_DB_ENCRYPTION_KEY`. `INFERENCE_PROXY_BASE_URL` is the public origin, +so legacy desktop model destinations never receive a private service address. +Only the singleton Den migrator owns schema upgrades; Gateway replicas never +migrate. Optional admin/webhook tokens and scheduled retention are not enabled by +these inputs. Native Gateway containers receive only database/encryption secrets, +not Den's auth/bootstrap/SMTP secrets. Azure attaches only an explicitly configured +registry-pull identity to the Gateway, not Den's Key Vault/integration identities. +The Helm wrapper retains the chart's shared Secret/envFrom behavior. No new secret generator or rotation counter is added. + +| Cloud | Den → AI Gateway | Desktop → AI Gateway | Operator prerequisites | +| --- | --- | --- | --- | +| AWS ECS | Private Cloud Map `http://ai-gateway..internal:8791`, tasks security group only | Existing HTTPS ALB host rule → target group 8791, `/health` | Cover the hostname in ACM and DNS; module-created ACM/Route 53 include it. Existing ALB/listener DNS/cert ownership stays external. `ai_gateway_listener_rule_priority` defaults to **50**, validated against other enabled module rules; check external rules too. | +| Google Cloud Run | **HTTPS load-balancer origin**, same as public URL; no private service-discovery guarantee | Dedicated serverless NEG/backend and host rule; default run.app URI disabled; separate managed AI certificate preserves the existing Den certificate | DNS A record to `load_balancer_ip`; Compute API and LB permissions. A Gateway-only hostname creates an LB even when Den keeps run.app URLs. With `ALL_TRAFFIC`, supply Cloud NAT/egress to the public HTTPS origin and model providers. | +| Azure Container Apps | Generated HTTPS FQDN, routed within the same environment | Dedicated HTTPS ingress 8791 and custom-domain SNI binding; `/health` and `/ready` probes | Preconfigure CNAME + `asuid` TXT using the known app/environment hostname; supply `custom_domain_certificate` PFX covering the Gateway (and Den custom hosts, if used). The environment must exist before DNS verification. Certificate material uses the existing state-backed provider path. | + +AWS private ALBs and Azure internal environments still require desktop VPN/private +network reachability and trusted HTTPS. This option does not punch through an +existing private boundary. AWS allows 8791 only from ALB/tasks security groups; +it does not open a public container port. Cloud Run delegates authentication to +the Gateway's member keys, not Google ID tokens, as desktops do not supply those. + +### Cloud SQL socket compatibility + +Existing Cloud SQL secrets encode `mysql://...@localhost/...?...socket=/cloudsql/...`. +Enabled-Gateway production validation rejects localhost even for a Unix socket. +The Cloud Run packaging entrypoint changes **only the unused hostname in process +memory** to `cloudsql`, before importing den-api or the Gateway server. It preserves +credentials, database, socket path, IAM-authenticated connector and TLS settings; +the stored secret is unchanged. TCP/non-Cloud-SQL localhost URLs remain invalid. +The adapter is tested against the real production validator and MySQL URL parser. +Default-off serving commands and the migration Job's bootstrap command are unchanged. + +## Secrets, upgrades and disabling + +Keep the existing secret mode, values, resource addresses and all version counters. +**Enabling the Gateway is not permission to convert secret modes, replace secrets +or rotate an encryption key.** Follow each module's existing preservation/recovery +procedure first if adopting older infrastructure. AWS/GCP reference existing cloud +secret versions; Azure reuses its existing ephemeral recovery and write-only PUT +payload mechanism. Azure may resend the same values when deployment configuration +changes; it does not advance rotation counters. Back up keys securely outside +Terraform state. Losing the encryption key loses access to encrypted providers. + +Disabling removes the optional service/routing and generated Gateway env settings; +runtime defaults back to disabled. It does not delete providers, keys, usage rows, +the shared database or its secrets. Existing externally managed DNS/certificates +remain operator-owned. GCP explicitly leaves deletion protection off for this +stateless optional service so disabling is possible; core protection is unchanged. + +## Kubernetes wrapper + +`openwork-self-host` adds `ai_gateway_enabled` (false), `ai_gateway_domain_name` +(empty) and `ai_gateway_url` (null). It **reuses the chart's deployment**, not a +second Gateway implementation. First-class enablement passes canonical Helm +values and creates one additional TLS ingress to the existing `-inference` +Service. Supply `ingress.tls_secret_name` covering the Gateway hostname. With +`ingress.enabled=false`, the operator must provide external HTTPS routing. +Use a chart/release containing the current `gateway.enabled` contract. + +Chart image/version policy and shared Secret ownership are unchanged; use the +chart's image overrides for mirrors. The wrapper does not add the native modules' +`ai_gateway_image` input. Existing advanced `extra_helm_values` Gateway installs +remain untouched with the first-class switch off (the output stays null). Do not +combine first-class enablement with chart enable/URL/name/port overrides; those +conflicts fail validation. Chart-level per-app env/secret validation still applies. +Generated wrapper secrets remain state-backed as before: preserve existing state +and keys. The wrapper remains a draft without live-cluster qualification. + +## Managed-install boundary + +The AWS and Google Cloud managed installers call the **native modules directly**; +they do not use Helm or `openwork-self-host`. These module capabilities alone do +not change a managed install. Their root calls keep the Gateway **off** in this +change. There are no bootstrap IAM, installer publication, feature rollout or +managed-default changes. Enabling support in a managed installer requires a +separately reviewed root/image/DNS/certificate/permission change, any required +customer re-approval, and a fresh live managed-install receipt. Module mock plans +are not proof that managed installation or upgrades work live. + +## Verification boundary + +See [recorded offline evidence](AI_GATEWAY_VERIFICATION.md) for dated tool/provider +versions, pass/fail counts and observed plan values. + +`tests/ai_gateway.tftest.hcl` in each module uses mocked cloud/Helm/Kubernetes +providers. **Mock applies** establish synthetic off/on state only; enable, +coexistence, unchanged, disable and invalid-input checks inspect **plans**. Distinct +mock backend IDs also verify route targets in the synthetic enabled state. +No cloud account or credentials are needed. The tests inspect actual planned service/image/port/route/env/secret +fields, not screenshots of absent services. Run `init -backend=false`, `validate` +(Azure through its provider-bound example) and `test` with **both Terraform and +OpenTofu**; the shared module contract still applies. The Cloud SQL adapter tests +run with `node --test scripts/cloud-sql-entrypoint.test.mjs` on Node 24. + +**Live AI Gateway conformance: NotRun.** A separately authorized disposable run +must prove `/health` returns `{"ok":true,"service":"gateway"}`, readiness, admin +`/v1/org` advertises `deploymentCapabilities.aiGateway=true`, provider creation, +Desktop receives and uses the public origin, unchanged apply preserves secrets +and has no diff, disable retains data, and destroy/independent inventory is clean. +DNS/certificate activation, real routing, cloud migration ordering, IAM, load, +state-secret absence and managed-installer behavior are not certified by mock +plans. Do not advertise a production-ready or live-tested cloud installation from +these checks. diff --git a/infra/terraform/AI_GATEWAY_VERIFICATION.md b/infra/terraform/AI_GATEWAY_VERIFICATION.md new file mode 100644 index 00000000000..aef1304aa2e --- /dev/null +++ b/infra/terraform/AI_GATEWAY_VERIFICATION.md @@ -0,0 +1,121 @@ +# AI Gateway Terraform — offline verification for #7400 + +**2026-11-13 · macOS arm64 · offline evidence only. Live conformance: NotRun.** + +Code tested: `1de30f7ed522fef7061359eb80703c7a0e79a70e`, based on remote dev +`1833adcdd10a8368b397f69f00d8bab9a9a4224c`. Subsequent documentation does not +change the tested implementation. Both isolated CLI copies were byte-compared +with the source/test/example files (126 file comparisons). + +No live apply, paid resource provisioning, production mutation, infrastructure +provider credential access, secret rotation, deployment, merge or issue comment +was performed. Native +`command = apply` blocks below run **against mocked cloud/Helm/Kubernetes +providers only** to establish synthetic off/on state. Random generation is local. +No AI Gateway image was pulled or run; its digests are synthetic syntax fixtures. +This report is not a release, production-readiness or managed-install receipt. + +## Results + +| Module | Terraform 1.16.5 | OpenTofu 1.11.14 | Failed / errored / skipped | +| --- | ---: | ---: | --- | +| `openwork-aws-ecs` | 53 passed | 53 passed | 0 / 0 / 0 on each | +| `openwork-gcp-cloud-run` | 50 passed | 50 passed | 0 / 0 / 0 on each | +| `openwork-azure-container-apps` | 45 passed | 45 passed | 0 / 0 / 0 on each | +| `openwork-self-host` | 8 passed | 8 passed | 0 / 0 / 0 on each | +| **Total** | **156 passed** | **156 passed** | **312 passes, no failures/errors/skips** | + +Both tools passed module `init -backend=false`, subsequent +`init -backend=false -lockfile=readonly`, format checks and native tests. +Library `validate` passed for AWS, GCP and the wrapper; Azure validation uses +its supported provider-bound `examples/complete` root, not a missing caller alias. +Every executable example passed registry-matching read-only initialization and +validation (14 validation checks total including supported bare modules). +Azure's separate OpenTofu root lock was selected; GCP's example lock was +regenerated in the isolated OpenTofu copy. No library locks/caches/state are +committed. + +The wrapper also passed `init -backend=false` and `validate` on **Terraform +1.5.7**, retaining its existing minimum. Mock-test language requires newer tools; +this is not a 1.5 native-test or live-cluster result. + +## Reviewer questions and observed facts + +The native facts are in each module's `tests/ai_gateway.tftest.hcl` (`assert` and +`expect_failures`). Additional resource-action inspection used Terraform's +`test -json -verbose` output; OpenTofu's verdict was independently obtained with +`test -json`, not inferred from Terraform. All values below are neutral fixtures. + +| Question | Observed fact | +| --- | --- | +| Does the default create anything or advertise capability? | Native modules: **0** AI Gateway services/routes; `ai_gateway_url=null`; **0** new `GATEWAY_*` / `INFERENCE_*` names in Den API. Wrapper: **0** AI ingresses and exactly the original **2** Helm values documents. | +| What does enabling add? | Each native module: services **0 → 1**, inference image@digest, port **8791**, `/health`, public origin **`null → https://models.example.com`**. Azure also has `/ready`. Distinct mock backend/app IDs verify routing to inference, not Den. | +| Where does Den send requests? | AWS: **`http://ai-gateway.openwork.internal:8791`** over Cloud Map. GCP: **`https://models.example.com`** through the HTTPS LB, explicitly **not private DNS**. Azure: **`https://openwork-ai-gateway.happy-sky-1234.eastus.azurecontainerapps.io`**, the generated same-environment FQDN. | +| Where do desktops go? | All three native modules advertise **`https://models.example.com`** as both `GATEWAY_PUBLIC_BASE_URL` and the legacy Models destination `INFERENCE_PROXY_BASE_URL`; no internal hostname leaks into that destination. | +| Are the database and encryption values shared? | Both serving containers reference the same existing `DATABASE_URL` and `DEN_DB_ENCRYPTION_KEY` store/version. AWS/GCP keep **4 → 4** core secret entries, with **no database, generator, secret-parent or secret-version actions** on enable/disable. Azure keeps encryption/database rotation counters **1 → 1** and only metadata in the persistent Gateway body; values use the existing ephemeral recovery/write-only path. Live Azure secret fingerprints were **not measured**. | +| Does adding inference replace Den's GCP certificate? | Existing two-host Den certificate action is **`["no-op"]`**. A separate AI certificate is attached. Supplied covering certificate IDs remain operator-owned, with no generated certificate. | +| Does Azure inference gain unrelated identities? | With two identities on Den, Gateway identity count is **1**, registry-pull only; the Key Vault/integration identity is absent. | +| Can it coexist with den-gateway? | AWS: both services present, **8791 vs 8788**, distinct hosts and listener priorities (**50 vs 30** by default). GCP/Azure do not deploy den-gateway; their three Den/AI hosts/backends coexist without adding Web support. | +| Does an unchanged enabled configuration drift? | **0 managed-resource changes** on all three native modules; protected database/secret mutations **0**. These are synthetic-state plans, **not live unchanged applies**. | +| What happens on disable? | Services **1 → 0**, URL **HTTPS → null**, generated Gateway env removed. Protected database/secret mutations **0**. Plan-only managed-resource change counts are AWS **11**, GCP **18**, Azure **6**, including stateless service/routing removal and serving configuration updates. No real database rows were created or deleted in these tests. | +| Do invalid inputs fail before deployment? | Expected failures observed for missing domains, conflicting Gateway/Inference/shared runtime overrides, mutable image references, duplicate hosts, AWS rule priorities, Azure missing certificates, and wrapper conflicting values/missing TLS/shared hosts. | +| Does first-class false break an advanced Helm installation? | Advanced `gateway.enabled=true` values remain present and the values list remains **2** documents while the first-class switch is false; the new output is null because the operator owns that advanced URL. | + +## Additional checks + +| Check | Result | What it proves / does not prove | +| --- | --- | --- | +| Exact inline Cloud SQL entrypoint against production Gateway validator and MySQL parser | **4 passed, 0 failed/skipped**, Node **24.22.1** | Cloud SQL socket hostname changes `localhost → cloudsql` only in memory; parsed credentials/socket/TLS unchanged. External strict-TLS URL unchanged. TCP/non-Cloud-SQL localhost still rejected. No DB/network connection. | +| Existing Helm Gateway render matrix | **78 passed, 0 failed/skipped**, Helm **4.2.2** | In-tree chart compatibility/validation; no cluster or published image qualification. | +| Actual planned wrapper values rendered through the in-tree chart (`helm template --skip-tests`) | Passed | **2** serving `GATEWAY_ENABLED` entries, **2** shared encryption references; private service origin and public Models origin present. This is not a Kubernetes apply. | +| GCP installer unit/packaging suite | **41 passed, 0 failed** | Synthetic installer/release archive checks, including payload references and backend-disabled validation. Installer console messages in its fixtures are **not live readiness**. | +| Trivy **0.75.0** default/example scan of all four modules | **4 gates passed**, **0 unaccepted HIGH/CRITICAL** | Existing file-scoped exceptions unchanged; no new ignores. Default/example static scan, not enabled-cloud or runtime certification. | +| Mintlify `pnpm dlx mint validate` and `broken-links` | Build/validation passed; no broken links | Documentation build/link structure only. CLI `--version` reports `unknown`; no version is invented. Nothing published. | +| Both CLI recursive format checks and `git diff --check` | Passed | Formatting/whitespace only. | + +Native library provider selections were identical by version on both registries: + +- AWS **6.73.0**, archive **2.8.1**, HTTP **3.6.3**, random **3.9.1**. +- GCP Google **8.11.0**, random **3.9.1**. +- AzureRM **5.14.0**, AzAPI **2.13.0**, external **2.4.2**, random **3.9.1**. +- Wrapper Helm provider **2.17.0**, Kubernetes **3.5.0**, random **3.9.1**. + +Executable roots retain their own provider selections, including GCP Google +8.7.0 and AzureRM 5.10.0. Root validation is not a live install with those versions. + +## Reproduction + +Use separate copied module roots/locks for Terraform and OpenTofu. In each module: + +```sh +terraform init -backend=false -input=false +terraform init -backend=false -input=false -lockfile=readonly +terraform validate # Azure: validate examples/complete with its caller alias +terraform test +# Repeat independently with tofu and its matching registry lock. +``` + +From the repository root: + +```sh +terraform fmt -check -recursive infra/terraform +tofu fmt -check -recursive infra/terraform +node --test infra/terraform/modules/openwork-gcp-cloud-run/scripts/cloud-sql-entrypoint.test.mjs +bash packaging/helm/openwork-ee/tests/gateway.sh +python3 -m unittest discover -s infra/managed-deployments/tests -p 'test_gcp_installer.py' +``` + +The installer packaging test requires backend-disabled initialization of its +managed GCP Terraform root first, so its offline provider mirror exists. No +credentials or real provider plan/apply are needed for that prerequisite. + +## Required live follow-up — NotRun + +Only a separately authorized disposable run can establish DNS/certificate +activation, actual private/public routing, `/health` and `/ready`, admin +`/v1/org` capability, provider creation and a desktop inference request using the +public origin. It must also measure secret fingerprints/versions and no-diff +unchanged apply, disable with retained data, and destroy with independent absence +verification. Repeat through the separately approved managed installers before +claiming managed readiness. No permission, bootstrap, installer-publication or +managed root/default change is included here. See [the operator guide](AI_GATEWAY.md). diff --git a/infra/terraform/MODULE_CONTRACT.md b/infra/terraform/MODULE_CONTRACT.md index c5e6ced72d1..d7a852fa83f 100644 --- a/infra/terraform/MODULE_CONTRACT.md +++ b/infra/terraform/MODULE_CONTRACT.md @@ -127,6 +127,13 @@ counts, and skips; do not count one tool's result for the other. - Expose `web_url`, `api_url`, `database_identifier`, and `migration_step_identifier`, retaining native outputs. Explain each identifier's cloud-specific meaning and the value returned for external infrastructure. +- Optional AI Gateway uses `ai_gateway_enabled` (false), + `ai_gateway_domain_name` (empty), `ai_gateway_image` (empty; explicit digest + required when enabled), and `ai_gateway_url` (null when disabled) on all three + native modules. It reuses database/encryption secrets without rotating them. + Distinguish it from OpenWork Web's den-gateway and Den API's MCP gateway. + See [AI_GATEWAY.md](AI_GATEWAY.md) for native networking exceptions, the thin + Helm wrapper interface, managed-installer boundaries and live NotRun checks. - Document all defaults in the README. These smoke-tested floors are not production load-sizing guarantees. diff --git a/infra/terraform/modules/openwork-aws-ecs/README.md b/infra/terraform/modules/openwork-aws-ecs/README.md index 5b96280a1a1..a6b36b7fbfe 100644 --- a/infra/terraform/modules/openwork-aws-ecs/README.md +++ b/infra/terraform/modules/openwork-aws-ecs/README.md @@ -25,6 +25,15 @@ Runs a private, single-organization OpenWork control plane (Den) on AWS | Logs | CloudWatch `/ecs//den-api` and `/ecs//den-web` | | Headless runner and Workbot (optional) | Managed Node runner (Service Connect, SQLite on EFS) or an existing celld/S3 fleet; diskless `workbot` at `chat.` (`headless_runner`, `workbot`) | | OpenWork Web (optional) | `den-gateway` Fargate service at `web.`, sandboxes in your Daytona organization (`openwork_web_enabled = true`) | +| AI Gateway (optional, off by default) | Separate `openwork-inference` service on 8791, private Cloud Map, HTTPS host rule (`ai_gateway_enabled`) | + +**Three different gateways:** the AI Gateway handles inference; **den-gateway** +handles OpenWork Web/sandboxes; Den API's MCP gateway handles connected tools. +See [AI Gateway inputs, networking and verification limits](../../AI_GATEWAY.md). +`ai_gateway_enabled=false`, `ai_gateway_domain_name=""`, `ai_gateway_image=""` +are the defaults; enablement requires a dedicated hostname and digest-pinned image. +`ai_gateway_listener_rule_priority` defaults to 50; `ai_gateway_url` is null when off. +The managed installer calls this module directly, not Helm; its defaults stay off. It sets the same environment the [`openwork-ee` Helm chart](../../../../packaging/helm/openwork-ee) sets on Kubernetes, so the chart's README and `ee/apps/den-api/.env.example` @@ -44,6 +53,7 @@ What has been applied end to end in an AWS test account with | Scenario | Result | Release / tools | Date | | --- | --- | --- | --- | +| AI Gateway live DNS/TLS, admin/provider/Desktop flow, unchanged apply, disable/data retention and teardown | NotRun: offline-only implementation; no infrastructure provisioned | See [AI Gateway verification boundary](../../AI_GATEWAY.md#verification-boundary) | 2026-11-13 | | Contract-v1 offline guards, including write-only existing DB and no legacy bootstrap output in write-only mode | Passed: 37 per tool | Terraform 1.16.5 / OpenTofu 1.11.14 | 2026-10-15 | | Contract-v1 release → dev, two API-only tasks, exactly one successful migrator per image, recognizable org preserved | Passed | 0.18.58 → dev ef455ca; AWS 6.69.1 | 2026-10-15 | | Same-version later apply with identical ephemeral inputs; secret hashes/version and pre-apply session unchanged | Passed: plan exit 0, no changes | 0.18.58 | 2026-10-15 | @@ -220,7 +230,10 @@ at the `alb_dns_name` output (CNAME) at your DNS provider. or set `allow_public_signup = true` for a while. - **OpenWork Web** (chat in the browser, each member's workspace in a Daytona sandbox) is off by default; see [OpenWork Web](#openwork-web). -- **Anything else** (SSO, proxies, Gateway, observability): add env vars with +- **AI Gateway:** use the first-class `ai_gateway_*` inputs above. Setting + `GATEWAY_ENABLED` through `extra_environment` alone does not deploy a service; + remove Gateway/Inference and shared runtime overrides before opting in. +- **Anything else** (SSO, proxies, observability): add non-secret env vars with `extra_environment` and secrets with `extra_secrets`. ## Upgrade notes @@ -586,6 +599,9 @@ several deployments can share one Daytona organization. ## Cost notes +Enabling the AI Gateway adds one 1 vCPU / 2 GiB Fargate task, logs and ALB traffic, +not a new database. Its live capacity and incremental costs have not been measured. + ### Small / trial baseline | Component | Default per copy | diff --git a/infra/terraform/modules/openwork-aws-ecs/acm.tf b/infra/terraform/modules/openwork-aws-ecs/acm.tf index f60fcb13ff0..ea798af4c30 100644 --- a/infra/terraform/modules/openwork-aws-ecs/acm.tf +++ b/infra/terraform/modules/openwork-aws-ecs/acm.tf @@ -6,7 +6,7 @@ locals { create_certificate = var.certificate_arn == "" # Keyed by hostname, known at plan time, so for_each never depends on # values ACM only returns after apply. - certificate_hosts = concat([var.domain_name, local.api_host], local.web_enabled ? [local.gateway_host] : [], local.workbot_enabled ? [local.workbot_host] : []) + certificate_hosts = concat([var.domain_name, local.api_host], local.web_enabled ? [local.gateway_host] : [], local.workbot_enabled ? [local.workbot_host] : [], var.ai_gateway_enabled ? [var.ai_gateway_domain_name] : []) certificate_arn = local.create_certificate ? aws_acm_certificate_validation.this[0].certificate_arn : var.certificate_arn } diff --git a/infra/terraform/modules/openwork-aws-ecs/ai-gateway.tf b/infra/terraform/modules/openwork-aws-ecs/ai-gateway.tf new file mode 100644 index 00000000000..11b72036a94 --- /dev/null +++ b/infra/terraform/modules/openwork-aws-ecs/ai-gateway.tf @@ -0,0 +1,205 @@ +# AI Gateway (ee/apps/gateway, published as openwork-inference), NOT den-gateway. +variable "ai_gateway_enabled" { + description = "Deploy the AI Gateway for administrator-managed inference. Installation configuration, not a product rollout flag; does not enable OpenWork Web." + type = bool + default = false + + validation { + condition = !var.ai_gateway_enabled || alltrue([for name in concat(keys(var.extra_environment), keys(var.extra_secrets)) : + !startswith(name, "GATEWAY_") && !startswith(name, "INFERENCE_") && !contains(["DB_MODE", "DATABASE_URL", "DATABASE_HOST", "DATABASE_USERNAME", "DATABASE_PASSWORD", "DEN_DB_ENCRYPTION_KEY", "NODE_ENV", "OPENWORK_DEV_MODE", "PORT"], name) + ]) + error_message = "With ai_gateway_enabled, remove Gateway/Inference and shared runtime/database overrides from extra_environment and extra_secrets; use the module inputs and existing app secrets." + } +} + +variable "ai_gateway_domain_name" { + description = "Dedicated desktop-reachable HTTPS hostname for the AI Gateway. Required when enabled; include it in externally managed DNS and certificates." + type = string + default = "" + + validation { + condition = !var.ai_gateway_enabled || ( + length(var.ai_gateway_domain_name) <= 253 && + can(regex("^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)+$", var.ai_gateway_domain_name)) && + !can(regex("(^localhost\\.|\\.(localhost|local|internal|svc)$|\\.[0-9]+$)", var.ai_gateway_domain_name)) + ) + error_message = "ai_gateway_enabled requires ai_gateway_domain_name: a qualified non-local DNS hostname, without scheme, port or path." + } +} + +variable "ai_gateway_image" { + description = "AI Gateway image pinned by digest, normally ghcr.io/different-ai/openwork-inference@sha256: from the same release as Den. Required when enabled." + type = string + default = "" + + validation { + condition = !var.ai_gateway_enabled || can(regex("@sha256:[a-f0-9]{64}$", var.ai_gateway_image)) + error_message = "ai_gateway_enabled requires ai_gateway_image pinned by @sha256 digest; mutable tags are not supported." + } +} + +variable "ai_gateway_listener_rule_priority" { + description = "AI Gateway host rule priority on the HTTPS listener. Must be unique, including rules managed outside this module." + type = number + default = 50 + + validation { + condition = var.ai_gateway_listener_rule_priority >= 1 && var.ai_gateway_listener_rule_priority <= 50000 && floor(var.ai_gateway_listener_rule_priority) == var.ai_gateway_listener_rule_priority + error_message = "ai_gateway_listener_rule_priority must be an integer from 1 to 50000." + } +} + +locals { + ai_gateway_url = "https://${var.ai_gateway_domain_name}" + # Cloud Map resolves private task IPs. No new public port or shared auth key. + ai_gateway_proxy_url = "http://ai-gateway.${aws_service_discovery_private_dns_namespace.this.name}:8791" + ai_gateway_environment = var.ai_gateway_enabled ? { + GATEWAY_ENABLED = "true" + GATEWAY_PROXY_BASE_URL = local.ai_gateway_proxy_url + GATEWAY_PUBLIC_BASE_URL = local.ai_gateway_url + # The legacy Models destination must also be desktop-reachable, not Cloud Map. + INFERENCE_PROXY_BASE_URL = local.ai_gateway_url + } : {} + ai_gateway_secrets = [for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : { + name = name, valueFrom = "${aws_secretsmanager_secret.app.arn}:${name}::" + }] +} + +resource "aws_cloudwatch_log_group" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + name = "/ecs/${var.name}/ai-gateway" + retention_in_days = var.log_retention_days + tags = var.tags +} + +resource "aws_service_discovery_service" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + name = "ai-gateway" + tags = var.tags + dns_config { + namespace_id = aws_service_discovery_private_dns_namespace.this.id + routing_policy = "MULTIVALUE" + dns_records { + type = "A" + ttl = 10 + } + } +} + +resource "aws_vpc_security_group_ingress_rule" "ai_gateway" { + for_each = var.ai_gateway_enabled ? { alb = local.alb_security_group_id, internal = aws_security_group.tasks.id } : {} + security_group_id = aws_security_group.tasks.id + referenced_security_group_id = each.value + from_port = 8791 + to_port = 8791 + ip_protocol = "tcp" +} + +resource "aws_lb_target_group" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + name = "${var.name}-ai-gateway" + port = 8791 + protocol = "HTTP" + target_type = "ip" + vpc_id = var.vpc_id + deregistration_delay = 30 + tags = var.tags + health_check { + path = "/health" + matcher = "200" + interval = 15 + healthy_threshold = 2 + unhealthy_threshold = 5 + } +} + +resource "aws_lb_listener_rule" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + listener_arn = local.listener_arn + priority = var.ai_gateway_listener_rule_priority + tags = var.tags + action { + type = "forward" + target_group_arn = aws_lb_target_group.ai_gateway[0].arn + } + condition { + host_header { values = [var.ai_gateway_domain_name] } + } + lifecycle { + precondition { + condition = !contains(concat([var.domain_name, local.api_host], local.web_enabled ? [local.gateway_host] : [], local.workbot_enabled ? [local.workbot_host] : []), var.ai_gateway_domain_name) + error_message = "The AI Gateway hostname must differ from Den, OpenWork Web (den-gateway) and Workbot." + } + precondition { + condition = !contains(concat([var.api_listener_rule_priority], !local.create_listener ? [var.web_listener_rule_priority] : [], local.web_enabled ? [var.openwork_web_listener_rule_priority] : [], local.workbot_enabled ? [var.workbot_listener_rule_priority] : []), var.ai_gateway_listener_rule_priority) + error_message = "The AI Gateway listener priority must differ from all other enabled host rules." + } + } +} + +resource "aws_ecs_task_definition" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + family = "${var.name}-ai-gateway" + requires_compatibilities = ["FARGATE"] + network_mode = "awsvpc" + cpu = "1024" + memory = "2048" + execution_role_arn = aws_iam_role.execution.arn + task_role_arn = aws_iam_role.task.arn + tags = var.tags + runtime_platform { + operating_system_family = "LINUX" + cpu_architecture = var.cpu_architecture + } + container_definitions = jsonencode([{ + name = "ai-gateway" + image = var.ai_gateway_image + essential = true + portMappings = [{ containerPort = 8791, protocol = "tcp" }] + environment = [for name, value in merge(local.ai_gateway_environment, { NODE_ENV = "production", OPENWORK_DEV_MODE = "0", DB_MODE = "mysql", PORT = "8791" }) : { name = name, value = value }] + secrets = local.ai_gateway_secrets + # Same CA setup as den-api; never run migrations from serving replicas. + entryPoint = ["/bin/sh", "-ec"] + command = ["${local.tls_setup}exec node --import /app/ee/apps/gateway/dist/instrumentation.js /app/ee/apps/gateway/dist/server.js"] + logConfiguration = { + logDriver = "awslogs" + options = { + "awslogs-group" = aws_cloudwatch_log_group.ai_gateway[0].name, "awslogs-region" = local.region, "awslogs-stream-prefix" = "ai-gateway" + } + } + }]) +} + +resource "aws_ecs_service" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + name = "${var.service_name_prefix}ai-gateway" + cluster = local.cluster_arn + task_definition = aws_ecs_task_definition.ai_gateway[0].arn + desired_count = 1 + launch_type = "FARGATE" + wait_for_steady_state = var.wait_for_steady_state + health_check_grace_period_seconds = 120 + propagate_tags = "SERVICE" + tags = var.tags + network_configuration { + subnets = var.service_subnet_ids + security_groups = [aws_security_group.tasks.id] + assign_public_ip = var.assign_public_ip + } + load_balancer { + target_group_arn = aws_lb_target_group.ai_gateway[0].arn + container_name = "ai-gateway" + container_port = 8791 + } + service_registries { registry_arn = aws_service_discovery_service.ai_gateway[0].arn } + deployment_circuit_breaker { + enable = true + rollback = true + } + depends_on = [terraform_data.migration_complete, aws_lb_listener_rule.ai_gateway, aws_secretsmanager_secret_version.app] +} + +output "ai_gateway_url" { + description = "AI Gateway HTTPS origin for desktops, or null when disabled. Not the OpenWork Web den-gateway URL." + value = var.ai_gateway_enabled ? local.ai_gateway_url : null +} diff --git a/infra/terraform/modules/openwork-aws-ecs/alb.tf b/infra/terraform/modules/openwork-aws-ecs/alb.tf index 0e4c5d1fbbb..d28da53f98a 100644 --- a/infra/terraform/modules/openwork-aws-ecs/alb.tf +++ b/infra/terraform/modules/openwork-aws-ecs/alb.tf @@ -261,7 +261,7 @@ resource "aws_lb_listener_certificate" "this" { # DNS ------------------------------------------------------------------------ resource "aws_route53_record" "this" { - for_each = local.create_alb && var.route53_zone_id != "" ? toset(concat([var.domain_name, local.api_host], local.web_enabled ? [local.gateway_host] : [], local.workbot_enabled ? [local.workbot_host] : [])) : toset([]) + for_each = local.create_alb && var.route53_zone_id != "" ? toset(concat([var.domain_name, local.api_host], local.web_enabled ? [local.gateway_host] : [], local.workbot_enabled ? [local.workbot_host] : [], var.ai_gateway_enabled ? [var.ai_gateway_domain_name] : [])) : toset([]) zone_id = var.route53_zone_id name = each.value diff --git a/infra/terraform/modules/openwork-aws-ecs/main.tf b/infra/terraform/modules/openwork-aws-ecs/main.tf index 809660fc9ae..11d17070c0c 100644 --- a/infra/terraform/modules/openwork-aws-ecs/main.tf +++ b/infra/terraform/modules/openwork-aws-ecs/main.tf @@ -90,7 +90,7 @@ locals { SMTP_HOST = var.smtp.host SMTP_PORT = tostring(var.smtp.port) SMTP_SECURE = local.bool[var.smtp.secure] - }, local.web_environment, local.agent_environment, var.extra_environment) + }, local.web_environment, local.agent_environment, local.ai_gateway_environment, var.extra_environment) # Empty values are dropped below, so these are unset while Web is off. web_environment = { diff --git a/infra/terraform/modules/openwork-aws-ecs/security.tf b/infra/terraform/modules/openwork-aws-ecs/security.tf index cb166537cc3..50ac35b634a 100644 --- a/infra/terraform/modules/openwork-aws-ecs/security.tf +++ b/infra/terraform/modules/openwork-aws-ecs/security.tf @@ -125,7 +125,8 @@ resource "terraform_data" "security_contract" { [local.den_api_image, local.den_web_image], local.web_enabled ? [local.den_gateway_image] : [], local.runner_managed ? [local.runner_image] : [], - local.workbot_enabled ? [local.workbot_image] : [] + local.workbot_enabled ? [local.workbot_image] : [], + var.ai_gateway_enabled ? [var.ai_gateway_image] : [] ) : can(regex("@sha256:[0-9a-f]{64}$", image))]) error_message = "Every enabled image must be pinned with @sha256:<64 hex characters>; pass digest overrides for optional services." } diff --git a/infra/terraform/modules/openwork-aws-ecs/tests/ai_gateway.tftest.hcl b/infra/terraform/modules/openwork-aws-ecs/tests/ai_gateway.tftest.hcl new file mode 100644 index 00000000000..9f764f70e30 --- /dev/null +++ b/infra/terraform/modules/openwork-aws-ecs/tests/ai_gateway.tftest.hcl @@ -0,0 +1,207 @@ +# OFFLINE ONLY. Mock applies establish synthetic off/on state for upgrade plans; +# every cloud provider is mocked. No resources are provisioned. +mock_provider "aws" { + mock_data "aws_iam_policy_document" { defaults = { json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" } } + mock_resource "aws_cloudwatch_log_group" { defaults = { arn = "arn:aws:logs:us-east-1:123456789012:log-group:openwork" } } + mock_resource "aws_ecs_cluster" { defaults = { arn = "arn:aws:ecs:us-east-1:123456789012:cluster/openwork-den" } } + mock_resource "aws_lb" { defaults = { arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/openwork/abc" } } + mock_resource "aws_lb_target_group" { defaults = { arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/openwork/abc" } } + mock_resource "aws_lb_listener" { defaults = { arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/openwork/abc/def" } } + mock_resource "aws_iam_role" { defaults = { arn = "arn:aws:iam::123456789012:role/openwork" } } + mock_resource "aws_secretsmanager_secret" { defaults = { arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:openwork" } } + mock_resource "aws_service_discovery_service" { defaults = { arn = "arn:aws:servicediscovery:us-east-1:123456789012:service/srv-abc" } } + mock_resource "aws_ecs_task_definition" { defaults = { arn = "arn:aws:ecs:us-east-1:123456789012:task-definition/openwork:1" } } +} +override_resource { + target = aws_lb_target_group.ai_gateway + values = { arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/ai-gateway/distinct" } +} +mock_provider "random" { + mock_resource "random_password" { defaults = { result = "SyntheticPreservedKey0123456789abcdef" } } +} +override_resource { + target = aws_lambda_invocation.migrate + values = { result = "{\"exitCode\":0,\"image\":\"ghcr.io/different-ai/openwork-den-api@sha256:0942e02a8e2033151e982f51b24926fe5090807b7f7480d77f2c4fc5b806737e\"}" } +} +variables { + openwork_version = "0.18.58" + write_only_secrets = false + database_ca_bundle = "-----BEGIN CERTIFICATE-----test" + vpc_id = "vpc-123" + alb_subnet_ids = ["subnet-a", "subnet-b"] + service_subnet_ids = ["subnet-a", "subnet-b"] + database_subnet_ids = ["subnet-c", "subnet-d"] + owner_emails = ["admin@example.com"] + domain_name = "openwork.example.com" + certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/abc" + route53_zone_id = "Z123" + ai_gateway_image = "ghcr.io/different-ai/openwork-inference@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" +} +run "off_by_default" { + command = apply # Mock state only: no provisioning, Lambda execution or SQL. + assert { + condition = length(aws_ecs_task_definition.ai_gateway) == 0 && length(aws_ecs_service.ai_gateway) == 0 && length(aws_lb_target_group.ai_gateway) == 0 && length(aws_lb_listener_rule.ai_gateway) == 0 && length(aws_vpc_security_group_ingress_rule.ai_gateway) == 0 && length(aws_service_discovery_service.ai_gateway) == 0 && output.ai_gateway_url == null + error_message = "Default: 0 AI Gateway tasks/services/routes/security rules/discovery; URL null." + } + assert { + condition = length([for e in jsondecode(aws_ecs_task_definition.api_v1.container_definitions)[0].environment : e.name if startswith(e.name, "GATEWAY_") || startswith(e.name, "INFERENCE_")]) == 0 && length(aws_route53_record.this) == 2 + error_message = "Default Den environment and two DNS hosts remain unchanged." + } +} +run "enable_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(aws_ecs_service.ai_gateway) == 1 && aws_lb_target_group.ai_gateway[0].port == 8791 && aws_lb_target_group.ai_gateway[0].health_check[0].path == "/health" && aws_lb_listener_rule.ai_gateway[0].listener_arn == aws_lb_listener.https[0].arn && output.ai_gateway_url == "https://models.example.com" + error_message = "Enabled: 0 -> 1 ECS service on 8791, /health, existing HTTPS listener, public models URL." + } + assert { + condition = jsondecode(aws_ecs_task_definition.ai_gateway[0].container_definitions)[0].image == var.ai_gateway_image && jsondecode(aws_ecs_task_definition.ai_gateway[0].container_definitions)[0].portMappings[0].containerPort == 8791 && strcontains(jsondecode(aws_ecs_task_definition.ai_gateway[0].container_definitions)[0].command[0], "/app/ee/apps/gateway/dist/server.js") + error_message = "Use the published inference image, correct server entrypoint and port, never the Web proxy or migrator." + } + assert { + condition = alltrue([for container in [jsondecode(aws_ecs_task_definition.api_v1.container_definitions)[0], jsondecode(aws_ecs_task_definition.ai_gateway[0].container_definitions)[0]] : alltrue([for k, v in { + GATEWAY_ENABLED = "true", GATEWAY_PROXY_BASE_URL = "http://ai-gateway.openwork.internal:8791", GATEWAY_PUBLIC_BASE_URL = "https://models.example.com", INFERENCE_PROXY_BASE_URL = "https://models.example.com", DB_MODE = "mysql", NODE_ENV = "production" + } : contains([for e in container.environment : "${e.name}=${e.value}"], "${k}=${v}")]) && alltrue([for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : contains([for e in container.secrets : "${e.name}=${e.valueFrom}"], "${name}=${aws_secretsmanager_secret.app.arn}:${name}::")])]) + error_message = "Den and Gateway share managed env, the existing app database/encryption secret and separate private/public URLs." + } + assert { + condition = random_password.db_encryption[0].result == "SyntheticPreservedKey0123456789abcdef" && jsondecode(aws_secretsmanager_secret_version.app.secret_string).DEN_DB_ENCRYPTION_KEY == random_password.db_encryption[0].result && length(keys(local.secret_values)) == 4 && var.secrets_version == 1 && length(aws_route53_record.this) == 3 && contains(local.certificate_hosts, "models.example.com") + error_message = "Encryption value and four app-secret keys preserved; version stays 1; hosts 2 -> 3, certificate includes AI Gateway." + } + assert { + condition = aws_vpc_security_group_ingress_rule.ai_gateway["alb"].from_port == 8791 && aws_vpc_security_group_ingress_rule.ai_gateway["internal"].referenced_security_group_id == aws_security_group.tasks.id + error_message = "8791 accepts only ALB/tasks security groups, never a new public CIDR ingress." + } +} +run "coexists_with_den_gateway" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + openwork_web_enabled = true + daytona_api_key = "synthetic" + daytona = { snapshot = "synthetic" } + den_gateway_image = "example.test/den-gateway@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + } + assert { + condition = length(aws_ecs_service.gateway) == 1 && length(aws_ecs_service.ai_gateway) == 1 && aws_lb_listener_rule.ai_gateway[0].priority != aws_lb_listener_rule.gateway_host[0].priority && local.gateway_host != var.ai_gateway_domain_name && aws_lb_target_group.gateway[0].port == 8788 + error_message = "AI Gateway and den-gateway have distinct services, hosts, ports and listener priorities." + } +} +run "module_managed_certificate" { + command = plan + override_resource { + target = aws_acm_certificate.this + values = { + arn = "arn:aws:acm:us-east-1:123456789012:certificate/managed" + domain_validation_options = [for host in ["openwork.example.com", "api.openwork.example.com", "models.example.com"] : { + domain_name = host, resource_record_name = "_x.${host}.", resource_record_type = "CNAME", resource_record_value = "_y.acm-validations.aws." + }] + } + } + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + certificate_arn = "" + } + assert { + condition = contains(aws_acm_certificate.this[0].subject_alternative_names, "models.example.com") && contains(keys(aws_route53_record.certificate_validation), "models.example.com") && tolist(tolist(aws_lb_listener_rule.ai_gateway[0].condition)[0].host_header[0].values)[0] == "models.example.com" + error_message = "Managed ACM SAN, Route 53 validation and listener host all include the dedicated AI Gateway hostname." + } +} +run "shared_listener" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + alb_listener_arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/shared/123/456" + alb_security_group_id = "sg-shared" + ai_gateway_listener_rule_priority = 150 + } + assert { + condition = aws_lb_listener_rule.ai_gateway[0].listener_arn == var.alb_listener_arn && aws_lb_listener_rule.ai_gateway[0].priority == 150 && length(aws_lb.this) == 0 + error_message = "Shared HTTPS listener receives only the requested AI Gateway rule; no replacement ALB." + } +} +run "missing_domain" { + command = plan + variables { ai_gateway_enabled = true } + expect_failures = [var.ai_gateway_domain_name] +} +run "mutable_image" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ai_gateway_image = "ghcr.io/different-ai/openwork-inference:latest" + } + expect_failures = [var.ai_gateway_image] +} +run "conflicting_environment" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + extra_environment = { GATEWAY_ENABLED = "false" } + } + expect_failures = [var.ai_gateway_enabled] +} +run "conflicting_legacy_alias" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + extra_environment = { INFERENCE_PROXY_BASE_URL = "https://other.example.com" } + } + expect_failures = [var.ai_gateway_enabled] +} +run "conflicting_priority" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ai_gateway_listener_rule_priority = 10 + } + expect_failures = [aws_lb_listener_rule.ai_gateway] +} +run "conflicting_host" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "openwork.example.com" + } + expect_failures = [aws_lb_listener_rule.ai_gateway] +} +run "enabled_mock_state" { + command = apply # Mock-only state for unchanged/disable plans below. + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(aws_ecs_service.ai_gateway) == 1 && random_password.db_encryption[0].result == "SyntheticPreservedKey0123456789abcdef" && tolist(aws_lb_listener_rule.ai_gateway[0].action)[0].target_group_arn == aws_lb_target_group.ai_gateway[0].arn && aws_ecs_service.ai_gateway[0].load_balancer != aws_ecs_service.api.load_balancer + error_message = "Synthetic enabled state routes to the distinct AI target group, not Den, and retains the original encryption value." + } +} +run "unchanged_enabled_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(aws_ecs_service.ai_gateway) == 1 && random_password.db_encryption[0].result == "SyntheticPreservedKey0123456789abcdef" && var.secrets_version == 1 + error_message = "Unchanged enabled plan retains one Gateway and the original encryption key/version." + } +} +run "disable_plan_keeps_data" { + command = plan + assert { + condition = length(aws_ecs_service.ai_gateway) == 0 && output.ai_gateway_url == null && length(aws_db_instance.this) == 1 && length(keys(local.secret_values)) == 4 && random_password.db_encryption[0].result == "SyntheticPreservedKey0123456789abcdef" && !contains(keys(local.environment), "GATEWAY_ENABLED") + error_message = "Disable: Gateway 1 -> 0, URL null and runtime defaults false; database and four original app secrets kept." + } +} diff --git a/infra/terraform/modules/openwork-aws-ecs/variables.tf b/infra/terraform/modules/openwork-aws-ecs/variables.tf index a4097b0fb84..50e625115fd 100644 --- a/infra/terraform/modules/openwork-aws-ecs/variables.tf +++ b/infra/terraform/modules/openwork-aws-ecs/variables.tf @@ -187,7 +187,7 @@ variable "api_domain_name" { } variable "certificate_arn" { - description = "ACM certificate (same region) covering domain_name, api_domain_name, and openwork_web_domain_name when openwork_web_enabled. Empty creates and DNS-validates one in route53_zone_id." + description = "ACM certificate (same region) covering Den and all enabled optional hosts, including ai_gateway_domain_name. Empty creates and DNS-validates one in route53_zone_id." type = string default = "" } diff --git a/infra/terraform/modules/openwork-azure-container-apps/README.md b/infra/terraform/modules/openwork-azure-container-apps/README.md index 0d955baace0..66de532ee21 100644 --- a/infra/terraform/modules/openwork-azure-container-apps/README.md +++ b/infra/terraform/modules/openwork-azure-container-apps/README.md @@ -14,13 +14,18 @@ A single-organization Den control plane, without Kubernetes: | Secrets | Container Apps managed secrets and optional existing Key Vault references | | Network / logs | New or existing Container Apps environment and Log Analytics workspace | | Deletion protection | Default `CanNotDelete` ARM lock on the managed database | +| AI Gateway (optional, off by default) | Separate `openwork-inference` Container App on 8791, HTTPS ingress and custom-domain SNI binding | +The **AI Gateway** handles inference, **den-gateway** handles OpenWork Web/sandboxes, +and Den API's **MCP gateway** handles tools. This module does not deploy den-gateway. +See [AI Gateway setup, shared secrets and networking limits](../../AI_GATEWAY.md). Workbot, runners, Slack and OpenWork Web are not deployed. Existing product flag inputs remain accepted. ## Tested scenarios | Date | Tools / providers | Release / scenario | Result | | --- | --- | --- | --- | +| 2026-11-13 | Offline-only implementation | AI Gateway live DNS/TLS, admin/provider/Desktop, unchanged apply, disable/data retention and teardown | **NotRun**; no infrastructure provisioned | | 2026-10-13 | Terraform 1.16, OpenTofu 1.11; AzureRM 5.0 / 5.9 | Legacy init-container module, 0.18.58, private MySQL, setup/browser/isolated desktop, destroy | Passed; not proof of the new Job topology | | 2026-10-14 | Terraform 1.16.5 / OpenTofu 1.11.14; AzureRM 5.9 | Legacy module, 0.18.58 to dev, one API replica, retained administrator/org, browser/desktop, destroy | Passed; not a two-replica result | | 2026-10-16 | Terraform 1.16.5 / OpenTofu 1.11.14; AzureRM 5.10, AzAPI 2.13, external 2.4, random 3.9 | Contract mocked tests, 33 runs on each tool; registry-specific example locks and fail-closed restricted-ingress guard | Passed | @@ -154,6 +159,7 @@ The reusable module has no lock file; the executable `examples/complete` root ** | `database_network` members | subnet required; VNet `""`; create DNS `true`; DNS ID `""` | Existing subnet/VNet, or an existing linked private DNS zone | | `deletion_protection` | `true` | Out-of-band database deletion lock; Terraform destroy caveat above | | `den_api`, `den_web` | 1 vCPU / 2 GiB / 1 replica; 0.5 vCPU / 1 GiB / 1 replica | App reservations and fixed min/max replicas | +| `ai_gateway_enabled`, `ai_gateway_domain_name`, `ai_gateway_image` | `false`, `""`, `""` | Dedicated inference service; enabled requires hostname, image@digest, preconfigured DNS and covering `custom_domain_certificate` | | `migration_timeout_seconds`, `migration_deployment_id` | `1800`, `""` | Bounded Job deadline (120–3600); deliberate release identity | | `existing_database_identifier` | `external` | Non-secret stable ID; change with existing database to rerun migration | @@ -162,6 +168,7 @@ The reusable module has no lock file; the executable `examples/complete` root ** | Output | Meaning | | --- | --- | | `web_url`, `api_url` | Public HTTPS origins | +| `ai_gateway_url`, `ai_gateway_dns_records` | AI Gateway HTTPS origin / prerequisite DNS records, null when disabled | | `database_identifier`, `migration_step_identifier`, `migration_job_id`, `migration_execution_id` | Shared contract database and exact migration identities | | `setup_url`, `bootstrap_secret_name`, `bootstrap_code` | Setup page, cloud secret name and deprecated null code output (no secret value in state) | | `default_hostnames`, `custom_domain_dns_records` | App CNAME targets and Azure DNS verification records | @@ -186,6 +193,9 @@ Mocks never contact Azure. `python3 scripts/test-wait-for-migration.py` runs fiv ## Cost notes +Enabling the AI Gateway adds one 1 vCPU / 2 GiB Container App replica and traffic, +not a new database. Incremental live Gateway costs and capacity are not measured. + Regional capacity may reject new environments with `ManagedEnvironmentCapacityHeavyUsageError`. The conformance run's first eastus attempt failed this way; after full failed-stack cleanup, one fresh eastus retry succeeded. No existing environment was reused and no region switch was needed. An alternate authorized region may be necessary for another deployment; clean up the failed attempt before switching. The earlier one-replica test measured under USD 1 estimated for a short session, not a settled bill. Rough always-on East US defaults were estimated at USD 2–4/day before logs/traffic; these are not production quotes. Two API replicas increase consumption. The migration Job is short-lived; Azure may retain execution history. Use Azure pricing and your plan's actual reservations before committing spend. diff --git a/infra/terraform/modules/openwork-azure-container-apps/ai-gateway.tf b/infra/terraform/modules/openwork-azure-container-apps/ai-gateway.tf new file mode 100644 index 00000000000..167962ec487 --- /dev/null +++ b/infra/terraform/modules/openwork-azure-container-apps/ai-gateway.tf @@ -0,0 +1,104 @@ +# AI Gateway (ee/apps/gateway / openwork-inference), not OpenWork Web's den-gateway. +variable "ai_gateway_enabled" { + description = "Deploy the AI Gateway for administrator-managed inference. Off by default; installation configuration, not a product flag." + type = bool + default = false + validation { + condition = !var.ai_gateway_enabled || alltrue([for name in concat(keys(var.extra_environment), keys(var.extra_secrets)) : + !startswith(name, "GATEWAY_") && !startswith(name, "INFERENCE_") && !contains(["DB_MODE", "DATABASE_URL", "DATABASE_HOST", "DATABASE_USERNAME", "DATABASE_PASSWORD", "DEN_DB_ENCRYPTION_KEY", "NODE_ENV", "OPENWORK_DEV_MODE", "PORT"], name) + ]) + error_message = "With ai_gateway_enabled, remove Gateway/Inference and shared runtime/database overrides from extra_environment and extra_secrets; use the module inputs and existing app secrets." + } +} +variable "ai_gateway_domain_name" { + description = "Dedicated desktop-reachable HTTPS hostname for the AI Gateway. Required when enabled; preconfigure CNAME/TXT records and a covering custom_domain_certificate." + type = string + default = "" + validation { + condition = !var.ai_gateway_enabled || ( + length(var.ai_gateway_domain_name) <= 253 && + can(regex("^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)+$", var.ai_gateway_domain_name)) && + !can(regex("(^localhost\\.|\\.(localhost|local|internal|svc)$|\\.[0-9]+$)", var.ai_gateway_domain_name)) + ) + error_message = "ai_gateway_enabled requires ai_gateway_domain_name: a qualified non-local DNS hostname, without scheme, port or path." + } +} +variable "ai_gateway_image" { + description = "AI Gateway image pinned by digest, normally ghcr.io/different-ai/openwork-inference@sha256: from the same release as Den. Required when enabled." + type = string + default = "" + validation { + condition = !var.ai_gateway_enabled || can(regex("@sha256:[a-f0-9]{64}$", var.ai_gateway_image)) + error_message = "ai_gateway_enabled requires ai_gateway_image pinned by @sha256 digest; mutable tags are not supported." + } +} +locals { + ai_gateway_app_name = "${var.app_name_prefix}${var.name}-ai-gateway" + ai_gateway_default_host = "${local.ai_gateway_app_name}.${local.environment_default_domain}" + ai_gateway_url = "https://${var.ai_gateway_domain_name}" + # Same-environment HTTPS FQDN is internally routed by Container Apps. + ai_gateway_environment = var.ai_gateway_enabled ? { + GATEWAY_ENABLED = "true" + GATEWAY_PROXY_BASE_URL = "https://${local.ai_gateway_default_host}" + GATEWAY_PUBLIC_BASE_URL = local.ai_gateway_url + INFERENCE_PROXY_BASE_URL = local.ai_gateway_url + } : {} + # Gateway reads core managed secrets from its write-only payload, not Key + # Vault. Attach only an explicitly configured registry-pull identity. + ai_gateway_identity_ids = var.registry == null ? [] : (var.registry.identity != "" ? [var.registry.identity] : []) + ai_gateway_secret_names = concat(["database-url", "den-db-encryption-key"], local.registry_password ? ["registry-password"] : []) + ai_gateway_secret_metadata = [for secret in local.secret_metadata : secret if contains(local.ai_gateway_secret_names, secret.name)] + ai_gateway_secret_payload = [for name in concat(["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"], local.registry_password ? ["REGISTRY_PASSWORD"] : []) : { + name = lower(replace(name, "_", "-")), value = local.secret_values[name] + }] +} +resource "azapi_resource" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + type = "Microsoft.App/containerApps@2025-01-01" + name = local.ai_gateway_app_name + parent_id = data.azurerm_resource_group.this.id + location = local.app_location + tags = var.tags + ignore_null_property = true + response_export_values = [] + dynamic "identity" { + for_each = length(local.ai_gateway_identity_ids) > 0 ? [1] : [] + content { + type = "UserAssigned" + identity_ids = local.ai_gateway_identity_ids + } + } + body = local.app_bodies.ai_gateway + sensitive_body = { properties = { configuration = { secrets = local.ai_gateway_secret_payload } } } + sensitive_body_version = local.app_write_versions.ai_gateway + depends_on = [data.external.migration_completion] + lifecycle { + precondition { + condition = var.custom_domain_certificate != null + error_message = "The AI Gateway needs custom_domain_certificate covering ai_gateway_domain_name and preconfigured DNS." + } + precondition { + condition = !contains([local.web_host, local.api_host], var.ai_gateway_domain_name) + error_message = "The AI Gateway hostname must differ from Den web and API hostnames." + } + } +} +resource "azurerm_container_app_custom_domain" "ai_gateway" { + count = var.ai_gateway_enabled && nonsensitive(var.custom_domain_certificate != null) ? 1 : 0 + name = var.ai_gateway_domain_name + container_app_id = azapi_resource.ai_gateway[0].id + container_app_environment_certificate_id = azurerm_container_app_environment_certificate.this[0].id + certificate_binding_type = "SniEnabled" +} +output "ai_gateway_url" { + description = "AI Gateway HTTPS origin for desktops, or null when disabled. Not the OpenWork Web den-gateway URL." + value = var.ai_gateway_enabled ? local.ai_gateway_url : null +} +output "ai_gateway_dns_records" { + description = "Preconfigure CNAME and asuid TXT before enabling the AI Gateway (environment must exist); null when disabled." + value = var.ai_gateway_enabled ? { + host = var.ai_gateway_domain_name + cname = local.ai_gateway_default_host + txt = { name = "asuid.${var.ai_gateway_domain_name}", value = local.environment_verification } + } : null +} diff --git a/infra/terraform/modules/openwork-azure-container-apps/apps.tf b/infra/terraform/modules/openwork-azure-container-apps/apps.tf index b4d57fc998c..20d3e7119f3 100644 --- a/infra/terraform/modules/openwork-azure-container-apps/apps.tf +++ b/infra/terraform/modules/openwork-azure-container-apps/apps.tf @@ -16,17 +16,19 @@ locals { plain_env = [for name, value in local.environment : { name = name, value = value }] secret_env_values = [for name, reference in local.secret_env : { name = name, secretRef = reference }] ingress_restrictions = [for index, cidr in var.allowed_ingress_cidrs : { name = "allow-${index}", action = "Allow", ipAddressRange = cidr }] - app_specs = { + app_specs = merge({ api = { name = "den-api", port = 8788, health = "/health", cpu = var.den_api.cpu, memory = var.den_api.memory, replicas = var.den_api.replicas, image = local.den_api_image } web = { name = "den-web", port = 3005, health = "/api/health", cpu = var.den_web.cpu, memory = var.den_web.memory, replicas = var.den_web.replicas, image = local.den_web_image } - } + }, var.ai_gateway_enabled ? { + ai_gateway = { name = "ai-gateway", port = 8791, health = "/health", cpu = 1, memory = "2Gi", replicas = 1, image = var.ai_gateway_image } + } : {}) app_bodies = { for key, app in local.app_specs : key => { properties = { managedEnvironmentId = local.environment_id workloadProfileName = var.workload_profile_name != "" ? var.workload_profile_name : null configuration = { activeRevisionsMode = "Single" - secrets = local.secret_metadata + secrets = key == "ai_gateway" ? local.ai_gateway_secret_metadata : local.secret_metadata registries = local.registries ingress = { external = true, targetPort = app.port, transport = "Auto", allowInsecure = false @@ -41,10 +43,13 @@ locals { containers = [{ name = app.name, image = app.image resources = { cpu = app.cpu, memory = app.memory } - env = concat([{ name = "PORT", value = tostring(app.port) }], local.plain_env, local.secret_env_values) + env = key == "ai_gateway" ? concat( + [for name, value in merge(local.ai_gateway_environment, { NODE_ENV = "production", OPENWORK_DEV_MODE = "0", DB_MODE = "mysql", PORT = "8791" }) : { name = name, value = value }], + [for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : { name = name, secretRef = local.secret_env[name] }], + ) : concat([{ name = "PORT", value = tostring(app.port) }], local.plain_env, local.secret_env_values) probes = concat( [{ type = "Liveness", httpGet = { path = app.health, port = app.port, scheme = "HTTP" }, initialDelaySeconds = 10, periodSeconds = 30 }], - key == "api" ? [{ type = "Readiness", httpGet = { path = "/ready", port = app.port, scheme = "HTTP" }, periodSeconds = 10 }] : [], + contains(["api", "ai_gateway"], key) ? [{ type = "Readiness", httpGet = { path = "/ready", port = app.port, scheme = "HTTP" }, periodSeconds = 10 }] : [], ) }] } @@ -54,7 +59,7 @@ locals { # Version this write-only property by NON-SECRET deployment configuration; the # payload is ephemerally recovered from the cloud unless a *_version rotates it. app_write_versions = { for key, body in local.app_bodies : key => { - "properties.configuration.secrets" = sha256(jsonencode({ body = body, tags = var.tags, identities = local.identity_ids, location = local.app_location })) + "properties.configuration.secrets" = sha256(jsonencode({ body = body, tags = var.tags, identities = key == "ai_gateway" ? local.ai_gateway_identity_ids : local.identity_ids, location = local.app_location })) } } } diff --git a/infra/terraform/modules/openwork-azure-container-apps/custom_domain.tf b/infra/terraform/modules/openwork-azure-container-apps/custom_domain.tf index 8001defb9fe..46c30b11576 100644 --- a/infra/terraform/modules/openwork-azure-container-apps/custom_domain.tf +++ b/infra/terraform/modules/openwork-azure-container-apps/custom_domain.tf @@ -5,11 +5,12 @@ locals { # Without a certificate nothing is bound; den-api's precondition explains why. - bind_custom_domains = local.custom_domains && nonsensitive(var.custom_domain_certificate != null) + bind_custom_domains = local.custom_domains && nonsensitive(var.custom_domain_certificate != null) + create_domain_certificate = (local.custom_domains || var.ai_gateway_enabled) && nonsensitive(var.custom_domain_certificate != null) } resource "azurerm_container_app_environment_certificate" "this" { - count = local.bind_custom_domains ? 1 : 0 + count = local.create_domain_certificate ? 1 : 0 name = "${var.name}-den" container_app_environment_id = local.environment_id diff --git a/infra/terraform/modules/openwork-azure-container-apps/main.tf b/infra/terraform/modules/openwork-azure-container-apps/main.tf index a1d3fb8920c..85ff7331c45 100644 --- a/infra/terraform/modules/openwork-azure-container-apps/main.tf +++ b/infra/terraform/modules/openwork-azure-container-apps/main.tf @@ -76,7 +76,7 @@ locals { SMTP_PORT = nonsensitive(tostring(var.smtp.port)) SMTP_USER = nonsensitive(var.smtp.username) SMTP_SECURE = nonsensitive(local.bool[var.smtp.secure]) - }, var.extra_environment) : k => v if v != "" } + }, local.ai_gateway_environment, var.extra_environment) : k => v if v != "" } # Container Apps secrets. Which ones exist must be known at plan time, so # the optional ones are chosen from their inputs, never from generated values. diff --git a/infra/terraform/modules/openwork-azure-container-apps/tests/ai_gateway.tftest.hcl b/infra/terraform/modules/openwork-azure-container-apps/tests/ai_gateway.tftest.hcl new file mode 100644 index 00000000000..0dbbcda17a6 --- /dev/null +++ b/infra/terraform/modules/openwork-azure-container-apps/tests/ai_gateway.tftest.hcl @@ -0,0 +1,183 @@ +# OFFLINE. All Azure/external calls mocked; ephemeral random is local-only. +# Mock applies seed synthetic off/on state, then Gateway plan checks. No provisioning. +mock_provider "azurerm" { + mock_data "azurerm_resource_group" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test", location = "eastus" } } + mock_resource "azurerm_container_app_environment" { + defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/managedEnvironments/openwork-den", default_domain = "happy-sky-1234.eastus.azurecontainerapps.io", custom_domain_verification_id = "VERIFY123" } + } + mock_resource "azurerm_log_analytics_workspace" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.OperationalInsights/workspaces/openwork-den-logs" } } + mock_resource "azurerm_mysql_flexible_server" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.DBforMySQL/flexibleServers/openwork", fqdn = "openwork-den.mysql.database.azure.com" } } + mock_resource "azurerm_container_app_environment_certificate" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/managedEnvironments/openwork-den/certificates/openwork-den" } } +} +# Required ephemeral schema; synthetic binding has use_cli=false, and absent +# existing-store mocks ensure the real secret-read provider is never called. +provider "azapi" { + alias = "secret_read" + subscription_id = "00000000-0000-0000-0000-000000000000" + tenant_id = "00000000-0000-0000-0000-000000000000" + client_id = "00000000-0000-0000-0000-000000000000" + client_secret = "fixture-not-used" + use_cli = false +} +mock_provider "azapi" { + mock_data "azapi_resource" { defaults = { exists = false, output = { versions = null } } } + mock_resource "azapi_resource" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/containerApps/app", output = { properties = { outboundIpAddresses = [] } } } } + mock_resource "azapi_resource_action" { defaults = { output = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/jobs/openwork-migrate/executions/test" } } } +} +override_resource { + target = azapi_resource.ai_gateway + values = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/containerApps/ai-gateway-distinct" } +} +override_resource { + target = azapi_resource.migration + values = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.App/jobs/openwork-migrate" } +} +mock_provider "external" { + mock_data "external" { defaults = { result = { status = "Succeeded" } } } +} +variables { + resource_group_name = "rg-test" + owner_emails = ["admin@example.com"] + ai_gateway_image = "ghcr.io/different-ai/openwork-inference@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" +} +run "off_by_default" { + command = apply # Mock state only; no cloud resources or Job execution. + assert { + condition = length(azapi_resource.ai_gateway) == 0 && length(azurerm_container_app_custom_domain.ai_gateway) == 0 && output.ai_gateway_url == null && length([for env in azapi_resource.api.body.properties.template.containers[0].env : env.name if startswith(env.name, "GATEWAY_") || startswith(env.name, "INFERENCE_")]) == 0 && length(keys(local.app_specs)) == 2 + error_message = "Default: 0 AI Gateway apps/bindings, null URL, unchanged Den env; only two apps." + } +} +run "enable_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + assert { + condition = length(azapi_resource.ai_gateway) == 1 && azapi_resource.ai_gateway[0].body.properties.template.containers[0].image == var.ai_gateway_image && azapi_resource.ai_gateway[0].body.properties.configuration.ingress.targetPort == 8791 && azapi_resource.ai_gateway[0].body.properties.configuration.ingress.external && !azapi_resource.ai_gateway[0].body.properties.configuration.ingress.allowInsecure && azapi_resource.ai_gateway[0].body.properties.template.containers[0].probes[0].httpGet.path == "/health" && azapi_resource.ai_gateway[0].body.properties.template.containers[0].probes[1].httpGet.path == "/ready" + error_message = "Enabled: 0 -> 1 inference app, HTTPS-only ingress to 8791, health/readiness probes." + } + assert { + condition = output.ai_gateway_url == "https://models.example.com" && azurerm_container_app_custom_domain.ai_gateway[0].name == "models.example.com" && azurerm_container_app_custom_domain.ai_gateway[0].certificate_binding_type == "SniEnabled" && output.ai_gateway_dns_records.cname == "openwork-ai-gateway.happy-sky-1234.eastus.azurecontainerapps.io" + error_message = "Dedicated SNI hostname and DNS target expose the HTTPS origin to desktops." + } + assert { + condition = alltrue([for c in [azapi_resource.api.body.properties.template.containers[0], azapi_resource.ai_gateway[0].body.properties.template.containers[0]] : alltrue([for k, v in { + GATEWAY_ENABLED = "true", GATEWAY_PROXY_BASE_URL = "https://openwork-ai-gateway.happy-sky-1234.eastus.azurecontainerapps.io", GATEWAY_PUBLIC_BASE_URL = "https://models.example.com", INFERENCE_PROXY_BASE_URL = "https://models.example.com", DB_MODE = "mysql", NODE_ENV = "production" + } : anytrue([for e in c.env : e.name == k && try(e.value == v, false)])]) && alltrue([for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : anytrue([for e in c.env : e.name == name && try(e.secretRef == lower(replace(name, "_", "-")), false)])])]) + error_message = "Den and Gateway share canonical env and database/encryption secretRefs; internal proxy differs from desktop origin." + } + assert { + condition = toset([for s in azapi_resource.ai_gateway[0].body.properties.configuration.secrets : s.name]) == toset(["database-url", "den-db-encryption-key"]) && alltrue([for s in azapi_resource.ai_gateway[0].body.properties.configuration.secrets : !contains(keys(s), "value")]) && length(azapi_resource.ai_gateway[0].sensitive_body_version) == 1 && var.secret_versions.db_encryption_key_version == 1 && var.secret_versions.database_password_version == 1 && length(azapi_resource.ai_gateway[0].body.properties.template.initContainers) == 0 + error_message = "Only existing database/encryption values enter the write-only payload; state holds metadata, version counters stay 1, replicas never migrate." + } +} +run "den_custom_domains_coexist" { + command = plan + variables { + domain_name = "openwork.example.com" + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + assert { + condition = length(keys(local.app_specs)) == 3 && azurerm_container_app_custom_domain.web[0].name != azurerm_container_app_custom_domain.ai_gateway[0].name && azurerm_container_app_custom_domain.api[0].name != azurerm_container_app_custom_domain.ai_gateway[0].name && length(azurerm_container_app_environment_certificate.this) == 1 + error_message = "Three separate Den/AI apps and hostnames reuse the supplied covering certificate; no den-gateway service is implied." + } +} +run "gateway_gets_only_registry_identity" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + key_vault_identity_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/vault" + user_assigned_identity_ids = [ + "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/vault", + "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/registry", + ] + registry = { server = "registry.example.com", identity = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/registry" } + } + assert { + condition = length(azapi_resource.ai_gateway[0].identity[0].identity_ids) == 1 && contains(azapi_resource.ai_gateway[0].identity[0].identity_ids, var.registry.identity) && !contains(azapi_resource.ai_gateway[0].identity[0].identity_ids, var.key_vault_identity_id) && length(azapi_resource.api.identity[0].identity_ids) == 2 + error_message = "Gateway gets only registry-pull identity, never Den's Key Vault/integration identities; Den retains both existing identities." + } +} +run "missing_domain" { + command = plan + variables { + ai_gateway_enabled = true + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + expect_failures = [var.ai_gateway_domain_name] +} +run "missing_certificate" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + expect_failures = [azapi_resource.ai_gateway] +} +run "conflicting_environment" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + extra_environment = { GATEWAY_ENABLED = "false" } + } + expect_failures = [var.extra_environment] +} +run "conflicting_host" { + command = plan + variables { + domain_name = "openwork.example.com" + ai_gateway_enabled = true + ai_gateway_domain_name = "openwork.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + expect_failures = [azapi_resource.ai_gateway] +} +run "mutable_image" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ai_gateway_image = "ghcr.io/different-ai/openwork-inference:latest" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + expect_failures = [var.ai_gateway_image] +} +run "enabled_mock_state" { + command = apply # Mock-only state, no ARM or migration execution. + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + assert { + condition = length(azapi_resource.ai_gateway) == 1 && var.secret_versions.db_encryption_key_version == 1 && azurerm_container_app_custom_domain.ai_gateway[0].container_app_id == azapi_resource.ai_gateway[0].id && azurerm_container_app_custom_domain.ai_gateway[0].container_app_id != azapi_resource.api.id + error_message = "Synthetic state binds the public domain to the distinct Gateway app, not Den; encryption counter remains 1." + } +} +run "unchanged_enabled_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + custom_domain_certificate = { pfx_base64 = "UEZYLXRlc3Q=", password = "fixture" } + } + assert { + condition = azapi_resource.ai_gateway[0].sensitive_body_version["properties.configuration.secrets"] == local.app_write_versions.ai_gateway["properties.configuration.secrets"] && var.secret_versions.db_encryption_key_version == 1 && var.secret_versions.database_password_version == 1 + error_message = "Unchanged deployment hashes/write-only gates and rotation counters preserve existing secret values." + } +} +run "disable_plan_keeps_data" { + command = plan + assert { + condition = length(azapi_resource.ai_gateway) == 0 && output.ai_gateway_url == null && length(azurerm_mysql_flexible_server.this) == 1 && length(local.secret_env_names) == 4 && var.secret_versions.db_encryption_key_version == 1 && !contains(keys(local.environment), "GATEWAY_ENABLED") + error_message = "Disable: Gateway 1 -> 0, runtime defaults false, database and original secret/version metadata kept." + } +} diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/README.md b/infra/terraform/modules/openwork-gcp-cloud-run/README.md index 0539824cef8..9b4ec8cf1b4 100644 --- a/infra/terraform/modules/openwork-gcp-cloud-run/README.md +++ b/infra/terraform/modules/openwork-gcp-cloud-run/README.md @@ -14,13 +14,22 @@ A single-organization OpenWork control plane (Den), without Kubernetes: | HTTPS | Default run.app URLs; optional global HTTPS load balancer and managed certificate | | Identity / logs | Dedicated runtime service account, per-secret accessor grants; Cloud Logging | | Optional network | Direct VPC egress into an existing VPC/subnet | +| AI Gateway (optional, off by default) | `openwork-inference` Cloud Run service on 8791, HTTPS LB host/NEG/backend and certificate | -Uses the environment described in [the manual Cloud Run guide](../../../../docs/gcp-cloud-run.md), `packaging/cloud-run`, and the Helm chart. Workbot, the runner, Slack, OpenWork Web/gateway, Redis and headless Automations are outside this module; their existing inputs are unchanged. +The **AI Gateway** handles inference, **den-gateway** handles OpenWork Web/sandboxes, +and Den API's **MCP gateway** handles tools; they are not interchangeable. +See [AI Gateway setup and networking limits](../../AI_GATEWAY.md), including the +Cloud SQL socket adapter and the HTTPS (not private DNS) service-call boundary. +A separate managed AI Gateway certificate avoids replacing the existing Den +certificate when adding its hostname; supplied certificates must cover it. + +Uses the environment described in [the manual Cloud Run guide](../../../../docs/gcp-cloud-run.md), `packaging/cloud-run`, and the Helm chart. Workbot, the runner, Slack, OpenWork Web/den-gateway, Redis and headless Automations are outside this module; their existing inputs are unchanged. ## Tested scenarios | Date | Tools | Images | Scenario / result | | --- | --- | --- | --- | +| 2026-11-13 | Offline-only implementation | Synthetic fixture digests | AI Gateway live DNS/TLS, admin/provider/Desktop, unchanged apply, disable/data retention and teardown: **NotRun**; no provisioning | | 2026-10-14 | Terraform 1.16.5 | 0.18.58 → dev b8efc5b (digests) | Earlier module: setup, browser, isolated desktop, same-database upgrade and destroy passed; one API instance | | 2026-10-15 | Terraform 1.16.5 / OpenTofu 1.11.14 | Mocked Google provider | Init, validate and 32 offline test runs per tool passed; write-only fields, protected DB, 2-instance singleton migration topology | | 2026-10-15 | Terraform 1.16.5; local testkit | 0.18.58 → dev f450b1a (digests) | Two API instances observed in both revisions; one migrator/task per deployment, exit 0; setup, browser, exact admin/org preservation, isolated source Electron, no secret rotation/state values, 21-resource destroy and independent empty inventory passed | @@ -94,6 +103,7 @@ Get the setup code directly from Secret Manager, without logging it, using the ` | `database.deletion_protection` | Deprecated nullable alias; when explicitly set it overrides the common input for Cloud SQL only | | `den_api` | CPU `1`, memory `2Gi`, min `1`, max `3` | | `den_web` | CPU `1`, memory `1Gi`, min `1`, max `3` (Cloud Run request-driven service floor uses one CPU) | +| `ai_gateway_enabled`, `ai_gateway_domain_name`, `ai_gateway_image` | `false`, `""`, `""`; enabled requires a dedicated hostname and mirrored inference image@digest; creates HTTPS LB even without Den custom domains | | `migration_run_id` | `"1"`; changing it deliberately runs another migration | | `write_only_secrets` | `true`; ephemeral generation and write-only provider arguments; false is a not-recommended legacy compatibility opt-out | | `database_password_wo_version` | `1`; deliberate database-password rotation counter, coordinate with the connection secret | @@ -112,7 +122,8 @@ Get the setup code directly from Secret Manager, without logging it, using the ` | `secret_ids`, `service_account_email`, `service_names` | Managed secret identifiers and runtime identities | | `images` | Final resolved image references | | `bootstrap_code` | Sensitive legacy-mode value; null in write-only mode | -| `load_balancer_ip`, `dns_records` | Optional IP and two A records | +| `load_balancer_ip`, `dns_records` | Optional IP and A records for enabled custom Den/AI Gateway hostnames | +| `ai_gateway_url` | AI Gateway HTTPS origin, null when disabled | ### Existing infrastructure and TLS @@ -192,4 +203,9 @@ For an S3 backend outside this GCP example, use native `use_lockfile = true`, no ## Cost notes +Enabling the AI Gateway keeps one 1 vCPU / 2 GiB instance warm (up to three), +and creates an HTTPS LB/IP if Den previously used only run.app. It adds no Cloud +SQL instance. Estimate those incremental regional charges before applying; no +live Gateway cost/capacity measurement is claimed. + Defaults keep one API (1 vCPU / 2 GiB) and one web (1 vCPU / 1 GiB) instance warm plus a db-g1-small (1.7 GB) with 20 GB disk, backups, secrets and image cache. This is a smoke-test baseline, **not** HA or production load sizing. SQL, warm-instance rates, storage, egress and optional load balancer/IP charges vary; estimate current regional prices before apply. The old micro-based daily estimate is no longer applicable. Min instances zero reduces Cloud Run idle cost but adds sign-in cold starts. Disposable `examples/complete` disables deletion protection and backups; do not copy those choices into production. Direct VPC egress can keep a subnet busy for up to roughly two hours after service deletion: budget teardown time, and never force-delete someone else's resources. diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/ai-gateway.tf b/infra/terraform/modules/openwork-gcp-cloud-run/ai-gateway.tf new file mode 100644 index 00000000000..ac96ed45489 --- /dev/null +++ b/infra/terraform/modules/openwork-gcp-cloud-run/ai-gateway.tf @@ -0,0 +1,175 @@ +# AI Gateway (ee/apps/gateway / openwork-inference), not OpenWork Web's den-gateway. +variable "ai_gateway_enabled" { + description = "Deploy the AI Gateway for administrator-managed inference. Off by default; installation configuration, not a product flag." + type = bool + default = false + validation { + condition = !var.ai_gateway_enabled || alltrue([for name in concat(keys(var.extra_environment), keys(var.extra_secrets)) : + !startswith(name, "GATEWAY_") && !startswith(name, "INFERENCE_") && !contains(["DB_MODE", "DATABASE_URL", "DATABASE_HOST", "DATABASE_USERNAME", "DATABASE_PASSWORD", "DEN_DB_ENCRYPTION_KEY", "NODE_ENV", "OPENWORK_DEV_MODE", "PORT"], name) + ]) + error_message = "With ai_gateway_enabled, remove Gateway/Inference and shared runtime/database overrides from extra_environment and extra_secrets; use the module inputs and existing app secrets." + } +} +variable "ai_gateway_domain_name" { + description = "Dedicated desktop-reachable HTTPS hostname for the AI Gateway. Required when enabled; point DNS to load_balancer_ip and include it in supplied certificates." + type = string + default = "" + validation { + condition = !var.ai_gateway_enabled || ( + length(var.ai_gateway_domain_name) <= 253 && + can(regex("^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)+$", var.ai_gateway_domain_name)) && + !can(regex("(^localhost\\.|\\.(localhost|local|internal|svc)$|\\.[0-9]+$)", var.ai_gateway_domain_name)) + ) + error_message = "ai_gateway_enabled requires ai_gateway_domain_name: a qualified non-local DNS hostname, without scheme, port or path." + } +} +variable "ai_gateway_image" { + description = "AI Gateway image in Artifact Registry/Docker Hub, pinned by digest and matching the Den release. Mirror ghcr.io/different-ai/openwork-inference; required when enabled." + type = string + default = "" + validation { + condition = !var.ai_gateway_enabled || can(regex("@sha256:[a-f0-9]{64}$", var.ai_gateway_image)) + error_message = "ai_gateway_enabled requires ai_gateway_image pinned by @sha256 digest; mutable tags are not supported." + } +} +locals { + ai_gateway_url = "https://${var.ai_gateway_domain_name}" + # Like den-web -> den-api, Cloud Run uses the HTTPS load balancer, NOT a + # private DNS name. Its default run.app URI stays disabled behind the LB. + ai_gateway_environment = var.ai_gateway_enabled ? { + GATEWAY_ENABLED = "true" + GATEWAY_PROXY_BASE_URL = local.ai_gateway_url + GATEWAY_PUBLIC_BASE_URL = local.ai_gateway_url + INFERENCE_PROXY_BASE_URL = local.ai_gateway_url + } : {} + ai_gateway_secret_refs = { for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : name => local.secret_refs[name] } + # Preserve the stored URL and all version counters. Only the unused hostname + # of an authenticated Unix-socket URL is normalized in the serving process. + # Inline like the AWS TLS entrypoint, so managed installer payloads (which + # include operational .tf files only) need no new file-allowlist exceptions. + ai_gateway_entrypoint = <<-JS + // Cloud SQL uses socketPath; only its unused localhost URL hostname changes. + // Never change stored secrets, credentials, socket paths or TLS settings. + import { pathToFileURL } from "node:url"; + const url = new URL(process.env.DATABASE_URL); + if (url.protocol === "mysql:" && url.hostname === "localhost" && url.searchParams.get("socket")?.startsWith("/cloudsql/")) { + url.hostname = "cloudsql"; + process.env.DATABASE_URL = url.href; + } + await import(pathToFileURL(process.argv[1]).href); + JS +} +resource "google_cloud_run_v2_service" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + project = var.project_id + name = "${var.name}-ai-gateway" + location = var.region + ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER" + default_uri_disabled = true + invoker_iam_disabled = true # Gateway authenticates member keys; no Cloud Run ID token in desktops. + deletion_protection = false # Optional stateless service; disabling must not delete data. + labels = var.labels + template { + service_account = local.service_account_email + timeout = "3600s" + scaling { + min_instance_count = 1 + max_instance_count = 3 + } + containers { + image = var.ai_gateway_image + command = ["node"] + args = ["--import", "/app/ee/apps/gateway/dist/instrumentation.js", "--input-type=module", "-e", local.ai_gateway_entrypoint, "/app/ee/apps/gateway/dist/server.js"] + ports { container_port = 8791 } + resources { limits = { cpu = "1", memory = "2Gi" } } + startup_probe { + http_get { path = "/health" } + period_seconds = 5 + failure_threshold = 24 + } + liveness_probe { + http_get { path = "/health" } + } + dynamic "env" { + for_each = merge(local.ai_gateway_environment, { NODE_ENV = "production", OPENWORK_DEV_MODE = "0", DB_MODE = "mysql" }) + content { + name = env.key + value = env.value + } + } + dynamic "env" { + for_each = local.ai_gateway_secret_refs + content { + name = env.key + value_source { + secret_key_ref { + secret = env.value.secret + version = env.value.version + } + } + } + } + dynamic "volume_mounts" { + for_each = local.cloud_sql + content { + name = "cloudsql" + mount_path = "/cloudsql" + } + } + } + dynamic "volumes" { + for_each = local.cloud_sql + content { + name = "cloudsql" + cloud_sql_instance { instances = [volumes.value] } + } + } + dynamic "vpc_access" { + for_each = local.vpc_access + content { + egress = var.vpc_egress + network_interfaces { + network = vpc_access.value.network + subnetwork = vpc_access.value.subnetwork + } + } + } + } + lifecycle { + precondition { + condition = !contains([var.domain_name, local.api_host, local.web_host], var.ai_gateway_domain_name) + error_message = "The AI Gateway hostname must differ from Den web and API hostnames." + } + } + depends_on = [google_cloud_run_v2_job.migrate] +} +resource "google_compute_region_network_endpoint_group" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + project = var.project_id + name = "${var.name}-ai-gateway" + region = var.region + network_endpoint_type = "SERVERLESS" + cloud_run { service = google_cloud_run_v2_service.ai_gateway[0].name } +} +resource "google_compute_backend_service" "ai_gateway" { + count = var.ai_gateway_enabled ? 1 : 0 + project = var.project_id + name = "${var.name}-ai-gateway" + load_balancing_scheme = "EXTERNAL_MANAGED" + protocol = var.managed_legacy ? "HTTP" : "HTTPS" + backend { group = google_compute_region_network_endpoint_group.ai_gateway[0].id } +} +# Separate certificate: enabling inference must not replace the existing Den +# certificate (managed certificate domain sets are immutable). Hash the host so +# a later hostname change can create its replacement before detaching the old one. +resource "google_compute_managed_ssl_certificate" "ai_gateway" { + count = var.ai_gateway_enabled && length(var.ssl_certificate_ids) == 0 ? 1 : 0 + project = var.project_id + name = "${var.name}-ai-${substr(sha256(var.ai_gateway_domain_name), 0, 8)}" + managed { domains = [var.ai_gateway_domain_name] } + lifecycle { create_before_destroy = true } +} +output "ai_gateway_url" { + description = "AI Gateway HTTPS origin for desktops, or null when disabled. Not the OpenWork Web den-gateway URL." + value = var.ai_gateway_enabled ? local.ai_gateway_url : null +} diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/cloudrun.tf b/infra/terraform/modules/openwork-gcp-cloud-run/cloudrun.tf index fa483562672..733022e5e4d 100644 --- a/infra/terraform/modules/openwork-gcp-cloud-run/cloudrun.tf +++ b/infra/terraform/modules/openwork-gcp-cloud-run/cloudrun.tf @@ -166,8 +166,8 @@ resource "google_cloud_run_v2_service" "api" { containers { image = local.den_api_image - command = var.managed_legacy ? ["node"] : null - args = var.managed_legacy ? ["/app/ee/apps/den-api/dist/main.js"] : null + command = var.ai_gateway_enabled || var.managed_legacy ? ["node"] : null + args = var.ai_gateway_enabled ? ["--input-type=module", "-e", local.ai_gateway_entrypoint, "/app/ee/apps/den-api/dist/main.js"] : (var.managed_legacy ? ["/app/ee/apps/den-api/dist/main.js"] : null) ports { container_port = 8788 diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/lb.tf b/infra/terraform/modules/openwork-gcp-cloud-run/lb.tf index a408b507c3f..0a6dec0072b 100644 --- a/infra/terraform/modules/openwork-gcp-cloud-run/lb.tf +++ b/infra/terraform/modules/openwork-gcp-cloud-run/lb.tf @@ -4,9 +4,9 @@ # HTTP redirects to HTTPS. Without it, the services' run.app URLs serve HTTPS. locals { - create_lb = local.custom_domain + create_lb = local.custom_domain || var.ai_gateway_enabled create_lb_address = local.create_lb && var.load_balancer_ip_address == "" - create_lb_cert = local.create_lb && length(var.ssl_certificate_ids) == 0 + create_lb_cert = local.custom_domain && length(var.ssl_certificate_ids) == 0 lb_ip = local.create_lb ? (local.create_lb_address ? google_compute_global_address.lb[0].address : var.load_balancer_ip_address) : null lb_services = local.create_lb ? { api = google_cloud_run_v2_service.api.name, web = google_cloud_run_v2_service.web.name } : {} } @@ -52,13 +52,28 @@ resource "google_compute_url_map" "https" { name = var.managed_legacy ? "${var.name}-routes" : "${var.name}-den" default_service = google_compute_backend_service.run["web"].id + dynamic "host_rule" { + for_each = var.ai_gateway_enabled ? [1] : [] + content { + hosts = [var.ai_gateway_domain_name] + path_matcher = "ai-gateway" + } + } + dynamic "path_matcher" { + for_each = var.ai_gateway_enabled ? [1] : [] + content { + name = "ai-gateway" + default_service = google_compute_backend_service.ai_gateway[0].id + } + } + host_rule { - hosts = [local.api_host] + hosts = [trimprefix(local.api_url, "https://")] path_matcher = "api" } host_rule { - hosts = [var.domain_name] + hosts = [local.web_host] path_matcher = "web" } @@ -87,10 +102,14 @@ resource "google_compute_managed_ssl_certificate" "this" { resource "google_compute_target_https_proxy" "this" { count = local.create_lb ? 1 : 0 - project = var.project_id - name = var.managed_legacy ? "${var.name}-https" : "${var.name}-den" - url_map = google_compute_url_map.https[0].id - ssl_certificates = local.create_lb_cert ? [google_compute_managed_ssl_certificate.this[0].id] : var.ssl_certificate_ids + project = var.project_id + name = var.managed_legacy ? "${var.name}-https" : "${var.name}-den" + url_map = google_compute_url_map.https[0].id + ssl_certificates = concat( + local.create_lb_cert ? [google_compute_managed_ssl_certificate.this[0].id] : [], + google_compute_managed_ssl_certificate.ai_gateway[*].id, + var.ssl_certificate_ids, + ) } resource "google_compute_global_forwarding_rule" "https" { diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/main.tf b/infra/terraform/modules/openwork-gcp-cloud-run/main.tf index 794a6b09b10..892089bd3ea 100644 --- a/infra/terraform/modules/openwork-gcp-cloud-run/main.tf +++ b/infra/terraform/modules/openwork-gcp-cloud-run/main.tf @@ -75,7 +75,7 @@ locals { SMTP_PORT = tostring(var.smtp.port) SMTP_USER = var.smtp.username SMTP_SECURE = local.bool[var.smtp.secure] - }, var.extra_environment) + }, local.ai_gateway_environment, var.extra_environment) # Cloud Run reserves PORT and sets it from the container port. environment_list = { for k, v in local.environment : k => v if v != "" && k != "PORT" } diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/outputs.tf b/infra/terraform/modules/openwork-gcp-cloud-run/outputs.tf index 9147e824583..cf0fde463cf 100644 --- a/infra/terraform/modules/openwork-gcp-cloud-run/outputs.tf +++ b/infra/terraform/modules/openwork-gcp-cloud-run/outputs.tf @@ -20,13 +20,13 @@ output "bootstrap_code" { } output "load_balancer_ip" { - description = "Load balancer IPv4 address: point domain_name and the API host (A records) at it. Null without domain_name." + description = "Load balancer IPv4 address for custom Den and AI Gateway hostnames. Null when neither is enabled." value = local.lb_ip } output "dns_records" { - description = "A records to create for the custom hostnames. Empty without domain_name." - value = local.create_lb ? { (var.domain_name) = local.lb_ip, (local.api_host) = local.lb_ip } : {} + description = "A records to create for custom Den and AI Gateway hostnames. Empty without custom hosts." + value = merge(local.custom_domain ? { (var.domain_name) = local.lb_ip, (local.api_host) = local.lb_ip } : {}, var.ai_gateway_enabled ? { (var.ai_gateway_domain_name) = local.lb_ip } : {}) } output "service_names" { diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/scripts/cloud-sql-entrypoint.test.mjs b/infra/terraform/modules/openwork-gcp-cloud-run/scripts/cloud-sql-entrypoint.test.mjs new file mode 100644 index 00000000000..96a1816f6a9 --- /dev/null +++ b/infra/terraform/modules/openwork-gcp-cloud-run/scripts/cloud-sql-entrypoint.test.mjs @@ -0,0 +1,57 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import test from "node:test"; + +// Test the exact inline deployment entrypoint, not a second implementation. +const terraform = readFileSync(new URL("../ai-gateway.tf", import.meta.url), "utf8"); +const script = terraform.match(/ai_gateway_entrypoint = <<-JS\n([\s\S]*?)^\s*JS$/m)?.[1]; +assert.ok(script, "Terraform entrypoint heredoc must exist"); +const root = new URL("../../../../../", import.meta.url); +const validation = new URL("ee/packages/utils/src/gateway-env.ts", root).href; +const mysql = new URL("ee/packages/den-db/src/mysql-config.ts", root).href; + +// A harmless data module inspects the environment exactly where the real server +// would import. No database driver, account, credential store or network is used. +function run(databaseUrl, expected) { + const verify = ` + import assert from 'node:assert/strict'; + import { parseGatewayDeploymentEnv } from ${JSON.stringify(validation)}; + import { parseMySqlConnectionConfig } from ${JSON.stringify(mysql)}; + const original = ${JSON.stringify(databaseUrl)}; + const current = process.env.DATABASE_URL; + assert.equal(new URL(current).hostname, ${JSON.stringify(expected)}); + const before = parseMySqlConnectionConfig(original); + const after = parseMySqlConnectionConfig(current); + assert.deepEqual({...before, host: after.host}, after); + assert.equal(parseGatewayDeploymentEnv(process.env).enabled, true); + console.log('validated enabled Gateway; credentials/socket/TLS unchanged'); + `; + // The entrypoint intentionally imports a filesystem path in deployed images. + // Replace just that final import for a zero-network test module. + const isolated = script.replace('await import(pathToFileURL(process.argv[1]).href);', `await import(${JSON.stringify("data:text/javascript," + encodeURIComponent(verify))});`); + return spawnSync(process.execPath, ["--input-type=module", "-e", isolated], { + env: { PATH: process.env.PATH, NODE_ENV: "production", OPENWORK_DEV_MODE: "0", GATEWAY_ENABLED: "true", GATEWAY_PROXY_BASE_URL: "https://models.example.com", GATEWAY_PUBLIC_BASE_URL: "https://models.example.com", DB_MODE: "mysql", DATABASE_URL: databaseUrl, DEN_DB_ENCRYPTION_KEY: "synthetic-encryption-key-0123456789abcdef" }, + encoding: "utf8", + }); +} + +test("Cloud SQL URL passes production Gateway validation without changing credentials or socket", () => { + const result = run("mysql://app:p%40ss@localhost/openwork_den?socket=%2Fcloudsql%2Ftest-project%3Aus-central1%3Aden", "cloudsql"); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /credentials\/socket\/TLS unchanged/); +}); +test("external verified-TLS database URL is unchanged", () => { + const result = run("mysql://app:fixture@db.example.com/openwork_den?sslaccept=strict", "db.example.com"); + assert.equal(result.status, 0, result.stderr); +}); +test("localhost TCP URL still fails production validation", () => { + const result = run("mysql://app:fixture@localhost/openwork_den?sslaccept=strict", "localhost"); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /production must not use localhost/); +}); +test("non-Cloud-SQL socket does not bypass localhost validation", () => { + const result = run("mysql://app:fixture@localhost/openwork_den?socket=%2Ftmp%2Fmysql.sock", "localhost"); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /production must not use localhost/); +}); diff --git a/infra/terraform/modules/openwork-gcp-cloud-run/tests/ai_gateway.tftest.hcl b/infra/terraform/modules/openwork-gcp-cloud-run/tests/ai_gateway.tftest.hcl new file mode 100644 index 00000000000..fb1c3771ac1 --- /dev/null +++ b/infra/terraform/modules/openwork-gcp-cloud-run/tests/ai_gateway.tftest.hcl @@ -0,0 +1,175 @@ +# OFFLINE: all Google APIs mocked; random is local-only (ephemeral schema). +# Mock applies establish synthetic off/on state; enable/invalid checks use plans. +mock_provider "google" { + mock_data "google_project" { defaults = { number = "123456789012" } } + mock_resource "google_sql_database_instance" { defaults = { connection_name = "test-project:us-central1:openwork-den" } } + mock_resource "google_secret_manager_secret_version" { defaults = { version = "1" } } + mock_resource "google_service_account" { defaults = { email = "runtime@test-project.iam.gserviceaccount.com" } } + mock_resource "google_compute_backend_service" { defaults = { id = "projects/test-project/global/backendServices/openwork" } } + mock_resource "google_compute_managed_ssl_certificate" { defaults = { id = "projects/test-project/global/sslCertificates/openwork-den" } } +} +override_resource { + target = google_compute_backend_service.ai_gateway + values = { id = "projects/test-project/global/backendServices/ai-gateway-distinct" } +} +override_resource { + target = google_compute_managed_ssl_certificate.ai_gateway + values = { id = "projects/test-project/global/sslCertificates/openwork-ai-gateway" } +} +provider "random" {} +variables { + project_id = "test-project" + openwork_version = "0.0.0-test" + owner_emails = ["admin@example.com"] + write_only_secrets = false + image_digests = { den_api = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", den_web = "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" } + ai_gateway_image = "us-central1-docker.pkg.dev/test-project/mirror/openwork-inference@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" +} +run "off_by_default" { + command = apply # Synthetic state only; no provisioning or migration execution. + assert { + condition = length(google_cloud_run_v2_service.ai_gateway) == 0 && length(google_compute_backend_service.ai_gateway) == 0 && length(google_compute_region_network_endpoint_group.ai_gateway) == 0 && length(google_compute_backend_service.run) == 0 && output.ai_gateway_url == null && length([for e in google_cloud_run_v2_service.api.template[0].containers[0].env : e.name if startswith(e.name, "GATEWAY_") || startswith(e.name, "INFERENCE_")]) == 0 + error_message = "Default: 0 AI Gateway services/backends, URL null, no new Den env; no load balancer." + } +} +run "enable_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + network = "projects/test-project/global/networks/shared" + subnetwork = "projects/test-project/regions/us-central1/subnetworks/run" + } + assert { + condition = length(google_cloud_run_v2_service.ai_gateway) == 1 && google_cloud_run_v2_service.ai_gateway[0].template[0].containers[0].image == var.ai_gateway_image && google_cloud_run_v2_service.ai_gateway[0].template[0].containers[0].ports[0].container_port == 8791 && google_cloud_run_v2_service.ai_gateway[0].template[0].containers[0].startup_probe[0].http_get[0].path == "/health" && output.ai_gateway_url == "https://models.example.com" + error_message = "Enabled: 0 -> 1 inference service on 8791 with /health; public models URL." + } + assert { + condition = google_cloud_run_v2_service.ai_gateway[0].default_uri_disabled && google_cloud_run_v2_service.ai_gateway[0].ingress == "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER" && google_cloud_run_v2_service.ai_gateway[0].invoker_iam_disabled && !google_cloud_run_v2_service.ai_gateway[0].deletion_protection && contains(google_compute_managed_ssl_certificate.ai_gateway[0].managed[0].domains, "models.example.com") && anytrue([for r in google_compute_url_map.https[0].host_rule : contains(r.hosts, "models.example.com") && r.path_matcher == "ai-gateway"]) + error_message = "HTTPS LB routes the dedicated host and certificate; no direct run.app bypass or Cloud Run ID-token requirement; stateless service can be disabled." + } + assert { + condition = alltrue([for c in [google_cloud_run_v2_service.api.template[0].containers[0], google_cloud_run_v2_service.ai_gateway[0].template[0].containers[0]] : alltrue([for k, v in { + GATEWAY_ENABLED = "true", GATEWAY_PROXY_BASE_URL = "https://models.example.com", GATEWAY_PUBLIC_BASE_URL = "https://models.example.com", INFERENCE_PROXY_BASE_URL = "https://models.example.com", DB_MODE = "mysql", NODE_ENV = "production" + } : contains([for e in c.env : "${e.name}=${e.value == null ? "" : e.value}"], "${k}=${v}")]) && alltrue([for name in ["DATABASE_URL", "DEN_DB_ENCRYPTION_KEY"] : anytrue([for e in c.env : e.name == name && try(e.value_source[0].secret_key_ref[0].secret == google_secret_manager_secret.app[name].secret_id && e.value_source[0].secret_key_ref[0].version == "1", false)])])]) + error_message = "Both serving apps get matching enabled/public/proxy env and exactly the existing database/encryption secret versions." + } + assert { + condition = google_cloud_run_v2_service.ai_gateway[0].template[0].vpc_access[0].network_interfaces[0].subnetwork == var.subnetwork && contains(google_cloud_run_v2_service.ai_gateway[0].template[0].volumes[0].cloud_sql_instance[0].instances, "test-project:us-central1:openwork-den") && strcontains(google_cloud_run_v2_service.api.template[0].containers[0].args[2], "url.hostname = \"cloudsql\"") && strcontains(google_cloud_run_v2_service.ai_gateway[0].template[0].containers[0].args[4], "url.hostname = \"cloudsql\"") && strcontains(google_secret_manager_secret_version.app["DATABASE_URL"].secret_data, "@localhost/") + error_message = "Gateway retains VPC and Cloud SQL socket wiring; only serving-process hostname is adapted, stored localhost URL unchanged." + } + assert { + condition = toset(values(output.secret_ids)) == toset(["openwork-den-database-url", "openwork-den-better-auth-secret", "openwork-den-den-db-encryption-key", "openwork-den-den-initial-admin-bootstrap-code"]) && length(google_secret_manager_secret.app) == 4 && google_secret_manager_secret_version.app["DEN_DB_ENCRYPTION_KEY"].secret_data == random_password.db_encryption[0].result && length(var.secret_data_wo_version) == 0 + error_message = "Same four app secrets and encryption value, no new version counters." + } +} +run "custom_den_mock_state" { + command = apply # Mock only: establish an existing Den TLS certificate. + variables { domain_name = "openwork.example.com" } + assert { + condition = toset(google_compute_managed_ssl_certificate.this[0].managed[0].domains) == toset(["openwork.example.com", "api.openwork.example.com"]) && length(google_compute_managed_ssl_certificate.ai_gateway) == 0 + error_message = "Synthetic Den baseline has its two-host certificate and no Gateway certificate." + } +} +run "with_den_custom_domains" { + command = plan + variables { + domain_name = "openwork.example.com" + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(google_compute_region_network_endpoint_group.run) + length(google_compute_region_network_endpoint_group.ai_gateway) == 3 && length(google_compute_url_map.https[0].host_rule) == 3 && toset(google_compute_managed_ssl_certificate.this[0].managed[0].domains) == toset(["openwork.example.com", "api.openwork.example.com"]) && google_compute_managed_ssl_certificate.this[0].id != "" && toset(google_compute_managed_ssl_certificate.ai_gateway[0].managed[0].domains) == toset(["models.example.com"]) + error_message = "Three separate backends/hosts coexist on one HTTPS LB; a separate Gateway certificate preserves the existing Den certificate and does not create den-gateway." + } +} +run "operator_managed_certificates" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ssl_certificate_ids = ["projects/test-project/global/sslCertificates/external"] + } + assert { + condition = length(google_compute_managed_ssl_certificate.ai_gateway) == 0 && length(google_compute_managed_ssl_certificate.this) == 0 && toset(google_compute_target_https_proxy.this[0].ssl_certificates) == toset(var.ssl_certificate_ids) + error_message = "Supplied covering certificates remain operator-owned; no replacement certificate is generated." + } +} +run "missing_domain" { + command = plan + variables { ai_gateway_enabled = true } + expect_failures = [var.ai_gateway_domain_name] +} +run "invalid_public_hostname" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.internal" + } + expect_failures = [var.ai_gateway_domain_name] +} +run "conflicting_environment" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + extra_environment = { GATEWAY_PUBLIC_BASE_URL = "https://other.example.com" } + } + expect_failures = [var.ai_gateway_enabled] +} +run "conflicting_secret" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + extra_secrets = { DEN_DB_ENCRYPTION_KEY = "other-secret" } + } + expect_failures = [var.ai_gateway_enabled] +} +run "conflicting_host" { + command = plan + variables { + domain_name = "openwork.example.com" + ai_gateway_enabled = true + ai_gateway_domain_name = "openwork.example.com" + } + expect_failures = [google_cloud_run_v2_service.ai_gateway] +} +run "mutable_image" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ai_gateway_image = "example.test/openwork-inference:latest" + } + expect_failures = [var.ai_gateway_image] +} +run "enabled_mock_state" { + command = apply # Mock-only state, no Cloud Run/SQL calls. + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(google_cloud_run_v2_service.ai_gateway) == 1 && length(google_secret_manager_secret.app) == 4 && contains(google_compute_target_https_proxy.this[0].ssl_certificates, google_compute_managed_ssl_certificate.ai_gateway[0].id) && anytrue([for matcher in google_compute_url_map.https[0].path_matcher : matcher.name == "ai-gateway" && matcher.default_service == google_compute_backend_service.ai_gateway[0].id]) + error_message = "Synthetic state routes the Gateway matcher to its distinct backend, not Den, and keeps the original four secrets." + } +} +run "unchanged_enabled_plan" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = length(google_cloud_run_v2_service.ai_gateway) == 1 && toset(values(output.secret_ids)) == toset(["openwork-den-database-url", "openwork-den-better-auth-secret", "openwork-den-den-db-encryption-key", "openwork-den-den-initial-admin-bootstrap-code"]) && length(var.secret_data_wo_version) == 0 + error_message = "Unchanged enabled plan retains original secret identities and version counters." + } +} +run "disable_plan_keeps_data" { + command = plan + assert { + condition = length(google_cloud_run_v2_service.ai_gateway) == 0 && output.ai_gateway_url == null && length(google_sql_database_instance.this) == 1 && toset(values(output.secret_ids)) == toset(["openwork-den-database-url", "openwork-den-better-auth-secret", "openwork-den-den-db-encryption-key", "openwork-den-den-initial-admin-bootstrap-code"]) && !contains(keys(local.environment), "GATEWAY_ENABLED") + error_message = "Disable: Gateway 1 -> 0, runtime defaults false, database and four original secrets kept." + } +} diff --git a/infra/terraform/modules/openwork-self-host/README.md b/infra/terraform/modules/openwork-self-host/README.md index 8baa1569804..8590e639f96 100644 --- a/infra/terraform/modules/openwork-self-host/README.md +++ b/infra/terraform/modules/openwork-self-host/README.md @@ -20,6 +20,18 @@ inputs and generated secrets. | `BETTER_AUTH_SECRET`, `DEN_DB_ENCRYPTION_KEY` | Generated when not supplied. Stored in Terraform state. | | Kubernetes Secret | Written by Terraform by default. Can instead be rendered by the chart or supplied by you (`existing_secret_name`). | | Helm release `openwork-ee` | Den API, Den web, and the migration Job that runs before each install or upgrade. Ingress is optional. | +| AI Gateway (optional, off by default) | The chart's **existing** inference deployment plus one Gateway TLS ingress; not another runtime implementation. | + +Use `ai_gateway_enabled=true` and `ai_gateway_domain_name="models.example.com"`; +`ingress.tls_secret_name` must cover that name (or disable ingress and supply HTTPS +routing externally). Output `ai_gateway_url` is null when disabled. Defaults are +false / empty; existing advanced Helm Gateway installs are not disabled by this +new default. The existing Terraform >= 1.5 floor is retained; native mock tests +run on Terraform/OpenTofu >= 1.11. See [AI Gateway guide](../../AI_GATEWAY.md) for +conflicts and limits. AI Gateway live-cluster conformance remains **NotRun**. +The inference AI Gateway, OpenWork Web's den-gateway and Den API's MCP gateway +are separate services. Native managed AWS/GCP installs call cloud modules, not +this wrapper. Nothing here changes managed defaults or establishes live readiness. ## What you provide @@ -95,7 +107,8 @@ without it. ## Anything else in the chart Pass any other chart value through `extra_helm_values`. Helm deep-merges it -over the module's values, for example to enable Gateway or change probes: +over the module's values, for example to change probes. When using first-class +AI Gateway enablement, do not override its enable/URL/name/port values here: ```hcl extra_helm_values = { @@ -111,6 +124,6 @@ every value. - Provisioning the cluster, MySQL, Redis, DNS or certificates for a specific cloud. - A tested apply on a live cluster, and an upgrade across chart versions. -- First-class inputs for Gateway, Daytona provisioning, the GitHub connector - and installer artifacts (use `extra_helm_values` for now). +- First-class inputs for Daytona provisioning, the GitHub connector and installer + artifacts (use `extra_helm_values` for now). - Helm provider 3.x (`kubernetes = {}` attribute syntax); pinned below 3.0. diff --git a/infra/terraform/modules/openwork-self-host/ai-gateway.tf b/infra/terraform/modules/openwork-self-host/ai-gateway.tf new file mode 100644 index 00000000000..9e1775fabe0 --- /dev/null +++ b/infra/terraform/modules/openwork-self-host/ai-gateway.tf @@ -0,0 +1,84 @@ +# Thin interface to the chart's existing gateway.enabled behavior. The chart +# remains the runtime/deployment authority; only a separate HTTPS ingress is new. +variable "ai_gateway_enabled" { + description = "Enable the chart's existing AI Gateway with first-class URL/ingress wiring. Off by default. Existing extra_helm_values installs remain supported when false." + type = bool + default = false +} +variable "ai_gateway_domain_name" { + description = "Dedicated desktop-reachable HTTPS hostname for the AI Gateway; required with ai_gateway_enabled." + type = string + default = "" + validation { + condition = var.ai_gateway_domain_name == "" || ( + length(var.ai_gateway_domain_name) <= 253 && + can(regex("^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)+$", var.ai_gateway_domain_name)) && + !can(regex("(^localhost\\.|\\.(localhost|local|internal|svc)$|\\.[0-9]+$)", var.ai_gateway_domain_name)) + ) + error_message = "ai_gateway_domain_name must be empty or a qualified non-local DNS hostname, without scheme, port or path." + } +} +locals { + # Keep the wrapper's existing Terraform >= 1.5 floor: cross-input checks go + # on helm_release preconditions, not post-1.9 cross-variable validation. + ai_gateway_overrides_valid = ( + !can(var.extra_helm_values.gateway.enabled) && !can(var.extra_helm_values.inference.enabled) && + !can(var.extra_helm_values.config.internal.gatewayProxyBaseUrl) && !can(var.extra_helm_values.config.internal.inferenceProxyBaseUrl) && + !can(var.extra_helm_values.config.public.gatewayPublicBaseUrl) && + !can(var.extra_helm_values.fullnameOverride) && !can(var.extra_helm_values.nameOverride) && + !can(var.extra_helm_values.gateway.containerPort) && !can(var.extra_helm_values.inference.containerPort) && + !can(var.extra_helm_values.gateway.service.port) && !can(var.extra_helm_values.inference.service.port) + ) + ai_gateway_url = "https://${var.ai_gateway_domain_name}" + # Same service name as the chart's inferenceServiceName helper. + ai_gateway_service_name = "${trimsuffix(substr(local.chart_fullname, 0, 63), "-")}-inference" + ai_gateway_values = yamldecode(var.ai_gateway_enabled ? yamlencode({ + gateway = { enabled = true } + config = { + internal = { gatewayProxyBaseUrl = "http://${local.ai_gateway_service_name}:8791", inferenceProxyBaseUrl = local.ai_gateway_url } + public = { gatewayPublicBaseUrl = local.ai_gateway_url } + } + }) : "{}") +} +resource "kubernetes_ingress_v1" "ai_gateway" { + count = var.ai_gateway_enabled && var.ingress.enabled ? 1 : 0 + metadata { + name = "${var.release_name}-ai-gateway" + namespace = var.namespace + labels = var.labels + annotations = var.ingress.annotations + } + spec { + ingress_class_name = var.ingress.class_name != "" ? var.ingress.class_name : null + tls { + secret_name = var.ingress.tls_secret_name + hosts = [var.ai_gateway_domain_name] + } + rule { + host = var.ai_gateway_domain_name + http { + path { + path = "/" + path_type = "Prefix" + backend { + service { + name = local.ai_gateway_service_name + port { number = 8791 } + } + } + } + } + } + } + lifecycle { + precondition { + condition = var.ingress.tls_secret_name != "" + error_message = "AI Gateway ingress needs ingress.tls_secret_name covering ai_gateway_domain_name; otherwise disable ingress and supply external HTTPS routing." + } + } + depends_on = [helm_release.openwork] +} +output "ai_gateway_url" { + description = "First-class AI Gateway HTTPS origin, or null when disabled. Advanced extra_helm_values URLs remain owned by the operator." + value = var.ai_gateway_enabled ? local.ai_gateway_url : null +} diff --git a/infra/terraform/modules/openwork-self-host/main.tf b/infra/terraform/modules/openwork-self-host/main.tf index 7b4fd03d0ec..3d35491286e 100644 --- a/infra/terraform/modules/openwork-self-host/main.tf +++ b/infra/terraform/modules/openwork-self-host/main.tf @@ -192,10 +192,25 @@ resource "helm_release" "openwork" { # Helm deep-merges these in order, so extra_helm_values can override any # single key without restating the whole block. - values = [ + values = concat([ yamlencode(local.chart_values), yamlencode(var.extra_helm_values), - ] + ], var.ai_gateway_enabled ? [yamlencode(local.ai_gateway_values)] : []) + + lifecycle { + precondition { + condition = !var.ai_gateway_enabled || var.ai_gateway_domain_name != "" + error_message = "ai_gateway_enabled requires ai_gateway_domain_name." + } + precondition { + condition = !var.ai_gateway_enabled || local.ai_gateway_overrides_valid + error_message = "First-class ai_gateway_enabled cannot be combined with Gateway enable/URL/name/port overrides in extra_helm_values. Remove those overrides or keep the advanced Helm interface." + } + precondition { + condition = !var.ai_gateway_enabled || !contains([var.ingress.web_host, var.ingress.api_host, trimsuffix(trimprefix(var.web_origin, "https://"), "/"), trimsuffix(trimprefix(var.api_origin, "https://"), "/")], var.ai_gateway_domain_name) + error_message = "The AI Gateway hostname must differ from Den web and API hostnames." + } + } depends_on = [ kubernetes_namespace_v1.this, diff --git a/infra/terraform/modules/openwork-self-host/tests/ai_gateway.tftest.hcl b/infra/terraform/modules/openwork-self-host/tests/ai_gateway.tftest.hcl new file mode 100644 index 00000000000..46a54271408 --- /dev/null +++ b/infra/terraform/modules/openwork-self-host/tests/ai_gateway.tftest.hcl @@ -0,0 +1,88 @@ +# Offline wrapper checks. Helm and Kubernetes are mocked; no cluster is used. +mock_provider "helm" {} +mock_provider "kubernetes" {} +mock_provider "random" { + mock_resource "random_password" { defaults = { result = "SyntheticPreservedKey0123456789abcdef" } } +} +variables { + openwork_version = "0.0.0-test" + web_origin = "https://openwork.example.com" + database_url = "mysql://app:fixture@mysql.example.com/openwork_den?sslaccept=strict" + owner_emails = ["admin@example.com"] + initial_admin_bootstrap_code = "SyntheticBootstrap012345" + ingress = { web_host = "openwork.example.com", tls_secret_name = "openwork-tls" } +} +run "off_by_default" { + command = apply # Mock state only; no chart download or cluster connection. + assert { + condition = length(kubernetes_ingress_v1.ai_gateway) == 0 && output.ai_gateway_url == null && length(keys(local.ai_gateway_values)) == 0 && length(helm_release.openwork.values) == 2 + error_message = "Default adds no ingress, Gateway values or public URL." + } +} +run "enabled_chart_and_ingress" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + } + assert { + condition = yamldecode(helm_release.openwork.values[2]).gateway.enabled && yamldecode(helm_release.openwork.values[2]).config.internal.gatewayProxyBaseUrl == "http://openwork-ee-inference:8791" && yamldecode(helm_release.openwork.values[2]).config.public.gatewayPublicBaseUrl == "https://models.example.com" && output.ai_gateway_url == "https://models.example.com" + error_message = "The wrapper delegates enabling and shared env/secrets to the existing chart; private service and desktop origins differ." + } + assert { + condition = kubernetes_ingress_v1.ai_gateway[0].spec[0].rule[0].http[0].path[0].backend[0].service[0].name == "openwork-ee-inference" && kubernetes_ingress_v1.ai_gateway[0].spec[0].tls[0].secret_name == "openwork-tls" && kubernetes_ingress_v1.ai_gateway[0].spec[0].rule[0].host == "models.example.com" && random_password.den_db_encryption_key[0].result == "SyntheticPreservedKey0123456789abcdef" + error_message = "One TLS ingress targets the chart's existing service; encryption generator/state unchanged." + } +} +run "preserves_advanced_values_interface" { + command = plan + variables { extra_helm_values = { gateway = { enabled = true } } } + assert { + condition = yamldecode(helm_release.openwork.values[1]).gateway.enabled && length(helm_release.openwork.values) == 2 + error_message = "False first-class default does not override existing advanced gateway.enabled installs." + } +} +run "operator_managed_https_routing" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ingress = { enabled = false } + } + assert { + condition = length(kubernetes_ingress_v1.ai_gateway) == 0 && yamldecode(helm_release.openwork.values[2]).gateway.enabled && output.ai_gateway_url == "https://models.example.com" + error_message = "External routing creates no ingress but still enables the chart and reports the required HTTPS origin." + } +} +run "rejects_shared_host_without_managed_ingress" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "openwork.example.com" + ingress = { enabled = false } + } + expect_failures = [helm_release.openwork] +} +run "missing_domain" { + command = plan + variables { ai_gateway_enabled = true } + expect_failures = [helm_release.openwork] +} +run "conflicting_values" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + extra_helm_values = { gateway = { enabled = false } } + } + expect_failures = [helm_release.openwork] +} +run "missing_tls" { + command = plan + variables { + ai_gateway_enabled = true + ai_gateway_domain_name = "models.example.com" + ingress = { web_host = "openwork.example.com" } + } + expect_failures = [kubernetes_ingress_v1.ai_gateway] +} diff --git a/packages/docs/self-host/deploy-to-your-cloud/aws.mdx b/packages/docs/self-host/deploy-to-your-cloud/aws.mdx index 37b3824e802..eaa35336603 100644 --- a/packages/docs/self-host/deploy-to-your-cloud/aws.mdx +++ b/packages/docs/self-host/deploy-to-your-cloud/aws.mdx @@ -42,6 +42,16 @@ If you run containers on ECS Fargate rather than Kubernetes, the Terraform modul API defaults to 1 vCPU / 2 GiB and web to 0.5 vCPU / 1 GiB. Managed RDS deletion protection defaults on. The 0.18.57/0.18.58 API/web defaults are digest-pinned; other versions and enabled optional services require digest-pinned image overrides. Database TLS verifies the certificate chain using the public regional RDS CA bundle, loaded before Node starts. The application's current MySQL parser does not enable mysql2's separate hostname identity check; full hostname verification remains an application follow-up, not a claim of this module. +The optional **AI Gateway** is a separate inference service (`openwork-inference`, +8791), not OpenWork Web's **den-gateway** proxy (8788) or Den API's MCP gateway. +Set `ai_gateway_enabled=true`, a dedicated `ai_gateway_domain_name`, and a matching +release's digest-pinned `ai_gateway_image`; `ai_gateway_url` returns its HTTPS +origin (null when off). The HTTPS ALB routes desktops while Den uses private Cloud +Map. Defaults remain off, including in the native managed installer. See the +[Terraform AI Gateway guide](https://github.com/different-ai/openwork/blob/dev/infra/terraform/AI_GATEWAY.md) +for DNS/ACM, shared-listener priorities, unchanged secrets and managed-install limits. +**Live AI Gateway conformance is NotRun**; mock plans do not qualify an installation. + ### Tested scenarios | Scenario | Tools / release | Result | Date | diff --git a/packages/docs/self-host/deploy-to-your-cloud/azure.mdx b/packages/docs/self-host/deploy-to-your-cloud/azure.mdx index ffe0b9a926a..02654080e7c 100644 --- a/packages/docs/self-host/deploy-to-your-cloud/azure.mdx +++ b/packages/docs/self-host/deploy-to-your-cloud/azure.mdx @@ -32,6 +32,18 @@ For exact Azure CLI commands, values, TLS options, migration troubleshooting, ve The module deploys Den API and web as HTTPS-only Container Apps, a singleton manual migration Job, MySQL Flexible Server 8.4 (or your existing database), managed Container Apps secrets with optional Key Vault references, and a new or existing environment and Log Analytics workspace. Application replicas do not run migrations. The native Job start is followed by a read-only observer of the exact execution; only `Succeeded` permits application rollout. Default image references and all overrides must be digest-pinned. The default API is 1 vCPU / 2 GiB and web is 0.5 vCPU / 1 GiB. +The optional **AI Gateway** (`openwork-inference`, 8791) is distinct from OpenWork +Web's **den-gateway** and Den API's MCP gateway. Set `ai_gateway_enabled=true`, a +dedicated `ai_gateway_domain_name`, a matching release's digest-pinned +`ai_gateway_image`, and a covering `custom_domain_certificate`. Preconfigure CNAME +and `asuid` TXT records using the environment's hostname/verification ID before +binding. `ai_gateway_url` returns the HTTPS origin (null when off); Den calls the +generated HTTPS FQDN within the same environment. Existing database/encryption +values are reused without advancing rotation counters. See the +[Terraform AI Gateway guide](https://github.com/different-ai/openwork/blob/dev/infra/terraform/AI_GATEWAY.md) +for prerequisites and limits. **Live AI Gateway conformance is NotRun**; mock plans +do not qualify an installation. + ### Tested scenarios | Date | Tools / release | Result | diff --git a/packages/docs/self-host/deploy-to-your-cloud/google-cloud.mdx b/packages/docs/self-host/deploy-to-your-cloud/google-cloud.mdx index 233b59330b4..a4031bba776 100644 --- a/packages/docs/self-host/deploy-to-your-cloud/google-cloud.mdx +++ b/packages/docs/self-host/deploy-to-your-cloud/google-cloud.mdx @@ -16,7 +16,7 @@ OpenWork supports two Google Cloud deployment shapes. Use GKE Autopilot and Helm ## Choose a deployment target -Choose GKE when you need Helm hooks, pod-level customization, custom CA mounts, installer artifact volumes, or the optional OpenWork Gateway. Choose Cloud Run when Den Web and Den API fit the request-driven service model and you prefer managed autoscaling without a Kubernetes cluster. +Choose GKE when you need Helm hooks, pod-level customization, custom CA mounts or installer artifact volumes. Choose Cloud Run when Den web, Den API and optional AI Gateway fit the request-driven service model and you prefer managed autoscaling without a Kubernetes cluster. ## What Google Cloud manages @@ -60,6 +60,18 @@ The common outputs are `web_url`, `api_url`, `database_identifier` (managed Clou Existing network/subnet, database, service account, certificates and reserved IP inputs remain supported. The applying account needs resource-creation and IAM-setting permissions, including Project IAM Admin, Secret Manager Admin and Cloud Run Admin; Editor alone is insufficient. Alternatively grant runtime IAM separately and use `grant_iam_roles = false`. Use application-default credentials or workload identity. +The optional **AI Gateway** (`openwork-inference`, 8791) is distinct from OpenWork +Web's **den-gateway** and Den API's MCP gateway. Set `ai_gateway_enabled=true`, a +dedicated `ai_gateway_domain_name`, and a digest-pinned `ai_gateway_image` mirrored +into Artifact Registry. `ai_gateway_url` is its HTTPS origin, null when off. +It creates a serverless backend/host rule and separate certificate on an HTTPS +load balancer, even when Den keeps run.app URLs, without replacing Den's certificate. Den calls that HTTPS origin, **not private DNS**; +the Gateway's direct run.app URI is disabled. Existing Cloud SQL socket secrets are +preserved by a serving-process-only hostname adapter. Native managed-install roots +remain off. See the [Terraform AI Gateway guide](https://github.com/different-ai/openwork/blob/dev/infra/terraform/AI_GATEWAY.md) +for DNS, networking, shared secrets and limits. **Live AI Gateway conformance is +NotRun**; mock plans do not qualify an installation. + ### Tested scenarios | Date | Tools | Release | Result | @@ -83,7 +95,7 @@ The live run used new-deployment write-only mode, db-g1-small and default HTTPS The [module](https://github.com/different-ai/openwork/tree/dev/infra/terraform/modules/openwork-gcp-cloud-run) is available on `dev` in **beta**. -Follow the shared [module contract](https://github.com/different-ai/openwork/blob/dev/infra/terraform/MODULE_CONTRACT.md) and pin to a reviewed commit. Requires Terraform or OpenTofu `>= 1.11, < 2.0`. The module's bounded providers are Google `~> 8.7` and Random `~> 3.9`. Workbot, the runner, Slack and OpenWork Web/gateway are outside this change. A dedicated job orders one deployment; it does not implement a database mutex, dirty marker or coordination across independent deployments. +Follow the shared [module contract](https://github.com/different-ai/openwork/blob/dev/infra/terraform/MODULE_CONTRACT.md) and pin to a reviewed commit. Requires Terraform or OpenTofu `>= 1.11, < 2.0`. The module's bounded providers are Google `~> 8.7` and Random `~> 3.9`. Workbot, the runner, Slack and OpenWork Web/den-gateway are outside this change. A dedicated job orders one deployment; it does not implement a database mutex, dirty marker or coordination across independent deployments. Defaults: API **1 vCPU / 2 GiB**, web **1 vCPU / 1 GiB**, one warm instance per service, maximum three; Cloud SQL **db-g1-small (1.7 GB)** with 20 GB disk and backups. The old micro tier exhausted its reported memory quota at idle. Database, services and job have deletion protection by default; explicitly turn it off only for disposable stacks. The deprecated nested `database.deletion_protection` override retains precedence.