Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ inventory/hosts.mapping.yml

# Sensitive templates (contain vault variable references)
**/Caddyfile.j2
**/Caddyfile.example.j2

# Local Development Tooling & Binaries
tools/
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ override.tf.json

# Caddyfile
Caddyfile
Caddyfile.j2
# =================================================
# SSH Keys & Certificates
# =================================================
Expand Down Expand Up @@ -121,6 +120,8 @@ inventory/hosts.ini.generated
inventory/hosts.ini
inventory/host_vars/*.yml
!inventory/host_vars/*.example
inventory/group_vars/all/ingress.yml
!inventory/group_vars/all/ingress.yml.example

# SOPS / age (secrets management)
# Operator age private key - backed up offline; NEVER commit
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,20 @@ status: active

# Changelog

## [Unreleased]

### Added
- **Deterministic Make Toolchain Contract**: Standardized developer and automation entrypoints with `make doctor`, `make tools`, `make test-security`, and `make release-ready`.
- **Operational Scale & Sizing Boundaries**: Documented capacity ceilings for Bare Metal / VPS topologies (Brain: 1–3 High-Spec, 3–5 Low-Spec; Muscle: 50–100 workers) prior to hyperscale K8s orchestration.

### Changed
- **Ingress & Secrets Decoupling**: Decoupled `ingress_services` catalog from SOPS encryption into tracked `ingress.yml.example` and gitignored `ingress.yml` to prevent merge conflict friction and protect client domain confidentiality without encrypting non-secret routing topology.
- **Unified Caddyfile SSOT**: Removed `.example.j2` copy requirement in favor of a single tracked `Caddyfile.j2` incorporating global `metrics` directive aligned with upstream.
- **Upstream Caddy WAF Upgrade**: Pinned `ghcr.io/developmi/caddy-waf:v3.6.2` (Coraza v3.8.1, CRS v4) with cryptographic multi-arch sha256 digest validation.

### Fixed
- **Preflight Toolchain Compliance**: Replaced legacy `pip` reference in roadmap with `uv tool run` and documented upstream container entrypoint requirement in Twenty CRM compose worker.

## [7.0.0] - 2026-10-27

### Added
Expand Down
27 changes: 27 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,32 @@ setup: ## Bootstrap local environment (sync + collections + pre-commit)
$(PKG) run pre-commit install
@echo "=== Toolchain Ready ==="

tools: setup ## Standard contract: Bootstrap and verify local toolchain

doctor: ## Diagnostic environment and playbook readiness check
@echo "=== System & Toolchain Doctor ==="
@echo "OS: $$(uname -s) $$(uname -m)"
@which uv >/dev/null && echo "uv: $$(uv --version)" || echo "uv: MISSING"
@which python3 >/dev/null && echo "python3: $$(python3 --version)" || echo "python3: MISSING"
@$(PKG) run ansible --version | head -n 1
@which sops >/dev/null && echo "sops: $$(sops --version)" || echo "sops: NOT FOUND in PATH (optional host tool)"
@which trivy >/dev/null && echo "trivy: $$(trivy --version | head -n 1)" || echo "trivy: NOT FOUND in PATH"
@which cosign >/dev/null && echo "cosign: $$(cosign version 2>&1 | head -n 1)" || echo "cosign: NOT FOUND in PATH"
@echo "=== Playbook & Inventory Validation ==="
@./scripts/setup.sh --validate

test-security: ## Audit Python dependencies (pip-audit) and workflow security (zizmor)
@echo "==> 1/2 Auditing Python dependencies (pip-audit)..."
@$(PKG) run --group dev pip-audit
@echo "==> 2/2 Auditing GitHub Actions workflows (zizmor)..."
@$(PKG) run zizmor .
@echo "=== Security Gates: ALL PASSED ==="

release-ready: lint test-security ## Comprehensive pre-release gate (lint + security + doctor)
@echo "==> Running environment & playbook validation..."
@$(MAKE) doctor
@echo "=== Release Readiness Gate: ALL CHECKS PASSED ==="

sync: ## Sync Python dependencies with uv
@echo "Running $(PKG) sync..."
$(PKG) sync
Expand Down Expand Up @@ -283,6 +309,7 @@ test-layer:
@$(MAKE) test LAYER=$(LAYER)

.PHONY: help setup sync lint precommit test show-inventory \
tools doctor test-security release-ready \
sops-init sops-edit sops-encrypt sops-view \
deploy-bootstrap deploy-hardening deploy-engine deploy-edge \
deploy-monitoring deploy-backups deploy-portainer deploy-platform \
Expand Down
1 change: 1 addition & 0 deletions apps/twenty-crm/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ services:
read_only: true
tmpfs:
- /tmp
# Upstream contract: Twenty CRM official image entrypoint requirement
command: ["yarn", "worker:prod"]
depends_on:
twenty-db:
Expand Down
26 changes: 26 additions & 0 deletions docs/architecture/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,32 @@ Determined by `group_vars/{brain,muscle,local}/main.yml`. A host's inventory gro
- `muscle/main.yml` → `hardening_profile: server`
- `local/main.yml` → `hardening_profile: workstation`

### Operational Scale & Sizing Boundaries

This repository is engineered under the **"Boring is Better"** philosophy: providing enterprise-grade security hardening (NIST SP 800-53, CIS OS/Docker benchmarks, kernel confinement, Zero Trust networking via Tailscale, and automated Restic disaster recovery) directly on Bare Metal hosts or VPS instances without requiring Kubernetes.

#### Hardware Profiles & Node Allocation

| Host Class | Hardware Profile | Recommended Count | Architectural Role & Workload Distribution |
| :--- | :--- | :--- | :--- |
| **`brain`** | **High-Spec** (8–16 vCPU, 32–64 GB RAM) | **1 – 3 nodes** | Centralized control plane: VictoriaMetrics, Loki, Grafana, Portainer. 3 nodes provide active quorum for HA. |
| **`brain`** | **Low-Spec** (2–4 vCPU, 4–8 GB RAM) | **3 – 5 nodes** | Role-partitioned control plane: dedicated nodes for Ingress/VPN, metrics ingestion, log indexing (Loki), and management across low-cost VPS instances. |
| **`muscle`** | **Any compute tier** | **1 – 50 nodes** | Standard push-based Ansible sweet spot. Full parallel execution with low drift and minimal latency. |
| **`muscle`** | **Any compute tier** | **50 – 100 nodes** | Upper operational ceiling for push-based Ansible. Deployments should target specific nodes or tags (`--limit muscle-X`). |

#### Scaling Beyond 100 Nodes (Ecosystem Positioning)

> [!NOTE]
> **Surpassing 100 worker nodes does NOT invalidate this repository.**
> This project serves as **Phase 1: Hardened Foundations**. For hyper-scale topologies (>100 muscle nodes), push-based SSH routing can be transitioned to dynamic service discovery (Consul, CoreDNS) or container orchestrators (Nomad, Kubernetes). This repository remains the foundational hardening and compliance baseline that configures OS security, sysctl, firewalls, and storage across bare metal before any higher-level orchestrator runs.

#### Ingress Routing Catalog & Secrets Architecture

Routing topology and secrets adhere to strict separation of concerns:
- **Routing Catalog** (`inventory/group_vars/all/ingress.yml`): Plain YAML catalog of domains, upstreams, headers, and security profiles. Ignored by `.gitignore` to prevent leaking private client infrastructure in open-source environments (template provided in `ingress.yml.example`).
- **Cryptographic Secrets** (`inventory/group_vars/all/secrets.sops.yml`): Strictly reserved for sensitive credentials (TLS private keys in `cloudflare_origin_certs`, `caddy_ui_token`, R2 access keys).
- **Ingress Template** (`roles/L4_networking/caddy/templates/Caddyfile.j2`): 100% data-driven, version-controlled Jinja2 template with zero hardcoded values, featuring automatic global `metrics` export and per-service WAF/TLS client routing.

---

## 8. Application Profile Schema (Reduced 6-Field Specification)
Expand Down
2 changes: 1 addition & 1 deletion docs/project/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ This project starts at v6.0.0. Prior development history (v1.0.0 - v5.5.0) is ma

### U2.4 Graphify Codebase Mapping (Evaluation)

- Evaluate `pip install graphifyy` for generating cross-reference documentation.
- Evaluate `uv tool run graphifyy` for generating cross-reference documentation.
- **Known limitation**: Like CodeGraph, Graphify treats YAML as DOCS type only (no symbolic analysis). Useful for general project topology, not Ansible-specific navigation.
- If evaluation passes: integrate into CI as optional documentation artifact.
- If evaluation fails: document the limitation and rely on `make graph` + structured docs.
Expand Down
6 changes: 3 additions & 3 deletions inventory/group_vars/all/images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
# -----------------------------------------------------------------------------
# Custom image namespace
image_namespace: "{{ vault_image_namespace }}" # AC-3: value from secrets.sops.yml only
# Caddy image (SSOT: caddy-waf repo releases; v3.6.1)
caddy_image: "ghcr.io/developmi/caddy-waf:v3.6.1"
# Caddy image (SSOT: caddy-waf repo releases; v3.6.2)
caddy_image: "ghcr.io/developmi/caddy-waf:v3.6.2"

# SHA256 digest (OCI index, multi-arch amd64+arm64) for supply chain verification.
# Enforced as a verification gate by the L4 role (docker_image_info digest match,
# tasks/caddy.yml 'Verify Caddy image digest if specified').
caddy_image_digest: "sha256:d6a824c83e7edbc97391907cb404c44782c75305ac9d07c1278797cffc17ed93"
caddy_image_digest: "sha256:572968506168150bed96e293c9461578d911a980ecb8ef91d6672f381dda2cbb"

# Caddy WAF UI sidecar image (SSOT: caddy-waf-ui repo releases; v1.4.0)
caddy_ui_image: "ghcr.io/developmi/caddy-waf-ui:v1.4.0"
Expand Down
54 changes: 54 additions & 0 deletions inventory/group_vars/all/ingress.yml.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
# =============================================================================
# Ingress Services & Security Profiles (Routing Catalog)
# =============================================================================
# Single Source of Truth for reverse proxy routing (Caddy / Nginx / Traefik).
# Copy this file to inventory/group_vars/all/ingress.yml (ignored by .gitignore).
#
# PRIVACY & SECURITY:
# Domain names, upstreams, and routing topology are uncommitted configuration
# to protect client privacy in this open-source repository.
# Sensitive cryptographic secrets (TLS private keys, UI tokens) stay in
# inventory/group_vars/all/secrets.sops.yml (SOPS + age encrypted).
#
# VARIABLE OWNERSHIP (four layers - NEVER mix):
# Layer 1 | SOPS (secrets.sops.yml) → certs (privkeys), API keys, UI tokens
# Layer 2 | Ingress (ingress.yml) → domains, upstreams, routes, security profiles
# Layer 3 | Inventory (hosts.ini) → caddy_node_id, waf_mode_default (per host)
# Layer 4 | Group vars (brain/muscle) → caddy_type, caddy_admin_enabled (group defaults)
# =============================================================================

# Security profiles referenced by ingress_services[*].security_profile.
# Each profile toggles the WAF (Coraza) and VPN-only restriction.
# Default baseline profiles can also be overridden here.
ingress_security_profiles:
public_waf:
import_waf: true
import_vpn_only: false

basicauth:
import_waf: false
import_vpn_only: false

vpn_only:
import_waf: true
import_vpn_only: true

# Routing catalog: one entry per domain Caddy must serve.
# Rendered by roles/L4_networking/caddy/templates/Caddyfile.j2.
# Filtered per node: target_group matching node group or caddy_node_id.
# Examples: "brain" | "muscle" | "muscle-1" | ["muscle-1", "brain-1"].
ingress_services:
- name: app-example # label only - used in logs/warnings
domain: app.example.invalid # required - public hostname (RFC 2606)
upstream: app:8080 # required - container:port
client: client_primary # required - cert source; MUST exist in
# cloudflare_origin_certs in secrets.sops.yml
target_group: brain # required - routing filter (node group or ID)
security_profile: public_waf # required - MUST exist in ingress_security_profiles
tls_client_auth: cloudflare # optional - enables Cloudflare Origin Pull mTLS
proxy_headers:
standard: true # forward Host + X-Real-IP to the backend
websocket: false # true = bypass WAF for WebSocket upgrades
headers:
X-Custom-Header: "PLACEHOLDER_HEADER_VALUE"
17 changes: 16 additions & 1 deletion inventory/group_vars/all/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,24 @@ caddy_admin_network_name: "caddy-admin-net"

# Ingress routing catalog (single-tenant). Each entry maps a domain
# to an upstream backend with WAF and cert isolation.
# Defined in the SOPS file: inventory/group_vars/all/secrets.sops.yml.
# Defined in inventory/group_vars/all/ingress.yml (see ingress.yml.example).
ingress_services: []

# Standard security profiles available to ingress services.
# Override or extend in inventory/group_vars/all/ingress.yml.
ingress_security_profiles:
public_waf:
import_waf: true
import_vpn_only: false

basicauth:
import_waf: false
import_vpn_only: false

vpn_only:
import_waf: true
import_vpn_only: true

# Default WAF mode for services without an explicit waf_mode.
# enforcement: block violations (return 403)
# detection_only: log violations but forward request (safe default for rollout)
Expand Down
52 changes: 10 additions & 42 deletions inventory/group_vars/all/secrets.sops.yml.example
Original file line number Diff line number Diff line change
Expand Up @@ -123,50 +123,18 @@ cloudflare_origin_certs:


# =============================================================================
# INGRESS SERVICES (routing catalog)
# INGRESS SERVICES & SECURITY PROFILES (Decoupled to ingress.yml)
# =============================================================================

# Security profiles referenced by ingress_services[*].security_profile.
# Each profile toggles the WAF (Coraza) and the VPN-only restriction.
# Used by roles/L4_networking/caddy/tasks/main.yml:27 and
# roles/L4_networking/caddy/templates/Caddyfile.example.j2:142.
# Required by L4 deployment (preflight asserts it is defined).
ingress_security_profiles:

public_waf:
import_waf: true
import_vpn_only: false

basicauth:
import_waf: false
import_vpn_only: false

vpn_only:
import_waf: true
import_vpn_only: true

# Routing catalog: one entry per domain Caddy must serve.
# Rendered by roles/L4_networking/caddy/templates/Caddyfile.example.j2:135,
# preflight-checked in roles/L4_networking/caddy/tasks/main.yml:22.
# Overrides the default [] in inventory/group_vars/all/main.yml:185.
# Required by L4 deployment - an empty catalog aborts deployment.
# Add one entry per domain; target_group examples: "brain" | "muscle" |
# "muscle-1" | ["muscle-1", "brain-1"].
ingress_services:

- name: app-example # label only - used in logs/warnings
domain: app.example.invalid # required - public hostname
upstream: app:8080 # required - container:port
client: client_primary # required - cert source; MUST exist in
# cloudflare_origin_certs (path /etc/caddy/certs/<client>/)
target_group: brain # required - routing filter (see above)
security_profile: public_waf # required - MUST exist in ingress_security_profiles
tls_client_auth: cloudflare # optional - enables Cloudflare Origin Pull mTLS
proxy_headers:
standard: true # forward Host + X-Real-IP to the backend
websocket: true # true = bypass WAF for WebSocket upgrades
headers:
X-Custom-Header: "PLACEHOLDER_HEADER_VALUE"
# Ingress routing configuration (domains, upstreams, headers, security profiles)
# has been decoupled from secrets.sops.yml to protect client privacy in Git
# while keeping routing configuration as plain YAML.
#
# Define your routes in:
# inventory/group_vars/all/ingress.yml (copy from ingress.yml.example)
#
# Only cryptographic secrets (TLS cert private keys in cloudflare_origin_certs,
# caddy_ui_token, etc.) stay in this SOPS-encrypted file.


# =============================================================================
Expand Down
9 changes: 4 additions & 5 deletions molecule/L4_networking/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,12 @@

post_tasks:
# ---------------------------------------------------------------------------
# Contract render: Caddyfile.example.j2 is the TRACKED SSOT (convention: live
# Caddyfile/Caddyfile.j2 are never tracked). Verify assertions run against
# this render so the contract test pins the tracked file, never the local one.
# Contract render: Caddyfile.j2 is the TRACKED SSOT. Verify assertions run against
# this render so the contract test pins the tracked file.
# ---------------------------------------------------------------------------
- name: "Post | Render Caddyfile.example.j2 (tracked SSOT) for contract assertions"
- name: "Post | Render Caddyfile.j2 (tracked SSOT) for contract assertions"
ansible.builtin.template:
src: ../../roles/L4_networking/caddy/templates/Caddyfile.example.j2
src: ../../roles/L4_networking/caddy/templates/Caddyfile.j2
dest: "{{ app_base_dir }}/ingress/Caddyfile.contract"
mode: "0644"

Expand Down
12 changes: 6 additions & 6 deletions molecule/L4_networking/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@
# ---------------------------------------------------------------------------
# CONTRACT: ui_managed overlay import contract (D2)
# Mirrors caddy-waf-ui tests/integration/caddyfile_test.go assertions against
# the tracked SSOT render (Caddyfile.example.j2 -> Caddyfile.contract).
# the tracked SSOT render (Caddyfile.j2 -> Caddyfile.contract).
# ---------------------------------------------------------------------------
- name: "GIVEN: Slurp the contract-rendered Caddyfile"
ansible.builtin.slurp:
Expand Down Expand Up @@ -383,18 +383,18 @@
fail_msg: "SDD Failed: UI service depends_on must reference the compose SERVICE key 'caddy' (container_name is caddy-waf) with service_healthy"
success_msg: "UI service depends_on caddy (service name) with service_healthy condition"

- name: "THEN: Audit log file exists with mode 0644"
- name: "THEN: Audit log file exists with mode 0640"
ansible.builtin.stat:
path: /opt/user/apps/testing/ingress/caddy-data/logs/coraza-audit.log
register: audit_log_stat

- name: "THEN: Audit log mode is 0644 (UI can read)"
- name: "THEN: Audit log mode is 0640 (hardened)"
ansible.builtin.assert:
that:
- audit_log_stat.stat.exists
- audit_log_stat.stat.mode == "0644"
fail_msg: "SDD Failed: audit log missing or mode is not 0644 (got {{ audit_log_stat.stat.mode | default('missing') }})"
success_msg: "Audit log exists with mode 0644"
- audit_log_stat.stat.mode == "0640"
fail_msg: "SDD Failed: audit log missing or mode is not 0640 (got {{ audit_log_stat.stat.mode | default('missing') }})"
success_msg: "Audit log exists with mode 0640"

# ---------------------------------------------------------------------------
# E2E ASSERTS (gated on ghcr images being present in local daemon)
Expand Down
Loading
Loading