Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/runbooks/argocd-webhook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Argo CD GitHub webhook

GitHub POSTs push events to `https://argocd-hook.arunanshu.dev/api/webhook`, and Argo CD refreshes the matching Applications in seconds instead of waiting for the poll (`timeout.reconciliation` 120s + up to 60s jitter, left at the chart default). Polling stays on: a dropped or rejected delivery only costs the old latency.

The UI host `argocd.arunanshu.dev` is unchanged and stays behind the wildcard Access OTP.

## Layers

| Layer | Where | What it admits |
|-------|-------|----------------|
| Cloudflare WAF custom rule | `infra/cloudflare_waf.tf` | Only GitHub's `hooks` CIDRs (read from `api.github.com/meta` on every plan), only `POST /api/webhook` |
| Cloudflare Access bypass | `infra/cloudflare_access.tf` | Only the `/api/webhook` path; any other path on the host gets the OTP page |
| Traefik HTTPRoute + RateLimit | `kubernetes/components/argocd-webhook` | Only `POST /api/webhook` (anything else is a 404); 30/min, burst 30, per Traefik pod |
| Argo CD | `configs.secret.githubSecret` in both argocd values files | Only deliveries whose `X-Hub-Signature-256` HMAC matches; body capped at 5MB |

The HMAC secret lives in tofu state (`random_password.argocd_webhook`), is sealed into `argocd-webhook-github` (labelled `app.kubernetes.io/part-of: argocd`, which Argo CD requires for a `$secret:key` reference), and is sent to GitHub through the API. It is never printed.

## First rollout

1. **Check for dashboard WAF rules.** `cloudflare_ruleset.zone_custom_firewall` takes over the zone's custom-rules entry point. If Security → WAF → Custom rules lists anything, import it into `infra/cloudflare_waf.tf` first (`cf-terraforming`), or the apply fails or replaces it.
2. `just infra-apply` — creates the secret, the WAF rule and the Access bypass.
3. `just seal-argocd-webhook` — writes `kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml`. Commit it; the `render` CI job (Argo CD renders the component) fails until this file exists.
4. Merge. The `argocd-webhook` Application auto-syncs. The `argocd` Application does **not** (self-management is manual): sync it so `argocd-secret` gets `webhook.github.secret` and `argocd-cm` gets `webhook.maxPayloadSizeMB` (argocd-server rolls on the cm checksum). Check the live objects, not just "Synced".
5. `just github-argocd-webhook` — creates the hook (push events only) and prints its id. Running it twice creates a second hook.
6. Push a trivial commit and watch an Application refresh within seconds. GitHub → Settings → Webhooks → Recent Deliveries shows every attempt with the response.

## When a delivery fails

Read the response in Recent Deliveries first, then Cloudflare Security → Events.

| Response | Cause | Fix |
|----------|-------|-----|
| 403, Cloudflare block page, event source "Custom rules" | WAF lockdown | GitHub changed ranges: `just infra-plan` shows the diff, `just infra-apply` |
| 403 / challenge, event source "Bot fight mode" | Bot Fight Mode (`cloudflare_hardening.tf`) runs outside the Ruleset Engine; no custom rule can skip it | Add IP Access *allow* rules for the `hooks` CIDRs (Bot Fight Mode does not trigger when an IP Access rule matches first), or turn Bot Fight Mode off. Both are zone-wide trade-offs; decide deliberately |
| 302 to `cloudflareaccess.com` | Access bypass missing or not matching | Check the `argocd_webhook_bypass` app path |
| 404 from Traefik | Route mismatch | URL path or method wrong; check the hook URL |
| 429 | Traefik rate limit | Look for a redelivery storm or push loop before raising the limit |
| 4xx from Argo CD about the signature | Secret mismatch | Re-run step 3 and the rotation PATCH below |

Docs: <https://developers.cloudflare.com/waf/feature-interoperability/>, <https://developers.cloudflare.com/bots/get-started/bot-fight-mode/>

## Rotate the secret

1. From `infra/`: `sops exec-env secrets.yaml "tofu apply -replace=random_password.argocd_webhook"`.
2. `just seal-argocd-webhook`, commit, merge (the `$ref` re-resolves; no argocd sync needed).
3. Update GitHub's copy (hook id from step 5 above, or `gh api repos/arunanshub/devops/hooks --jq '.[] | [.id, .config.url]'`):

```bash
just _tofu-output argocd_webhook_secret \
| jq -Rn --arg url https://argocd-hook.arunanshu.dev/api/webhook -f infra/github-webhook.jq \
| jq .config | gh api --method PATCH repos/arunanshub/devops/hooks/<id>/config --input -
```

The body carries the whole `config` (url, content type, TLS check, secret), so nothing depends on how the endpoint treats omitted fields: <https://docs.github.com/en/rest/repos/webhooks#update-a-webhook-configuration-for-a-repository>.

Between steps 2 and 3, deliveries fail the HMAC check and polling covers the gap.
37 changes: 37 additions & 0 deletions infra/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

61 changes: 61 additions & 0 deletions infra/argocd_webhook.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# --------------------------- Argo CD GitHub webhook ---------------------------
#
# GitHub POSTs push events to argocd-hook.arunanshu.dev/api/webhook, so Argo CD
# refreshes on merge instead of on its 3-minute poll. Polling stays on as the
# fallback for dropped deliveries.
#
# The UI host (argocd.arunanshu.dev) stays behind the wildcard Access OTP. The
# hook host is public, and four independent layers keep it narrow:
# 1. Cloudflare WAF (cloudflare_waf.tf): only GitHub's hook IPs, only
# `POST /api/webhook`; everything else is blocked at the edge.
# 2. Cloudflare Access (cloudflare_access.tf): the bypass covers only the
# /api/webhook path; any other path on the host still gets the OTP page.
# 3. Traefik (components/argocd-webhook): the HTTPRoute matches only
# `POST /api/webhook` and is rate-limited.
# 4. Argo CD: drops any delivery whose X-Hub-Signature-256 HMAC does not match
# webhook.github.secret.
#
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/
# https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks

locals {
argocd_webhook_host = "argocd-hook.arunanshu.dev"
argocd_webhook_path = "/api/webhook"

# GitHub's webhook source ranges (IPv4 and IPv6). GitHub changes these
# occasionally and says to query the API instead of pinning, so every plan
# re-reads them; a changed range shows up as a WAF rule diff.
# https://docs.github.com/en/rest/meta/meta#get-github-meta-information
github_hook_cidrs = jsondecode(data.http.github_meta.response_body).hooks
}

data "http" "github_meta" {
url = "https://api.github.com/meta"

request_headers = {
Accept = "application/vnd.github+json"
X-GitHub-Api-Version = "2022-11-28"
}

retry {
attempts = 2
}

lifecycle {
# An empty list would render `ip.src in {}` and block every delivery; fail
# the plan instead of shipping that.
postcondition {
condition = self.status_code == 200 && length(jsondecode(self.response_body).hooks) > 0
error_message = "GET https://api.github.com/meta did not return a non-empty `hooks` list."
}
}
}

# HMAC key shared by GitHub (signs) and Argo CD (verifies). Never printed: it
# goes from state straight into kubeseal (`just seal-argocd-webhook`) and the
# GitHub API (`just github-argocd-webhook`). Hex-safe so neither side needs
# quoting. Taint and re-run both recipes to rotate.
resource "random_password" "argocd_webhook" {
length = 64
special = false
}
24 changes: 24 additions & 0 deletions infra/cloudflare_access.tf
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,30 @@ resource "cloudflare_zero_trust_access_application" "libby_hbd_2026_bypass" {
]
}

# Public bypass for the Argo CD GitHub webhook — GitHub cannot answer an OTP
# and webhooks cannot carry a service-token header. Scoped to the one path: the
# more specific application wins, so every other path on the host (and the UI
# host argocd.arunanshu.dev) still falls to the wildcard OTP policy below. The
# WAF lockdown (cloudflare_waf.tf) and the HMAC check in Argo CD are what
# actually authenticate a delivery. See argocd_webhook.tf.
# https://developers.cloudflare.com/cloudflare-one/access-controls/policies/app-paths/
resource "cloudflare_zero_trust_access_application" "argocd_webhook_bypass" {
account_id = var.cloudflare_account_id
name = "${local.argocd_webhook_host}${local.argocd_webhook_path} — GitHub webhook bypass"
domain = "${local.argocd_webhook_host}${local.argocd_webhook_path}"
type = "self_hosted"
session_duration = "24h"

policies = [
{
name = "bypass"
decision = "bypass"
precedence = 1
include = [{ everyone = {} }]
},
]
}

# Cloudflare Access — single wildcard app gates all *.arunanshu.dev traffic.
# New services only need an HTTPRoute; no Terraform change required unless
# per-app access control is needed (different users, different session length).
Expand Down
39 changes: 39 additions & 0 deletions infra/cloudflare_waf.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# ------------------------------ WAF custom rules ------------------------------
#
# This resource is the zone's entry-point ruleset for the
# http_request_firewall_custom phase, and OpenTofu owns it whole: "Terraform
# assumes that it has complete control over account and zone rulesets."
# Dashboard-made custom rules must be imported here first (cf-terraforming), or
# the first apply fails or replaces them. Free plan: 5 custom rules per zone.
# https://developers.cloudflare.com/terraform/additional-configurations/waf-custom-rules/
# https://developers.cloudflare.com/waf/custom-rules/
#
# Known gap: Bot Fight Mode (cloudflare_hardening.tf) runs outside the Ruleset
# Engine, so no rule here can skip it. If it challenges GitHub's deliveries
# (Security > Events, or "Recent Deliveries" on the GitHub hook), see
# docs/runbooks/argocd-webhook.md.
# https://developers.cloudflare.com/waf/feature-interoperability/

resource "cloudflare_ruleset" "zone_custom_firewall" {
zone_id = var.cloudflare_zone_id
name = "Zone custom rules"
description = "Entry point for http_request_firewall_custom; managed by OpenTofu."
kind = "zone"
phase = "http_request_firewall_custom"

rules = [
{
ref = "argocd_webhook_lockdown"
description = "argocd-hook: allow only GitHub hook IPs sending POST ${local.argocd_webhook_path}"
action = "block"
enabled = true
expression = join(" ", [
"(http.host eq \"${local.argocd_webhook_host}\" and (",
"not ip.src in {${join(" ", local.github_hook_cidrs)}}",
"or http.request.method ne \"POST\"",
"or http.request.uri.path ne \"${local.argocd_webhook_path}\"",
"))",
])
},
]
}
16 changes: 16 additions & 0 deletions infra/github-webhook.jq
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Request body for POST /repos/{owner}/{repo}/hooks, used by
# `just github-argocd-webhook`. The secret arrives on stdin (jq -R `input`) so
# it is never on a command line or in shell history; the URL arrives as $url.
# Only push events: Argo CD refreshes on push and ignores everything else.
# https://docs.github.com/en/rest/repos/webhooks#create-a-repository-webhook
{
name: "web",
active: true,
events: ["push"],
config: {
url: $url,
content_type: "json",
insecure_ssl: "0",
secret: input
}
}
6 changes: 6 additions & 0 deletions infra/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ terraform {
source = "hashicorp/random"
version = "~> 3.6"
}

# Reads GitHub's webhook IP ranges for the WAF lockdown (argocd_webhook.tf).
http = {
source = "hashicorp/http"
version = "~> 3.6"
}
}

encryption {
Expand Down
11 changes: 11 additions & 0 deletions infra/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,14 @@ output "libby_hbd_2026_cache_purge_token_id" {
description = "Cloudflare account token id for the libby-hbd-2026 cache-purge token (safe to log)."
value = cloudflare_account_token.libby_hbd_2026_cache_purge.id
}

output "argocd_webhook_secret" {
description = "HMAC secret for the Argo CD GitHub webhook. Seal via `just seal-argocd-webhook`; register on GitHub via `just github-argocd-webhook`."
value = random_password.argocd_webhook.result
sensitive = true
}

output "argocd_webhook_url" {
description = "Payload URL GitHub delivers push events to (safe to log)."
value = "https://${local.argocd_webhook_host}${local.argocd_webhook_path}"
}
16 changes: 16 additions & 0 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,22 @@ seal-tunnel-token:
--from-file=token=/dev/stdin --dry-run=client -o yaml \
| just _seal kubernetes/components/cloudflared/resources/tunnel-token.sealedsecret.yaml

# Seal the Argo CD GitHub webhook HMAC secret from tofu state (label: Argo CD's $ref lookup)
[group('secrets')]
seal-argocd-webhook:
just _tofu-output argocd_webhook_secret \
| kubectl create secret generic argocd-webhook-github -n argocd \
--from-file=secret=/dev/stdin --dry-run=client -o yaml \
| kubectl label --local -f - app.kubernetes.io/part-of=argocd -o yaml \
| just _seal kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml

# Create the GitHub push webhook once (gh needs Webhooks: write); prints the hook id
[group('secrets')]
github-argocd-webhook:
just _tofu-output argocd_webhook_secret \
| jq -Rn --arg url "$(just _tofu-output argocd_webhook_url)" -f infra/github-webhook.jq \
| gh api --method POST repos/arunanshub/devops/hooks --input - --jq .id

# Seal an app's Cloudflare cache-purge token (app: arunanshu-dev | libby-hbd-2026)
[group('secrets')]
seal-cache-purge app output:
Expand Down
14 changes: 14 additions & 0 deletions kubernetes/base/infra/argocd/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,20 @@ configs:
resource.manifest.compression.enabled: "true"
resource.manifest.storage: msgpack
resource.manifest.compression: gzip-bestspeed
# Cap GitHub webhook bodies well under the 50MB default: the endpoint has no
# rate limiting of its own. A push payload is KBs; an oversized one is
# rejected and the 3-minute poll picks the change up instead. Changing
# argocd-cm rolls argocd-server (chart checksum/cm annotation).
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/
webhook.maxPayloadSizeMB: "5"
secret:
# Not the secret itself: a reference Argo CD resolves at runtime to key
# `secret` of the Secret argocd-webhook-github, which must carry
# app.kubernetes.io/part-of=argocd (components/argocd-webhook, sealed by
# `just seal-argocd-webhook`). Safe in public git; the chart writes it to
# argocd-secret as webhook.github.secret.
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/
githubSecret: "$argocd-webhook-github:secret"

controller:
resources:
Expand Down
14 changes: 14 additions & 0 deletions kubernetes/bootstrap/values/argocd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,20 @@ configs:
resource.manifest.compression.enabled: "true"
resource.manifest.storage: msgpack
resource.manifest.compression: gzip-bestspeed
# Cap GitHub webhook bodies well under the 50MB default: the endpoint has no
# rate limiting of its own. A push payload is KBs; an oversized one is
# rejected and the 3-minute poll picks the change up instead. Changing
# argocd-cm rolls argocd-server (chart checksum/cm annotation).
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/
webhook.maxPayloadSizeMB: "5"
secret:
# Not the secret itself: a reference Argo CD resolves at runtime to key
# `secret` of the Secret argocd-webhook-github, which must carry
# app.kubernetes.io/part-of=argocd (components/argocd-webhook, sealed by
# `just seal-argocd-webhook`). Safe in public git; the chart writes it to
# argocd-secret as webhook.github.secret.
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/
githubSecret: "$argocd-webhook-github:secret"

controller:
resources:
Expand Down
27 changes: 27 additions & 0 deletions kubernetes/components/argocd-webhook/application.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd-webhook
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: infra

source:
repoURL: git@github.com:arunanshub/devops.git
targetRevision: master
path: kubernetes/components/argocd-webhook/resources

destination:
server: https://kubernetes.default.svc
namespace: argocd

syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
4 changes: 4 additions & 0 deletions kubernetes/components/argocd-webhook/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- application.yaml
Loading
Loading