From 24209346553fb36d2a4799d41154166b43c85a45 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 19:55:17 +0000 Subject: [PATCH 1/2] feat(argocd): GitHub push webhook on a locked-down public host Argo CD now refreshes on push instead of waiting up to ~3 minutes for its poll. GitHub delivers to argocd-hook.arunanshu.dev/api/webhook; the UI host stays behind the wildcard Access OTP. Four independent layers keep the public host narrow: - Cloudflare WAF custom rule: only GitHub's `hooks` CIDRs (read from api.github.com/meta each plan), only POST /api/webhook. - Cloudflare Access: path-scoped bypass for /api/webhook only. - Traefik: HTTPRoute matches only POST /api/webhook, behind a RateLimit middleware (per-host bucket; remote addr is cloudflared's). - Argo CD: HMAC check via webhook.github.secret, resolved from the sealed argocd-webhook-github Secret through a $ref; body capped at 5MB. The secret is generated in tofu state and flows to kubeseal (`just seal-argocd-webhook`) and the GitHub API (`just github-argocd-webhook`) without being printed. Polling stays at the chart default as the fallback. Rollout, triage and rotation are in docs/runbooks/argocd-webhook.md. The sealed secret is not in this commit: it must be generated from tofu state after `just infra-apply`, so check-manifests fails until it lands. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01ApgDFwSZDFKWW75JMu3hXu --- docs/runbooks/argocd-webhook.md | 56 +++++++++++++++++ infra/.terraform.lock.hcl | 37 +++++++++++ infra/argocd_webhook.tf | 61 +++++++++++++++++++ infra/cloudflare_access.tf | 24 ++++++++ infra/cloudflare_waf.tf | 39 ++++++++++++ infra/github-webhook.jq | 16 +++++ infra/main.tf | 6 ++ infra/outputs.tf | 11 ++++ justfile | 16 +++++ kubernetes/base/infra/argocd/values.yaml | 14 +++++ kubernetes/bootstrap/values/argocd.yaml | 14 +++++ .../argocd-webhook/application.yaml | 27 ++++++++ .../argocd-webhook/kustomization.yaml | 4 ++ .../argocd-webhook/resources/httproute.yaml | 41 +++++++++++++ .../resources/kustomization.yaml | 8 +++ .../argocd-webhook/resources/middleware.yaml | 26 ++++++++ kubernetes/overlays/prod/kustomization.yaml | 1 + 17 files changed, 401 insertions(+) create mode 100644 docs/runbooks/argocd-webhook.md create mode 100644 infra/argocd_webhook.tf create mode 100644 infra/cloudflare_waf.tf create mode 100644 infra/github-webhook.jq create mode 100644 kubernetes/components/argocd-webhook/application.yaml create mode 100644 kubernetes/components/argocd-webhook/kustomization.yaml create mode 100644 kubernetes/components/argocd-webhook/resources/httproute.yaml create mode 100644 kubernetes/components/argocd-webhook/resources/kustomization.yaml create mode 100644 kubernetes/components/argocd-webhook/resources/middleware.yaml diff --git a/docs/runbooks/argocd-webhook.md b/docs/runbooks/argocd-webhook.md new file mode 100644 index 00000000..09602486 --- /dev/null +++ b/docs/runbooks/argocd-webhook.md @@ -0,0 +1,56 @@ +# Argo CD GitHub webhook + +GitHub POSTs push events to `https://argocd-hook.arunanshu.dev/api/webhook`, and Argo CD refreshes the matching Applications in seconds instead of waiting for the poll (`timeout.reconciliation` 120s + up to 60s jitter, left at the chart default). Polling stays on: a dropped or rejected delivery only costs the old latency. + +The UI host `argocd.arunanshu.dev` is unchanged and stays behind the wildcard Access OTP. + +## Layers + +| Layer | Where | What it admits | +|-------|-------|----------------| +| Cloudflare WAF custom rule | `infra/cloudflare_waf.tf` | Only GitHub's `hooks` CIDRs (read from `api.github.com/meta` on every plan), only `POST /api/webhook` | +| Cloudflare Access bypass | `infra/cloudflare_access.tf` | Only the `/api/webhook` path; any other path on the host gets the OTP page | +| Traefik HTTPRoute + RateLimit | `kubernetes/components/argocd-webhook` | Only `POST /api/webhook` (anything else is a 404); 30/min, burst 30, per Traefik pod | +| Argo CD | `configs.secret.githubSecret` in both argocd values files | Only deliveries whose `X-Hub-Signature-256` HMAC matches; body capped at 5MB | + +The HMAC secret lives in tofu state (`random_password.argocd_webhook`), is sealed into `argocd-webhook-github` (labelled `app.kubernetes.io/part-of: argocd`, which Argo CD requires for a `$secret:key` reference), and is sent to GitHub through the API. It is never printed. + +## First rollout + +1. **Check for dashboard WAF rules.** `cloudflare_ruleset.zone_custom_firewall` takes over the zone's custom-rules entry point. If Security → WAF → Custom rules lists anything, import it into `infra/cloudflare_waf.tf` first (`cf-terraforming`), or the apply fails or replaces it. +2. `just infra-apply` — creates the secret, the WAF rule and the Access bypass. +3. `just seal-argocd-webhook` — writes `kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml`. Commit it; CI's manifest check fails until this file exists. +4. Merge. The `argocd-webhook` Application auto-syncs. The `argocd` Application does **not** (self-management is manual): sync it so `argocd-secret` gets `webhook.github.secret` and `argocd-cm` gets `webhook.maxPayloadSizeMB` (argocd-server rolls on the cm checksum). Check the live objects, not just "Synced". +5. `just github-argocd-webhook` — creates the hook (push events only) and prints its id. Running it twice creates a second hook. +6. Push a trivial commit and watch an Application refresh within seconds. GitHub → Settings → Webhooks → Recent Deliveries shows every attempt with the response. + +## When a delivery fails + +Read the response in Recent Deliveries first, then Cloudflare Security → Events. + +| Response | Cause | Fix | +|----------|-------|-----| +| 403, Cloudflare block page, event source "Custom rules" | WAF lockdown | GitHub changed ranges: `just infra-plan` shows the diff, `just infra-apply` | +| 403 / challenge, event source "Bot fight mode" | Bot Fight Mode (`cloudflare_hardening.tf`) runs outside the Ruleset Engine; no custom rule can skip it | Add IP Access *allow* rules for the `hooks` CIDRs (Bot Fight Mode does not trigger when an IP Access rule matches first), or turn Bot Fight Mode off. Both are zone-wide trade-offs; decide deliberately | +| 302 to `cloudflareaccess.com` | Access bypass missing or not matching | Check the `argocd_webhook_bypass` app path | +| 404 from Traefik | Route mismatch | URL path or method wrong; check the hook URL | +| 429 | Traefik rate limit | Look for a redelivery storm or push loop before raising the limit | +| 4xx from Argo CD about the signature | Secret mismatch | Re-run step 3 and the rotation PATCH below | + +Docs: , + +## Rotate the secret + +1. From `infra/`: `sops exec-env secrets.yaml "tofu apply -replace=random_password.argocd_webhook"`. +2. `just seal-argocd-webhook`, commit, merge (the `$ref` re-resolves; no argocd sync needed). +3. Update GitHub's copy (hook id from step 5 above, or `gh api repos/arunanshub/devops/hooks --jq '.[] | [.id, .config.url]'`): + + ```bash + just _tofu-output argocd_webhook_secret \ + | jq -Rn --arg url https://argocd-hook.arunanshu.dev/api/webhook -f infra/github-webhook.jq \ + | jq .config | gh api --method PATCH repos/arunanshub/devops/hooks//config --input - + ``` + +The body carries the whole `config` (url, content type, TLS check, secret), so nothing depends on how the endpoint treats omitted fields: . + +Between steps 2 and 3, deliveries fail the HMAC check and polling covers the gap. diff --git a/infra/.terraform.lock.hcl b/infra/.terraform.lock.hcl index 3fec298a..d4e404c3 100644 --- a/infra/.terraform.lock.hcl +++ b/infra/.terraform.lock.hcl @@ -25,6 +25,43 @@ provider "registry.opentofu.org/cloudflare/cloudflare" { ] } +provider "registry.opentofu.org/hashicorp/http" { + version = "3.6.3" + constraints = "~> 3.6" + hashes = [ + "h1:4Qcn3THnwpzEH7EJvhnDYw82YM/+Fs4x5rjtaOdHIGg=", + "h1:DWXJhjYQ66frFZ2edx766sUWDPoPTsWcO8VeL1TQmj0=", + "h1:ISETodakZAv3gUBLFHm1+JpKB0vUXzOSmJGlyTvap0A=", + "h1:IWCGxH32+DkaMeH7xd1SCggVeuhKnKOlkKxlRfQnSlc=", + "h1:JrmuoX6FTlDFJDqveHpyO0xCOyWd40BVTAEFGmwbsPI=", + "h1:OqnVFnmMgeKLZ0l32P80o6DdAhd2ug0nJmEcngf6u/U=", + "h1:Se19S4Etttxlm6u/fT0JbvqfsTJjNusWK5pe2Hb1Aso=", + "h1:Sj+741LS543TKq6nf2MXCKzhzEygGgE8ptZKZhWh1N0=", + "h1:T8pVs0L1oME182/NXYte0G65aOxNod8fhKd6Gi7l+04=", + "h1:VYtHHeZQ+jzrqFppK/eAzdj3dMXn4wPYvfWn9STSaKg=", + "h1:Xkog5MBnip5VSOe8gkMAg/s+fBSpuc2yOPtoMY78mVQ=", + "h1:Xoun8WNhPBF/Acv6vvBJcOtPS30rWL8bDbPBXAZkWXA=", + "h1:b8JSDcp7TH6fQYII+ZZSrMlyAyyP8oSA6FPLs+2Hm3g=", + "h1:bK4m/bKTU8PjLG4G3wEdoRjqmFSB222h1ue0Jxdsp1U=", + "h1:sIiP0wiUT/sXTzwxUoAcEnTgxDkWQZUspChLs/SbXI8=", + "zh:0551a71cf3c063c8c4925fb3bc95e9145da6440887e8513040ea49ea1ecf61ed", + "zh:19f379bb524434d05f5c917371c8d993491058cf577a6c66ebbf1b26a9924003", + "zh:2da5794d9f33f5ba8bdc74de7422b3289f1862be7583fcd86328d28a35a533d9", + "zh:3e3d463e11d3a0d1eb54d4a4936d5de42395271e00a0aa884b27f00ca98d246d", + "zh:476a140b360acb9ca589b7de6a5a8a379aabab4e034fd2dbf4be033c21749907", + "zh:47f46c2a71075a414049f9c3382dad1b6a0fa3b02a67ed8b7db1e7e24dac1960", + "zh:5845ccfc22524f8b2b76a8b790569183e70962b05f8767738edcd4a1b86c0c23", + "zh:80b341609f5f962f709bae424860334e69e1fe6d12b9027f9c0355a1da63eb02", + "zh:94cd85fd04a0b36a8e622efc63e08c34a6ba135713dd9c7a60059af55aea6876", + "zh:abee98f71480a9485f85dee138d9c120d0a0bccb77c700e03fe9585ad0538673", + "zh:ba330dc3fd22cb646d225b0e5e1263a1aa5493c7e880f97b22788e3702a93acd", + "zh:c3534ef0998dba2a3b79d4c993197ed86f1482dd451479ae8244d92cf7b3b3a9", + "zh:d2a2d76caa8256430cae2238f91ca4d5cdb8aa50cccd0c057c97c229ae8508d5", + "zh:ec3f1132210103816061bcff2043e852d9dc2add1857af4620f49059771e1646", + "zh:f3294fc74d18ada739f968b4e0b5ec65c0acb6a821c777909c6c54e3c54cf771", + ] +} + provider "registry.opentofu.org/hashicorp/random" { version = "3.9.1" constraints = "~> 3.6" diff --git a/infra/argocd_webhook.tf b/infra/argocd_webhook.tf new file mode 100644 index 00000000..c70def72 --- /dev/null +++ b/infra/argocd_webhook.tf @@ -0,0 +1,61 @@ +# --------------------------- Argo CD GitHub webhook --------------------------- +# +# GitHub POSTs push events to argocd-hook.arunanshu.dev/api/webhook, so Argo CD +# refreshes on merge instead of on its 3-minute poll. Polling stays on as the +# fallback for dropped deliveries. +# +# The UI host (argocd.arunanshu.dev) stays behind the wildcard Access OTP. The +# hook host is public, and four independent layers keep it narrow: +# 1. Cloudflare WAF (cloudflare_waf.tf): only GitHub's hook IPs, only +# `POST /api/webhook`; everything else is blocked at the edge. +# 2. Cloudflare Access (cloudflare_access.tf): the bypass covers only the +# /api/webhook path; any other path on the host still gets the OTP page. +# 3. Traefik (components/argocd-webhook): the HTTPRoute matches only +# `POST /api/webhook` and is rate-limited. +# 4. Argo CD: drops any delivery whose X-Hub-Signature-256 HMAC does not match +# webhook.github.secret. +# +# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/ +# https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks + +locals { + argocd_webhook_host = "argocd-hook.arunanshu.dev" + argocd_webhook_path = "/api/webhook" + + # GitHub's webhook source ranges (IPv4 and IPv6). GitHub changes these + # occasionally and says to query the API instead of pinning, so every plan + # re-reads them; a changed range shows up as a WAF rule diff. + # https://docs.github.com/en/rest/meta/meta#get-github-meta-information + github_hook_cidrs = jsondecode(data.http.github_meta.response_body).hooks +} + +data "http" "github_meta" { + url = "https://api.github.com/meta" + + request_headers = { + Accept = "application/vnd.github+json" + X-GitHub-Api-Version = "2022-11-28" + } + + retry { + attempts = 2 + } + + lifecycle { + # An empty list would render `ip.src in {}` and block every delivery; fail + # the plan instead of shipping that. + postcondition { + condition = self.status_code == 200 && length(jsondecode(self.response_body).hooks) > 0 + error_message = "GET https://api.github.com/meta did not return a non-empty `hooks` list." + } + } +} + +# HMAC key shared by GitHub (signs) and Argo CD (verifies). Never printed: it +# goes from state straight into kubeseal (`just seal-argocd-webhook`) and the +# GitHub API (`just github-argocd-webhook`). Hex-safe so neither side needs +# quoting. Taint and re-run both recipes to rotate. +resource "random_password" "argocd_webhook" { + length = 64 + special = false +} diff --git a/infra/cloudflare_access.tf b/infra/cloudflare_access.tf index 0734a0f2..998bd0cd 100644 --- a/infra/cloudflare_access.tf +++ b/infra/cloudflare_access.tf @@ -38,6 +38,30 @@ resource "cloudflare_zero_trust_access_application" "libby_hbd_2026_bypass" { ] } +# Public bypass for the Argo CD GitHub webhook — GitHub cannot answer an OTP +# and webhooks cannot carry a service-token header. Scoped to the one path: the +# more specific application wins, so every other path on the host (and the UI +# host argocd.arunanshu.dev) still falls to the wildcard OTP policy below. The +# WAF lockdown (cloudflare_waf.tf) and the HMAC check in Argo CD are what +# actually authenticate a delivery. See argocd_webhook.tf. +# https://developers.cloudflare.com/cloudflare-one/access-controls/policies/app-paths/ +resource "cloudflare_zero_trust_access_application" "argocd_webhook_bypass" { + account_id = var.cloudflare_account_id + name = "${local.argocd_webhook_host}${local.argocd_webhook_path} — GitHub webhook bypass" + domain = "${local.argocd_webhook_host}${local.argocd_webhook_path}" + type = "self_hosted" + session_duration = "24h" + + policies = [ + { + name = "bypass" + decision = "bypass" + precedence = 1 + include = [{ everyone = {} }] + }, + ] +} + # Cloudflare Access — single wildcard app gates all *.arunanshu.dev traffic. # New services only need an HTTPRoute; no Terraform change required unless # per-app access control is needed (different users, different session length). diff --git a/infra/cloudflare_waf.tf b/infra/cloudflare_waf.tf new file mode 100644 index 00000000..d3fe7911 --- /dev/null +++ b/infra/cloudflare_waf.tf @@ -0,0 +1,39 @@ +# ------------------------------ WAF custom rules ------------------------------ +# +# This resource is the zone's entry-point ruleset for the +# http_request_firewall_custom phase, and OpenTofu owns it whole: "Terraform +# assumes that it has complete control over account and zone rulesets." +# Dashboard-made custom rules must be imported here first (cf-terraforming), or +# the first apply fails or replaces them. Free plan: 5 custom rules per zone. +# https://developers.cloudflare.com/terraform/additional-configurations/waf-custom-rules/ +# https://developers.cloudflare.com/waf/custom-rules/ +# +# Known gap: Bot Fight Mode (cloudflare_hardening.tf) runs outside the Ruleset +# Engine, so no rule here can skip it. If it challenges GitHub's deliveries +# (Security > Events, or "Recent Deliveries" on the GitHub hook), see +# docs/runbooks/argocd-webhook.md. +# https://developers.cloudflare.com/waf/feature-interoperability/ + +resource "cloudflare_ruleset" "zone_custom_firewall" { + zone_id = var.cloudflare_zone_id + name = "Zone custom rules" + description = "Entry point for http_request_firewall_custom; managed by OpenTofu." + kind = "zone" + phase = "http_request_firewall_custom" + + rules = [ + { + ref = "argocd_webhook_lockdown" + description = "argocd-hook: allow only GitHub hook IPs sending POST ${local.argocd_webhook_path}" + action = "block" + enabled = true + expression = join(" ", [ + "(http.host eq \"${local.argocd_webhook_host}\" and (", + "not ip.src in {${join(" ", local.github_hook_cidrs)}}", + "or http.request.method ne \"POST\"", + "or http.request.uri.path ne \"${local.argocd_webhook_path}\"", + "))", + ]) + }, + ] +} diff --git a/infra/github-webhook.jq b/infra/github-webhook.jq new file mode 100644 index 00000000..c03928b7 --- /dev/null +++ b/infra/github-webhook.jq @@ -0,0 +1,16 @@ +# Request body for POST /repos/{owner}/{repo}/hooks, used by +# `just github-argocd-webhook`. The secret arrives on stdin (jq -R `input`) so +# it is never on a command line or in shell history; the URL arrives as $url. +# Only push events: Argo CD refreshes on push and ignores everything else. +# https://docs.github.com/en/rest/repos/webhooks#create-a-repository-webhook +{ + name: "web", + active: true, + events: ["push"], + config: { + url: $url, + content_type: "json", + insecure_ssl: "0", + secret: input + } +} diff --git a/infra/main.tf b/infra/main.tf index cc739e79..5c107e5e 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -16,6 +16,12 @@ terraform { source = "hashicorp/random" version = "~> 3.6" } + + # Reads GitHub's webhook IP ranges for the WAF lockdown (argocd_webhook.tf). + http = { + source = "hashicorp/http" + version = "~> 3.6" + } } encryption { diff --git a/infra/outputs.tf b/infra/outputs.tf index 25a1c192..40ccdf71 100644 --- a/infra/outputs.tf +++ b/infra/outputs.tf @@ -55,3 +55,14 @@ output "libby_hbd_2026_cache_purge_token_id" { description = "Cloudflare account token id for the libby-hbd-2026 cache-purge token (safe to log)." value = cloudflare_account_token.libby_hbd_2026_cache_purge.id } + +output "argocd_webhook_secret" { + description = "HMAC secret for the Argo CD GitHub webhook. Seal via `just seal-argocd-webhook`; register on GitHub via `just github-argocd-webhook`." + value = random_password.argocd_webhook.result + sensitive = true +} + +output "argocd_webhook_url" { + description = "Payload URL GitHub delivers push events to (safe to log)." + value = "https://${local.argocd_webhook_host}${local.argocd_webhook_path}" +} diff --git a/justfile b/justfile index d32ba8a6..248ea7a3 100644 --- a/justfile +++ b/justfile @@ -78,6 +78,22 @@ seal-tunnel-token: --from-file=token=/dev/stdin --dry-run=client -o yaml \ | just _seal kubernetes/components/cloudflared/resources/tunnel-token.sealedsecret.yaml +# Seal the Argo CD GitHub webhook HMAC secret from tofu state (label: Argo CD's $ref lookup) +[group('secrets')] +seal-argocd-webhook: + just _tofu-output argocd_webhook_secret \ + | kubectl create secret generic argocd-webhook-github -n argocd \ + --from-file=secret=/dev/stdin --dry-run=client -o yaml \ + | kubectl label --local -f - app.kubernetes.io/part-of=argocd -o yaml \ + | just _seal kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml + +# Create the GitHub push webhook once (gh needs Webhooks: write); prints the hook id +[group('secrets')] +github-argocd-webhook: + just _tofu-output argocd_webhook_secret \ + | jq -Rn --arg url "$(just _tofu-output argocd_webhook_url)" -f infra/github-webhook.jq \ + | gh api --method POST repos/arunanshub/devops/hooks --input - --jq .id + # Seal an app's Cloudflare cache-purge token (app: arunanshu-dev | libby-hbd-2026) [group('secrets')] seal-cache-purge app output: diff --git a/kubernetes/base/infra/argocd/values.yaml b/kubernetes/base/infra/argocd/values.yaml index b6d2229f..76747dcb 100644 --- a/kubernetes/base/infra/argocd/values.yaml +++ b/kubernetes/base/infra/argocd/values.yaml @@ -12,6 +12,20 @@ configs: resource.manifest.compression.enabled: "true" resource.manifest.storage: msgpack resource.manifest.compression: gzip-bestspeed + # Cap GitHub webhook bodies well under the 50MB default: the endpoint has no + # rate limiting of its own. A push payload is KBs; an oversized one is + # rejected and the 3-minute poll picks the change up instead. Changing + # argocd-cm rolls argocd-server (chart checksum/cm annotation). + # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/ + webhook.maxPayloadSizeMB: "5" + secret: + # Not the secret itself: a reference Argo CD resolves at runtime to key + # `secret` of the Secret argocd-webhook-github, which must carry + # app.kubernetes.io/part-of=argocd (components/argocd-webhook, sealed by + # `just seal-argocd-webhook`). Safe in public git; the chart writes it to + # argocd-secret as webhook.github.secret. + # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/ + githubSecret: "$argocd-webhook-github:secret" controller: resources: diff --git a/kubernetes/bootstrap/values/argocd.yaml b/kubernetes/bootstrap/values/argocd.yaml index e50df756..7d341b7c 100644 --- a/kubernetes/bootstrap/values/argocd.yaml +++ b/kubernetes/bootstrap/values/argocd.yaml @@ -16,6 +16,20 @@ configs: resource.manifest.compression.enabled: "true" resource.manifest.storage: msgpack resource.manifest.compression: gzip-bestspeed + # Cap GitHub webhook bodies well under the 50MB default: the endpoint has no + # rate limiting of its own. A push payload is KBs; an oversized one is + # rejected and the 3-minute poll picks the change up instead. Changing + # argocd-cm rolls argocd-server (chart checksum/cm annotation). + # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/ + webhook.maxPayloadSizeMB: "5" + secret: + # Not the secret itself: a reference Argo CD resolves at runtime to key + # `secret` of the Secret argocd-webhook-github, which must carry + # app.kubernetes.io/part-of=argocd (components/argocd-webhook, sealed by + # `just seal-argocd-webhook`). Safe in public git; the chart writes it to + # argocd-secret as webhook.github.secret. + # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/ + githubSecret: "$argocd-webhook-github:secret" controller: resources: diff --git a/kubernetes/components/argocd-webhook/application.yaml b/kubernetes/components/argocd-webhook/application.yaml new file mode 100644 index 00000000..01061807 --- /dev/null +++ b/kubernetes/components/argocd-webhook/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: argocd-webhook + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "0" + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: infra + + source: + repoURL: git@github.com:arunanshub/devops.git + targetRevision: master + path: kubernetes/components/argocd-webhook/resources + + destination: + server: https://kubernetes.default.svc + namespace: argocd + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - ServerSideApply=true diff --git a/kubernetes/components/argocd-webhook/kustomization.yaml b/kubernetes/components/argocd-webhook/kustomization.yaml new file mode 100644 index 00000000..c91420d2 --- /dev/null +++ b/kubernetes/components/argocd-webhook/kustomization.yaml @@ -0,0 +1,4 @@ +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component +resources: + - application.yaml diff --git a/kubernetes/components/argocd-webhook/resources/httproute.yaml b/kubernetes/components/argocd-webhook/resources/httproute.yaml new file mode 100644 index 00000000..fba75529 --- /dev/null +++ b/kubernetes/components/argocd-webhook/resources/httproute.yaml @@ -0,0 +1,41 @@ +# Public route for GitHub's push webhook — and nothing else. The UI and API +# stay on argocd.arunanshu.dev behind Cloudflare Access (argocd-route). Any +# request that is not exactly `POST /api/webhook` matches no rule and gets +# Traefik's 404, so this host can never reach the rest of argocd-server even if +# the edge rules in infra/cloudflare_waf.tf and infra/cloudflare_access.tf were +# loosened. Method matching is a Gateway API extended feature; Traefik declares +# HTTPRouteMethodMatchingFeature and turns it into a Method() rule: +# https://gateway-api.sigs.k8s.io/guides/http-method-matching/ +# https://github.com/traefik/traefik/blob/master/pkg/provider/kubernetes/gateway/features.go +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: argocd-webhook + namespace: argocd +spec: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: traefik-gateway + namespace: traefik + sectionName: web + hostnames: + - argocd-hook.arunanshu.dev + rules: + - matches: + - path: + type: Exact + value: /api/webhook + method: POST + backendRefs: + - group: "" + kind: Service + name: argocd-server + port: 80 + weight: 1 + filters: + - type: ExtensionRef + extensionRef: + group: traefik.io + kind: Middleware + name: argocd-webhook-ratelimit diff --git a/kubernetes/components/argocd-webhook/resources/kustomization.yaml b/kubernetes/components/argocd-webhook/resources/kustomization.yaml new file mode 100644 index 00000000..6a628c39 --- /dev/null +++ b/kubernetes/components/argocd-webhook/resources/kustomization.yaml @@ -0,0 +1,8 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - httproute.yaml + - middleware.yaml + # Generated by `just seal-argocd-webhook` from tofu state; see + # docs/runbooks/argocd-webhook.md. + - github-secret.sealedsecret.yaml diff --git a/kubernetes/components/argocd-webhook/resources/middleware.yaml b/kubernetes/components/argocd-webhook/resources/middleware.yaml new file mode 100644 index 00000000..5049d166 --- /dev/null +++ b/kubernetes/components/argocd-webhook/resources/middleware.yaml @@ -0,0 +1,26 @@ +# Backstop rate limit for the public webhook route. The Cloudflare WAF already +# admits only GitHub's hook IPs, so this guards argocd-server against a burst +# from that side (a redelivery storm, a runaway push loop), not the internet. +# +# One bucket for the whole host (requestHost), not per client IP: behind +# cloudflared every request arrives from a cloudflared pod, so the remote +# address is meaningless, and GitHub is the only legitimate client anyway. +# Buckets are in-memory per Traefik pod (KEDA runs 2-5), so the effective +# ceiling is this limit times the replica count — fine for a backstop. +# +# A push produces one delivery; 30/min with a burst of 30 is far above a +# Renovate merge wave and still caps a flood well below what argocd-server's +# refresh path would notice. +# https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/ratelimit/ +apiVersion: traefik.io/v1alpha1 +kind: Middleware +metadata: + name: argocd-webhook-ratelimit + namespace: argocd +spec: + rateLimit: + average: 30 + period: 1m + burst: 30 + sourceCriterion: + requestHost: true diff --git a/kubernetes/overlays/prod/kustomization.yaml b/kubernetes/overlays/prod/kustomization.yaml index 7a72211a..4e3bb87d 100644 --- a/kubernetes/overlays/prod/kustomization.yaml +++ b/kubernetes/overlays/prod/kustomization.yaml @@ -23,6 +23,7 @@ components: - ../../components/system-upgrade-controller - ../../components/system-upgrade-plans - ../../components/argocd-route + - ../../components/argocd-webhook - ../../components/etcd-snapshot-health - ../../components/velero-restore-drill - ../../components/network-policies From 2e7e068ffb42d73aeddbe72eb4f87c277bb83c57 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 20:00:54 +0000 Subject: [PATCH 2/2] docs(argocd-webhook): name the CI job that waits on the sealed secret The prod overlay only holds the Application, so check-manifests never builds the component's resources/; the render job does, and that is the one that fails until the SealedSecret is committed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01ApgDFwSZDFKWW75JMu3hXu --- docs/runbooks/argocd-webhook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/runbooks/argocd-webhook.md b/docs/runbooks/argocd-webhook.md index 09602486..113fb5ed 100644 --- a/docs/runbooks/argocd-webhook.md +++ b/docs/runbooks/argocd-webhook.md @@ -19,7 +19,7 @@ The HMAC secret lives in tofu state (`random_password.argocd_webhook`), is seale 1. **Check for dashboard WAF rules.** `cloudflare_ruleset.zone_custom_firewall` takes over the zone's custom-rules entry point. If Security → WAF → Custom rules lists anything, import it into `infra/cloudflare_waf.tf` first (`cf-terraforming`), or the apply fails or replaces it. 2. `just infra-apply` — creates the secret, the WAF rule and the Access bypass. -3. `just seal-argocd-webhook` — writes `kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml`. Commit it; CI's manifest check fails until this file exists. +3. `just seal-argocd-webhook` — writes `kubernetes/components/argocd-webhook/resources/github-secret.sealedsecret.yaml`. Commit it; the `render` CI job (Argo CD renders the component) fails until this file exists. 4. Merge. The `argocd-webhook` Application auto-syncs. The `argocd` Application does **not** (self-management is manual): sync it so `argocd-secret` gets `webhook.github.secret` and `argocd-cm` gets `webhook.maxPayloadSizeMB` (argocd-server rolls on the cm checksum). Check the live objects, not just "Synced". 5. `just github-argocd-webhook` — creates the hook (push events only) and prints its id. Running it twice creates a second hook. 6. Push a trivial commit and watch an Application refresh within seconds. GitHub → Settings → Webhooks → Recent Deliveries shows every attempt with the response.