Skip to content

Infrastructure

Jump to bottom
Ingwald edited this page Aug 15, 2026 · 2 revisions

Infrastructure

FastAPI Monitoring

File: app/monitoring.py

A FastAPI app served by uvicorn alongside the aiogram polling loop in the same process and event loop. Port: 9090.

Endpoint Type Description
GET /health Liveness Always 200 — process is alive, no external calls
GET /ready Readiness 200 if PostgreSQL pool reachable, 503 otherwise
GET /metrics Prometheus All bot_* metrics in text exposition format

run_bot() races polling task and uvicorn task with asyncio.wait(FIRST_COMPLETED) — either failing triggers clean shutdown.

Prometheus Metrics

Defined in app/utils/metrics.py and emitted by app/bot/rate_limit.py, app/utils/http_retry.py, app/monitoring.py.

Metric Type Labels Description
bot_external_api_requests_total Counter service, outcome HTTP requests to Deezer/Spotify/Genius
bot_external_api_latency_seconds Histogram service Request latency
bot_circuit_breaker_open Gauge service 1 = breaker open, 0 = closed
bot_search_cache_hits_total Counter — PostgreSQL search cache hits
bot_search_cache_misses_total Counter — PostgreSQL search cache misses
bot_tls_cert_expiry_days Gauge — Days until TLS cert expires (-1 if unreadable)
bot_rate_limit_blocked_total Counter — Requests blocked by rate limiter

Scrape config for Prometheus:

scrape_configs:
  - job_name: find-music-bot
    static_configs:
      - targets: ['<server-ip>:9090']

Prometheus Alert Rules

File: deploy/alerts.yml

Alert Condition Severity
CircuitBreakerOpen bot_circuit_breaker_open > 0 for 1m warning
TLSCertExpiringSoon bot_tls_cert_expiry_days < 7 warning
HighExternalAPIErrorRate error rate > 10% over 5m warning
HighRateLimitBlockRate blocks > 1/s over 10m warning

Grafana Dashboard

File: deploy/grafana-dashboard.json

Importable via Grafana UI (Dashboards → Import → Upload JSON). UID: find-music-bot-v1.

8 panels:

  • Circuit Breaker state (stat)
  • TLS Cert Expiry (gauge)
  • Rate Limit Blocks (stat)
  • Request Rate by Service (timeseries)
  • Error Rate by Service (timeseries)
  • API Latency p50/p95/p99 (timeseries)
  • Search Cache Hits vs Misses (timeseries)
  • Cache Hit Ratio (gauge)

Docker Image

deploy/Dockerfile is multi-stage (v3.7.8): pip and the wheels it downloads stay in the build stage and never reach the runtime image. Measured: 368 MB → 347 MB.

The image is scanned by Trivy in CI on every build — fixable HIGH/CRITICAL findings fail the build. ignore-unfixed: true, since unpatched python:3.12-slim base CVEs would otherwise block every PR with no action the project can take.

Docker Compose Stack

File: docker-compose.yml (project root)

Service Image Port Notes
music-bot ghcr.io/ingwalde/find-music-bot:latest 9090 depends on postgres + redis healthy
postgres postgres:16-alpine 5432 named volume postgres-data, pg_isready healthcheck
redis redis:7-alpine 6379 healthcheck ping
test-postgres postgres:16-alpine 5433 profile test, ephemeral, healthcheck
test-redis redis:7-alpine 6380 profile test, ephemeral, healthcheck

All services carry healthchecks, so docker compose up -d --wait blocks until they actually accept connections rather than letting a client race a starting container.

.env is auto-loaded from project root — no -f or --env-file flags needed.

Webhook Mode

Optional. Set BOT_MODE=webhook in .env. Bot terminates TLS itself with a self-signed certificate (no reverse proxy).

Required .env values:

BOT_MODE=webhook
WEBHOOK_PUBLIC_URL=https://<public-ip>:8443
WEBHOOK_SECRET_PATH=<random-string>
WEBHOOK_SECRET_TOKEN=<random-string>
WEBHOOK_CERT_PATH=/app/certs/cert.pem
WEBHOOK_KEY_PATH=/app/certs/key.pem

Generate self-signed cert:

openssl req -newkey rsa:2048 -sha256 -nodes -keyout certs/key.pem -x509 -days 365 \
  -out certs/cert.pem -subj "/CN=<public-ip>"

docker-compose.yml mounts ./certs read-only at /app/certs. Polling mode (default) needs none of this.

Clone this wiki locally