docker compose up starts the full system. No additional setup required.
Four independently running agents communicate exclusively through Redis Streams. No agent calls another directly.
flowchart LR
API -->|raw.feedback| T[Triage Agent]
T -->|triaged.feedback| P[Pattern Agent]
P -->|patterns.ready| A[Action Agent]
T -.->|urgent poll| A
S[Supervisor] -.->|requeue stuck/failed| T
S -.->|monitors heartbeats| T & P & A
A -->|POST| GH[Mock GitHub]
A -->|POST| SL[Mock Slack]
T & P & A & S -->|heartbeat + events| DB[(PostgreSQL)]
| Stream | Producer | Consumer | Payload |
|---|---|---|---|
raw.feedback |
API / Supervisor | Triage | Raw feedback item + metadata |
triaged.feedback |
Triage | Pattern | feedback_id, categories, severity, embedding |
patterns.ready |
Pattern | Action | cluster_id, theme, confidence, feedback_ids |
The Supervisor consumes no streams — it operates on database state only (heartbeat table + feedback status columns).
Input: raw.feedback stream
Output: triage_records table → publishes to triaged.feedback
Decision loop (hybrid):
flowchart TD
A[Raw feedback] --> B{Keyword regex match?}
B -- yes --> C[Fast path: deterministic classification\nconfidence 0.85 single / 0.70 multi]
B -- no --> D[LLM: llama-3.3-70b via Groq]
D -- fails --> E[Force COMPLAINT\nconfidence 0.3\nrequires_human_review=true]
C & D & E --> F[Compute 384-dim embedding\nall-MiniLM-L6-v2]
F -- embedding fails --> G[Zero vector fallback]
F & G --> H[Persist triage_record\nPublish to triaged.feedback]
Multi-label handling: A message matching both BUG_REPORT and CHURN_SIGNAL gets both categories. Exception: FEATURE_REQUEST always overrides PRAISE — positive phrasing doesn't change the core ask.
Self-correction loop (every 5 min): Re-evaluates records with confidence < 0.65 and requires_human_review = true created in the last 2 hours. If LLM produces a better result, the record is updated and a self_correction event is logged.
Input: triaged.feedback stream (accumulator)
Output: pattern_clusters table → publishes to patterns.ready
Decision loop:
flowchart TD
A[Triaged feedback] --> B[Accumulate batch]
B --> C{BATCH_SIZE=10 OR 60s elapsed?}
C -- no --> B
C -- yes --> D[DBSCAN on embeddings\neps=0.25, min_samples=3]
D --> E{Significant cluster?}
E -- noise / too small --> F[Discard]
E -- yes --> G[LLM: name theme + assess significance]
G -- fails --> H[Derive theme from text\nconfidence=0.4]
G & H --> I{Conflicting signals?\nprase + bugs in same cluster}
I -- yes --> J[theme = CONFLICTING SIGNALS ...\ntrend = conflicting]
I -- no --> K[Persist cluster\nPublish to patterns.ready]
J --> K
Clusters require ≥ 3 items (DBSCAN min_samples). Signal vs noise distinction is handled by DBSCAN density — background noise is naturally excluded as label -1.
Input: patterns.ready stream + urgent polling loop (every 30s)
Output: action_records table, external API calls
Decision loop:
flowchart TD
A[Pattern cluster] --> B{>50% overlap with\nexisting action?}
B -- yes --> C[Skip — deduplicated]
B -- no --> D[LLM: decide action type]
D --> E{Action type}
E -->|bug, 2+ reports| F[POST GitHub issue]
E -->|critical or churn| G[POST Slack alert]
E -->|feature request| H[Upsert feature_requests\nincrement vote_count]
E -->|praise| I[Flag testimonial]
F & G & H & I --> J[Persist action_record\nMark feedback actioned]
K[Urgent poll: critical/churn\ncreated < 1h ago] --> B
Feature request deduplication: Fuzzy title match against existing feature_requests. Merges feedback_ids and increments vote_count rather than creating a duplicate entry.
Input: None (database polling only)
Output: Requeues to raw.feedback, updates feedback status
Three independent async loops:
| Loop | Interval | Logic |
|---|---|---|
| Agent monitor | 30s | Checks agent_heartbeats — warns if last_seen > 60s and status ≠ stopped |
| Stuck requeue | 60s | status = processing AND updated_at > 5min → reset to pending, republish |
| Failed retry | 120s | status = failed AND retry_count < 3 → exponential backoff (1min × 2^retry_count), republish |
Max 3 retries per item. After that, items remain as failed for manual inspection.
All agent state lives in PostgreSQL. Agents are stateless processes — restart any agent and it resumes by reading from its Redis consumer group (Redis Streams track per-consumer offsets).
feedback_items — canonical item state (pending → processing → triaged → actioned | failed)
triage_records — classification output, confidence, requires_human_review
pattern_clusters — DBSCAN clusters with theme, confidence, trend_direction
action_records — executed actions with external_id/url
feature_requests — deduplicated feature tracker with vote_count
agent_events — append-only audit log (every agent decision + reasoning + duration_ms)
agent_heartbeats — liveness table (upserted every 10s per agent)
feedback_items.retry_count and last_error are written on each failure. The Supervisor's retry loop reads these directly — no separate queue needed.
flowchart TD
A[Message consumed from stream] --> B[process message]
B -- success --> C[ACK stream message]
B -- exception --> D[Log to agent_events\nSet status=failed\nIncrement retry_count]
D --> E{retry_count < 3?}
E -- yes --> F[Supervisor republishes\nwith exponential backoff]
E -- no --> G[Abandoned — manual review]
F --> A
LLM failures by agent:
| Agent | LLM failure behaviour |
|---|---|
| Triage | Force COMPLAINT, confidence=0.3, requires_human_review=true — item is never dropped |
| Pattern | Derive theme from raw text, confidence=0.4 — cluster still published |
| Action | Skip action — no forced fallback, item stays triaged |
Graceful degradation:
- Triage fast path (deterministic regex) handles ~80% of items without touching the LLM
- Embedding failure → zero vector stored (pgvector query still works, just less accurate)
- Rate limiting at API layer: 500 req/min global, 100/min per source; 503 if pending queue > 1000
- External API calls (GitHub, Slack) have 10s timeout; failure logged, item retried by Supervisor
Why 4 agents vs 2 or 6?
Triage and Pattern have fundamentally different cadences — Triage is per-item and latency-sensitive; Pattern batches and clusters, which requires accumulation. Merging them would force a shared processing loop with conflicting timing requirements. Action is separated because it owns external side-effects and deduplication state. Supervisor is separated to avoid any single agent being responsible for its own liveness — a crashed Triage agent can't requeue its own stuck items.
Hybrid Triage (regex + LLM)
Pure LLM classification is ~300ms/item and costs money. Keyword regex covers the high-confidence cases (bug crash language, churn language) in <1ms. LLM is reserved for genuinely ambiguous feedback. This also means the system keeps working — with degraded accuracy — if the LLM API is down.
DBSCAN over k-means
k-means requires specifying cluster count upfront. Feedback volume is unpredictable and some batches produce no clusters. DBSCAN finds clusters of arbitrary shape and explicitly labels noise as -1, which maps cleanly to "no pattern detected."
Supervisor as database poller, not stream consumer
Requeue logic needs to inspect item age and retry count — data that lives in Postgres, not in the stream message. A stream-based supervisor would need to join across Postgres anyway. Polling Postgres every 30–120s for stuck/failed items is simpler and sufficient at this scale.
Redis Streams over Kafka
Consumer groups give us at-least-once delivery and per-consumer offset tracking without Kafka's operational overhead. At feedback volumes under ~10K/day, Streams are sufficient. The trade-off is that Redis Streams have no built-in message retention policy — a long outage could accumulate a large backlog.
For production readiness:
- Circuit breaker on external calls — GitHub and Slack calls currently fail and retry naively. A circuit breaker would open after N consecutive failures and stop hammering a degraded endpoint.
- Dead letter queue — Items that exhaust 3 retries are abandoned in-place. A separate DLQ table (or stream) with alerting would make failure visibility explicit rather than requiring a manual query.
- Supervisor auto-restart — The supervisor detects stale heartbeats but does not restart agents. In production this would integrate with a process supervisor (systemd, Kubernetes liveness probes) rather than doing it in application code.
- Action agent LLM fallback — The only agent with no forced fallback. A rules-based fallback (e.g. severity=critical → always Slack alert) would prevent silent drops.
- Embedding model upgrade —
all-MiniLM-L6-v2is fast but 384 dimensions limits clustering accuracy on short text.text-embedding-3-smallor a domain-tuned model would improve cluster quality. - Pattern window logic — Pattern detection currently batches by count or time elapsed, not by a proper sliding window. A 7-day rolling window with daily recalculation would make trend comparisons meaningful ("up 500% this week vs last week").
- Structured LLM outputs with retry — Pydantic AI validates LLM output schema but if the model returns malformed JSON, we fall through to the fallback immediately. A retry-with-repair loop (re-prompt with the validation error) would reduce unnecessary fallbacks.
- Observability —
agent_eventsis an audit log but there's no alerting on error rate spikes. Adding Prometheus metrics (processed/s, error rate, LLM latency p99) would make the system operable.