EDAS — Event-Driven Agentic System

High-Concurrency WhatsApp AI with Custom Raft Broker

WhatsApp Cloud API ──► Gateway ──► Raft Broker Cluster ──► Agent Workers ──► WhatsApp Cloud API
                         │             │                         │
                      HMAC-SHA256   3-node                 Claude claude-sonnet-4-20250514
                      validation    consensus              Tool-use loop
                                    BLAKE3 dedup           RAG + Episodic memory

---

What's Inside

Component	Language	Purpose
broker/	Rust + Tokio	Custom Raft consensus broker with exactly-once delivery
gateway/	Rust + Axum	WhatsApp webhook ingestion with HMAC validation
agents/	Python + Anthropic	Multi-agent AI workers with RAG and tool-use
k8s/	YAML	Production Kubernetes manifests
observability/	YAML + JSON	Prometheus + Grafana + Jaeger
.github/	YAML	Full CI/CD pipeline

---

Quick Start

# 1. Install dependencies
brew install protobuf rust python@3.12 docker

# 2. Configure secrets
cp .env.example .env
# Edit .env with your API keys

# 3. Start the full stack
docker compose up --build

# 4. Send a test message
curl -X POST http://localhost:3000/webhook \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature-256: sha256=$(echo -n '{"entry":[]}' | openssl dgst -sha256 -hmac "$WHATSAPP_APP_SECRET" | cut -d' ' -f2)" \
  -d '{"entry":[{"changes":[{"field":"messages","value":{"messages":[
    {"id":"test-1","from":"15551234567","type":"text","text":{"body":"Hello!"}}
  ]}}]}]}'

# 5. View dashboards
open http://localhost:3001   # Grafana (admin/admin)
open http://localhost:9090   # Prometheus
open http://localhost:16686  # Jaeger

---

Architecture

Custom Raft Broker (Rust)

• Raft consensus with pre-vote, leader election, log replication, snapshots
• Exactly-once delivery: Layer 1 = BLAKE3 fingerprint cache (RocksDB); Layer 2 = per-consumer-group sequential offset tracking
• gRPC transport via tonic with HTTP/2 multiplexing and 100ms deadline
• Pub/Sub with broadcast channels per topic (65,536 capacity)
• Metrics: 10+ Prometheus metrics exposed at :9090/metrics

Agent Workers (Python)

• Agentic loop: Claude claude-sonnet-4-20250514 with 6 tools, max 5 iterations
• Intent classifier: Claude Haiku + regex fallback + LRU cache (10k entries)
• RAG: ChromaDB with all-MiniLM-L6-v2 embeddings + Redis session history
• Episodic memory: Semantic recall of past interactions per user
• Multi-modal: Images (Vision), Audio (Whisper), Documents (PyPDF2/python-docx), Video (OpenCV)
• Rich messaging: Interactive lists, quick-reply buttons, templates, typing indicators, read receipts

Production Kubernetes

• Broker: StatefulSet with PodAntiAffinity (1 pod/node) + PodDisruptionBudget (minAvailable=2)
• Gateway: Deployment + HPA (2–10 replicas, CPU 60%)
• Agents: Deployment + HPA + KEDA ScaledObject (event-driven on queue depth)
• 7 Prometheus alerting rules (NoRaftLeader, HighErrorRate, CircuitBreakerOpen, ...)
• CI/CD: GitHub Actions with staging deploy → manual approval → production rolling deploy

---

Running Tests

# Broker (Rust) — 13 dedup unit tests + 12 integration tests
cd broker && cargo test --all-features -- --nocapture

# Agents (Python)
cd agents && pytest tests/ -v --asyncio-mode=auto

# Chaos (requires live staging cluster)
pytest agents/chaos/chaos_tests.py -v --timeout=120

---

Environment Variables

# Required
ANTHROPIC_API_KEY=sk-ant-...
WHATSAPP_VERIFY_TOKEN=your-verify-token
WHATSAPP_APP_SECRET=your-app-secret
WHATSAPP_API_TOKEN=your-api-token
WHATSAPP_PHONE_NUMBER_ID=1234567890

# Optional (have sensible defaults)
BROKER_ADDRS=broker-0:8001,broker-1:8001,broker-2:8001
REDIS_URL=redis://redis:6379/0
CHROMA_HOST=chromadb
NUM_WORKERS=8
MAX_AGENTIC_ITERATIONS=5

---

File Structure

edas/
├── broker/                    # Rust: custom Raft broker
│   ├── src/
│   │   ├── raft/
│   │   │   ├── mod.rs         # Core types: NodeRole, LogEntry, PersistentState
│   │   │   ├── node.rs        # RaftNode struct with all state
│   │   │   ├── election.rs    # Leader election (pre-vote + vote phases)
│   │   │   ├── log.rs         # AppendEntries + propose + heartbeat
│   │   │   ├── rpc.rs         # tonic gRPC server + PeerClientPool
│   │   │   └── cluster.rs     # Cluster orchestration
│   │   ├── broker/
│   │   │   ├── dedup.rs       # Two-layer exactly-once engine (13 tests)
│   │   │   ├── pubsub.rs      # Pub/Sub on top of Raft log
│   │   │   └── server.rs      # BrokerService gRPC server
│   │   ├── snapshot/
│   │   │   ├── mod.rs         # Snapshot create/restore/install
│   │   │   └── state_machine.rs
│   │   ├── config.rs          # CLI + env config
│   │   ├── metrics.rs         # Prometheus metrics
│   │   └── main.rs            # Entry point
│   ├── proto/raft.proto        # Protobuf definitions
│   ├── tests/raft_cluster.rs  # 12 integration tests
│   └── Dockerfile
├── gateway/                   # Rust: WhatsApp webhook gateway
│   ├── src/
│   │   ├── main.rs
│   │   ├── webhook.rs         # Verify + receive handlers
│   │   ├── hmac_validator.rs  # HMAC-SHA256 validation
│   │   └── broker_publisher.rs
│   └── Dockerfile
├── agents/                    # Python: AI agent workers
│   ├── src/
│   │   ├── classifier/intent.py     # Two-stage intent classifier
│   │   ├── memory/
│   │   │   ├── manager.py           # Redis + ChromaDB memory
│   │   │   └── episodic.py          # Episodic memory
│   │   ├── tools/registry.py        # 6 production tools
│   │   ├── multimodal/handler.py    # Image/audio/doc/video
│   │   ├── messaging/rich_messaging.py  # WhatsApp rich messages
│   │   ├── telemetry/telemetry.py   # 20+ Prometheus metrics + OTel
│   │   ├── worker/
│   │   │   ├── base.py              # BaseAgentWorker (retry + DLQ)
│   │   │   └── whatsapp_agent.py    # Full agentic loop
│   │   ├── broker_client.py         # gRPC broker client
│   │   ├── config.py                # Pydantic settings
│   │   └── orchestrator.py          # Worker pool supervisor
│   ├── tests/test_agents.py         # Unit tests
│   ├── chaos/chaos_tests.py         # 10 chaos scenarios
│   └── Dockerfile
├── k8s/
│   ├── broker/statefulset.yaml      # StatefulSet + PDB
│   ├── gateway/deployment.yaml      # Deployment + HPA + Ingress
│   ├── agents/deployment.yaml       # Deployment + HPA + KEDA
│   └── monitoring/stack.yaml        # Prometheus + 7 alert rules
├── observability/
│   ├── grafana/dashboard.json       # 15-panel Grafana dashboard
│   ├── prometheus.yml               # Scrape config
│   └── otel-collector.yaml          # OTel → Jaeger
├── .github/workflows/ci.yml         # CI/CD pipeline
├── docker-compose.yml
├── IMPLEMENTATION_PLAN.md           # Step-by-step build guide
└── README.md