RedisForge

RedisForge is a production-shaped Go service for learning Redis deeply.

It uses one intentionally small domain, Items, to make Redis architecture visible: RedisJSON, RedisBloom, RediSearch, Streams, Pub/Sub, Sentinel, Cluster-ready clients, cache-aside repositories, graceful shutdown, metrics, and integration tests with real Redis Stack containers.

If Redis feels messy, this repo is meant to be the place you reopen and revise from. If you are reviewing it from GitHub, it should read like a practical Redis field guide with code you can run.

Why Star This

• Learn Redis patterns through working Go code, not isolated snippets.
• Compare RedisJSON, Bloom filters, RediSearch, Streams, and Pub/Sub in one service.
• See where Redis fits in a backend: config, handlers, repositories, workers, metrics, tests, and deployment files.
• Use the docs as revision notes for interviews, system design, production debugging, and future Redis projects.
• Follow the project journal and roadmap-style docs to see how a real open-source repo evolves over time.

What You Can Learn Here

Topic	Where to Start
Project tour	docs/README.md
How the service is wired	docs/implementation/architecture.md
Redis module choices	docs/implementation/redis-patterns.md
Failure and delivery semantics	docs/failure-semantics.md
Phase-by-phase build history	docs/REDISFORGE_BUILD_GUIDE.md
Redis configuration tradeoffs	docs/redis-decisions.md
Profiling and tuning	docs/profiling-results.md
Demo and social launch flow	docs/demo_workflow.md
Commit cadence and repo growth	docs/project-journal.md

Architecture At A Glance

graph TD
    Client([Client]) --> API[HTTP API]
    API --> Middleware[chi middleware]
    Middleware --> Handlers[Item Handlers]
    
    subgraph Data Flow
        Handlers --> Repo[Cache-Aside Repository]
        Repo --> RedisJSON[(RedisJSON Cache)]
        Repo -. "Fallback" .-> InMem[(In-Memory Store)]
    end
    
    subgraph Redis Modules
        Handlers -- "Idempotency Check" --> RedisBloom[(RedisBloom)]
        Handlers -- "Best-effort Audit Event" --> AuditEmitter[Audit Emitter]
        AuditEmitter -- "XADD" --> Streams[(Redis Streams)]
        Handlers -- "Search Query" --> RediSearch[(RediSearch)]
        RediSearch -. "Indexes" .-> RedisJSON
    end
    
    Streams --> Worker[Background Audit Worker]
    
    subgraph Observability
        AuditEmitter -.-> Metrics["/healthz and /metrics"]
        API -.-> Metrics
        API -.-> OTel[OpenTelemetry Tracing]
    end

Loading

The important design choice: the domain stays tiny so Redis remains the main thing you are studying.

Feature Map

Redis Feature	What RedisForge Uses It For
RedisJSON	Store Item documents and support partial updates
RedisBloom	Idempotency pre-checks with no false negatives
RediSearch	Full-text search, category filters, tag filters, score ranges
Streams	Best-effort audit producer; at-least-once recovery for entries retained in the approximately 100,000-entry stream
Pub/Sub	Ephemeral real-time notifications
Sentinel	High-availability topology support
Cluster client	Horizontal-scale topology support and hash-tag discipline
SLOWLOG/LATENCY/MEMORY	Profiling drills and production-style tuning notes

Project Structure

redisforge/
|-- cmd/redisforge/              # application entrypoint
|-- internal/
|   |-- app/                     # dependency wiring and lifecycle
|   |-- audit/                   # audit producer, timeout, logging, metrics
|   |-- config/                  # typed environment configuration
|   |-- domain/                  # Item model and sentinel errors
|   |-- handlers/                # HTTP handlers for CRUD and search
|   |-- logging/                 # slog setup
|   |-- observability/           # metrics and tracing hooks
|   |-- redisx/                  # Redis client plus JSON/Bloom/Search/Streams wrappers
|   |-- repo/                    # in-memory store and cache-aside decorator
|   `-- workers/                 # audit stream worker
|-- deployments/
|   |-- docker-compose.yml       # app + Redis Stack + monitoring stack
|   |-- prometheus/              # Prometheus config
|   |-- grafana/                 # Grafana provisioning and dashboards
|   |-- redis-sentinel/          # Sentinel HA demo topology
|   `-- redis-cluster/           # 6-node Cluster horizontal-scale demo
|-- docs/
|   |-- implementation/          # architecture and Redis pattern notes
|   |-- README.md                # docs index and learning path
|   |-- failure-semantics.md     # explicit degraded-mode and delivery guarantees
|   |-- REDISFORGE_BUILD_GUIDE.md
|   |-- redis-decisions.md
|   |-- profiling-results.md
|   |-- demo_workflow.md
|   `-- project-journal.md
|-- scripts/                     # benchmark and demo scripts
|-- Makefile                     # Linux/macOS tasks
`-- make.ps1                     # Windows tasks

Quick Start

Prerequisites:

• Go 1.25+
• Docker Desktop
• PowerShell on Windows or make on Linux/macOS

Start the app, Redis Stack, Prometheus, and Grafana:

.\make.ps1 up

The API is now available on http://localhost:8080.

Create an item:

Invoke-RestMethod -Uri "http://localhost:8080/v1/items" -Method Post -ContentType "application/json" -Body '{
  "name": "Widget Pro",
  "category": "electronics",
  "score": 9.5,
  "tags": ["bestseller", "new"],
  "idempotency_key": "req-123"
}'

Search items:

Invoke-RestMethod "http://localhost:8080/v1/items/search?q=Widget"

Check health and metrics:

Invoke-RestMethod "http://localhost:8080/healthz"
Invoke-RestMethod "http://localhost:8080/metrics"

Stop the stack:

.\make.ps1 down

Linux/macOS equivalents:

make up
make test
make down

API Surface

Method	Path	Purpose
GET	/healthz	Health check
GET	/metrics	Prometheus metrics
POST	/v1/items	Create an item and schedule a best-effort created audit event
GET	/v1/items	List items
GET	/v1/items/{id}	Fetch one item through cache-aside lookup
PUT	/v1/items/{id}	Update an item and schedule a best-effort updated audit event
DELETE	/v1/items/{id}	Delete an item and schedule a best-effort deleted audit event
GET	/v1/items/search?q=...	Search through RediSearch

Tests

.\make.ps1 test

The tests use testcontainers-go where Redis behavior matters. Handler integration tests run against a pinned Redis Stack image, while Streams recovery tests run against a pinned Redis image and prove group bootstrap replay, cursor-complete stale claiming, and ACK-failure handling. Producer tests also prove create/update/delete action emission and synchronous append error propagation.

Performance & Allocation Profile

1. Repository Orchestration & Concurrency Micro-Benchmarks

Measured via go test -run=^$ -bench=. -benchmem -cpu=1,4,8 ./internal/repo on Linux amd64 (13th Gen Intel Core i5-13450HX). Isolates the Go cache-aside state machine, lock contention, and heap allocation overhead from network TCP latency:

Benchmark Path	Cores (GOMAXPROCS)	Latency (ns/op)	Throughput (In-Process)	Bytes / Op	Heap Allocs / Op
Cache Hit (GetByID_CacheHit)	8	36.98 ns/op	~27.0M ops/sec	0 B/op	0 allocs/op
Parallel Contention (GetByID_Contention)	8	42.99 ns/op	~23.3M ops/sec	0 B/op	0 allocs/op
Cache Miss Fast-Path (GetByID_CacheMiss)	8	73.99 ns/op	~13.5M ops/sec	0 B/op	0 allocs/op
Cold Miss + Cache Backfill (GetByID_BackfillPath)	8	418.90 ns/op	~2.39M ops/sec	352 B/op	2 allocs/op

Key Takeaway: Hot-path cache lookups and parallel read contention execute with zero heap allocations (0 B/op, 0 allocs/op), eliminating Go garbage collector (GC) pressure during read spikes. Heap allocations (352 B/op, 2 allocs/op) are strictly confined to cold cache-miss backfill writes.

2. End-to-End Multi-Module Verification Harness

Full integration verification across live Redis Stack containers (RedisJSON, RediSearch, RedisBloom, and Redis Streams consumer-group recovery via testcontainers-go):

• Verification Suite: 752 composite multi-module transactions | 0 failed | 100% stream recovery on worker crash

Development Rhythm

This repo is intended to grow in public. Good future commits are small, reviewable, and educational:

• Add one Redis lesson at a time.
• Add a failing test before fixing subtle behavior.
• Improve one doc after each implementation change.
• Record benchmark numbers when performance claims change.
• Keep docs/project-journal.md updated so visitors can see progress over time.

Suggested commit style:

docs: add redis streams recovery notes
test: cover bloom duplicate idempotency path
feat: expose cache hit ratio metric
perf: record search latency baseline

Status

RedisForge currently implements the planned learning phases from bootstrap through profiling notes:

• HTTP API, config, logging, graceful shutdown
• Redis client abstraction for single-node, Sentinel, and Cluster modes
• RedisJSON, RedisBloom, RediSearch, Streams, and Pub/Sub wrappers
• Cache-aside repository pattern
• Best-effort audit producer for create/update/delete with structured failure logs and Prometheus metrics
• Audit stream worker with at-least-once recovery for appended events while retained in its approximately 100,000-entry bounded stream
• Prometheus metrics and OpenTelemetry hooks
• Integration tests and benchmark/demo scripts

The item handlers intentionally return independently of audit XADD. A failed producer append does not roll back the item write, but it is visible through logs plus audit_events_emitted_total{action,status} and audit_emit_latency_ms{action,status}. After append, the worker can recover pending events at least once while their entries remain in the approximately 100,000-entry bounded stream; duplicates remain possible. The worker currently logs and ACKs events rather than persisting them to an external audit sink. See Failure Semantics for the exact contract.

Repository Goal

RedisForge should be useful in two modes:

1. Revision mode: reopen the repo when Redis concepts feel scattered, then follow the docs and implementation notes until the system clicks again.
2. Visitor mode: land on GitHub, understand the value in under a minute, run the project quickly, and save or star it as a Redis learning reference.

That is the bar for every future change.