A high-performance message middleware SDK built on Redis Stream + Redisson, offering a RocketMQ-like programming experience.
StreamMQ is an MIT-licensed open-source message middleware SDK built on Redis Stream and Redisson. It wraps Redis Stream's native capabilities in a RocketMQ-style, business-developer-friendly API so that — without spinning up a heavy MQ cluster — you get annotation-driven consumption, transactional messages, delayed messages, ordered messages, and other enterprise-grade features.
StreamMQ 0.1.2 hard-depends on JDK 21+ (enforced in pom.xml via maven-enforcer-plugin and requireJavaVersion [21,)). This is intentional:
- Virtual threads (JEP 444) are the default execution model for consume loops —
Executors.newVirtualThreadPerTaskExecutor()is only GA in JDK 21. We refuse to fall back to a platform-thread pool because the consumer thread count would multiply against the Redis connection pool in a 1:N relationship. - Pattern matching + Record patterns simplify core glue code in
ConsumeLoopTaskandConsumeAction. - We will not downgrade to JDK 17 in the 0.2.0 roadmap. If you are currently on JDK 17 LTS, please evaluate whether you can use JDK 21 within this project. Spring Boot 3.3–3.5 support both 17 and 21, but StreamMQ bets on 21 to avoid writing two thread models for old JDKs.
- Why StreamMQ
- Architecture
- Comparison
- Benchmarks
- Quick Start
- Deployment
- Core Features
- Broadcast Consumption — Operational Notes
- Troubleshooting: a consumer that never consumes
- Modules
- Configuration Reference
- Minimum dependency matrix
- Extension Points
- Observability
- Sample Projects
- Documentation
- Roadmap
- Contributing
- Security
- License
Already running Redis? You already have a message broker. StreamMQ reuses your existing Redis infrastructure — no NameServer, no Broker, no ZooKeeper. One Redis = one MQ cluster.
Aligned with RocketMQ's RocketMQTemplate / @RocketMQMessageListener programming model. If you know RocketMQ, you already know StreamMQ.
Transactional messages, 18-level delayed delivery, ordered messages, batch sending, dead-letter queue, message filtering, message compression — all out of the box.
Auto-configuration, configuration binding, Actuator endpoints, Micrometer metrics — just add the starter.
Serializers, converters, filters, interceptors, retry policies, rebalance strategies, compression codecs, DLQ failure strategies, management authenticators, trace collectors — almost everything is pluggable. They are resolved via the annotation's Class attribute or a Spring Bean override — not Java ServiceLoader.
- 1,509 tests measured on a full local gate run (
mvn clean verify -Djacoco.check.skip=false), split into 1,212 unit (surefire) + 297 integration (failsafe) tests, with 0 failures / 0 errors. The only skips are the 11 Cluster ITs, which skip loudly (printing cluster startup instructions) when no 3-master cluster is reachable — they are never silently green - CI enforces the integration side as a tripwire rather than a total: per module
streammq-redisson ≥ 100,streammq-spring-boot-starter ≥ 30,streammq-test ≥ 40,streammq-samples/* ≥ 16, global≥ 230, and a skip rate≤ 20%— so "Redis silently unavailable" cannot pass as green. Integration tests need a local Redis (localhost:6379); the Cluster ITs additionally need a 3-master cluster on127.0.0.1:7000-7002(or-Dstreammq.it.cluster.nodes=...) and otherwise skip loudly with startup instructions instead of failing.
┌─────────────────────────────────────────────────────────────────────────┐
│ StreamMQ Architecture │
├─────────────────────────────────────────────────────────────────────────┤
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Spring Boot Application │ │
│ │ ┌─────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │
│ │ │ auto-config │ │@StreamMQConsumer│ │ StreamMessageTemplate │ │ │
│ │ │ (auto-config)│ │(declarative │ │ (unified send entry) │ │ │
│ │ └──────┬──────┘ └──────┬───────┘ └───────────┬──────────────┘ │ │
│ └─────────┼─────────────────┼─────────────────────┼────────────────┘ │
│ ┌─────────▼─────────────────▼─────────────────────▼────────────────┐ │
│ │ streammq-core │ │
│ │ Message Builder │ Template │ Consumer │ Producer │ Transaction │ │
│ │ Filter Chain │ Interceptor Chain │ Retry Policy │ Rebalance │ │
│ │ Serializer (SPI) │ Converter (SPI) │ Codec (SPI) │ Trace │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ ┌─────────▼────────────────────────────────────────────────────────┐ │
│ │ streammq-redisson │ │
│ │ Redisson Producer │ Stream Listener │ Delay Scheduler │ PEL │ │
│ │ Claimer │ Tx Scanner │ Tx Lock │ Retry Scheduler │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ ┌─────────▼────────────────────────────────────────────────────────┐ │
│ │ Redis 7.2+ │ │
│ │ Stream (msg storage) ZSet (delay queue) Hash (tx state) │ │
│ └──────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
| Capability | StreamMQ | Redisson RStream | Spring Data Redis Stream | RocketMQ | Kafka |
|---|---|---|---|---|---|
| Underlying storage | Redis Stream | Redis Stream | Redis Stream | NameServer+Broker | Broker+KRaft |
| Deployment complexity | Low (Redis only) | Low | Low | High | High |
| Annotation-driven consumer | Yes | No | Partial | Yes | No |
| Template API | Yes | No | No | Yes | Yes |
| Transactional messages | Yes | No | No | Yes | No |
| Delayed messages | Yes (18 levels + any ms) | No | No | Yes (18 levels) | No |
| Ordered messages | Yes | No | No | Yes | Yes (per partition) |
| Dead-letter queue | Yes (incl. secondary DLQ) | No | No | Yes | Yes (spring-kafka DLT) |
| Tag + SQL92 filtering | Yes | No | No | Yes | No |
| Message compression | Yes (GZIP) | No | No | Yes | Yes |
| Backpressure | Yes (InflightQueue) | No | No | Yes | Yes |
| Spring Boot 3 integration | Deep | Average | Average | Average | Average |
| Extension points | 18 (user-facing + internal assembly) | 0 | 0 | Few | Few |
| Management interface | REST + Actuator | None | None | Dashboard | None |
| Tracing | Yes (TraceCollector SPI + OTel) | No | No | Yes | No |
| Recommended scale | Medium/small (< 100M/day) | Medium/small | Medium/small | Large | Very large |
Backpressure is OFF by default:
streammq.consumer.inflight-capacitydefaults to0(disabled). Set a positive value to enable throttling (the in-flight queue decouples fetch from processing and blocks fetches when full).
Why not Redisson
RTopic/RReliableTopic? Topics are pub/sub: no consumer group, no per-group offset, no PEL/recovery, no ordered or delayed delivery, and a slow subscriber either drops messages or grows an unbounded listener queue. StreamMQ is built on Redis Streams precisely to get at-least-once delivery with replay, DLQ, retry, ordering and transactions.Why not plain
XADD+XREADGROUP? That is exactly what StreamMQ does underneath; the SDK adds what the raw commands leave to you — consumer-group lifecycle & rebalance, ACK/PEL orphan recovery, retry with backoff, DLQ (including a secondary DLQ), delayed messages, transaction half-messages with check-back, tag/SQL92 filtering, backpressure, metrics, tracing and a management API. If you need none of those and want zero dependencies, rawXADD/XREADGROUPis a perfectly reasonable choice.
Important: the numbers below are locally measured benchmarks, all re-run on 2026-09-20 against the current code and the current JMH annotation profile (dedicated localhost Redis 8.8.0, JDK 21.0.11, Intel i7-14700KF / 32 GB / Windows 10 — desktop-grade hardware, not production hardware):
- Serialization benchmarks use JMH
Blackholeconsumers (prevents JIT dead-code elimination from inflating throughput)- Send and consume benchmarks run with
JacksonJsonSerializer— the 0.1.2 production default — so these are not "old default" figures- Consumer benchmark drives the raw Redisson read path (XREADGROUP → field decode → callback → batched XACK, 1 ACK per 100 messages) against a pre-loaded backlog, with no continuous refeeding during the measurement (only a low-watermark keep-alive feeder) and a validity gate that fails the run when the backlog runs dry. It deliberately bypasses the listener container, so the filter/interceptor chains, metrics, retry/DLQ handling and per-message ACK of the production path are not included — expect real container throughput to be lower. A container-driven benchmark is tracked as follow-up work.
- The previous README number "Stream consume ~269,760 ops/s" was removed because it measured an empty XREADGROUP roundtrip — a broken benchmark
- Error bars are 99.9% CI (much wider than the usual 99% one); results are for reference only — measure on your own production hardware
We openly acknowledge that before 0.1.0 we published methodology-flawed benchmark numbers. This transparency matters more than "pretending it didn't happen". Use your own environment's measurements for production capacity planning.
Full report with per-iteration raw values, latency percentiles and the machine-readable validity evidence:
streammq-benchmark/BENCHMARK_REPORT.md.
1KB message body, with Blackhole consumer. JMH profile declared in SerializationBenchmark — the class annotations are the single source of truth (main() no longer overrides them, and neither does CI): @Fork(1), @Warmup(2×1s), @Measurement(3×2s), @BenchmarkMode(Throughput, SampleTime). All six built-in serializers, including FlatBuffersSerializer (FlexBuffers) and SbeSerializer (SBE envelope).
| Serializer | Serialize (ops/s) | Deserialize (ops/s) | RoundTrip (ops/s) | Single Serialize | Single Deserialize |
|---|---|---|---|---|---|
| Fury | ~4,323,664 | ~4,374,579 | ~2,090,991 | ~4,442,831 | ~4,322,816 |
| Protostuff | ~352,662 | ~4,024,571 | ~333,142 | ~354,637 | ~3,777,286 |
| FlatBuffers (FlexBuffers) | ~694,202 | ~736,299 | ~374,788 | ~687,661 | ~807,548 |
| SBE (envelope) | ~358,537 | ~789,301 | ~242,670 | ~355,722 | ~795,995 |
| Jackson (default) | ~419,588 | ~899,889 | ~279,826 | ~414,844 | ~906,830 |
| JDK | ~428,211 | ~116,017 | ~86,815 | ~386,110 | ~115,415 |
Reading the numbers.
- Fury is fastest overall (~10.3× Jackson on serialize, ~4.9× on deserialize, ~7.5× on round-trip) but requires the class-registration whitelist (safety) and pulls Guava. Its p99 latency (~0.6 µs) is roughly an order of magnitude below Jackson's (1.8–5.2 µs).
- Protostuff deserializes very fast (~4.5× Jackson) yet serializes slowly (~0.84× Jackson) — good for read-heavy paths.
- FlatBuffers (FlexBuffers, schema-less, safe — pure data, no gadget RCE surface) is balanced. Its zero-copy read applies per field; because
FlatBuffersSerializerstill materializes a POJO via reflection, end-to-end deserialize (~736K) is comparable to Jackson, not dramatically faster.- SBE (envelope mode) is bounded by its inner Jackson codec (~0.86× Jackson both ways) plus the fixed 8-byte header +
varDataframing (~10–16 B). Its value is the framed, schema-versioned container; true low-latency gains need schema-first bodies (codegen readers), not the envelope.- JDK serializes on par with Jackson but deserializes ~7.8× slower (~116K vs ~900K); its deserialization filter accepts the benchmark payload because the target type is allow-listed for the call. Not recommended for production (see SECURITY.md).
- Laptop/desktop-grade hardware → reference only. An earlier snapshot with different CLI-override parameters lives in
docs/benchmarks/serialization-2026-09-17.md; it is the same order of magnitude but not item-by-item comparable.
Single instance, localhost Redis. JMH profile declared in StreamMessageTemplateBenchmark (single source of truth): @Fork(1), @Warmup(2×1s), @Measurement(3×2s), @BenchmarkMode(Throughput, SampleTime). Batch methods send 100 messages per invocation (asyncSendThroughput issues all 100 concurrently and then joins them; syncSendThroughput sends them serially) and declare @OperationsPerInvocation(100).
| Send Mode | 100B | 1KB | 10KB |
|---|---|---|---|
| Async (100 concurrent, all joined) | ~17,654 | ~16,944 | ~12,527 |
| Sync 100 serial per op | ~3,949 | ~3,499 | ~2,692 |
| Sync single | ~3,932 | ~3,886 | ~2,859 |
Async beats sync-single by ~4.5×; 10KB costs ~29% versus 100B.
SampleTimep99 for the async path stays at ~0.11–0.13 ms.
Rebuilt harness (raw read path: XREADGROUP + field decode + callback + batched XACK, 1 ACK per 100 messages) with:
- a pre-loaded backlog (
-Dstreammq.benchmark.backlog, default 50000) built before the measurement, whose build rate is only reported as a feeder reference metric; - no continuous refeeding during the measurement — only a low-watermark keep-alive feeder (
-Dstreammq.benchmark.feederThreads, default 2) so the consumer is the single bottleneck; - a validity gate: if the backlog runs dry at any point (an empty XREADGROUP), the run prints
INVALID RUNand the process exits non-zero (raise-Dstreammq.benchmark.backlog), otherwise it exits 0. Per-payload evidence lands intarget/consume-validity-<payload>.json.
| Benchmark | Description | 1KB | 10KB |
|---|---|---|---|
consumeThroughput |
Full path (network RTT, deserialization, callback, batched ACK) | ~12,572 | ~7,521 |
serializationRoundTrip |
Local Jackson round-trip (no Redis) | ~312,685 | ~21,813 |
messageCreateAndConsume |
Message build + callback dispatch (no Redis) | ~7,651,608 | ~8,047,889 |
Validity evidence for this run:
starvedReads=0,avgBatchSize=100.0,supplyTight=false,valid=truefor both payload sizes — the consumer never waited on an empty stream, and (same window) it consumed more than the feeder replenished (1KB 160,400 > 123,500; 10KB 88,900 > 52,000), the difference coming from the pre-loaded backlog. Under the old feeder-capped harness the same benchmark reported only ~2,383 / ~2,018 ops/s, so the rebuilt number is 5.3× / 3.7× higher — and it is still a lower bound of the SDK container path: it omits per-message ACK and the filter/interceptor/metrics chains, and the feeder shares the Redis instance during the measurement (conservative direction).
consumeThroughputmeasures the consume path, not an empty read roundtrip. Numbers vary significantly across hardware, Redis instances, and network latency.
Run yourself (same reproducible command as the benchmark report — run the JMH CLI directly so test-compile is not skipped and stale bytecode cannot be measured):
mvn -q -pl streammq-benchmark test-compile dependency:build-classpath \
-Dmdep.includeScope=test -Dmdep.outputFile=target/cp.txt
cd streammq-benchmark
java -Djmh.ignoreLock=true \
-cp "target/test-classes:target/classes:$(cat target/cp.txt)" \
org.openjdk.jmh.Main "SerializationBenchmark" \
-rf json -rff target/jmh-serialization.json(no -f/-wi/-i overrides: the class annotations are the declared profile — the table above was produced exactly this way,
plus the -Dstreammq.redis.mode=local -Dstreammq.benchmark.allowFlush=true properties when pointed at a non-container Redis.)
StreamMessageTemplateBenchmark runs the same way with org.openjdk.jmh.Main "StreamMessageTemplateBenchmark".
For the consumer benchmark prefer the class entry point (it enforces the validity gate; the raw JMH CLI bypasses it):
java -Djmh.ignoreLock=true \
-Dstreammq.benchmark.backlog=50000 -Dstreammq.benchmark.feederThreads=2 \
-cp "target/test-classes:target/classes:$(cat target/cp.txt)" \
io.github.streammq.benchmark.StreamConsumerBenchmark(the classpath separator is : on Linux/macOS and ; on Windows; add -Dstreammq.redis.mode=local -Dstreammq.benchmark.allowFlush=true to point at an existing Redis — the flushdb guard requires that explicit authorization for non-container targets, while a Testcontainers-owned instance is flushed without it).
Or trigger the CI benchmark job defined in
.github/workflows/benchmark.yml; its JMH artifacts are uploaded per run
and are what the report tables are back-filled from.
| Component | Minimum | Recommended |
|---|---|---|
| JDK | 21 | 21+ |
| Maven | 3.9 | 3.9+ |
| Redis | 7.2 | 7.2+ |
| Spring Boot | 3.3 – 3.5 | 3.5.16 |
⚠️ mvn verifyrequires a local Redis (localhost:6379). Without Redis, IT auto-skips; CI uses Docker service.⚠️ Build prerequisites: JDK 21+ is required (requireJavaVersion [21,)) and Maven 3.9+. The build runsspotless:checkat theverifyphase — runmvn spotless:applyfirst, or skip with-Dspotless.check.skip=true.⚠️ Default serializer isJacksonJsonSerializer(strict types, no gadget RCE surface). As of 0.1.2 the default flipped from Fury (unrestricted mode) to Jackson for safe-by-default publishing: a library's default deserializer must not expose an RCE surface to every downstream app. For higher throughput, opt in toFurySerializer(streammq.producer.serializer=io.github.streammq.adapter.redisson.serializer.FurySerializer, then pre-register your payload types (FurySerializerenforces the class-registration whitelist by default)), orProtostuffSerializer. Fory (org.apache.fory:fory-core, >= 1.1.0 — the underlying library is Apache Fory, formerly Apache Fury) and Protostuff areoptionaldependencies ofstreammq-redisson— add them to your classpath only when you use them, so Guava/Protostuff are not force-pulled into every app. See SECURITY.md.
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.github.streammq</groupId>
<artifactId>streammq-bom</artifactId>
<version>0.1.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.github.streammq</groupId>
<artifactId>streammq-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson-spring-boot-starter</artifactId>
</dependency>
</dependencies>
⚠️ You must also addredisson-spring-boot-starterto provide aRedissonClientBean; missing it will fail at startup withNoSuchBeanDefinitionException. See Minimum dependency matrix for which coordinates each usage path must supply itself.
spring:
application:
name: streammq-demo
# Redis connection: the Redisson Spring Boot Starter builds the client from spring.data.redis.*
data:
redis:
host: 127.0.0.1
port: 6379
database: 0
# password: ${REDIS_PASSWORD:} # uncomment when authentication is required
streammq:
enabled: true
namespace: streammqNote:
redisson.singleServerConfig.*has no property binding in this starter (it only bindsspring.redis.redisson.config/spring.redis.redisson.file), so writing it has no effect — do not use it. For Sentinel / master-replica and other advanced topologies, supply Redisson's native configuration throughspring.redis.redisson.config.
⚠️ Redis Cluster is NOT supported in 0.1.x — stated as measured behavior, not an inference. StreamMQ's data plane deliberately uses multi-key atomic operations (PEL claimXACK+XADD, transaction commitXRANGE+XADD+HSET, retry transfer batches, DLQ requeue). Those keys carry no{...}hash tag — the key family is partitioned by design so that a namespace cannot be pinned to a single slot (seeCHANGELOG.md→ "事务 key 结构与 Redis Cluster(hash tag)定型声明"). Measured on a real 3-master cluster (RedisClusterCompatibilityIT), which pins down both failure shapes:
- producing and basic consume/ACK (single-key
XADD/XREADGROUP/XACK) do work;- multi-key Lua fails deterministically with
CROSSSLOTand zero side effects (the message stays in the PEL/stream and is never falsely ACKed) — e.g.streammq:{ns}:msg:{topic}andstreammq:{ns}:retry:{topic}:{group}hash to different slots;- multi-key
REDIS_WRITE_ATOMICbatches (delayed enqueue/transfer, retry/DLQ scheduling and transfer, transaction metadata) are placement-dependent: keys owned by the same master but different slots are grouped into a singleMULTI/EXECthat the server rejects withCROSSSLOT; keys on different masters are split per node — every part commits, the batch reports success, and cross-key atomicity is silently gone (partial writes possible, no error raised).Because "silently degraded" is worse than "refused", StreamMQ 0.1.2 fails fast: with a cluster-configured client, every cross-key atomic path (delayed message enqueue/transfer, retry & DLQ scheduling/transfer, transaction prepare/commit, cross-stream PEL claim) throws an actionable
StreamMQExceptionbefore the first key is written, and the listener container and producer emit a one-time actionable WARN when a cluster topology is detected — so this surfaces at startup instead of as a bareCROSSSLOTin production. Scheduling state that already exists in the cluster is never dropped or half-written. Use a single instance or Sentinel / master-replica for now. Cluster support is tracked for a later release.
Just add the starter dependency — StreamMQ auto-configures all core Beans via META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports when the starter is on the classpath (and streammq.enabled=true, the default). No @Enable* annotation is required.
@SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}💡 The historical
@EnableStreamMQmarker annotation (empty, no@Import) did not trigger extra configuration and was removed in 0.1.2. Auto-configuration is standalone.
@Component
public class OrderService {
private final StreamMessageService messageService;
public OrderService(StreamMessageService messageService) {
this.messageService = messageService;
}
public SendResult sendOrder(String orderId, String content) {
return messageService.send(
"order-topic",
content,
MessageMetadataBuilder.create()
.tag("created")
.keys(orderId)
.userProperty("traceId", "t-001"));
}
}Power users: when you need access to interceptors / filters / SPIs, inject
StreamMessageTemplatedirectly (see Extension Points).
@Component
@StreamMQConsumer(topic = "order-topic", consumerGroup = "order-consumer-group")
public class OrderConsumer implements StreamMessageConcurrentlyConsumer<String> {
@Override
public ConsumeAction onMessage(Message<String> message, ConsumeContext context) {
System.out.println("Received order: " + message.getKeys() + ", content: " + message.getBody());
return ConsumeAction.SUCCESS;
}
}That's it! Start the app, send a message, the consumer will pick it up and process it.
StreamMQ 0.1.x is designed for these Redis topologies:
| Topology | Supported | Notes |
|---|---|---|
| Single instance | ✅ | Default; everything in this README applies |
| Master-replica | ✅ | Reads/writes go to the master via Redisson |
| Sentinel | ✅ | Provide Redisson's native config through spring.redis.redisson.config |
| Redis Cluster | ❌ in 0.1.x | See below — fails fast, never silently degrades |
Redis Cluster is not supported in 0.1.x because the data plane depends on cross-key atomicity
(multi-key Lua for PEL claim and transaction commit, REDIS_WRITE_ATOMIC batches for delayed/retry/DLQ
scheduling). StreamMQ key families deliberately carry no {...} hash tag, so a multi-key request is either
rejected with CROSSSLOT or split per node with the atomicity silently lost. Rather than let that happen:
- the producer and listener container run a one-time topology check at startup and log an actionable
WARNwhen a cluster is detected (RedisClusterCompatibility.warnIfCluster); - every cross-key atomic path (delayed enqueue/transfer, retry & DLQ scheduling/transfer, transaction
prepare/commit, cross-stream PEL claim) throws an actionable
StreamMQExceptionbefore writing the first key (RedisClusterCompatibility.requireCrossKeyAtomicity). Producing and basic consume/ACK keep working; scheduling state already stored in the cluster is never dropped or half-written.
Details, the measured failure matrix and the raw-cluster evidence: docs/REPORT.md
("Redis Cluster 实测") and CHANGELOG.md ("事务 key 结构与 Redis Cluster(hash tag)定型声明").
Cluster support requires a hash-tag key redesign and is tracked for a later release.
One-line annotation, declaratively defines the consumer; supports concurrent, ordered, broadcast, and DLQ consumption models.
// Concurrent (default)
@StreamMQConsumer(topic = "order-topic", consumerGroup = "order-group")
// Ordered
@StreamMQConsumer(topic = "order-topic", consumerGroup = "order-group",
messageModel = MessageModel.ORDERLY, shardCount = 8)
// Broadcast
@StreamMQConsumer(topic = "order-topic", consumerGroup = "order-group",
consumeMode = ConsumeMode.BROADCASTING)
// DLQ
@StreamMQConsumer(topic = "order-topic", consumerGroup = "order-group", dlqMode = true)
⚠️ Broadcast consumption creates a separate Redis consumer group per container instance, and the group name is derived from a persistent instance identity (0.1.2+). With a persistent identity the group name is stable across typical restarts and the PEL is preserved; only the random-UUID fallback (Redis and the local identity file both unavailable) produces a new group per restart. Before using broadcast mode in production, read Broadcast Consumption — Operational Notes.
StreamMQ provides two send APIs: StreamMessageService (facade) and StreamMessageTemplate (full SPI). Most users should inject StreamMessageService — it has simpler ergonomics for topic + body + metadata and delegates to the template underneath.
StreamMessageService covers all send patterns with three orthogonal dimensions: send mode × carrier form (full Message or topic+body+MessageMetadataBuilder) × parameters (SendOptions / metadata-inlined timeout/retry).
Half message + local transaction + check-back mechanism; final consistency guaranteed.
TransactionCallback<String> callback = (message, ctx) -> {
try {
executeLocalTransaction(message.getBody());
return LocalTransactionState.COMMIT_MESSAGE;
} catch (Exception e) {
return LocalTransactionState.ROLLBACK_MESSAGE;
}
};
SendResult result = template.executeInTransaction(message, callback);Built-in 18 fixed delay levels + arbitrary millisecond delays.
Message<String> msg = MessageBuilder.<String>withTopic("delay-topic")
.body("content")
.delayLevel(DelayLevel.MINUTE_5)
.build();
⚠️ Delay ceiling: 7 days.delayTimeMillismust not exceedStreamMQConstants.MAX_DELAY_TIME_MILLIS(7 days); a larger value fails fast at send time (StreamMQException), it is not silently truncated. The 18 fixed levels are far below this ceiling (maxHOUR_2= 2h); arbitrary millisecond delays are bounded by the same 7-day ceiling.
ShardingKey-based sharded ordered consumption; strict ordering within a shard.
BatchMessage batch delivery, uses Redis Pipeline for throughput.
Failed messages after retry exhaustion auto-enter DLQ; supports secondary DLQ + custom failure strategies.
Tag expression filtering and SQL92 expression filtering.
GZIP via CompressionCodec SPI; auto-compresses when payload exceeds threshold.
Read this before using ConsumeMode.BROADCASTING.
Redis consumer groups are inherently "competing consumers within a group". To implement broadcast
(every instance receives every message), StreamMQ gives each container instance its own Redis
consumer group identified by a persistent instance token (BroadcastInstanceIdResolver,
introduced in 0.1.2). The token is resolved via a five-level fallback chain:
- Explicit configuration:
streammq.consumer.broadcast-instance-id(or system propertystreammq.instance.id/ environment variableSTREAMMQ_INSTANCE_ID) - Local persistent file:
${user.home}/.streammq/instance-id-<namespace>_<group>— scoped per application/consumer group (multiple StreamMQ apps under the same OS user never share an identity file). The file stores one record per process (id pid timestamp), so a restart reuses the identity of an exited process and never overwrites the identity of a still-running one. Path overridable viastreammq.consumer.broadcast-instance-id-file(a single file for all consumers;none/falsedisables the local file entirely) - Redis broadcast registry reclaiming this host's previous slot (persists across restarts, keeps the PEL)
- Redis broadcast registry allocating a fresh slot for a brand-new instance
- Random UUID (last resort, non-persistent — the pre-0.1.2 behaviour)
This means the broadcast group name is stable across typical restarts when any of the persistent identity sources (1–4) are configured or available. However, if only the random fallback applies, the old behaviour applies:
- With persistent identity: restarted instances resume the same consumer group → PEL is
preserved, messages produced during downtime are delivered on reconnect, no new group created.
Stale groups still exist only when instances are decommissioned permanently (slot reclaimed and
eventually swept after the reclaim grace period, default
streammq.consumer.broadcast-reclaim-grace= 7d). - Without persistent identity (random UUID): every restart creates a new group; the abandoned
groups (a group is "stale" once its heartbeat stops) are destroyed by
RedissonBroadcastGroupRegistryafter a fixed 10-minute TTL (BROADCAST_GROUP_STALE_TTL_MS, not configurable) — identity slots, which carry the PEL, are reclaimed only after the 7-day reclaim grace. Total group count ≈ instance-count × restarts within that 10-minute window. - Each group holds its own PEL and occupies Redis memory.
- Configure a persistent identity source in production (explicit
broadcast-instance-id, a writable local file, or a reachable Redis at startup) — this ensures restart-safe offset persistence; only a random UUID fallback produces a new group per restart. - Do not use broadcast mode for workloads that restart frequently (CI environments, Pods that repeatedly OOM) when relying on the UUID fallback.
- Alert on
broadcastGroups: investigate above "instance count × 3". - Broadcast groups without persistent identity cannot resume a previous consumption offset — after a UUID-fallback restart the new group starts from the current point in time.
A consumer that "registers successfully but never consumes" is the symptom most often mistaken for "messages are being lost". Work through these in order:
- Check health:
GET /actuator/health→ thestreammqcomponent. If any consume loop failed to start, it reportsDOWNwithlistenerContainer.consumeLoopFailures(loopKey → reason) in the details. - Check the overview:
GET /actuator/streammq→ thestatusfield reflects the same state. - Check the logs: the
Failed to create consumer for listenerERROR line carries topic/group and the root cause — most commonly wrong Redis credentials, an illegal consumer group name, or a namespace mismatch. - Check container state: the
containerRunningfield in/actuator/streammq/groups.
A failed consume loop is not retried automatically, and the rebalance endpoint cannot recover it:
POST /actuator/streammq/rebalance/{group}only re-assigns shards for already-running ORDERLY containers (StreamMQAdminEndpoint#triggerRebalance→container.rebalanceGroup(group)), so it does nothing when the consume loop never started. After fixing the root cause you must restart the application.
⚠️ In 0.1.x onlystreammq-bom,streammq-core,streammq-redisson,streammq-spring-boot-starter(plusstreammq-test) are published to Maven Central. Rows marked source-only are compiled and tested in this repository but cannot be resolved from Central — build them from source (mvn install) if you need them.
| Module | Description |
|---|---|
| streammq-bom | Bill of Materials, unified version management |
| streammq-core | Core abstractions, message model, API, SPI interfaces (no Spring dependency) |
| streammq-redisson | Redisson adapter, implements core capabilities on Redis Stream |
| streammq-spring-boot-starter | Spring Boot 3 auto-configuration, configuration binding, Actuator integration |
| streammq-tracing-opentelemetry | OpenTelemetry tracing integration — source-only in 0.1.x |
| streammq-diagnostics | Message profiling, slow-consume, backlog, DLQ diagnostics — source-only in 0.1.x |
| streammq-kubernetes | K8s health checks, HPA, graceful shutdown, CRD operator (experimental, default off) — source-only in 0.1.x |
| streammq-spring-cloud-stream-binder | Spring Cloud Stream Binder implementation (partitioned production is not supported and fails fast at startup) — source-only in 0.1.x |
| streammq-benchmark | JMH benchmarks — source-only in 0.1.x |
| streammq-test | Test utilities: containerized Redis (Testcontainers, requires a Docker daemon), Redis availability probe, assertions, mocks. Import with test scope |
| streammq-samples | Sample projects covering all features — source-only in 0.1.x |
streammq:
enabled: true
namespace: streammq
producer:
group: default-producer
send-message-timeout: 3000
retry-times: 2
compress-threshold: 0
consumer:
batch-size: 32
pull-interval: 0
inflight-capacity: 0
retry:
max-reconsume-times: 16
transaction:
check-interval: 60s
max-check-times: 15
dlq:
max-dlq-retry-attempts: 3
admin:
enabled: true
tracing:
enabled: falsestreammq.diagnostics.enabled(defaultfalse) — turns on thestreammq-diagnosticsmodule (message profiling, slow-consume, backlog and DLQ diagnostics). It lives in a separate module: you must addstreammq-diagnosticsto the classpath and set the key totrue; aStreamMQTraceServicebean is also required for the analyzers to bind (enable it viastreammq.trace.enabled=true+streammq.trace.storage=redis; without a Redisson client the backlog probe silently falls back to trace-window estimation). If the module is not on the classpath, Spring silently ignoresstreammq.diagnostics.*— no warning, no failure: nothing binds the key (an unknown property is ignored by default). The same applies tostreammq.tracing.otel.*(needsstreammq-tracing-opentelemetry) and to any other module-scoped key. Troubleshooting when the key seems to have no effect: confirm the coordinate is really on the classpath (mvn dependency:tree | grep streammq-diagnostics) and thatenabled=truetook effect (the auto-configuration is gated by@ConditionalOnProperty(..., matchIfMissing=false)).@StreamMQConsumer(enable = false)(annotation attribute, defaulttrue) — the consumer is skipped entirely at registration time:StreamMQListenerRegistrarlogsSkip disabled @StreamMQConsumer: bean=..., topic=..., no listener/consume loop is created, and the(topic, group)registration key is not occupied (so another bean may claim it meanwhile). The consumer Bean itself is still created by Spring. Useful for gating a consumer behind an environment/flag (e.g. feature toggle) without removing the class. It is per-consumer only; there is no global counterpart. (The same attribute exists on@StreamMQDlqConsumer.)
0.1.2 ships with 18 extension points (user-facing + internal assembly points):
| Extension Point | Purpose | Default Implementation |
|---|---|---|
MessageSerializer |
Message serialization | JacksonJsonSerializer (default, strict types / no gadget RCE surface) / FurySerializer (opt-in, high throughput) / ProtostuffSerializer / FlatBuffersSerializer (FlexBuffers, zero-copy field reads) / SbeSerializer (fixed-header envelope) / JdkSerializer / StringSerializer / ByteArraySerializer |
MessageConverter |
Message-body ↔ business object | DefaultMessageConverter / CompactMessageConverter / PassThroughMessageConverter |
ProducerFilter |
Producer filter chain | none built-in — implement it and register a Bean |
ConsumerFilter |
Consumer filter chain | TagSelectorFilter / SqlSelectorFilter |
ProducerInterceptor |
Producer interceptor chain | TraceContextProducerInterceptor / OpenTelemetryProducerInterceptor |
ConsumerInterceptor |
Consumer interceptor chain | TraceContextConsumerInterceptor / OpenTelemetryConsumerInterceptor |
RetryPolicy |
Retry strategy | FixedArrayRetryPolicy / FixedIntervalRetryPolicy / ExponentialBackoffRetryPolicy / NoRetryPolicy |
RebalanceStrategy |
Consumer rebalance strategy | AverageRebalanceStrategy / ConsistentHashRebalanceStrategy / RangeRebalanceStrategy |
CompressionCodec |
Message compression | GzipCompressionCodec / Lz4CompressionCodec (classpath detection) |
TraceCollector |
Trace context collection | NoopTraceCollector / Slf4jTraceCollector / RedisTraceCollector |
ManagementAuthenticator |
Admin endpoint authentication | AllowAllAuthenticator / BasicAuthAuthenticator / TokenAuthenticator / DenyAllAuthenticator |
DlqFailureStrategy |
DLQ failure handling | LogAndDropDlqFailureStrategy / LimitedRetryDlqFailureStrategy / SecondaryDlqFailureStrategy |
ExpressionSelectorFilter |
Message filtering | TagSelectorFilter / SqlSelectorFilter |
ConsumerFilterResolver |
Per-consumer filter resolver | ReflectiveConsumerFilterResolver (default) / Spring container resolver |
OrderlyShardLockManager |
Ordered-consume shard distributed lock | RedissonOrderlyShardLockManager |
ConsumerGroupManager |
Consumer group instance management | RedissonConsumerGroupManager |
BroadcastInstanceRegistry |
Broadcast instance identity registry (stable slot + reclaim/lease, 0.1.2) | RedisBroadcastInstanceRegistry |
CompressionCodecRegistry |
Codec registry (register/lookup by name; consumers resolve the codec to decompress with) | DefaultCompressionCodecRegistry (GZIP built in, LZ4 by classpath detection) |
streammq-core and streammq-redisson declare their adapters/optional libraries as provided / optional
on purpose, so you can pick the client version. That also means you must supply some coordinates yourself:
| Path | Coordinates you must add yourself | Scope facts (verified in the module POMs) |
|---|---|---|
| core-only (framework-agnostic use) | org.slf4j:slf4j-api |
streammq-core declares SLF4J as provided (the logging facade is not shipped transitively); Jackson is optional in core and core's runtime path does not use it |
| redisson-only (no Spring) | org.redisson:redisson (add the Jackson/SLF4J coordinates explicitly only if you override or exclude them) |
redisson is provided; org.slf4j:slf4j-api, com.fasterxml.jackson.core:jackson-databind and com.fasterxml.jackson.datatype:jackson-datatype-jsr310 are compile dependencies of streammq-redisson and therefore arrive transitively (the two Jackson artifacts back the default JacksonJsonSerializer and the props field codec). Serializers that are optional are needed only when you opt in: org.apache.fory:fory-core (>= 1.1.0) / io.protostuff:protostuff-core+protostuff-runtime / com.google.flatbuffers:flatbuffers-java / org.agrona:agrona |
| starter (Spring Boot 3) | org.redisson:redisson-spring-boot-starter (its version is managed by streammq-bom) + your own Spring Boot parent/BOM |
the starter declares redisson-spring-boot-starter and spring-boot-starter as provided; spring-boot-starter-actuator and micrometer-core are optional (add them for health checks/metrics); spring-boot-autoconfigure is a regular compile dependency |
Optional-but-recommended extras: io.micrometer:micrometer-registry-prometheus for /actuator/prometheus,
and streammq-diagnostics / streammq-tracing-opentelemetry (source-only in 0.1.x) when you want their endpoints.
| Metric | Type | Description |
|---|---|---|
streammq.send.total |
Counter | Send total (tag: success) |
streammq.send.duration |
Timer | Send latency |
streammq.consume.total |
Counter | Consume total |
streammq.consume.duration |
Timer | Consume latency |
streammq.retry.total |
Counter | Retry count |
streammq.dlq.total |
Counter | DLQ count |
streammq.delay.total |
Counter | Delayed delivery count |
streammq.transaction.commit.total |
Counter | Transaction commit count |
streammq.transaction.rollback.total |
Counter | Transaction rollback count |
streammq.transaction.check.total |
Counter | Transaction check count |
| Endpoint | Description |
|---|---|
/actuator/health |
Health check (incl. StreamMQ component status) |
/actuator/metrics |
Micrometer metrics |
/actuator/prometheus |
Prometheus format — requires io.micrometer:micrometer-registry-prometheus on the classpath (not a StreamMQ dependency; the starter only brings micrometer-core). Add it yourself or the endpoint returns 404 |
Exposure note:
/actuator/streammq(and the management REST API below) is a Spring Boot@WebEndpoint. Spring Boot only exposeshealthandinfoby default, so without the setting below the endpoint returns 404. Explicitly include it:management: endpoints: web: exposure: include: streammq
All under /actuator/streammq, dispatched by HTTP method + path segment:
| Endpoint | Method | Description |
|---|---|---|
/actuator/streammq |
GET | Overview (status, consumer groups, topics) |
/actuator/streammq/groups |
GET | List consumer groups |
/actuator/streammq/topics |
GET | List topics |
/actuator/streammq/pending/{group}/{topic} |
GET | Pending messages |
/actuator/streammq/dlq/{group} |
GET | DLQ messages |
/actuator/streammq/dlq/{group}?messageId&targetTopic |
POST | Requeue a DLQ message |
/actuator/streammq/dlq/{group}/{messageId} |
DELETE | Delete a DLQ message (confirm={messageId} required) |
/actuator/streammq/stats/{group}/{topic} |
GET | Runtime stats |
/actuator/streammq/ack/{group}/{topic}?messageId |
POST | Manual ACK |
/actuator/streammq/rebalance/{group} |
POST | Trigger rebalance |
/actuator/streammq/topics?topic= |
POST | Create a topic |
/actuator/streammq/topics/{topic} |
DELETE | Delete a topic (confirm={topic} required) |
/actuator/streammq/config/{group} |
POST | Update consumer-group config at runtime |
All require ManagementAuthenticator. Default is DenyAllAuthenticator (rejects everything, returns 401). Register AllowAllAuthenticator / BasicAuthAuthenticator / TokenAuthenticator Bean to open access. Authentication failures are rate-limited (10 failures per 60s → 5-minute lockout per client), and clients are aggregated by the non-spoofable remoteAddr by default. X-Forwarded-For is not trusted unless you opt in with streammq.admin.trust-forwarded-headers=true and declare streammq.admin.trusted-proxies (IPv4/IPv6 CIDR, e.g. 10.0.0.0/8; loopback is trusted by default) — only a direct peer that matches the list may contribute the first XFF value, because an unvalidated XFF lets a single header bypass the lockout. Two more exposure caveats:
/streammq/diagnostics/**(thestreammq-diagnosticsmodule's plain MVC endpoints, mounted on the application's main port) is not governed bymanagement.endpoints.web.exposure.*— once the module is on the classpath and enabled, those endpoints stay reachable even if Actuator only exposeshealth.- If JMX exposure is on, exclude the StreamMQ endpoints from it (
management.endpoints.jmx.exposure.exclude: streammq).Restrict both endpoint families by network layer (security group / Ingress) and see SECURITY.md for the full policy.
| Sample | Description |
|---|---|
| streammq-sample-quickstart | Quick start |
| streammq-sample-transaction | Transactional messages |
| streammq-sample-delay | Delayed messages |
| streammq-sample-orderly | Ordered messages |
| streammq-sample-dlq | Dead-letter queue |
| streammq-sample-interceptor | Interceptors |
| streammq-sample-diagnostics | Diagnostics & slow-consume |
| streammq-sample-tracing | OpenTelemetry tracing |
| Document | Description |
|---|---|
| This README | Authoritative user manual |
| Configuration Reference | Every streammq.* key with its real default value |
| Javadoc | Bundled with Maven Central artifacts (sources/javadoc jars) |
| CHANGELOG | Version change log |
| CONTRIBUTING | Contribution process & dev conventions |
| SECURITY | Security policy & vulnerability disclosure |
⚠️ docs/archived-historical/contains V0.1/V1.0 draft design documents; their class names, configuration keys, and some mechanism descriptions have become outdated. Treat as archaeology only.
Note: The following features are available in the 0.1.x releases. The current version is 0.1.2 (feature preview). APIs may still evolve before 1.0.0 based on community feedback. Production use is recommended after gray-scale validation on non-critical paths.
- Annotation-driven consumption (
@StreamMQConsumer) -
StreamMessageTemplateprogramming model (sync/async/oneway/batch/tx) - Cluster + broadcast consumption (broadcast supports persistent instance identity)
- Ordered messages (ShardingKey)
- Transactional messages (half + check)
- Delayed messages (18 levels + any ms)
- Dead-letter queue (incl. secondary DLQ)
- Tag + SQL92 message filtering
- GZIP message compression
- Backpressure control (InflightQueue)
- Consume-timeout auto-cancel
- Micrometer metrics + MDC logging
- TraceCollector SPI
- Management REST API
- 18 extension points (user-facing + internal assembly points)
- Spring Boot 3 auto-config + Actuator
- Spring Cloud Stream Binder (source-only in 0.1.x)
- Kubernetes integration (experimental; source-only in 0.1.x)
- Message profiling & topology visualization (source-only in 0.1.x)
- OpenTelemetry distributed tracing (source-only in 0.1.x)
- Multi-backend abstraction layer (BackendProvider SPI for Redis / Kafka / RabbitMQ / Pulsar)
- Kafka backend implementation (BackendProvider on Kafka Client)
- Cross-datacenter replication (async, RPO ≤ 1s)
- Kafka wire-protocol compatibility (zero-code migration for native Kafka Client)
Welcome to StreamMQ! Please read CONTRIBUTING for details.
# 1. Fork & clone
git clone https://github.com/HK-hub/StreamMQ.git
cd streammq
# 2. Create branch
git checkout -b feature/your-feature
# 3. Code & test
mvn clean test
# 4. Commit (Conventional Commits)
git commit -m "feat: add your feature"
# 5. Open PR- Already have Redis, want to reuse it as a message bus
- Medium/small business (< 100M msgs/day per cluster)
- Need transactional/delayed/ordered messages but don't want a separate MQ cluster
- Lightweight async communication in Spring Boot 3 microservices
- E-commerce order state, payment callbacks, inventory deduction, notification push
- Very large streaming pipelines (> 1M TPS sustained) — use Kafka
- Financial-grade strict ACID transactions — use RocketMQ transactional messages
- 100M+ message accumulation — use Kafka (disk storage)
- Multi-datacenter deployment — use RocketMQ Cluster / Pulsar Geo-Replication
- Embedded IoT devices — use MQTT broker
| Tech | Version | Purpose |
|---|---|---|
| Java | 21+ | Runtime |
| Spring Boot | 3.5.16 | Framework |
| Redisson | 3.52.0 | Redis client |
| Jackson | 2.21.4 | JSON serialization (default serializer; upgraded from 2.17.2 for GHSA-r7wm-3cxj-wff9 / GHSA-72hv-8253-57qq — the jackson-bom import is declared before the Spring Boot BOM so Boot's managed version cannot override it) |
| Apache Fory (formerly Apache Fury) | 1.7.3 | High-perf serialization (optional; opt-in for throughput; org.apache.fory:fory-core, required >= 1.1.0 — CVE-2026-50076) |
| Protostuff | 1.8.0 | Protobuf serialization (optional alternative) |
| FlatBuffers | 24.3.25 | FlexBuffers dynamic binary serialization (optional; com.google.flatbuffers:flatbuffers-java, for FlatBuffersSerializer) |
| Agrona | 1.17.1 | Runtime library for SBE generated code (optional; org.agrona:agrona) |
| SBE (sbe-tool) | 1.18.0 | Build-time code generation for the SBE envelope (uk.co.real-logic:sbe-tool, runs at generate-sources) |
| Lombok | - | Code simplification |
| Micrometer | - | Metrics |
| SLF4J | - | Logging facade |
StreamMQ takes your security seriously. Best practices:
- No credentials in logs: StreamMQ never outputs Redis passwords to logs; recommend injecting via environment variables.
- Configuration security: Never hardcode Redis passwords. Use environment variables, config center (Nacos/Apollo), or secret management (Vault/AWS Secrets Manager).
- Least privilege: Redis accounts should have minimum necessary permissions.
FurySerializer(opt-in, high throughput;streammq.producer.serializermust be explicitly set; underlying library Apache Fory (formerly Apache Fury) 1.7.3, Maven coordinatesorg.apache.fory:fory-core, >= 1.1.0 required — earlier versions are affected by CVE-2026-50076) — enforces class registration by default (requireClassRegistration=true): only explicitly registered types can be deserialized and unregistered POJOs are rejected. Pre-register your payload types:Register application payloads up front withstreammq: producer: fury-require-class-registration: true fury-registered-classes: com.acme.OrderCreated,com.acme.Payment
new FurySerializer<>(OrderCreated.class)ornew FurySerializer().register(OrderCreated.class). From plain Java:new FurySerializer()is the whitelist mode (same asnew FurySerializer(true));new FurySerializer(false)is the unrestricted mode and requires-Dstreammq.security.allowUnrestrictedSerializer=true(otherwise it throwsSecurityException; with the property set it still logs a WARN). Note the post-deserializeisInstancecheck only validates the result type — it cannot stop gadget execution, so the whitelist is the real defense.- When the default serializer (
JacksonJsonSerializer, nested-object safe) is in effect, the underlying Redisson client codec must still be secured separately — see SECURITY.md for the full security surface. JdkSerializerhas a JEP 290 class name whitelist filter (target type + JDK basics); useJdkSerializer.unrestricted()only as a last resort — gated by-Dstreammq.security.allowUnrestrictedSerializer=true.
For shared/multi-tenant Redis, keep the FurySerializer class-registration whitelist enabled (it is the default). See SECURITY.md for the full security policy.
This project is licensed under the MIT License.
StreamMQ · Turn Redis into your message bus
If this project helps you, please give it a ⭐ Star!