Skip to content

feat: create topics on a running environment - #11

Closed
MetallFoX wants to merge 1 commit into
mainfrom
feature/embedded-kafka-topics
Closed

MetallFoX wants to merge 1 commit into
mainfrom
feature/embedded-kafka-topics

Conversation

@MetallFoX

Copy link
Copy Markdown
Collaborator

Topic creation is the environment's job, and the declared case already works: EmbeddedKafkaSystem("topic-a", "topic-b") passes the topics to the broker, so they exist before anything connects.

What was missing is the same guarantee for a topic that shows up later, and leaving it to the broker's on-demand creation does not provide it. Measured on the embedded KRaft broker: the topic is created, but it is absent from the response to the first partitionsFor call — that call is what triggers the creation. A client that assigns partitions right after connecting therefore keeps an empty assignment, and every later poll or seek fails with:

IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions
IllegalStateException: No current assignment for partition <topic>-0

The embedded ZK broker used to win that race, so specs that never declared their topics worked by accident; on KRaft they die in class initialisation, before a single assertion.

addTopics(vararg String) waits for the creation to finish and fails with the broker's own error if it cannot, so a spec can add a topic mid-flight and use it at once. topics() exposes what the environment holds, which is what a spec asserts against.

Verified twice: a test here adds a topic to a running environment and checks both topics() and AdminClient.listTopics; and downstream in the platform, an Exam KafkaConsumeAndSendTester on a topic that the environment did not declare fails without this change and passes with it.

Declared topics were already created before anything connects — the convenience
constructor passes them to the broker. What was missing is the same guarantee for a topic
that appears later, and relying on the broker to create it on demand does not give it: the
broker does create the topic, but not in the response to the first metadata request, so a
client that assigns partitions right after connecting is left with an empty assignment and
every later poll or seek throws.

addTopics waits for the creation to complete and fails loudly with the broker's own error,
so a spec can add a topic mid-flight and use it immediately. topics() exposes what the
environment has, which is what a spec asserts against.

Creating topics belongs to the environment, not to whoever consumes it — that is why this
lands here rather than in a wait loop inside the Exam tester.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@MetallFoX
MetallFoX force-pushed the feature/embedded-kafka-topics branch from 368e911 to 36cd2c4 Compare August 13, 2026 08:46
@MetallFoX

Copy link
Copy Markdown
Collaborator Author

Closing: not needed, and adding it would be the wrong shape.

The declared path already covers the contract — EmbeddedKafkaSystem("topic-a", "topic-b") creates the topics before anything connects. A second entry point for adding topics to a running environment invites specs to depend on order of operations instead of declaring what they need up front, and it does not fix the case it looks like it fixes: a topic nobody declared still is not there when a client asks for its metadata the first time.

So the rule stays simple and is the one we document for consumers: declare the topics your specs use when you build the environment. Auto-creation by the broker is not a supported path for specs — it worked on the ZK broker by winning a race, and KRaft does not.

@MetallFoX MetallFoX closed this Aug 13, 2026
@MetallFoX
MetallFoX deleted the feature/embedded-kafka-topics branch August 13, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant