[labrenbe]

Part of stackabletech/issues#888.

Stacked on #1045.

Description

Adds the KafkaTopic custom resource, which the agent reconciles into topics of its KafkaCluster.

• New KafkaTopic CRD, declaring a topic by name, partitions, replication factor and topic-level configs for a KafkaCluster in the same namespace.
• The agent creates missing topics, adopts existing ones, replaces their topic-level configs with the declared ones, and increases partitions. Changes Kafka cannot apply, fewer partitions or another replication factor, are reported instead of applied.
• A deletion policy decides whether deleting the KafkaTopic deletes the topic in Kafka. With Delete, a finalizer holds the KafkaTopic until the topic is gone.
• The status shows the topic as it is in Kafka, with Ready and Synced conditions that explain why a sync did not happen: an invalid spec, a request Kafka rejected, a lost connection or a paused reconciliation.
• When several KafkaTopics declare the same topic, the oldest one manages it and the others report a conflict. The reconciliation of a single KafkaTopic can be paused.
• The operator takes over when there is no agent: it reports this in the conditions and removes the finalizers once the KafkaCluster is deleted, so the deletion of a cluster or a namespace does not get stuck. The operator's readiness now also waits for the KafkaTopic CRD.
• Documentation for management access and topics, the changelog entries for the whole stack, and kuttl tests for the topic lifecycle and the authentication methods of the agent.

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

Release Notes

Highlight: Declarative management of Apache Kafka topics with the new KafkaTopic custom resource.

New platform features, Apache Kafka:

• The Stackable Data Platform can now manage resources inside an Apache Kafka cluster.
  When spec.clusterConfig.managementAccess is enabled on a KafkaCluster, the operator deploys an agent next to the cluster, which authenticates to Kafka with a TLS client certificate that the user provides through a SecretClass or a Secret.
  This is supported for clusters in KRaft mode without Kerberos authentication.
  Consult the management access documentation page for more details.
  Implemented in kafka-operator#1028.
• Topics can now be managed declaratively with the new KafkaTopic custom resource.
  The agent creates, adopts, changes and optionally deletes the topic in Kafka, and reports changes Kafka cannot apply in the status.
  Consult the topics documentation page for more details.
  Implemented in kafka-operator#1045 and kafka-operator#1050.

[sbernauer]

I very much like the extensive kuttl topic test! I only reviewed that.

One problem I noticed: I often times do --skip-delete to end up with a working example I can play around with. This PR always deletes everything at the end. Maybe you can ask AI if we can easily add something like stackabletech/airflow-operator#841 here as well. If not possible, I think this case is ok, as there is much deletion testing going on.

[labrenbe]

I very much like the extensive kuttl topic test! I only reviewed that.

One problem I noticed: I often times do --skip-delete to end up with a working example I can play around with. This PR always deletes everything at the end. Maybe you can ask AI if we can easily add something like stackabletech/airflow-operator#841 here as well. If not possible, I think this case is ok, as there is much deletion testing going on.

Added something similar in 45d37f3.

[sbernauer]

Thanks!