feat: system test framework proposal#111
Conversation
Introduces layered abstractions (ProxyScenario, FilterSpec, ProxyFixture, ProxyHandle) that separate test intent from deployment mechanism, enabling deployment-agnostic feature tests and test-first development. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
OperatorCapability now models the operator's externally observable state via generation-based reconciliation observation rather than the checksum annotation. Tests that assert on specific operator mechanisms (e.g. checksum change detection) observe resource state directly. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
OperatorCapability handles convergence and deployment agnosticism. Tests that assert on specific resource state (e.g. checksum annotations) use an injected KubernetesClient to observe resources directly, keeping the two concerns separate. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Three distinct test categories become separate modules with different compile-time dependencies: systemtest-feature (no K8s dependency), systemtest-operator (K8s client and CRD types), and systemtest-installer (one test per install method). All three are TCK-consumable. Installer is a public interface — the primary downstream extension point. CrdProxyFixture replaces OperatorProxyFixture to reflect that the fixture uses the CRD API, not the operator's internals. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Fixture/installer composition is extension-internal, driven by system properties. Installer tests use a single smoke test — the test does not vary between installers, CI provides the matrix. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Summary now mentions all four test categories and the Installer - "Three categories/modules" → "Four" throughout - Add systemtest-webhook to TCK module list with description - Feature test example: remove namespace parameter, use kafkaClient - Tags section: clarify module boundaries as primary separation - Rejected alternatives: @operator tag "ensures present", not "selects fixture" - Webhook section: rewrite as two-module story (installer + behaviour) - Affected projects: add systemtest-webhook module - Fix grammar: "An CrdProxyFixture" → "A CrdProxyFixture" Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
|
|
||
| 2. **No convergence contract**: the framework does not define when the proxy is ready. Each test class independently polls for readiness, with varying strategies and varying reliability. | ||
|
|
||
| 3. **Operator coupling**: every test implicitly requires the operator. Feature tests — which care only that a correctly-configured proxy is serving traffic — cannot run without the full operator installation. This conflates feature correctness with operator correctness and prevents fast local iteration. |
There was a problem hiding this comment.
The purpose of a System Tests is having all installed as you want on a real environment. We decided to go only for Kubernetes installation as we didn't have enough resources and time to cover everything. Ideally, we should have all kind of installations, such as bare-metal, Kubernetes...
Totally agree with system tests decoupling with the installation. The test cases should not care about how Kroxylicious has been installed, just check what is needed. Having a quicker installation will easy the Test-first development as we don't need to depend on the operator.
|
|
||
| A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. An `Installer` — the primary downstream extension point — handles getting the operator, CRDs, and RBAC into the cluster independently of how proxies are deployed. | ||
|
|
||
| Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. |
There was a problem hiding this comment.
Aside: I'd like to get Kroxylicious into Operatorhub soon. So having test that can are agnostic to their install methodology will be an enabler.
| These move to a `KubernetesClientCapability`: | ||
|
|
||
| ```java | ||
| interface KafkaClient { |
There was a problem hiding this comment.
One thing I dislike about the system tests is that all our client interactions are via a client CLI spawned in a separate process.
CLIs expose pretty basic operators: they boil down to produce or fetch from a topic. Consumer groups are possible but testing things like produce transactions + commit offsets is pretty much impossible. This will become a real impediment when we come to test things like Router implementations.
I think we need to be able to program the KafkaClient from the test so we are freer to express the fully richness of the Kafka.
A secondary issue is that relying on a CLI is slow: you pay the process startup time.
I have had an itch to scratch for a while which I think I've expressed to you both. Could we expose a common Java interface for Librdkafka and Sarama using Java Foreign Functions? We could even partially implement the Kafka Consumer, Producer and Admins client interfaces. The system tests could then be written in term of the Java Kafka API and exercise java/go/librdkafka using the same test.
Now we've got AI assistance I think writing a POC for this is in reach.
There was a problem hiding this comment.
Agreed — the CLI-based approach is too limiting, especially as we get into transactions, consumer groups, and richer Kafka interactions. The KafkaClient interface in the proposal is intended as the seam where a better client strategy plugs in, but I think working out what that strategy is (FFI, direct Java clients, something else) is a separate concern from the framework layering here. Happy to pick it up as follow-on work once this lands.
|
I like the ideas you are expressing in the proposal. I think it is the right way to go. |
Replace "delegated to QE" with community-neutral language and frame setup cost as a perceived barrier rather than a statement of fact. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
The same separation of concerns that motivates the fixture model applies to how tests interact with Kafka. Three independent axes: test intent (what), client driver (which implementation), and execution environment (where). The KafkaClient interface shows a richer target shape including transactions and consumer group management, enabled by in-process drivers (Java client, librdkafka/Sarama via FFI). CLI drivers implement produce/consume only. Produce and consume are the starting point; richer operations are the target. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
henryZrncik
left a comment
henryZrncik
left a comment
There was a problem hiding this comment.
Looks Great to me.
I think most of the proposed solution suits perfectly. I think to to gret extent current situation of System tests addressed mainly testing assuming operator and subsequently using resource manager mostly on test level instead of inside function calls. Because there is also mention of different enviroments (minikube, ocp etc) we need to also address the need to make http calls and client calls regardless of acessing local cluster or remote, but this represents minimal changes that are to be part of this. 👍
5 participants
Summary
ProxyScenario) from deployment mechanism (ProxyFixture) with convergence gating (ProxyHandle)systemtest-feature,systemtest-operator,systemtest-webhook,systemtest-installerInstalleras the primary downstream extension point — downstream varies by installation method, not proxy deploymentTest plan
🤖 Generated with Claude Code