feat: notify Metrics when a controller's event processor starts#3485
Open
jiangzho wants to merge 1 commit into
Open
feat: notify Metrics when a controller's event processor starts#3485jiangzho wants to merge 1 commit into
jiangzho wants to merge 1 commit into
Conversation
Add Metrics.eventProcessingStarted(Controller), invoked in Controller whenever the event processor begins accepting events (both on inline start and after a deferred start, e.g. once leader election succeeds), so implementations can measure operator startup latency. Wire it through AggregatedMetrics so composed Metrics instances all receive the callback, and implement it in MicrometerMetrics/V2 as a per-controller gauge recording JVM uptime at the moment processing starts.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new lifecycle callback to the SDK’s monitoring API to signal when a controller’s event processor actually begins accepting events, enabling measurement of operator/controller startup latency (including deferred-start scenarios such as leader election).
Changes:
- Introduces
Metrics.eventProcessingStarted(Controller)and wires it throughControllerandAggregatedMetrics. - Adds unit tests covering notification behavior around inline vs deferred event processor start.
- Implements the callback in Micrometer metrics (V1 and V2) as a per-controller gauge set to JVM uptime at the moment processing starts.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/Controller.java | Emits the new metrics callback when event processing starts (inline and deferred). |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/monitoring/Metrics.java | Adds the eventProcessingStarted default method and Javadoc contract. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/monitoring/AggregatedMetrics.java | Forwards the new callback to all aggregated metrics instances. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/ControllerTest.java | Adds coverage for the new callback behavior. |
| micrometer-support/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java | Implements callback as a per-controller gauge. |
| micrometer-support/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetrics.java | Implements callback as a per-controller gauge (legacy naming style). |
Comment on lines
+150
to
+157
| @Override | ||
| public void eventProcessingStarted(Controller<? extends HasMetadata> controller) { | ||
| final var name = controller.getConfiguration().getName(); | ||
| final var tags = new ArrayList<Tag>(); | ||
| addControllerNameTag(name, tags); | ||
| registry.gauge( | ||
| PROCESSING_STARTED_LATENCY_GAUGE, tags, ManagementFactory.getRuntimeMXBean().getUptime()); | ||
| } |
Comment on lines
+152
to
+160
| @Override | ||
| public void eventProcessingStarted(Controller<? extends HasMetadata> controller) { | ||
| final var configuration = controller.getConfiguration(); | ||
| final var name = configuration.getName(); | ||
| final var tags = new ArrayList<Tag>(3); | ||
| addGVKTags(GroupVersionKind.gvkFor(configuration.getResourceClass()), tags, false); | ||
| registry.gauge( | ||
| PROCESSING_STARTED_LATENCY + name, tags, ManagementFactory.getRuntimeMXBean().getUptime()); | ||
| } |
Comment on lines
+72
to
+90
| @Test | ||
| void notifiesMetricsWhenEventProcessorStarts() { | ||
| final var client = MockKubernetesClient.client(Secret.class); | ||
| final var metrics = mock(Metrics.class); | ||
| final var configurationService = | ||
| ConfigurationService.newOverriddenConfigurationService( | ||
| new BaseConfigurationService(), o -> o.withMetrics(metrics)); | ||
| final var configuration = | ||
| MockControllerConfiguration.forResource(Secret.class, configurationService); | ||
| final var controller = new Controller<Secret>(reconciler, configuration, client); | ||
|
|
||
| // Inline start (event processing started synchronously, e.g. no leader election). | ||
| controller.start(true); | ||
| verify(metrics, times(1)).eventProcessingStarted(controller); | ||
|
|
||
| // Deferred start (event processing started later, e.g. after acquiring leadership). | ||
| controller.startEventProcessing(); | ||
| verify(metrics, times(2)).eventProcessingStarted(controller); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We have been observing logs like
While this is expected during startup, it would be beneficial for us to measure the startup delay
Add Metrics.eventProcessingStarted(Controller), invoked in Controller
whenever the event processor begins accepting events (both on inline
start and after a deferred start, e.g. once leader election succeeds),
so implementations can measure operator startup latency.
Wire it through AggregatedMetrics so composed Metrics instances all
receive the callback, and implement it in MicrometerMetrics/V2 as a
per-controller gauge recording JVM uptime at the moment processing
starts.
Add Metrics.eventProcessingStarted(Controller), invoked in Controller
whenever the event processor begins accepting events (both on inline
start and after a deferred start, e.g. once leader election succeeds),
so implementations can measure operator startup latency.
Wire it through AggregatedMetrics so composed Metrics instances all
receive the callback, and implement it in MicrometerMetrics/V2 as a
per-controller gauge recording JVM uptime at the moment processing
starts.