You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of #523 (Track 2: non-empirical gap 3 + empirical gap 4).
Goal
Replace the optimizer cost model's hardcoded N_g = 1 with label-group cardinality inferred from live Prometheus data.
Scope
Add label_group_count to CandidateConfig, not AQE, so the profiled count belongs to the candidate configuration and remains compatible with future Phase 3 sharing.
Use N_g = 1 for subpopulation-aware sketches and the profiled distinct grouping-key count for non-subpopulation-aware sketches.
Apply the count to ingest memory and query CPU/memory costs; ingest CPU remains based on total arrival rate.
Profile once per profile key (metric, spatial_filter, grouping_labels) before candidate enumeration, and reuse the result across the AQE candidate grid.
Use the existing Prometheus /api/v1/series path and shared retry/fetch logic. Construct a selector from the metric and spatial filter, then count distinct grouping-label tuples client-side.
Cache series data so multiple AQEs with the same metric/filter do not repeat the expensive fetch.
An empty grouping-label set has N_g = 1. Empty or unavailable Prometheus data must never become N_g = 0.
If cardinality is unavailable, retain EXACT and subpopulation-aware candidates while dropping non-subpopulation-aware candidates, consistent with the missing-data policy in Optimizer: param-aware atomic costs from sketch-bench #524.
Follow-up
The offline, dataset-backed implementation is tracked separately in #693. Both implementations should feed the same optimizer profile interface; #693 should not require Prometheus.
No new PromQL query type needed: fetch_labels_for_metric already calls /api/v1/series with match[]=<metric> and gets back every matching series' full label set. Add profile_label_group_count(url, metric, spatial_filter, grouping_labels): same /api/v1/series call (match[] = metric + spatial_filter_normalized), group the returned series client-side by their grouping_labels values, count distinct groups. Refactor the shared HTTP-fetch-with-retry loop out so this doesn't duplicate the 503-retry logic. Empty grouping_labels → trivially 1 (matches today's hardcoded value, no regression for ungrouped queries).
Profiled once per AQE (batched in pipeline.rs's run_pipeline shell right after extract_aqes), threaded through as Vec<(AQE, u64)> (paired, not a separately-indexed parallel Vec) rather than re-profiled per candidate.
Total miss (no live Prometheus data AND no metrics: hint): drop candidates using non-subpopulation-aware sketches for that AQE, consistent with Optimizer: param-aware atomic costs from sketch-bench #524's "missing data → drop the candidate" policy — defaulting to 1.0 would silently understate a capacity-planning number. Subpopulation-aware candidates (CMS, MultipleSum, etc.) and EXACT are unaffected.
Re-scope note: with the MILP (#794), N_g is groups[].cardinality in the --workload-facts file. Inferring that file from Prometheus also needs each sketch-served grouping's data shape (zipf_s, distinct_keys, tail_index); for that we can use sketch-bench's data-shape fitter.
Activity
milindsrivastava1997 commentedon Jul 13, 2026
Implementation plan discussed:
fetch_labels_for_metricalready calls/api/v1/serieswithmatch[]=<metric>and gets back every matching series' full label set. Addprofile_label_group_count(url, metric, spatial_filter, grouping_labels): same/api/v1/seriescall (match[] = metric + spatial_filter_normalized), group the returned series client-side by theirgrouping_labelsvalues, count distinct groups. Refactor the shared HTTP-fetch-with-retry loop out so this doesn't duplicate the 503-retry logic. Emptygrouping_labels→ trivially 1 (matches today's hardcoded value, no regression for ungrouped queries).pipeline.rs'srun_pipelineshell right afterextract_aqes), threaded through asVec<(AQE, u64)>(paired, not a separately-indexed parallelVec) rather than re-profiled per candidate.metrics:hint): drop candidates using non-subpopulation-aware sketches for that AQE, consistent with Optimizer: param-aware atomic costs from sketch-bench #524's "missing data → drop the candidate" policy — defaulting to 1.0 would silently understate a capacity-planning number. Subpopulation-aware candidates (CMS, MultipleSum, etc.) and EXACT are unaffected.[-]Optimizer: real N_g via Prometheus profiling, hints fallback[/-][+]Optimizer: auto-infer N_g from Prometheus[/+]milindsrivastava1997 commentedon Oct 26, 2026
Re-scope note: with the MILP (#794), N_g is
groups[].cardinalityin the--workload-factsfile. Inferring that file from Prometheus also needs each sketch-served grouping's datashape(zipf_s,distinct_keys,tail_index); for that we can use sketch-bench's data-shape fitter.