Skip to content

feat: speed up exact filters - #3695

Merged
knudtty merged 3 commits into
claude/filter-keys-fetch-limitfrom
aaron/faster-exact-filter-values
Nov 19, 2026
Merged

knudtty merged 3 commits into
claude/filter-keys-fetch-limitfrom
aaron/faster-exact-filter-values

Conversation

@knudtty

@knudtty knudtty commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Speeds up exact filters immensely. Should help on both the search and for dashboard filters.

The old queries sent a query asking for many different individual map columns. ClickHouse's behavior is not set up well for that. Keep in mind, Maps are really stored as an array of key value pairs. ClickHouse will

  1. Load a row
  2. For the first map key column, walk the array until it finds the value. If value is not found, you just walked the whole array.
  3. Repeat for each map key column
  4. Move on to the next row

This leads to tons of extra reads, roughly one read per queried key per row. The new behavior is

  1. Load a row
  2. Filter for only specific keys you're looking for
  3. Move on to the next row

This means just one walk through the array per row.

Screenshots or video

Before

export-1794928260568-before.mp4

After

export-1794926363019-after.mp4

References

  • Linear Issue: Closes HDX-6053

@knudtty
knudtty requested a review from a team as a code owner November 17, 2026 15:24
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 3:33pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 3:33pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c0a8f48

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 2
  • Production lines changed: 216 (+ 221 in test files, excluded from tier calculation)
  • Branch: aaron/faster-exact-filter-values
  • Author: knudtty

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

5 new clones introduced by this PR.

  • packages/api/src/models/source.ts:180-206 ↔ packages/api/src/models/source.ts:237-263 — 27 lines
  • packages/common-utils/src/core/metadata.ts:1345-1369 ↔ packages/common-utils/src/core/metadata.ts:1393-1414 — 25 lines
  • packages/common-utils/src/core/metadata.ts:1867-1876 ↔ packages/common-utils/src/core/metadata.ts:1996-2005 — 10 lines
  • packages/common-utils/src/core/metadata.ts:1876-1889 ↔ packages/common-utils/src/core/metadata.ts:2005-2019 — 14 lines
  • packages/common-utils/src/types.ts:2871-2885 ↔ packages/common-utils/src/types.ts:2920-2934 — 15 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 107 in total (1.12%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 1 P0/P1 · 🟡 1 P2 · 🔵 1 P3

2 posted as inline comment(s) on the changed lines. 1 P3 folded at the bottom.

🔵 1 P3 — no action needed before merge
  • 🔵 P3 .changeset/faster-exact-filter-values.md:12 — Changeset says each JSON column gets its own query, but dot-path JSON keys share the plain-column query → getMapColumn only splits off bracket keys (parseKeyPath length > 1). JSON keys in Body.\user`form go into the plain-column group, and the new test inuseMetadata.test.tsx` expects exactly that. Fix: either change the changeset text to say only Map and bracket-form keys get their own query, or group dot-path JSON keys by their root column too.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: base accc11ea → PR head (aaron/faster-exact-filter-values), 6 files (2 production, 3 test, 1 changeset).
Intent: Speed up 'exact' filter-value lookups by (a) reading multiple keys of one Map column with a single groupUniqArrayMap(mapFilter(k IN (…))) aggregate instead of one groupUniqArray(Map['k']) per key, and (b) splitting client queries into one native-column group plus one per Map column, streamed in completion order.

✅ No critical issues found.

This is a focused, well-tested change. Both SQL paths (disableRowLimit true and the sampled CTE path) are unit-tested, including the mapParam/param alias wiring and faceted-key exclusion; an integration test exercises the real groupUniqArrayMap read against ClickHouse. Verified independently: per-key limit semantics are preserved by the -Map combinator, getColumn resolves through the cached getColumns so the per-column awaits in groupMapSubscripts add no real round-trips, facet consumers look up by .key so the new completion-order streaming does not break them, and inCompletionOrder surfaces the first rejection without orphaning unhandled rejections.

🟡 P2 -- recommended

  • packages/common-utils/src/core/metadata.ts:3398 -- The Map subscript literal is built by raw interpolation ('${path[1]}') of the parsed key segment and reused in both mapFilter(k IN (…)) and the [literal] lookup, so correctness depends on an undocumented contract that incoming key subscripts are already SQL-escaped (the b\'c test encodes this implicitly). A key whose subscript carries an unescaped quote would produce malformed SQL; the same raw-interpolation shape predates this PR for rendered keys, so this is a latent pattern, not a new exploit path.
    • Fix: Add an assertion or a comment at groupMapSubscripts stating that path[1] must arrive pre-escaped, or escape it explicitly before embedding so the filter and subscript share one escaping path.
🔵 P3 nitpicks (1)
  • packages/app/src/hooks/useMetadata.tsx:482 -- "Group keys by their Map column" is now implemented twice: client-side groupKeysByMapColumn/getMapColumn (to split queries) and server-side groupMapSubscripts (to share the SQL read), both deriving the column via parseKeyPath + unquoteIdentifier(path[0]). The two apply different rules (the server additionally verifies the column is a Map type and needs ≥2 keys), so they can drift as key-shape handling evolves.
    • Fix: Extract the shared "map column of a key" derivation into one helper imported by both layers, or add a cross-reference comment noting the two must stay consistent.

Reviewers (9): correctness, security, adversarial, performance, maintainability, testing, kieran-typescript, project-standards, previous-comments.

Testing gaps: none substantive — the diff adds unit coverage for both the direct and CTE SQL paths, faceted-key handling, special-character escaping, the native-columns-uncapped change, and completion-order error propagation, plus an integration test for the shared Map read.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 481 passed • 1 skipped • 1178s

Status Count
✅ Passed 481
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but left one thing to improve reusability

Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@knudtty
knudtty force-pushed the aaron/faster-exact-filter-values branch from 6fa2485 to 1e116d7 Compare November 19, 2026 14:04
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@knudtty
knudtty merged commit 1a4c116 into main Nov 19, 2026
28 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — c0a8f484 Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — c0a8f484 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants