Skip to content

feat: let SQL query dashboard filters use an optional source - #3691

Merged
karl-power merged 4 commits into
mainfrom
sql-dashboard-variables-source
Nov 19, 2026
Merged

karl-power merged 4 commits into
mainfrom
sql-dashboard-variables-source

Conversation

@karl-power

@karl-power karl-power commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Lets SQL query dashboard filters pick an optional source, so their SQL can use $__sourceTable and $__filters like raw SQL tiles. Follows up on review feedback on #3674.

  • Adds an optional source to the filter schema. Without one, $__sourceTable and $__filters are still rejected.
  • $__filters expands to the dashboard's broadcast selections for that source, via a new renderSourceFiltersToSql wrapper around the existing renderFiltersToSql.
  • The editor adds a clearable Source picker beside Connection. Autocomplete, validation and Preview values follow the selected source.
  • Moves resolveConnectionSourceSync from RawSqlChartEditor.tsx to utils/connectionSourceSync.ts so the filter editor can reuse it.
  • The API rejects missing or PromQL sources. The external API and MCP use sourceId and also check that the source is on the filter's connection.
  • Export and import map the source by name.

Screenshots or video

Screenshot 2026-11-17 at 14 16 13

How to reproduce

Starting point: yarn dev with a "Local ClickHouse" connection and a Logs source on default.otel_logs.

On main:

  1. Add a SQL query values filter with SELECT DISTINCT ServiceName FROM $__sourceTable. The editor rejects it.

On this branch:

  1. Add a Queried values filter Severity on Logs with expression SeverityText.
  2. Add a SQL query values filter Service with connection "Local ClickHouse", Source Logs, variable svc, and:
    SELECT DISTINCT ServiceName FROM $__sourceTable
    WHERE $__timeFilter(Timestamp) AND $__filters
    ORDER BY 1
  3. Save. The Service dropdown lists services.
  4. Pick a value in Severity. Service narrows to matching services.
  5. Edit Service and clear Source. Saving is blocked with "SQL uses $__sourceTable and $__filters but no source is selected".

References

@karl-power
karl-power requested a review from a team as a code owner November 17, 2026 13:02
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 8:57am UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 8:57am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 20d5442

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 17, 2026 13:03
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 4 P3

No inline comments. 3 P3 folded at the bottom. 1 unchanged from an earlier push (already inline above).

🔵 3 P3 — no action needed before merge
  • 🔵 P3 packages/app/src/hooks/__tests__/useSqlQueryFilterValues.test.tsx:2239 — The new renderSourceFiltersToSql is mocked here and has no test of its own → Add a common-utils test for renderSourceFiltersToSql that runs it on a log source (check the implicit-column and body expressions), a trace source and a metric source with an empty tableName (check the raw-condition branch). Right now the hook test replaces it with jest.fn(), so a regression in how $__filters is rendered for a source would still pass.
  • 🔵 P3 packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:1547 — The effect that keeps connection and source in sync is copied, not shared → Only the pure resolveConnectionSourceSync moved to a shared module. The effect around it is still duplicated in RawSqlChartEditor.tsx:152-178: it calls the helper, skips connection updates for a connection variable, and dispatches setValue. Move that wiring into a shared useConnectionSourceSync({ source, connection, storedConnection, sources, setValue }) hook in utils/connectionSourceSync.ts so both editors use it.
  • 🔵 P3 packages/common-utils/src/core/renderChartConfig.ts:2679 — Duplicates the existing mapping of source fields to raw SQL config fields → The isLogOrTrace choice of implicitColumnExpression, bodyExpression and useTextIndexForImplicitColumn restates the mapping already in packages/app/src/components/ChartEditor/utils.ts:412-428 (the raw SQL formToChartConfig path) and DBDashboardPage.tsx:807-817. Move it into one exported helper in common-utils, for example getSourceFilterRenderFields(source). Use it here and in the raw SQL tile path so $__filters on a SQL query filter and on a raw SQL tile can't drift apart.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 107 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 473 passed • 1 skipped • 1018s

Status Count
✅ Passed 473
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Additional context: touches the query rendering engine lightly (47 lines, under the 150-line bar for Tier 4)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 26
  • Production lines changed: 884 (+ 1045 in test files, excluded from tier calculation)
  • Critical-path lines changed: 38
  • Branch: sql-dashboard-variables-source
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Comment thread packages/common-utils/src/filters.ts Resolved
@karl-power
karl-power requested a review from pulpdrew November 17, 2026 14:11
Comment thread packages/common-utils/src/filters.ts Resolved
// The source is optional, so drop it rather than export an empty name
// the template schema would reject.
if (filter.source) {
filter.source = getSourceName(filter.source) || undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Export silently drops a deleted source, so the imported SQL filter can't be remapped

When the source is gone, the filter is exported with source: undefined but its sqlTemplate still uses $__sourceTable/$__filters. On import, isSourceBackedTemplateFilter is false, so there is no 'Data source' mapping row, and validateDashboardFilterQueries only shows a non-blocking warning. The dashboard imports with a filter that always errors with 'no source is selected'. Export a placeholder name, or keep a marker that the filter had a source, so the import UI shows a mapping row and requires one.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* Returns an error message if a SQL_QUERY filter's source is a kind its
* $__sourceTable macro can't expand (PromQL), or null.
*/
function getSqlQueryFilterSourceKindError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — SQL filter source rules are implemented twice, once per API

getSqlQueryFilterSourceKindError plus the new filter branch in getSourceConnectionMismatches restate what getFilterSourceRefs/validateFilterSources in packages/api/src/routers/api/dashboards.ts already enforce: an allowed kind and the same connection. The internal version also skips unchanged refs, which the external one doesn't, so the two can already disagree when a source has since been deleted or changed. Move the kind and connection check into one shared helper (for example in packages/api/src/utils/) that both routers call.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

await user.click(screen.getByRole('button', { name: 'Save' }));

expect(
screen.queryByText(/no source is selected|cannot be used/),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Absence assertion runs before async validation, so 'accepts the source macros' can't fail

handleSubmit validates asynchronously, and queryByText(...) runs right after the click, before any error could render. The test would still pass if the form rejected $__sourceTable with a source selected. Assert on a positive outcome instead: pass an onSubmit spy to the Harness and await waitFor(() => expect(onSubmit).toHaveBeenCalled()), or flush the validation first, as the sibling test does with findByText.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

stretches it to the full width. */}
<div
data-testid="filter-sql-source-select"
style={{ display: 'grid' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Raw inline style used where a Mantine style prop exists

<div style={{ display: 'grid' }}> breaks the convention 'Prefer individual Mantine style props over raw styles'. Use <Box display="grid" data-testid="filter-sql-source-select"> instead.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: 39 files vs base 2f61137. Optional source on SQL_QUERY dashboard filters, enabling $__sourceTable/$__filters in filter SQL, with API/MCP/external-API source validation and export/import name mapping.

Direct verification traced the high-risk surfaces: server-side source/connection validation (filterSources.ts, internal + external API), the public API/MCP contract changes, the $__filters rendering path (renderSourceFiltersToSql), and the editor/hook/import frontend flows. No ship-blocking defect was found: the connection-mismatch guard is enforced on the internal API, external API, and import mapping; $__filters safely defaults to NO_FILTERS when no source is selected; the optional source round-trips through reference resolution without touching the undefined case (mapReferenceFields guards on typeof value === 'string'); and the new behavior is broadly covered by added unit and integration tests.

✅ No critical issues found.

🔵 P3 nitpicks (2)
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:214 — isWaitingForSources is true when any filter has a source, and the following loop then stamps { isLoading: true } on every filter without a cached entry, including sourceless SQL filters that do not depend on the sources list, so their dropdowns show a transient loading state until sources resolve.
    • Fix: gate the per-filter loading placeholder on filter.source so only source-backed filters are marked loading while sources load.
  • packages/common-utils/src/filters.ts:863 — the 'unknown' string sentinel for sourceTables is mixed into a field that otherwise holds a structured SqlQueryFilterSourceTables object, which is easy to mishandle in future callers that destructure the context.
    • Fix: consider a discriminated shape (e.g. { kind: 'unknown' } | { kind: 'loaded'; tables }) or a named constant to make the sentinel self-documenting.

Reviewers (9): correctness, kieran-typescript, security, api-contract, adversarial, testing, maintainability, previous-comments, performance.

Testing gaps:

  • No test exercises the runtime path in useSqlQueryFilterValues where a source variable resolves to a source on a different connection than filter.connection (the Source X is not on the filter's connection error branch).
  • The mixed-dashboard loading state (a sourceless SQL filter coexisting with a source-backed one while the sources list is still loading) is not covered.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

pulpdrew
pulpdrew previously approved these changes Nov 17, 2026

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

queried?.source ??
promqlLabel?.source ??
sqlQuery?.source ??
presetSourceId ??

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — New SQL query filters silently pick up the dashboard's preset source, and it is now saved

A new filter starts as QUERY_EXPRESSION with source: presetSourceId (DashboardFilterEditForm.tsx:127). When the user switches the type to SQL query, the SQL form keeps that log source, because its effect only clears sources of an incompatible kind. resolveConnectionSourceSync then also moves the connection to that source's connection. With source: values.source || undefined in toSavedFilter, this source is now persisted, where before it was dropped as a leftover. So filters get a source reference the user never picked. That reference is then validated by the API, exported, and can break the dropdown later. Fix: don't seed presetSourceId for SQL_QUERY, and clear source when the type switches to SQL_QUERY unless it came from a stored SQL_QUERY filter.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

if (filter.source && !source) {
// Every source looks missing until the list arrives.
if (!isLoadingSources) {
errors.set(filter.id, `Source ${filter.source} not found`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — A missing or mismatched source breaks the filter even when its SQL uses no source macros

The filter errors with 'Source X not found' or 'not on the filter's connection' whenever filter.source is set, even if the SQL never uses $__sourceTable/$__filters. Example: a SQL filter that just reads default.envs and has a stored source. If that source is deleted, the dropdown stops working. The API deliberately skips re-checking since-deleted sources (filterSources.ts), so the dashboard still saves while the filter is broken. Fix: only resolve and require the source when getSourceDependentMacrosUsed(filter.sqlTemplate) is non-empty. Otherwise ignore it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@karl-power
karl-power force-pushed the sql-dashboard-variables-source branch from d6d0be2 to 20d5442 Compare November 19, 2026 08:53
@karl-power
karl-power merged commit 2cec94f into main Nov 19, 2026
28 checks passed
kodiakhq Bot pushed a commit that referenced this pull request Nov 19, 2026
…ype-switch fix, E2E tests (#3719)

## Summary

Follow-ups from review on #3678: autocomplete table and column names in the SQL query filter editor, rename "Queried values", fix a validation bug when switching filter types, and add E2E coverage for SQL query filters. One commit per change:

- **feat: autocomplete table and column names in the SQL query filter editor.** The editor now suggests the tables and columns of the selected connection's sources, like the Raw SQL tile. A connection variable suggests the tables of the connection it currently selects. The tile's table-list logic moves to a shared `getTableConnectionsForConnection()` in `utils.ts`, which both editors use.
- **feat: rename the Queried values filter type to SQL expression values.** This tells it apart from "SQL query values" and "PromQL label values". The two SQL types are now next to each other in the type dropdown.
- **fix: clear the previous filter type's rules when switching types.** react-hook-form keeps a field's validation rules after its input unmounts, and merges the next input's rules over them. Switching from "PromQL label values" to "SQL query values" failed with "Select a PromQL source", and a new SQL query filter couldn't be saved without a source because "SQL expression values" (the default type) requires one. Changing the type now clears the old rules before the next editor registers its own.
- **test: add E2E tests for SQL query dashboard filters.** Five tests: values listed in query order and selected by variable name, values narrowed by another filter's variable using `$__sourceTable` and `$__filter`, a connection variable, a data source variable, and a query error shown in the preview and on the dropdown.

### Screenshots or video

Not captured. The visible changes are autocomplete suggestions in the SQL query filter editor and the renamed, reordered type dropdown.

### How to reproduce

**Starting point:** `yarn dev`, any dashboard, with a Logs source on the local connection.

**On `main`:**

1. Click **Edit filters and variables**, then **Add new filter**, and choose **PromQL label values**.
2. Switch the type to **SQL query values**, fill in a name, connection and query, and click **Save filter**. It fails with "Select a PromQL source".
3. In the SQL editor, start typing a table name. Only keywords, functions and macros are suggested.

**On this branch:**

1. Repeat step 2. The filter saves.
2. In the SQL editor, type `otel`. The tables of the connection's sources are suggested, and after `FROM`, their columns.
3. Open the **Type** dropdown. It lists **SQL expression values** then **SQL query values**, and "Queried values" is gone.

### How to test on Vercel preview

**Preview routes:** /dashboards

**Steps:**

1. Create a new dashboard and click **Edit filters and variables**, then **Add new filter**.
2. Open the **Type** dropdown and verify the first two options are "SQL expression values" and "SQL query values".
3. Choose **SQL query values**, pick the demo connection, and type `SELECT * FROM otel` in the SQL editor. Verify table names appear in the suggestions.

### References

- Linear Issue: [HDX-5524](https://linear.app/clickhouse/issue/HDX-5524/support-dashboard-variables-populated-from-arbitrary-clickhouse-sql)
- Related PRs: follows #3691. Addresses [this comment](#3678 (comment)) and [this one](#3678 (comment)) on #3678.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 20d54429 Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — 20d54429 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants