Repository navigation
feat: let SQL query dashboard filters use an optional source - #3691
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 20d5442 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
PR Review
4 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 4 P3 No inline comments. 3 P3 folded at the bottom. 1 unchanged from an earlier push (already inline above). 🔵 3 P3 — no action needed before merge
Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides. |
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
E2E Test Results✅ All tests passed • 473 passed • 1 skipped • 1018s
Tests ran across 4 shards in parallel. |
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Additional context: touches the query rendering engine lightly (47 lines, under the 150-line bar for Tier 4) Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
| // The source is optional, so drop it rather than export an empty name | ||
| // the template schema would reject. | ||
| if (filter.source) { | ||
| filter.source = getSourceName(filter.source) || undefined; |
There was a problem hiding this comment.
🔵 minor — Export silently drops a deleted source, so the imported SQL filter can't be remapped
When the source is gone, the filter is exported with source: undefined but its sqlTemplate still uses $__sourceTable/$__filters. On import, isSourceBackedTemplateFilter is false, so there is no 'Data source' mapping row, and validateDashboardFilterQueries only shows a non-blocking warning. The dashboard imports with a filter that always errors with 'no source is selected'. Export a placeholder name, or keep a marker that the filter had a source, so the import UI shows a mapping row and requires one.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| * Returns an error message if a SQL_QUERY filter's source is a kind its | ||
| * $__sourceTable macro can't expand (PromQL), or null. | ||
| */ | ||
| function getSqlQueryFilterSourceKindError( |
There was a problem hiding this comment.
🔵 minor — SQL filter source rules are implemented twice, once per API
getSqlQueryFilterSourceKindError plus the new filter branch in getSourceConnectionMismatches restate what getFilterSourceRefs/validateFilterSources in packages/api/src/routers/api/dashboards.ts already enforce: an allowed kind and the same connection. The internal version also skips unchanged refs, which the external one doesn't, so the two can already disagree when a source has since been deleted or changed. Move the kind and connection check into one shared helper (for example in packages/api/src/utils/) that both routers call.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| await user.click(screen.getByRole('button', { name: 'Save' })); | ||
|
|
||
| expect( | ||
| screen.queryByText(/no source is selected|cannot be used/), |
There was a problem hiding this comment.
🔵 minor — Absence assertion runs before async validation, so 'accepts the source macros' can't fail
handleSubmit validates asynchronously, and queryByText(...) runs right after the click, before any error could render. The test would still pass if the form rejected $__sourceTable with a source selected. Assert on a positive outcome instead: pass an onSubmit spy to the Harness and await waitFor(() => expect(onSubmit).toHaveBeenCalled()), or flush the validation first, as the sibling test does with findByText.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| stretches it to the full width. */} | ||
| <div | ||
| data-testid="filter-sql-source-select" | ||
| style={{ display: 'grid' }} |
There was a problem hiding this comment.
🔵 minor — Raw inline style used where a Mantine style prop exists
<div style={{ display: 'grid' }}> breaks the convention 'Prefer individual Mantine style props over raw styles'. Use <Box display="grid" data-testid="filter-sql-source-select"> instead.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Deep ReviewScope: 39 files vs base Direct verification traced the high-risk surfaces: server-side source/connection validation ( ✅ No critical issues found. 🔵 P3 nitpicks (2)
Reviewers (9): correctness, kieran-typescript, security, api-contract, adversarial, testing, maintainability, previous-comments, performance. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
4ca7b11 to
682543b
Compare
682543b to
69a40fe
Compare
789ca90 to
4cede23
Compare
| queried?.source ?? | ||
| promqlLabel?.source ?? | ||
| sqlQuery?.source ?? | ||
| presetSourceId ?? |
There was a problem hiding this comment.
🟡 P2 — New SQL query filters silently pick up the dashboard's preset source, and it is now saved
A new filter starts as QUERY_EXPRESSION with source: presetSourceId (DashboardFilterEditForm.tsx:127). When the user switches the type to SQL query, the SQL form keeps that log source, because its effect only clears sources of an incompatible kind. resolveConnectionSourceSync then also moves the connection to that source's connection. With source: values.source || undefined in toSavedFilter, this source is now persisted, where before it was dropped as a leftover. So filters get a source reference the user never picked. That reference is then validated by the API, exported, and can break the dropdown later. Fix: don't seed presetSourceId for SQL_QUERY, and clear source when the type switches to SQL_QUERY unless it came from a stored SQL_QUERY filter.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| if (filter.source && !source) { | ||
| // Every source looks missing until the list arrives. | ||
| if (!isLoadingSources) { | ||
| errors.set(filter.id, `Source ${filter.source} not found`); |
There was a problem hiding this comment.
🟡 P2 — A missing or mismatched source breaks the filter even when its SQL uses no source macros
The filter errors with 'Source X not found' or 'not on the filter's connection' whenever filter.source is set, even if the SQL never uses $__sourceTable/$__filters. Example: a SQL filter that just reads default.envs and has a stored source. If that source is deleted, the dropdown stops working. The API deliberately skips re-checking since-deleted sources (filterSources.ts), so the dashboard still saves while the filter is broken. Fix: only resolve and require the source when getSourceDependentMacrosUsed(filter.sqlTemplate) is non-empty. Otherwise ignore it.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
a909c6a to
5a69ee5
Compare
5a69ee5 to
d6d0be2
Compare
d6d0be2 to
20d5442
Compare
…ype-switch fix, E2E tests (#3719) ## Summary Follow-ups from review on #3678: autocomplete table and column names in the SQL query filter editor, rename "Queried values", fix a validation bug when switching filter types, and add E2E coverage for SQL query filters. One commit per change: - **feat: autocomplete table and column names in the SQL query filter editor.** The editor now suggests the tables and columns of the selected connection's sources, like the Raw SQL tile. A connection variable suggests the tables of the connection it currently selects. The tile's table-list logic moves to a shared `getTableConnectionsForConnection()` in `utils.ts`, which both editors use. - **feat: rename the Queried values filter type to SQL expression values.** This tells it apart from "SQL query values" and "PromQL label values". The two SQL types are now next to each other in the type dropdown. - **fix: clear the previous filter type's rules when switching types.** react-hook-form keeps a field's validation rules after its input unmounts, and merges the next input's rules over them. Switching from "PromQL label values" to "SQL query values" failed with "Select a PromQL source", and a new SQL query filter couldn't be saved without a source because "SQL expression values" (the default type) requires one. Changing the type now clears the old rules before the next editor registers its own. - **test: add E2E tests for SQL query dashboard filters.** Five tests: values listed in query order and selected by variable name, values narrowed by another filter's variable using `$__sourceTable` and `$__filter`, a connection variable, a data source variable, and a query error shown in the preview and on the dropdown. ### Screenshots or video Not captured. The visible changes are autocomplete suggestions in the SQL query filter editor and the renamed, reordered type dropdown. ### How to reproduce **Starting point:** `yarn dev`, any dashboard, with a Logs source on the local connection. **On `main`:** 1. Click **Edit filters and variables**, then **Add new filter**, and choose **PromQL label values**. 2. Switch the type to **SQL query values**, fill in a name, connection and query, and click **Save filter**. It fails with "Select a PromQL source". 3. In the SQL editor, start typing a table name. Only keywords, functions and macros are suggested. **On this branch:** 1. Repeat step 2. The filter saves. 2. In the SQL editor, type `otel`. The tables of the connection's sources are suggested, and after `FROM`, their columns. 3. Open the **Type** dropdown. It lists **SQL expression values** then **SQL query values**, and "Queried values" is gone. ### How to test on Vercel preview **Preview routes:** /dashboards **Steps:** 1. Create a new dashboard and click **Edit filters and variables**, then **Add new filter**. 2. Open the **Type** dropdown and verify the first two options are "SQL expression values" and "SQL query values". 3. Choose **SQL query values**, pick the demo connection, and type `SELECT * FROM otel` in the SQL editor. Verify table names appear in the suggestions. ### References - Linear Issue: [HDX-5524](https://linear.app/clickhouse/issue/HDX-5524/support-dashboard-variables-populated-from-arbitrary-clickhouse-sql) - Related PRs: follows #3691. Addresses [this comment](#3678 (comment)) and [this one](#3678 (comment)) on #3678.
Summary
Lets SQL query dashboard filters pick an optional source, so their SQL can use
$__sourceTableand$__filterslike raw SQL tiles. Follows up on review feedback on #3674.sourceto the filter schema. Without one,$__sourceTableand$__filtersare still rejected.$__filtersexpands to the dashboard's broadcast selections for that source, via a newrenderSourceFiltersToSqlwrapper around the existingrenderFiltersToSql.resolveConnectionSourceSyncfromRawSqlChartEditor.tsxtoutils/connectionSourceSync.tsso the filter editor can reuse it.sourceIdand also check that the source is on the filter's connection.Screenshots or video
How to reproduce
Starting point:
yarn devwith a "Local ClickHouse" connection and a Logs source ondefault.otel_logs.On
main:SELECT DISTINCT ServiceName FROM $__sourceTable. The editor rejects it.On this branch:
Severityon Logs with expressionSeverityText.Servicewith connection "Local ClickHouse", Source Logs, variablesvc, and:References