Skip to content

fix(analyzer): stop promising NOT NULL from rows a join or view may not have - #63

Merged
lbguilherme merged 1 commit into
mainfrom
fix/join-soundness
Oct 3, 2026
Merged

lbguilherme merged 1 commit into
mainfrom
fix/join-soundness

Conversation

@lbguilherme

Copy link
Copy Markdown
Member

The join and row-guarantee narrowing promised NOT NULL for values PostgreSQL 18 returns as NULL. A soundness review reproduced six ways it happens:

  1. Set-returning calls outside the select list were ignored. PG adds an ORDER BY / GROUP BY / DISTINCT ON item as a resjunk target entry, so its SRF runs in the level's ProjectSet: SELECT (SELECT 1 ORDER BY generate_series(1,0)) is NULL, and SELECT generate_series(1,2) AS g ORDER BY generate_series(1,3) pads g with NULL.
  2. A foreign key was followed through any =. For char(3) referenced by text, the key compares with bpchareq (trailing blanks ignored), but the query's pb.k = cb.k resolves to text = text and can miss.
  3. A foreign key whose equality depends on TimeZone was followed. timestamptz = timestamp is STABLE, so rows checked under UTC can stop matching in another zone.
  4. View queries were re-analyzed by name. PG binds a view to OIDs and attnums when it is created. After renaming and recreating a table, creating the view under another search_path, or swapping column names, re-analysis read a different table or column. That fed FK following, view origins, view nullability refreshes and rows written through views.
  5. Row locking inside a view was not counted. With CREATE VIEW cv AS SELECT * FROM c FOR UPDATE, EvalPlanQual re-fetches a concurrently updated row whose new parent the snapshot can't see.
  6. VALUES (1) ORDER BY generate_series(1,0) was accepted, but every execution fails.

(Item 5, DISABLE TRIGGER ALL on partitions, is handled in a separate PR.)

Fix

  • level_srf_calls collects the set-returning calls of the select list plus those in sort, grouping and DISTINCT ON items that don't repeat a select-list expression. It is used wherever only the select list was counted before: row guarantees, FROM-less subqueries, lockstep padding, EXCEPT's always-NULL arm, and the FOR UPDATE check.
  • A foreign key is followed only when the query's = resolves to the key's pfeqop (or to its commutator when written the other way round), and only when that operator is immutable. pfeqop is derived as ATAddForeignKeyConstraint does, from the referenced unique index's operator family; only no-op casts are allowed for the referencing value. If several unique indexes cover the key, they must all agree, because conindid isn't recorded.
  • At CREATE VIEW, each view and materialized view stores the objects its query resolved to, in analysis order, with columns stored by attnum. Re-analysis (reanalyze_view) is trusted only while the query still resolves to the same objects; otherwise it returns None, which is the conservative path every caller already had. View ASTs are still not rewritten on RENAME. A benign rename therefore still loses narrowing through that view, as it did before.
  • A statement counts as locking rows when it reads a view whose query (or a nested view's query, followed through the stored bindings) has a locking clause. Nested analyses inherit the statement's locking.
  • Also fixed: a statement that locks rows no longer trusts a view column NOT NULL only because of a foreign key. SELECT vs.name FROM vs FOR UPDATE returned NULL on PG under a concurrent re-key, so such columns are now re-derived under row locking.
  • Statements that always fail are rejected with PG's 0A000 message. This applies only where the VALUES is sure to be planned: the statement's own VALUES, its set-operation arms, an INSERT source, EXPLAIN, and a scalar or ARRAY subquery that is a whole select-list entry. The planner drops the VALUES (and the statement runs) under WHERE false, an unreferenced CTE, EXISTS or CASE WHEN false; those cases stay accepted and are tested.

Tests

typedpg_analyzer/tests/query/join_soundness.rs has 17 tests. Before the fix, all 13 regression tests fail. The 4 near-miss tests pass both before and after: cross-type, domain, varchar and date keys are still followed; a sort key with no new SRF still narrows; and a view whose names still resolve the same way still narrows.

Verification:

  • cargo nextest run --release -p typedpg_analyzer: all pass
  • scripts/run-pg-sanity.sh --no-fail-fast (full oracle on PG 18): 2266 passed, no "nullability unsound"
  • cargo nextest run --release --workspace: 2522 passed
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo clippy -p typedpg_analyzer --all-targets --features pg_sanity -- -D warnings: clean

Implemented by Claude Opus 5.5 (claude-opus-5-5) in Claude Code via T3 Code.

- Count the set-returning calls of ORDER BY, GROUP BY and DISTINCT ON
  items that aren't select-list expressions with the select list's:
  PG runs them in the level's ProjectSet, where they can leave the level
  without rows or pad the select list's calls with NULL. This covers the
  row guarantees (finish_level, FROM-less subqueries), the lockstep
  padding, EXCEPT's always-NULL arm and the FOR UPDATE check.
- Reject a VALUES list sorted by a set-returning call where it is sure
  to be planned: every execution fails with "set-valued function called
  in context that cannot accept a set".
- Follow a foreign key only through the equality it enforces: the
  query's `=` must resolve to the key's pfeqop (derived from the
  referenced unique index's operator family) or its commutator, and that
  operator must be immutable (`timestamptz = timestamp` depends on the
  TimeZone).
- Re-analyze a view's stored query only while its names still resolve
  to the relations, columns, functions, operators and types they did at
  CREATE VIEW (stored per view), for view origins, view nullability
  refreshes and rows written through views.
- Treat a statement reading a view with a locking clause (directly or
  through nested views) as locking rows, and re-derive a view's column
  nullability under row locking when the statement locks rows.

Co-Authored-By: Claude <noreply@anthropic.com>
@lbguilherme
lbguilherme merged commit b3e3dec into main Oct 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant