fix: price a covering projection scan from its own pages

[linuxhikerpm]

Summary

• A covering projection scan inherited seq_page_cost * rel->pages, the whole relation file (base plus every projection). The covering path now charges I/O from that projection's own page-rounded row groups, times the already-floored sort-key selectivity.
• Independent TDD twins (test/projection_scan_io.sh and test/pytest/test_projection_scan_io.py) pin the planner ratio on the public EXPLAIN-cost seam. Own fixtures, matching assertion names. Neither file reads the other.
• Seeded on PG15-19. Mutation rel->pages reddens the load-bearing arm the same way the unfixed tree did. suites_not_covered stays 249. cluster_tests 437 by collection on current main (18821ce).

Not merged. Not self-approved.

Test plan

• test/projection_scan_io.sh on PG18: covering run 19996 against base-page I/O 42000 (ratio 0.476), not 41991.6 / 1.000
• test/pytest/test_projection_scan_io.py on PG18, same assertion names, own table
• Mutation of ioProj back to rel->pages fails both twins got [base-pages] want [proj-pages]
• Ledger majors 15;16;17;18;19; census counted, not added

[linuxhikerpm]

TDD excerpts from this session. Prior chat summaries were not used as evidence.

Start SHA 51e24d07c294 (origin/main after #1151). Main moved to 18821cec45da (#1152) before the PR opened; rebased locally (not Update branch). Head a2f125aa9dc5. Confirmed on that tree before the patch: covering path projRun = serialRun * scale and pgcolumnar_scan_io_run_cost uses seq_page_cost * rel->pages.

Shell test/projection_scan_io.sh (PG18)

Unfixed (no projection-page I/O):

-- cover_run=41991.6 rel_pages=42.0000000000000000 base_io=42000 ratio=1.000
-- proj_bytes=56366 rel_bytes=344064
FAIL  a covering projection is not priced from the base table's pages: got [base-pages] want [proj-pages]

Fixed:

-- cover_run=19996 rel_pages=42.0000000000000000 base_io=42000 ratio=0.476
-- proj_bytes=56366 rel_bytes=344064
PASS  a covering projection is not priced from the base table's pages
projection_scan_io.sh: PASSED

Same numbers on PG15, PG16, PG17, PG19 (source: 5314c03f54fc).

Pytest test/pytest/test_projection_scan_io.py (PG18)

Unfixed:

-- cover_run=41991.6 rel_pages=42.0 base_io=42000.0 ratio=1.000
-- proj_bytes=83575 rel_bytes=344064
AssertionError: a covering projection is not priced from the base table's pages: got 'base-pages' want 'proj-pages'

Fixed: checks run: 6 / 6 pass + 0 fail. Own table pciot / onck / 36000 rows.

Causation

ioProj = seq_page_cost * (double) rel->pages * sel (was projPages):

-- cover_run=41991.6 rel_pages=42.0000000000000000 base_io=42000 ratio=1.000
FAIL  a covering projection is not priced from the base table's pages: got [base-pages] want [proj-pages]

Pytest the same got 'base-pages' want 'proj-pages'. Restored to projPages; both green again.

Ledger: six projection_scan_io rows, majors 15;16;17;18;19. Load-bearing arm last-red 2026-09-19 mutation rel->pages. Census counted: checks_never_observed_red 1426. suites_not_covered stays 249. cluster_tests 437 by collection after the reseat.

Not merged. Not self-approved.

[OffgridwithJD]

Your test work is verified, not read. I ran your removal proof here and it reproduces
exactly the numbers your description claims:

CONTROL   cover_run=19996.0  rel_pages=42.0  base_io=42000.0  ratio=0.476   .so 245cd2b659f1
MUTATED   cover_run=41991.6  rel_pages=42.0  base_io=42000.0  ratio=1.000   .so 7fcfd0357aba
          a covering projection is not priced from the base table's pages:
              got 'base-pages' want 'proj-pages'

Different .so per cell, so each measured its own binary. Six ledger rows across all five
majors, both twins with matching names and neither reading the other, cluster_tests
436 → 437 which matches a collection here. Everything you were asked for last time is
present without being asked again.

One blocker, in the C rather than the tests, and it is four lines.

A projection the lookup cannot find is priced at one page

if (projSid == 0)
    return 1;

That is a fail-OPEN default. One page is essentially free, so ioProj collapses to
nothing, the covering path becomes the cheapest thing available, and the planner takes it
— on the strength of a lookup that just failed. The direction is exactly backwards: a
lookup failure should make the path look expensive, not free.

It should not happen, I agree — the path is only built when a covering projection was
found. But "should not happen" is the state that gets reached by a route nobody modelled,
and the cost of being wrong here is silent: the plan changes, nothing errors, and the
number that caused it is unreachable from SQL. rel->pages as the fallback keeps the old
behaviour for that case and cannot make the path look better than the base scan; an
elog(ERROR) would also be defensible since you believe it unreachable. Either is fine —
returning 1 is the one I would not ship.

Same shape one line down:

if (pages < 1)
    pages = 1;

That one I would keep: a real projection occupying less than a page is genuinely close to
free, and the clamp is arithmetic rather than an error path. Worth a word in the comment
saying the two 1s mean different things, because they read identically.

A question rather than a blocker: this scans every row group at plan time

pgcolumnar_projection_pages calls PgColumnarReadRowGroupList and sums every group, on
every planning of a relation with a covering projection. There is explicit precedent
against that in the tree, in pgcolumnar_written_stripe_row_limit:

The exact quantity is the real group count, and reading it is a scan proportional to the
number of groups on every plan, which is too much to spend refining a term that is
approximate by construction.

Your case is not identical — that comment is about refining an approximate term, while
this is the I/O estimate itself, so the trade may well be worth it. But the two decisions
now point opposite ways in the same planner for the same reason, and whoever reads them
next deserves to know that was considered rather than missed. I would put a sentence in
your function's header saying why this one earns the scan, and I would want @jdatcmd's
view on the planning cost on a table with many row groups, since that is an owner call
rather than mine.

Smaller

Your description says cluster_tests 437 "by collection on current main (18821ce)". 437
is on your branch; main is 436. The number is right and the derivation is right — only the
sentence attributes it to the wrong tree, and I mention it because a reader checking your
work against main will find 436 and wonder which of you is wrong.

CI is UNSTABLE, 2 of 14 outstanding, so I would not have approved this round regardless.
Fix the fallback and I will re-run the proof against the new head.

[jdatcmd]

Your two red legs are one defect, and the fix will change under you unless you rebase first. Both, plus the numbers you will need, below.

The two failures are the same assertion

pytest (harness guards, no database)
  FAILED test_docs_cover_the_corpus.py::test_the_contents_list_is_numbered_in_order
    with one contents entry per section: got 66 want 67

pytest (cluster tests, with the driver)
  FAILED test_harness_deps.py::test_the_guard_half_of_the_corpus_runs_without_a_database_driver
    the no-cluster files pass with psycopg absent: ...got 66 want 67...: got 1 want 0

The second leg runs the guard half in a subprocess and asserts it exits 0. It did not, because of the first. So there is one thing to fix, not two — the cluster leg goes green on its own when the guard leg does. Worth saying because two red legs naturally read as two problems, and chasing the second one leads nowhere.

The defect itself: TESTS.md gained a section and the contents list at the top did not gain its entry. 67 numbered ## N. headings, 66 - [N. ...] lines.

But do not fix the number yet — rebase first

main has taken three merges since your run at 18:27 today:

18821ce  #1152  test_temporal.py                                      section 66
f12caa2  #1156  test_native_parquet_dict_oob.py, test_advisory_lock_class.py   67, 68
7a1095f  #1147  test_index_am_support.py                              69

So on current main:

TESTS.md        69 numbered sections, 69 contents entries
guard_tests     382
cluster_tests   441

Your branch carries cluster_tests 437, which was right for the tree you derived it on and is wrong for this one. Rebase, then re-derive by collection — never by adding your delta to 441. Three branches did exactly that today and every one of them landed on a number no tree collects; the file itself says so four times and it was still true twice more this afternoon. The recipe is in expected_tests.txt:

cd test/pytest
F="$(python3 -c 'import sys, pathlib; sys.path.insert(0, "."); from test_harness_deps import NO_CLUSTER;
    print(" ".join(p.name for p in sorted(pathlib.Path(".").glob("test_*.py")) if p.name not in NO_CLUSTER))')"
PYTHONPATH=. pytest --collect-only -q --pg-config <your pg_config> $F | tail -1

Re-derive guard_tests in the same run even though you expect it not to move. It is two seconds and it is the only way to know.

Your new section will be 70, and it needs both the heading and the contents entry — that is the pair the guard checks.

Running the guard corpus locally, which would have caught both

Neither leg is reachable from run_all_versions.sh — nothing in the matrix runs pytest. That gap cost me two CI rounds on #1147 this afternoon for exactly this class of failure, so it is not a criticism of your workflow:

python3 -m venv /tmp/pgcvenv
/tmp/pgcvenv/bin/pip install -q $(grep -E '^(pytest|pytest-xdist)==' test/pytest/requirements-test.txt)
cd test/pytest
G="$(python3 -c 'import sys; sys.path.insert(0, "."); from test_harness_deps import NO_CLUSTER; print(" ".join(NO_CLUSTER))')"
PYTHONPATH=. /tmp/pgcvenv/bin/pytest -q --pgc-expect-tests "$(awk '$1=="guard_tests"{print $2}' expected_tests.txt)" $G

That needs no cluster and no driver, and it runs in about twelve seconds. It reproduces the first failure exactly.

The C blocker is unchanged

@OffgridwithJD's if (projSid == 0) return 1; point still stands and is separate from all of the above. Their reasoning is right: a lookup failure should make the path look expensive, not free, and rel->pages or an elog(ERROR) both do that while returning 1 does not.

Your test work was verified rather than read — they reproduced your removal proof with a different .so per cell — and that is the part that usually needs another round here. It did not.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

[jdatcmd]

Updating the numbers in my previous comment — main has taken #1160 and #1161 since I wrote it.

main is now e7eafcfe:

TESTS.md         70 numbered sections, 70 contents entries
cluster_tests   442
guard_tests     382

So your new section is 71, not 70, and the cluster_tests you derive will be against 442 rather than 441.

The diagnosis itself is unchanged and is still the whole of what is wrong:

one defect  -- TESTS.md gained a section and the contents list did not gain its entry
two reds    -- the cluster leg runs the guard half in a subprocess, so it fails on the first

Fix the contents entry and both legs go green. Derive the counts with the recipe rather than from my figures — I have now had two sets go stale inside an hour, and that is exactly why the file says re-derive rather than carry.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK