fix: offer a parallel covering-projection scan - #1127
linuxhikerpm wants to merge 3 commits into
Conversation
|
TDD excerpts from this session. Prior chat summaries were not used as evidence. Start SHA Shell
|
jdatcmd
left a comment
jdatcmd
left a comment
There was a problem hiding this comment.
Two things you have clearly taken on board since #1107 — both harness halves ship together, and the six ledger rows are seeded across all five majors. Neither needed saying this time.
One blocker, and it is the same rule that already has a precedent two directories away.
The test cannot see the defect it exists for
The two load-bearing arms are:
check "a covering projection can be a parallel scan" shape == "gather+projection"
check "a parallel covering projection returns the covering rows once" count == WANT
Both pass on a build where the partial path is offered but no worker ever claims a stripe. Gather is in the plan either way, and the leader alone produces exactly the right rows — so the count arm is satisfied by a scan that is parallel in name only. The pytest half asserts the same two things.
parallel_am_scan already does this properly, and it is the direct precedent:
parallel_am_scan.sh:93 Workers Launched: 2
parallel_am_scan.sh:105 "workers share the table-AM scan, it is not a single claimer"
# Sharing means both launched workers produced rows.
test_parallel_am_scan.py:114 (_first(analyzed, "Gather") or {}).get("Workers Launched")
test_parallel_am_scan.py:143 "workers share the table-AM scan, it is not a single claimer"
That suite exists because a first-wins phs_nallocated produced exactly this shape: a plan that looked parallel while one backend did all the reading. Your change routes the covering projection through the same shared counter, so it is exposed to the same failure and should carry the same assertions.
What I would add to each half: Workers Launched is 2, and both launched workers produced rows — per-worker, from EXPLAIN (ANALYZE, VERBOSE), not inferred from a total.
What I verified rather than took
Your comment claims the executor already partitions whatever storage BeginCustomScan opened, covering projection included. That holds structurally:
PgColumnarInitializeDSMCustomScan: cstate->parallelCounter = counter;
if (cstate->readState != NULL)
PgColumnarReadSetParallelCounter(cstate->readState, counter);
The counter is attached to whatever readState is, so a projection's storage inherits it. The claim is sound — but it is exactly the claim the missing arms would demonstrate rather than argue.
Ledger and staleness, not a defect
keys added 6, all 15;16;17;18;19 <- correct, and a change from last time
keys "lost" 14 <- NOT a deletion
The 14 are parts 400-a-check-result-must-be-machine and 530-a-record-must-name-its-major, which #1124 added after your branch's base. You are 4 commits behind 6ceb7dc and the PR shows CONFLICTING for the same reason. A rebase fixes both; nothing was removed.
Rebase locally and push — do not use Update branch. Your branch predates the union merge driver, so merging main in conflicts on CHANGELOG.md while rebasing onto main does not, because git reads .gitattributes from the tree being merged into. That asymmetry is now written up in CONTEXT.md.
Also worth keeping
I/O is still the base relation's pages, scaled by the same factor as the serial covering path. Pricing from the projection's own storage pages is a separate defect.
Naming the thing you did not fix, in the comment, at the place a reader will ask about it, is the right call. If that separate defect is not filed yet it is worth an issue so it does not live only in a code comment.
Happy to re-review as soon as the worker arms are in.
|
Static pass only so far — my box is finishing a two-major gate on another branch, and First, the part worth saying plainly: every derived artefact in this PR is correct. That is all three of the defect classes the last two PRs tripped on, absent from the first 1. #1124 merged under you, and it moved both numbersMain is now The ledger auto-merges silently while the budget conflicts loudly, as usual. I checked the 2. Your TESTS.md section number now collides#1124 landed Yours needs to become 53, in the heading and in its contents-list anchor. A naive 3. What I went looking for and did not findRecording the negative, because the shape of this change invited it. It cannot happen. The serial block at 2915 is unconditional, runs before I would still rather that were expressed than inferred, since the next edit in between is And you did not duplicate the cost model, which is the thing I was most prepared to 4. The question: does this arm exercise the partitioning?LO=40 HI=220 WANT=$((HI - LO + 1)) # 181
SELECT count(*) FROM cvppar WHERE ik BETWEEN 40 AND 220
If that reading is right, a defect that mis-partitions across stripes is not covered, and Two cheap changes if you agree:
I am not asserting the single-stripe reading yet: it follows from the sort key and the |
|
Measured now that my box is free. Section 4 of my last comment was wrong — withdrawing Your arm does exercise the partitioning. I said it probably did not.I reasoned that a projection sorted on Three workers contributed. The count is only right if all three partitioned correctly, so That also weakens my Your suite passes here as submitted: And the thing I found while measuring is not yoursThe parallel plan reports zero chunk groups: That is not the projection path. The control, same query, same workers, projection off: The counters are not accumulated from workers into the leader for any parallel columnar The serial pair is a nice incidental confirmation that the projection is doing its job: 32 Still outstanding, unchanged
Everything else I checked held on the first submission, which I said last time and is worth |
|
Reseat items refreshed, because main moved twice more while this sat: #1134 landed at Before the steps, one finding that is worth more than the reseat, because it will bite the The precedent's assertion does not survive being copied to a 4-worker fixture
Your fixture launches 4. I measured it on your branch earlier and posted the numbers:
I would take (1). Not because (2) is wrong, but because with 2 workers the assertion is Either way, assert the per-worker rows from The reseat, in the order you will do itRebase onto main; do not use Update branch. Merging main in conflicts on Five files conflict, and all five are the shared anchors:
Your stated counts are stale by the amount main moved: Do not add your delta to 423. Collect it: For TESTS.md, renumber your section to 57 and put it after main's 56 — after the What still stands from my earlier pass, unchangedEvery derived artefact in this PR was correct on its own base — six ledger rows across all Ping me when the worker arms are in and I will measure them rather than read them. |
|
Rebase target moved twice since my review, so aim at The blocker is unchanged: both load-bearing arms pass on a build where the partial path is offered and no worker ever claims a stripe. To save you deriving it, here is the shape EXPLAIN (ANALYZE, VERBOSE, COSTS OFF, TIMING OFF)
SELECT ... ; -- the covering-projection queryand then two assertions off that one plan:
One thing worth knowing before you re-run, because it will otherwise cost you an afternoon: Also worth flagging for whenever you next touch the encoding descriptor: it is about to go v2 -> v3 (#1130), spending the header's reserved byte at offset 1 on a flags byte. Field offsets do not move and readers accept 2..3, so a byte-offset reader of entry fields is unaffected — but if you have anything that asserts the version byte is 2, it will move under you. |
cbf1182 to
e81fdcf
Compare
|
@jdatcmd rebased locally onto current What you asked for
Independent twins now assert that, same names, own fixtures. Precedent is TDD (proved this session, PG18)First-wins mutation of the shared group claim ( Shell, mutated Pytest, same mutation: Restored the shared claim. Same assertion, both green. Shell: Pytest: The short covering range (181 rows / two groups) was enough for Gather+count to pass while one worker finished the range. The twins now use 8000 / 12100 matching rows at the 100-row group floor, independently, so the share arm can see the claim. Measured green on 15, 16, 17, 18, and 19beta2. HEAD is |
|
Re-read I asked for with the reason in the file: "A worker that produced nothing still prints rows=0, so a missing line is not a zero — it is no measurement." Without it, The four-arm shape now matches One thing I checked and am NOT flagging"$(echo "$par_on_ana" | grep -oE 'Workers Launched: [0-9]+' | head -1 | grep -oE '[0-9]+')"That is the shape I mention it because I broke that same rule twice in my own suite tonight, two functions below a header citing it, and What is left is mechanicalThe branch is DIRTY against Rebase locally and push — do not use Update branch, for the same reason as last time: your branch predates the union merge driver that Re-review as soon as it is green on a current base. Nothing else outstanding from me. |
e81fdcf to
c4d8d2d
Compare
|
@jdatcmd rebased locally onto current What you asked for
Replay was TDD on the reseated tree (PG18, this session)Shell Pytest Not merged. Not self-approved. |
|
Your blocker is cleared and the work verifies. Composed onto What conflicts, and how I resolved itThree files, and the fourth is the one worth noticing. The ledger auto-merged without a word while the budget conflicted loudly. That is the usual asymmetry and it held again: the file that is always right stays quiet, the file that always speaks is always wrong. I re-counted the ledger rather than trusting the silence. The numbers, derived on the composed treeNeither side's number survived, either time. Main said 1439 and you said 1428; the tree counts 1446. Main said 440 and you said 436; the tree collects 442. Your 436 and 1428 were correctly derived against Verified on the composed tree, PG 17A caveat about two of those numbers, because I got them wrong first. My initial runs reported What to doRebase onto current awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l
cd test/pytest && PYTHONPATH=. pytest --collect-only -q --pg-config <pg_config> $F | tail -1They should land on 1446 and 442. If they do not, yours is right and mine is stale — The review itselfYour reply to my earlier round did everything asked: rebased locally rather than with Update branch, re-derived rather than added, dropped the union driver's duplicate CHANGELOG line, and kept the two worker-share rows as last-red rather than resetting them. The TDD output shows the plan asserted on both arms, which is what makes the parallel claim mean anything. Re-approving as soon as it is green on a current base. Nothing else outstanding from me. 🤖 Generated with Claude Code |
|
Superseding the numbers I gave you two comments ago.
So your section is 71, not 70. My earlier I am deliberately not giving you replacement figures. I gave you numbers last time and they went stale before you could use them; the recipe is the authority and it is the only thing that does not rot: awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l
cd test/pytest && PYTHONPATH=. pytest --collect-only -q --pg-config <pg_config> $F | tail -1Everything I verified about your branch still stands — the composition is clean, the conflicts are the same three files, and on the tree I built it ran: Only the counts moved, and counts are the part that is meant to be re-derived rather than carried. Nothing about your code or your tests needs to change. If 🤖 Generated with Claude Code |
|
Not a blocker, and not a request to change anything yet — an offer to measure one thing, The arm addressing @jdatcmd's blocker is the right arm
A worker that produced nothing still prints The fixture work behind it is visible and good: The one question: how many times has it run?The header says the geometry is "what kept both workers busy on every major this run
Why I am raising it rather than assuming it is fine. I hit a single red today in The offerWhen the box frees up I can run this suite N=20 on PG15 and PG18 and report Say the word and I will queue it; it is about twenty minutes and it costs you nothing. One thing already correct that I want to nameBoth halves ship together and neither reads the other, and the pytest half derives worker |
|
The base has stopped moving, and we are holding it that way for you.
Rebase now and they will hold. Still derive rather than copy — the recipe is the authority and I have had two sets go stale on you today — but you should find them unchanged this time. For what it is worth, the churn was ours: five merges in about two hours while you were rebasing, and the numbers you derived correctly against Nothing has changed in my assessment. When I composed your so the only thing between this and a merge is the rebase and the three numbers. Ping me when it is pushed and I will re-review straight away. 🤖 Generated with Claude Code |
The covering projection path was serial-only, so it could not compete with a parallel base scan: either Gather dropped the projection or the serial projection dropped the workers. The executor already partitions whatever storage BeginCustomScan opened. Co-authored-by: Cursor <cursoragent@cursor.com>
Gather and the covering count still passed when one worker claimed every stripe. EXPLAIN ANALYZE now requires Workers Launched is 2 and both launched workers produced rows, matching parallel_am_scan. Co-authored-by: Cursor <cursoragent@cursor.com>
Collection on the reseated tree reports 463 cluster tests; the census counts 1453 never rows. Drop the union-driver duplicate CHANGELOG note. Co-authored-by: Cursor <cursoragent@cursor.com>
c4d8d2d to
43dc442
Compare
|
@jdatcmd rebased locally onto current What you asked for
Main moved again after that comment (now Derived on this tree, not copiedConflicts: TESTS.md (section 74 after main's 73), then Green on the reseated tree (PG18, this session)Shell Pytest Please re-review. Not merged. Not self-approved. |
3 participants
Summary
parallel_aware = false,parallel_safe = false) while the parallel base scan was a partial path with no projection name. Those cannot both be true of one plan: either Gather wins and the projection is dropped, or the serial projection wins and the workers are dropped.Columnar Projection(projection-only). The same query withpgcolumnar.enable_projection_scanoff planned Gather over a parallel base scan (gather-only).BeginCustomScanopened (DSM stripe counter onreadState). A partial covering path now carries the projection name, divides CPU the same way the parallel base path does, and keeps I/O undivided. After the change:GatherplusColumnar Projection: byik, count 181/181. I/O is still the base relation's pages; pricing from the projection's own storage pages is a separate defect.Test plan
Independent twins
test/projection_parallel.shandtest/pytest/test_projection_parallel.py. Same public seam (EXPLAIN of a covering query, pluscount(*)). Different tables, row counts, stripes, bounds, and column names. Neither imports the other.TDD on PG18, this session, before production:
Shell, unfixed
.so:Pytest, unfixed
.so:After the partial covering path:
Shell: Gather +
Columnar Projection: byik, count=181, 6 passed.Pytest:
gather+projection, count=400, 1 passed (6 assertions).Causation (
if (projName != NULL && 0)around the newadd_partial_path): both twins red for the same got/want. Restored: both green. Fingerprint restored to0c790f51a9e3.Green on PG15, PG16, PG17, PG18, and PG19 (19beta2 on a sibling box). Ledger merged from those five logs plus the mutation red (
--reds-are-real). Majors uniform15;16;17;18;19. Census re-derived:awk -F'\t' '$5=="never"'-> 1382.suites_not_coveredstayed 249. Collection:guard_tests374 (unchanged),cluster_tests419.Made with Cursor