docs: name the index methods that work, and what a range predicate does not prune - #1147
Conversation
OffgridwithJD
left a comment
There was a problem hiding this comment.
You asked for a page edit that changes what the sentence CLAIMS while the extractor
returns the old set. There is one, it defeats both harnesses, and it is the edit a
person would actually make.
The attack
The extractor harvests backticked lowercase names from the window. Negation lives in the
prose, and neither reader parses prose. Put the denial inside the window:
-- `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a
+- `CREATE INDEX` builds `btree`, `hash` and `gist` but NOT `spgist` indexes over a
columnar table.Both halves, on your branch, nothing else changed:
shell -- docs/features.md claims: btree gist hash spgist
19 passed + 0 failed index_am_support.sh: PASSED
pytest -- docs/features.md claims: btree gist hash spgist
19 pass + 0 fail 1 passed
The page now says SP-GiST is not supported. The suite proves it works, by name, and
reports green. That is worse than no coupling: the tracker says the page is pinned, so
nobody re-reads it.
A second, weaker variant shows the two halves can also silently disagree. With the
denial placed after columnar table instead:
shell claims: btree gist hash spgist (awk appends the whole line, then stops)
pytest claims: btree gist hash (the non-greedy regex stops before it)
Both pass. Your comment says "neither inherits the other's blind spots", which is true and
is exactly why they can return different answers to the same question and neither notices.
Nothing compares the two extractions.
What I would do about it
Not "parse the negation" — that is a losing game. Refuse the sentence you cannot read,
which is the pattern compare_to_bash.py already uses for a helper whose name argument it
cannot resolve:
if the captured window contains not | never | except | unsupported | rather than
REFUSE: "the CREATE INDEX sentence has a shape this cannot read; it names
btree, hash, gist, spgist but also qualifies them. Split the claim
and the exclusion into separate sentences."
A refusal is honest where a guess is not, it costs about four lines, and it turns the
dangerous edit into a loud one. The alternative — stop the window at the first comma-list
terminator — is more fragile, because the next person writes the list differently.
Worth asserting the two extractors agree, too, but that is a cross-harness comparison and
CONTEXT.md forbids one half invoking the other. The honest version is that each half
refuses independently on the same rule.
Your GIN/BRIN question: "builds" is not enough, but not on features.md
Leaving them off the features page is right — that page is what works, and a method no
plan chooses does not. But silence is what caused this PR. A reader ran CREATE INDEX ... USING gin, it succeeded, and nothing told them the planner would never choose it;
that is the same shape as the reader who concluded GiST was unsupported because the page
did not name it.
So I would put one line in docs/limitations.md, where you already added "Which
predicates prune":
A GIN or BRIN index can be created on a columnar table and will build. No plan has been
observed to choose GIN. BRIN summarises physical block ranges, which do not correspond
to columnar storage layout, so a BRIN index over a columnar table is not known to be
meaningful — whether it should be refused atCREATE INDEXis #1143.
That documents the state without claiming it is a feature, and it makes the open question
visible to the person most likely to hit it. You made the judgement one way; I would make
it the other, and the reason is that the cost of silence here has already been paid once.
The rest, checked
extractor premise present, and reddens when the sentence is stripped
per-method premise a name with no probe fails rather than skipping
doc_parallel_premise the right precedent, and cited
gist / spgist `&&` 119 = 119 and `@>` 60 = 60 against the scan path,
so the arms compare an index answer to a known truth
The three documentation defects you found are all real and the fix for each is in the
right file. I am blocking only on the extraction, because a doc guard that reads names
instead of claims is a guard that will certify its own contradiction.
9a075eb to
cd20e32
Compare
@OffgridwithJD broke the extractor, and the second way was worse than the firstI asked them to find a page edit that changes what the sentence claims while the extractor still returns the old set. They found one, and then found a second that defeated something I had not thought to look at. The edit: A name-harvesting extractor reads The second edit exposed a defect in the pair, not the page. Writing the denial after The fix: refuse what you cannot readTheir suggestion, and it is the right one — do not parse negation, that is a losing game. The rule is the one Both halves now also read the same unit — the whole list item — while still reading it differently (awk walks lines, the regex takes the bullet). Verified they agree on the current page: 5 lines, the same four names, from each side.
And I took their judgement over mine on GIN and BRINI had left them off the page and said nothing anywhere. They argued that silence is what caused this whole PR: a reader ran
Verification on
|
cd20e32 to
2739684
Compare
|
Reseated onto Everything re-derived on the merged tree rather than carried:
Full PG17 matrix on The review fix from the last round — the negation refusal, with both of @OffgridwithJD's page edits as removal proofs — is unchanged by the reseat; only the section number and the collection count moved. |
OffgridwithJD
left a comment
There was a problem hiding this comment.
The refusal stops the edit I demonstrated, in both halves. It does not stop the same
edit written with "cannot", and I think the lesson is that a denylist was the wrong shape
rather than that it needs another token.
Measured on 2739684
A the page as written shell 20 passed pytest 20 pass
B "builds `btree`, `hash` and `gist`
but NOT `spgist` indexes" shell REFUSES pytest REFUSES
C "...over a columnar table, though `spgist`
cannot be chosen by the planner." shell 20 passed pytest 20 pass
B is fixed — -- refusing the claim window; it contains: NOT, one arm red in each half,
nothing else moved. C is the same inversion in a more natural sentence, and
\b(not|never|except|unsupported|rather than|but no)\b does not match inside cannot
because there is no word boundary before the not.
Why I would not just add cannot to the list
Because the next one is isn't, then no longer, then save for, then with the exception of, then a sentence that inverts the claim with no negation token at all
("spgist is listed for completeness"). A denylist of negations is the same losing game
as parsing the prose — I said that about parsing and then proposed a denylist, which was
my error.
The robust shape is a whitelist of the claim, not a denylist of its inversions. The
sentence this suite depends on has exactly one legitimate form:
- `CREATE INDEX` builds <a comma/and list of backticked names> indexes over a columnar table.
Refuse anything the window contains BEYOND that — a second sentence, a subordinate clause,
any prose after the full stop. Then C is refused for the right reason ("there is more here
than the claim this suite can read") rather than because someone predicted the word. It is
about the same amount of code, it cannot be walked around by vocabulary, and it fails in
the safe direction: a page edit that adds legitimate prose to that bullet gets a loud
refusal asking for it to be moved to its own sentence, which is a five-second fix and
exactly the discipline the suite is trying to impose on the page.
If you would rather keep the denylist and add cannot, say so and I will approve that —
it is your suite and B was the edit that motivated the work. I am flagging the shape
because the guard's whole value is that the page cannot drift, and a guard that certifies
a page saying "spgist cannot be chosen" while proving spgist works is the same defect
as the one you just fixed, not a smaller one.
Everything else re-checked
both halves now read the same unit the whole bullet, verified they agree on the
current page: same four names from each
refusal wording names the offending token and says what to do
per-method premise a name with no probe still fails rather than skips
extractor premise still reddens when the sentence is stripped
GIN/BRIN in limitations.md, "build, and nothing has been
seen to use them", pointing at #1143
The GIN/BRIN line reads exactly as I would want it: it documents the state without
claiming a feature, and it puts the open question where the person most likely to hit it
will find it.
CI is green on 2739684 — 14 of 14, non-green 0 — so this is the only thing between me
and an approval.
2739684 to
ee07c8c
Compare
|
Pushed The whitelist round found two defects in my own work, both caught by the harness rather than by me
claim_line | grep -qE "$CLAIM_FORM" || return 0
printf '%s' "$CLAIM" | grep -qE "$CLAIM_FORM"
That is the answer to why #486 is a suite and not a convention. I wrote "piping a captured string into a reader that exits early is what selftest/080 refuses" at the top of this file and then did it while fixing something else.
And the rename left an orphan the tooling would not let me discard quietlyRenaming the check made its ledger row an orphan carrying history, and
I took the second path, because this is not the old check renamed: the old one screened for negation tokens and was defeated by One more silent merge worth namingThe reseat onto |
|
The whitelist fixes both attacks. Four cells on C is the one that mattered — it walked through the denylist untouched and the whitelist D is not a mechanism problem, it is a message problemThe page in cell D reads: Both halves pass, and I do not think they should refuse — a contradiction in a But that state is exactly what your refusal message asks the author to produce:
So a reader who hits the refusal, follows the instruction to the letter, and splits the Suggested wording, and it costs one clause:
That tells the author both what to do and what they are giving up, which is the part the Not approving yet, and it is CI rather than the codeApproving into that would be approving something I have not seen finish. Ping me when it On your two harness catches
And recording a fresh red against the new check name before letting the orphan row go is |
ee07c8c to
a4a50ed
Compare
|
Round 4 on The blocker: whitelist, not denylistYour Measured on D is refused on purpose and the message says to move it to its own sentence. And I took your own correction one step further, because that instruction steers the author into the one arrangement the guard is blind to. It reads ONE bullet, so an exclusion in a NEIGHBOURING bullet contradicts the claim and nothing sees it. Widening the window makes every unrelated edit to the section a refusal, so instead the refusal now names where an exclusion belongs — Two CI guards my local matrix cannot reachCI was red on The second is the interesting one. I had written the premise as expect.at_least(len(ams), 1,
"premise: the features page names at least one index access method")Same check name, so parity and the ledger are untouched. Removal proof, run rather than reasoned — the document cannot reach this arm any more (a readable bullet names at least one method by construction), so the mutation is of the extractor: I asserted the mutant still parses and that the restored file is byte-identical to the tree, after the stale- A third defect, found by running the guards rather than the matrixFixing the first one, I wrote The rebase onto main carrying #1152, which is the file's own warning happeningBoth sides said 436, so git resolved the VALUE silently and conflicted only on the comment above it. A reader who trusted that resolution ships 436. Re-derived by collection on the merged tree: Your section took 66, so mine is 67 in TESTS.md and the contents entry moved with it. Everything re-run on the rebased tree, PG 17One thing I will state as unverified rather than claim: I did not run the ledger gate locally. It needs 🤖 Generated with Claude Code |
|
Correcting the provenance of one table in the comment above, and the correction is the interesting part. I wrote the A/B/C/D table from the earlier round's record rather than from a run on The mutation had not applied. The bullet wraps across two source lines, the extractor joins them, and my Re-run with a mutator that asserts its own anchor ( So the table I posted was right, and I had no business posting it before the run. Reaching the right answer from the record and reaching it from a run are the same sentence on the page and different claims. One asymmetry worth naming, since it is visible in the output. The shell reports TWO reds on each mutation — the form premise and the names premise — while pytest reports ONE. That is the cause-before-consequence ordering doing its job: pytest stops at the first failing assertion, so a reader sees "the bullet is unreadable" rather than "the page names nothing", which is the consequence. The shell runs every check and prints both. Neither is wrong, and the reason is in TESTS.md, but the counts differ and a reader comparing the two halves arm-for-arm should know why. 🤖 Generated with Claude Code |
…es not prune
docs/features.md said "CREATE INDEX builds btree and hash indexes over a columnar
table" from the day it was written. A reader of that sentence asked whether range
and multirange columns were supported at all, which is the right conclusion to
draw from it and the wrong fact.
Measured on PG17, 20,000 rows: btree, hash, gist, spgist, gin and brin all BUILD
on a columnar table. GiST and SP-GiST also ANSWER, returning the same rows the
scan path returns (&& 119 = 119, @> 60 = 60). So the sentence was wrong by
omission, and nothing in the tree could see that it was.
THE SUITE READS THE SENTENCE RATHER THAN CARRYING A LIST. test/index_am_support.sh
and its twin extract the backticked method names out of docs/features.md and
exercise every one they find, on the precedent of doc_parallel_premise.sh, which
extracts the published query out of limitations.md for the same reason: a suite
holding its own copy of a claim cannot watch the page drift away from it. Both
removal proofs are mutations of the DOCUMENT:
add `gin` to the sentence with no probe the per-method premise fails,
19 passed + 1 failed
remove every name from the sentence the extractor premise fails; the
loop runs zero times and cannot
pass vacuously
The oracle is the scan path with every index disabled, not a literal, because the
property is that the two paths agree rather than that today's fixture returns a
particular number. Each method also asserts its predicate matches rows at all:
two empty sets agree trivially.
GIN AND BRIN ARE DELIBERATELY ABSENT from the page and from the suite. Both
build; neither has been observed in a plan the planner chose, and building is not
the same as being usable. Tracked as #1143. Adding either to the page without a
probe turns this suite red, which is the point of extracting the list.
Two further documentation gaps this closes:
- Temporal constraints were undocumented. PostgreSQL 18 WITHOUT OVERLAPS keys and
19's UPDATE ... FOR PORTION OF work on a columnar table, and test/temporal.sh
has asserted heap-identical behaviour for both since #448. No user-facing page
said so. The entry names the btree_gist prerequisite, which is real.
- Nothing said which predicates prune a native scan. docs/limitations.md now has
"Which predicates prune": the scan resolves an operator through the column
type's btree family and takes the five btree strategies, so overlap and
containment are filtered after decode. Measured, 200,000 rows, one tstzrange
column: && and @> push down 0 filters and probe 0 zone maps, while > pushes
down 1 and probes 2. Zone maps ARE written for range columns and cannot answer
overlap, because the maximum under the range btree ordering is not the range
with the greatest upper bound. Tracked as #1144.
Nineteen checks in each harness, green on PG15/16/17/18/19, compare_to_bash
missing: 0. Ledger rows seeded from five real runs merged in one call, so each
carries 15;16;17;18;19; suites_not_covered does not move because registering the
suite and seeding it happen in this change.
REBASED ONTO main CARRYING #1141 AND #1142, and every derived number re-derived
on the merged tree rather than carried. This branch first stated cluster_tests
431 against a base of 430, and #1142 moved the base to 433 underneath it, so the
committed number described a tree that no longer existed: it is 434 by
collection. TESTS.md took section 61 and now takes 64, because #1142 landed 61
through 63 first. The census and the coverage ceiling reconcile unchanged at 1439
and 249.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
a4a50ed to
03f25d5
Compare
|
Reseated onto main carrying your #1156, now The collision I flagged in my review of #1156 landed exactly as described, and this time git conflicted. Neither survives. Your two sections took 67 and 68, so Re-run on the reseated tree, PG 17Two issues out of the #1156 review, both reproduced here before filing:
The whitelist blocker and the two CI guards are addressed in the two comments above. Ready for round 4 whenever you are. 🤖 Generated with Claude Code |
|
Round 4 run on B and C are fixed and the whitelist is the right shape. D does not refuse here, and I Mine puts the exclusion in a new bullet: That cannot refuse, structurally, and not by accident: _BULLET = re.compile(r"^- `CREATE INDEX` builds.*?(?=^- |^\s*$)", re.S | re.M)The window stops at the next This is not a defect and I am not asking you to close it. D passing is exactly what I I raise it because the difference matters for what the guard is documented to cover. If Tell me which D you ran and I will re-run whichever of us is testing the other thing. Not approving this round, and it is CIThe review itself is done: B and C refuse in both halves, the loop-coverage premise is now On your two findings#1158 is the more valuable one and it explains something I reported to you as a #1157: you are right and my docstring undersells it. I claimed the contention arm |
|
Correcting the D row of my table. @OffgridwithJD is right, our D cells are different edits, and the way I wrote mine would have put a false property into the record. I ran their D here: It passes, in both halves, and it is meant to. The mutator asserted its anchor matched once and that the file changed, so this is not another anchor that missed. So the row should have been two rows: The window is one list item. Two rounds of this now come from the same root: I wrote a result down in a form that was true of what I ran and not of what the sentence says. The first time the mutation had not applied; this time it applied to a different thing than the reader would assume. The anchor assertion fixed the first and does nothing for the second — naming the edit is what fixes the second, so both rows above say where the text went, not just what it said. On the CI state
🤖 Generated with Claude Code |
OffgridwithJD
left a comment
There was a problem hiding this comment.
Approving 03f25d5. Four rounds, and the thing that changed between round 2 and round 3
is the reason this is worth approving rather than the wording.
What I ran, on this exact commit
A control shell 20 passed pytest 20 pass
B "...and `gist` but NOT `spgist`..." REFUSED REFUSED
C "...though `spgist` cannot be chosen..." REFUSED REFUSED
D2 exclusion in a separate `- ` bullet 20 passed 20 pass (by design)
Mutator asserting its own anchor matched exactly once and that the file changed, so
this table is not the artefact the earlier one was.
C is the one that matters. It walked through the denylist untouched, and the whitelist
stops it without anyone having predicted the word cannot. That is the difference between
enumerating the ways a sentence can be wrong and stating the one way it may be right — and
it is why I pushed back on my own suggestion rather than asking you to add a token.
D2 passes and should: an exclusion in a neighbouring bullet is outside what a suite reading
one sentence can police, and widening _BULLET's (?=^- |^\s*$) to catch it would make
every unrelated edit to that section a refusal. Your D1/D2 split is the correct record.
The three sentences I asked for already existed
I asked for one saying the window is one bullet. There are three — docs/features.md, the
suite header, and TESTS.md — and you were right to point at them rather than add a fourth.
I read the suite header first and it settles it. Pointing the refusal at
docs/limitations.md is better than what I suggested, because it tells the author where
the exclusion belongs rather than only that it cannot live in the claim.
CI, since I made a point of it
03f25d5 14 of 14 completed/success, non-green 0, MERGEABLE/CLEAN
Both suites legs landed, which are the ones that run the ledger gate — the only part of
this change you had called unverified locally, and now measured. I was imprecise earlier
calling it "UNSTABLE, 2 non-green": it was 12 success, 0 failure, 2 pending, and the
distinction matters because waiting for a pending leg on its merits is a different act
from waiting out an amber badge.
The loop-coverage premise
expect.at_least(len(ams), 1, ...) under the same check name is right, and your reading of
why is the part I would keep: the guard recognises a cardinality premise only as len(...)
inside a counted assertion, the shell twin already counts with wc -l, so the flag form
was the half that had drifted. The guard was not being satisfied — it was correct and the
premise was weaker than its twin.
Two rounds of a reporting failure, and the second fix is the transferable one
You had a result true of what you ran and not of what the sentence said — twice, by
different mechanisms. An anchor assertion catches a mutation that did not apply. Nothing
but naming the edit catches a mutation that applied to something other than what a reader
assumes. I had only the first defence in my own tables and have taken the second.
I was wrong to call the merge's rewrite churn and to restore it. `pgc_ledger.py` documents the last column as a `;`-separated SET and writes `";".join(sorted(muts))`. The reordering is the tool's canonical form, not a side effect, so restoring the unsorted rows preserved a non-canonical state the next merge would have fixed anyway. The leading space was the real defect and it is DATA, not the tool: one element of that set literally begins with a space, so sorting puts it first. The row is mine, recorded under #1147. Both rows are now sorted with each element stripped. The file's delta against main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the ledger starts an element with whitespace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
I was wrong to call the merge's rewrite churn and to restore it. `pgc_ledger.py` documents the last column as a `;`-separated SET and writes `";".join(sorted(muts))`. The reordering is the tool's canonical form, not a side effect, so restoring the unsorted rows preserved a non-canonical state the next merge would have fixed anyway. The leading space was the real defect and it is DATA, not the tool: one element of that set literally begins with a space, so sorting puts it first. The row is mine, recorded under #1147. Both rows are now sorted with each element stripped. The file's delta against main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the ledger starts an element with whitespace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
I was wrong to call the merge's rewrite churn and to restore it. `pgc_ledger.py` documents the last column as a `;`-separated SET and writes `";".join(sorted(muts))`. The reordering is the tool's canonical form, not a side effect, so restoring the unsorted rows preserved a non-canonical state the next merge would have fixed anyway. The leading space was the real defect and it is DATA, not the tool: one element of that set literally begins with a space, so sorting puts it first. The row is mine, recorded under #1147. Both rows are now sorted with each element stripped. The file's delta against main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the ledger starts an element with whitespace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
A reader asked whether the recent encoding work helps temporal keys and range indexing. It does not, and answering properly turned up three documentation defects. This fixes all three and covers the new claim with a suite.
The defect that prompted it
docs/features.mdhas said this since it was written:A reader of that sentence concludes range columns are unsupported. Measured on PG17, 20,000 rows — all six access methods build, and GiST and SP-GiST also answer, returning the same rows the scan path returns:
The sentence was wrong by omission, and nothing in the tree could see that it was.
The suite reads the sentence rather than carrying a list
test/index_am_support.shand its twin extract the backticked method names out offeatures.mdand exercise every one they find. The precedent isdoc_parallel_premise.sh, which extracts the published query out oflimitations.mdfor the same reason: a suite holding its own copy of a claim cannot watch the page drift away from it.So both removal proofs are mutations of the document:
`gin`to the sentence, with no probe for itpremise: gin ... has a probe in this suitefails — 19 passed + 1 failedThe oracle is the scan path with every index path disabled, not a literal — the property is that the two paths agree, not that today's fixture returns a particular number. Each method also asserts its predicate matches rows at all, because two empty sets agree trivially.
GIN and BRIN are deliberately absent from the page and the suite. Both build; neither has been observed in a plan the planner chose, and building is not being usable. Tracked as #1143. Adding either to the page without a probe turns this suite red, which is the point of extracting the list.
Two further gaps closed
Temporal constraints were undocumented. PostgreSQL 18
WITHOUT OVERLAPSkeys and 19'sUPDATE ... FOR PORTION OFwork on a columnar table, andtest/temporal.shhas asserted heap-identical behaviour for both since #448. No user-facing page said so. The new entry names thebtree_gistprerequisite, which is real.Nothing said which predicates prune a native scan.
limitations.mdnow has Which predicates prune: the scan resolves an operator through the column type's btree family and takes the five btree strategies, so overlap and containment are filtered after decode. Measured, 200,000 rows, onetstzrangecolumn:span && tstzrange(...)span @> timestamptzspan > tstzrange(...)Zone maps are written for range columns and cannot answer overlap: the maximum under the range btree ordering is not the range with the greatest upper bound. The statistic that would work is the greatest upper bound per chunk, which nothing records. Tracked as #1144.
Verification
census stated 1439, ledger holds 1439: they agreeandcoverage: registered=262 | covered=13, not covered=249, ceiling=249.compare_to_bash.py:missing: 0.Rebased onto main carrying #1141 and #1142, with every derived number re-derived rather than carried. This branch first stated
cluster_tests 431against a base of 430; #1142 moved the base to 433 underneath it, so the committed number described a tree that no longer existed. It is 434 by collection.TESTS.mdtook section 61 and now takes 64, because #1142 landed 61 through 63 first.🤖 Generated with Claude Code
https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK