Skip to content

docs: name the index methods that work, and what a range predicate does not prune - #1147

Merged
jdatcmd merged 1 commit into
mainfrom
docs/index-am-and-temporal
Sep 19, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
docs/index-am-and-temporal

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

A reader asked whether the recent encoding work helps temporal keys and range indexing. It does not, and answering properly turned up three documentation defects. This fixes all three and covers the new claim with a suite.

The defect that prompted it

docs/features.md has said this since it was written:

CREATE INDEX builds btree and hash indexes over a columnar table.

A reader of that sentence concludes range columns are unsupported. Measured on PG17, 20,000 rows — all six access methods build, and GiST and SP-GiST also answer, returning the same rows the scan path returns:

btree (id)    BUILDS     gist (span)    BUILDS + ANSWERS   && -> 119, scan 119
hash (id)     BUILDS     spgist (span)  BUILDS + ANSWERS   @> ->  60, scan  60
gin (j)       BUILDS     brin (id)      BUILDS

The sentence was wrong by omission, and nothing in the tree could see that it was.

The suite reads the sentence rather than carrying a list

test/index_am_support.sh and its twin extract the backticked method names out of features.md and exercise every one they find. The precedent is doc_parallel_premise.sh, which extracts the published query out of limitations.md for the same reason: a suite holding its own copy of a claim cannot watch the page drift away from it.

So both removal proofs are mutations of the document:

mutation result
add `gin` to the sentence, with no probe for it premise: gin ... has a probe in this suite fails — 19 passed + 1 failed
remove every method name from the sentence the extractor premise fails; the loop runs zero times and cannot pass vacuously

The oracle is the scan path with every index path disabled, not a literal — the property is that the two paths agree, not that today's fixture returns a particular number. Each method also asserts its predicate matches rows at all, because two empty sets agree trivially.

GIN and BRIN are deliberately absent from the page and the suite. Both build; neither has been observed in a plan the planner chose, and building is not being usable. Tracked as #1143. Adding either to the page without a probe turns this suite red, which is the point of extracting the list.

Two further gaps closed

Temporal constraints were undocumented. PostgreSQL 18 WITHOUT OVERLAPS keys and 19's UPDATE ... FOR PORTION OF work on a columnar table, and test/temporal.sh has asserted heap-identical behaviour for both since #448. No user-facing page said so. The new entry names the btree_gist prerequisite, which is real.

Nothing said which predicates prune a native scan. limitations.md now has Which predicates prune: the scan resolves an operator through the column type's btree family and takes the five btree strategies, so overlap and containment are filtered after decode. Measured, 200,000 rows, one tstzrange column:

predicate pushed-down filters zone map probes
span && tstzrange(...) 0 0
span @> timestamptz 0 0
span > tstzrange(...) 1 2

Zone maps are written for range columns and cannot answer overlap: the maximum under the range btree ordering is not the range with the greatest upper bound. The statistic that would work is the greatest upper bound per chunk, which nothing records. Tracked as #1144.

Verification

  • Full PG17 matrix on this exact commit: ALL VERSIONS PASSED, 257 of 262 suites ran, 5 skipped, 0 incomplete. Ledger gate in the same run: census stated 1439, ledger holds 1439: they agree and coverage: registered=262 | covered=13, not covered=249, ceiling=249.
  • 19 checks in each harness, green on PG15/16/17/18/19 — five separate runs, which is where the ledger rows came from.
  • compare_to_bash.py: missing: 0.

Rebased onto main carrying #1141 and #1142, with every derived number re-derived rather than carried. This branch first stated cluster_tests 431 against a base of 430; #1142 moved the base to 433 underneath it, so the committed number described a tree that no longer existed. It is 434 by collection. TESTS.md took section 61 and now takes 64, because #1142 landed 61 through 63 first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You asked for a page edit that changes what the sentence CLAIMS while the extractor
returns the old set. There is one, it defeats both harnesses, and it is the edit a
person would actually make.

The attack

The extractor harvests backticked lowercase names from the window. Negation lives in the
prose, and neither reader parses prose. Put the denial inside the window:

-- `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a
+- `CREATE INDEX` builds `btree`, `hash` and `gist` but NOT `spgist` indexes over a
   columnar table.

Both halves, on your branch, nothing else changed:

shell    -- docs/features.md claims: btree gist hash spgist
         19 passed + 0 failed        index_am_support.sh: PASSED
pytest   -- docs/features.md claims: btree gist hash spgist
         19 pass + 0 fail            1 passed

The page now says SP-GiST is not supported. The suite proves it works, by name, and
reports green. That is worse than no coupling: the tracker says the page is pinned, so
nobody re-reads it.

A second, weaker variant shows the two halves can also silently disagree. With the
denial placed after columnar table instead:

shell    claims: btree gist hash spgist      (awk appends the whole line, then stops)
pytest   claims: btree gist hash             (the non-greedy regex stops before it)

Both pass. Your comment says "neither inherits the other's blind spots", which is true and
is exactly why they can return different answers to the same question and neither notices.
Nothing compares the two extractions.

What I would do about it

Not "parse the negation" — that is a losing game. Refuse the sentence you cannot read,
which is the pattern compare_to_bash.py already uses for a helper whose name argument it
cannot resolve:

if the captured window contains  not | never | except | unsupported | rather than
    REFUSE: "the CREATE INDEX sentence has a shape this cannot read; it names
             btree, hash, gist, spgist but also qualifies them. Split the claim
             and the exclusion into separate sentences."

A refusal is honest where a guess is not, it costs about four lines, and it turns the
dangerous edit into a loud one. The alternative — stop the window at the first comma-list
terminator — is more fragile, because the next person writes the list differently.

Worth asserting the two extractors agree, too, but that is a cross-harness comparison and
CONTEXT.md forbids one half invoking the other. The honest version is that each half
refuses independently on the same rule.

Your GIN/BRIN question: "builds" is not enough, but not on features.md

Leaving them off the features page is right — that page is what works, and a method no
plan chooses does not. But silence is what caused this PR. A reader ran CREATE INDEX ... USING gin, it succeeded, and nothing told them the planner would never choose it;
that is the same shape as the reader who concluded GiST was unsupported because the page
did not name it.

So I would put one line in docs/limitations.md, where you already added "Which
predicates prune":

A GIN or BRIN index can be created on a columnar table and will build. No plan has been
observed to choose GIN. BRIN summarises physical block ranges, which do not correspond
to columnar storage layout, so a BRIN index over a columnar table is not known to be
meaningful — whether it should be refused at CREATE INDEX is #1143.

That documents the state without claiming it is a feature, and it makes the open question
visible to the person most likely to hit it. You made the judgement one way; I would make
it the other, and the reason is that the cost of silence here has already been paid once.

The rest, checked

extractor premise      present, and reddens when the sentence is stripped
per-method premise     a name with no probe fails rather than skipping
doc_parallel_premise   the right precedent, and cited
gist / spgist          `&&` 119 = 119 and `@>` 60 = 60 against the scan path,
                       so the arms compare an index answer to a known truth

The three documentation defects you found are all real and the fix for each is in the
right file. I am blocking only on the extraction, because a doc guard that reads names
instead of claims is a guard that will certify its own contradiction.

@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD broke the extractor, and the second way was worse than the first

I asked them to find a page edit that changes what the sentence claims while the extractor still returns the old set. They found one, and then found a second that defeated something I had not thought to look at.

The edit:

- `CREATE INDEX` builds `btree`, `hash` and `gist` but NOT `spgist` indexes over a

A name-harvesting extractor reads NOT spgist as a claim that SP-GiST is supported. The page would then say a method does not work while this suite proves it does, by name, and reports green — worse than no coupling at all, because the suite says the page is pinned, so nobody re-reads it.

The second edit exposed a defect in the pair, not the page. Writing the denial after columnar table gave the shell four names and the pytest regex three. Both passed. They disagreed. Nothing compared them. I had written "neither inherits the other's blind spots" as a virtue in the header, and it was concealing a divergence.

The fix: refuse what you cannot read

Their suggestion, and it is the right one — do not parse negation, that is a losing game. The rule is the one compare_to_bash already applies to a helper whose name argument it cannot resolve. If the claim window holds not, never, except, unsupported, rather than or but no, the premise fails and prints "split the claim and the exclusion into separate sentences". Each half applies it independently; no cross-harness call.

Both halves now also read the same unit — the whole list item — while still reading it differently (awk walks lines, the regex takes the bullet). Verified they agree on the current page: 5 lines, the same four names, from each side.

mutation shell pytest
but NOT spgist in the sentence 19 passed + 1 failed same arm, nothing else
the denial after columnar table 19 passed + 1 failed same arm, nothing else
restored 20/20 20/20

And I took their judgement over mine on GIN and BRIN

I had left them off the page and said nothing anywhere. They argued that silence is what caused this whole PR: a reader ran CREATE INDEX, it succeeded, and nothing said the planner would never choose it — the same shape as the reader who concluded GiST was unsupported because the page did not name it.

limitations.md now has "GIN and BRIN build, and nothing has been seen to use them", with the GIN question open and the BRIN one named as a design question rather than a tuning one, both pointing at #1143. It documents the state without claiming a feature.

Verification on cd20e32

  • Full PG17 matrix: ALL VERSIONS PASSED, 257 of 262 ran, census stated 1439, ledger holds 1439: they agree, coverage: registered=262 | covered=13, not covered=249, ceiling=249.
  • 20 checks in each harness, green on PG15/16/17/18/19 — re-seeded across all five after the arm was added.
  • Two of the twenty now carry an observed red in the ledger, both from mutating the document. The third mutation (gin with no probe) is deliberately not merged: it reddens a check whose name exists only under that mutation, and a ledger row no clean run can produce is what the orphan scan refuses.

TESTS.md section 64 collides with #1148, which took 64 and 65 first. I will reseat, not them — this branch has already been rebased once tonight and theirs should not wait on mine.

@jdatcmd
jdatcmd force-pushed the docs/index-am-and-temporal branch from cd20e32 to 2739684 Compare September 19, 2026 16:27
@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Reseated onto 3a2ef2f (main took #1148 while this was in review) and pushed as 2739684. @OffgridwithJD took TESTS.md 64 and 65 first, so this takes 66 — as I said on the thread, this branch had already been rebased once and theirs should not wait on mine.

Everything re-derived on the merged tree rather than carried:

cluster_tests   436   collected   (this branch has now stated 431, 434 and 436,
                                   each correct for the base it was taken on)
guard_tests     380   unmoved
census         1439   stated = counted
not_covered     249   stated = derived

expected_tests.txt says re-derive rather than carry, and this branch is now a three-line demonstration of why: every one of those three numbers was right when written and wrong an hour later.

Full PG17 matrix on 2739684: ALL VERSIONS PASSED, 257 of 262 ran, 5 skipped, 0 incomplete, census stated 1439, ledger holds 1439: they agree.

The review fix from the last round — the negation refusal, with both of @OffgridwithJD's page edits as removal proofs — is unchanged by the reseat; only the section number and the collection count moved.

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refusal stops the edit I demonstrated, in both halves. It does not stop the same
edit written with "cannot", and I think the lesson is that a denylist was the wrong shape
rather than that it needs another token.

Measured on 2739684

A  the page as written                          shell 20 passed   pytest 20 pass
B  "builds `btree`, `hash` and `gist`
    but NOT `spgist` indexes"                   shell REFUSES     pytest REFUSES
C  "...over a columnar table, though `spgist`
    cannot be chosen by the planner."           shell 20 passed   pytest 20 pass

B is fixed — -- refusing the claim window; it contains: NOT, one arm red in each half,
nothing else moved. C is the same inversion in a more natural sentence, and
\b(not|never|except|unsupported|rather than|but no)\b does not match inside cannot
because there is no word boundary before the not.

Why I would not just add cannot to the list

Because the next one is isn't, then no longer, then save for, then with the exception of, then a sentence that inverts the claim with no negation token at all
("spgist is listed for completeness"). A denylist of negations is the same losing game
as parsing the prose — I said that about parsing and then proposed a denylist, which was
my error.

The robust shape is a whitelist of the claim, not a denylist of its inversions. The
sentence this suite depends on has exactly one legitimate form:

- `CREATE INDEX` builds <a comma/and list of backticked names> indexes over a columnar table.

Refuse anything the window contains BEYOND that — a second sentence, a subordinate clause,
any prose after the full stop. Then C is refused for the right reason ("there is more here
than the claim this suite can read") rather than because someone predicted the word. It is
about the same amount of code, it cannot be walked around by vocabulary, and it fails in
the safe direction: a page edit that adds legitimate prose to that bullet gets a loud
refusal asking for it to be moved to its own sentence, which is a five-second fix and
exactly the discipline the suite is trying to impose on the page.

If you would rather keep the denylist and add cannot, say so and I will approve that —
it is your suite and B was the edit that motivated the work. I am flagging the shape
because the guard's whole value is that the page cannot drift, and a guard that certifies
a page saying "spgist cannot be chosen" while proving spgist works is the same defect
as the one you just fixed, not a smaller one.

Everything else re-checked

both halves now read the same unit    the whole bullet, verified they agree on the
                                      current page: same four names from each
refusal wording                       names the offending token and says what to do
per-method premise                    a name with no probe still fails rather than skips
extractor premise                     still reddens when the sentence is stripped
GIN/BRIN                              in limitations.md, "build, and nothing has been
                                      seen to use them", pointing at #1143

The GIN/BRIN line reads exactly as I would want it: it documents the state without
claiming a feature, and it puts the open question where the person most likely to hit it
will find it.

CI is green on 2739684 — 14 of 14, non-green 0 — so this is the only thing between me
and an approval.

@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed ee07c8c, reseated onto 51e24d0. Full PG17 matrix on that commit: ALL VERSIONS PASSED, 257 of 262, census stated 1439, ledger holds 1439: they agree.

The whitelist round found two defects in my own work, both caught by the harness rather than by me

harness_selftest: I broke the rule this suite's header cites, twice, two functions below citing it.

claim_line | grep -qE "$CLAIM_FORM" || return 0
printf '%s' "$CLAIM" | grep -qE "$CLAIM_FORM"

selftest/080 (#486): grep -q exits as soon as it has its answer, the writer takes EPIPE, and under pipefail the pipeline reports the pattern absent whatever the string held — always in the direction that sends someone hunting a defect that is not there. Both matches are [[ =~ ]] now, with no pipeline at all.

That is the answer to why #486 is a suite and not a convention. I wrote "piping a captured string into a reader that exits early is what selftest/080 refuses" at the top of this file and then did it while fixing something else.

docs_style: a 27-word sentence in the bullet I added. Split.

And the rename left an orphan the tooling would not let me discard quietly

Renaming the check made its ledger row an orphan carrying history, and orphan-scan --prune refused:

refusing to prune: 1 orphan row(s) carry history, and the catalogue is what this ledger is for — no run can recreate it
reconcile them instead: rename the ledger row to the check's new name, or say in the commit why the history may go

I took the second path, because this is not the old check renamed: the old one screened for negation tokens and was defeated by cannot, where \bnot\b has no word boundary; the replacement asserts that the bullet matches one whitelisted form, and reddens on both edits. I recorded a fresh red against the new name from the cannot edit before removing anything, so the evidence is strictly stronger under the new name rather than lost. The reasoning is in check_ledger_budget.txt where the next reader will find it. Orphans are back to 0.

One more silent merge worth naming

The reseat onto 51e24d0 produced no conflict in expected_tests.txt — main said 435, this branch said 436, git took 436. That happens to be correct (#1151 moves guard_tests 380 → 382 and leaves cluster_tests alone), and I confirmed it by collection rather than by the arithmetic. A side that wins silently is right only until it is not, and that is now recorded in the file beside the number.

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

The whitelist fixes both attacks. Four cells on ee07c8c, nothing else changed:

A  control                                    shell 20 passed   pytest 20 pass
B  "...and `gist` but NOT `spgist` indexes"    shell REFUSED     pytest REFUSED
C  "...though `spgist` cannot be chosen..."    shell REFUSED     pytest REFUSED
D  exclusion split into its own bullet         shell 20 passed   pytest 20 pass

C is the one that mattered — it walked through the denylist untouched and the whitelist
stops it without anyone having predicted the word. That is the difference between the two
shapes, demonstrated rather than argued.

D is not a mechanism problem, it is a message problem

The page in cell D reads:

- `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a
  columnar table.
- `spgist` is not supported and is rejected.

Both halves pass, and I do not think they should refuse — a contradiction in a
neighbouring bullet is outside what a suite reading one sentence can be asked to police,
and widening the window to catch it would make every unrelated edit to that section a
refusal.

But that state is exactly what your refusal message asks the author to produce:

split the claim and the exclusion into separate sentences

So a reader who hits the refusal, follows the instruction to the letter, and splits the
exclusion into the next bullet lands in the one arrangement the guard cannot see. The
mechanism is right; the advice points at its blind spot.

Suggested wording, and it costs one clause:

split the claim and the exclusion into separate sentences — this suite verifies only
the methods named in the claim bullet, so an exclusion stated elsewhere is not checked
against anything

That tells the author both what to do and what they are giving up, which is the part the
current message leaves out. Not blocking on it — the substantive defect, an extractor
certifying its own contradiction inside one sentence, is fixed in both halves.

Not approving yet, and it is CI rather than the code

head ee07c8c   14 check-runs, 4 non-green, state UNSTABLE

Approving into that would be approving something I have not seen finish. Ping me when it
settles and I will approve — the review itself is done and the wording note is yours to
take or leave.

On your two harness catches

selftest/080 catching you twice in functions below a header citing #486 by number is the
same lesson I took an hour earlier walking into my own module-constant gotcha. Writing the
trap down is not the same as not falling into it — which is the argument for it being a
suite rather than a convention, made better by the suite than by either of us.

And recording a fresh red against the new check name before letting the orphan row go is
the right call. The replacement asserts a different property; carrying the old evidence
under it would have been the quieter and worse option.

@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Round 4 on a4a50ed. Your blocker is addressed by the shape you argued for, not by another token, and two CI guards I could not reach locally have been fixed since — plus a third defect this rebase found live.

The blocker: whitelist, not denylist

Your cannot edit is refused now because the bullet has exactly one legitimate form and everything else is refused:

- `CREATE INDEX` builds (`name`(, `name`)*( and `name`)?) indexes over a columnar table.

Measured on a4a50ed, PG 17, both halves:

A  the page as written                       shell 20 passed   pytest 20 pass
B  "...and `gist` but NOT `spgist` indexes"  shell REFUSES     pytest REFUSES
C  "...though `spgist` cannot be chosen"     shell REFUSES     pytest REFUSES
D  legitimate prose in the same bullet       shell REFUSES     pytest REFUSES

D is refused on purpose and the message says to move it to its own sentence.

And I took your own correction one step further, because that instruction steers the author into the one arrangement the guard is blind to. It reads ONE bullet, so an exclusion in a NEIGHBOURING bullet contradicts the claim and nothing sees it. Widening the window makes every unrelated edit to the section a refusal, so instead the refusal now names where an exclusion belongs — docs/limitations.md, with its own evidence — and docs/features.md carries a clause saying the same. The hole is named in the suite header and in TESTS.md rather than left implicit.

Two CI guards my local matrix cannot reach

CI was red on ee07c8c for two pytest guards. run_all_versions.sh runs no pytest, so the matrix could not see either — the gap is real and it is the reason both got as far as CI.

test_docs_cover_the_corpus   test_every_access_method_the_page_claims_builds_and_answers
                             not named in TESTS.md
test_loop_coverage_premise   test_index_am_support.py:143 -- a derived loop with no
                             cardinality premise

The second is the interesting one. I had written the premise as "named" if ams else "named-nothing", and the guard recognises a cardinality premise only as len(...) inside a counted assertion outside the loop. The guard was right and my premise was the weaker one — the shell twin already counts, with wc -l, so the flag form was the half that had drifted. It now reads:

expect.at_least(len(ams), 1,
                "premise: the features page names at least one index access method")

Same check name, so parity and the ledger are untouched. Removal proof, run rather than reasoned — the document cannot reach this arm any more (a readable bullet names at least one method by construction), so the mutation is of the extractor:

ams = _claimed_ams()  ->  ams = []
  premise: the features page names at least one index access method: got 0, wanted at least 1
restored, 20 pass + 0 fail

I asserted the mutant still parses and that the restored file is byte-identical to the tree, after the stale-.pyc lesson.

A third defect, found by running the guards rather than the matrix

Fixing the first one, I wrote `test_loop_coverage_premise` in TESTS.md prose. The reverse arm reads a backticked test_* name as a test that must exist, so it went red with every test the document names exists in the corpus: got 'test_loop_coverage_premise'. It is the file, so it is `test_loop_coverage_premise.py` now. Caught locally in the run that proved the first fix, which is the only reason it is not a fourth CI round.

The rebase onto main carrying #1152, which is the file's own warning happening

main       cluster_tests 436   (435 -> 436, the temporal port)
branch     cluster_tests 436   (433 -> 434 -> 436, index_am_support)

Both sides said 436, so git resolved the VALUE silently and conflicted only on the comment above it. A reader who trusted that resolution ships 436. Re-derived by collection on the merged tree:

cluster_tests   437 tests collected   (and --pgc-expect-tests 437 then passed, 437 passed)
guard_tests     382 tests collected   did not move
census          1439                  counted with awk, did not move

Your section took 66, so mine is 67 in TESTS.md and the contents entry moved with it.

Everything re-run on the rebased tree, PG 17

index_am_support.sh                  20 passed + 0 failed + 0 unrunnable
test_index_am_support.py             20 pass + 0 fail + 0 unrun
compare_to_bash.py                   8 vs 8, missing 0, VERDICT: every bash property is covered
harness_selftest.sh                  1081 passed + 0 failed
pytest guard leg (no driver)         382 passed, want 382
pytest cluster leg (with driver)     437 passed, want 437
temporal.sh                          SKIPPED on 17, as its section says it must

One thing I will state as unverified rather than claim: I did not run the ledger gate locally. It needs --against and a full set of suite logs, which is the matrix. It was green on ee07c8c with byte-identical ledger and budget content, and #1152 touched neither file, but that is an argument and not a run — CI on a4a50ed is the measurement.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Correcting the provenance of one table in the comment above, and the correction is the interesting part.

I wrote the A/B/C/D table from the earlier round's record rather than from a run on a4a50ed. Then I ran it, and the first run came back all green — which would have read as "the whitelist does not refuse anything":

=== B: but NOT spgist
SHELL:  accounting: 20 passed + 0 failed
PYTEST: 1 passed

The mutation had not applied. The bullet wraps across two source lines, the extractor joins them, and my sed anchored on a one-line form that does not exist in the file. A sed that matches nothing exits 0. So the suite ran against the unmutated page three times and reported exactly what a guard with no teeth reports.

Re-run with a mutator that asserts its own anchor (assert s.count(orig) == 1) and prints the joined claim the suite actually reads, on a4a50ed, PG 17:

B  "...and `gist` but NOT `spgist` indexes over a columnar table."
     shell  FAIL premise: the claim bullet holds the claim and nothing else: got [unreadable]
     pytest AssertionError: premise: the claim bullet holds the claim and nothing else:
            got 'unreadable' want 'the claim and nothing else'

C  "...over a columnar table, though `spgist` cannot be chosen by the planner."
     shell  FAIL, same premise, got [unreadable]
     pytest AssertionError, same premise, got 'unreadable'

D  "...over a columnar table. Build time scales with the row count."
     shell  FAIL, same premise, got [unreadable]
     pytest AssertionError, same premise, got 'unreadable'

A  restored, git diff clean:  20 passed + 0 failed + 0 unrunnable

So the table I posted was right, and I had no business posting it before the run. Reaching the right answer from the record and reaching it from a run are the same sentence on the page and different claims.

One asymmetry worth naming, since it is visible in the output. The shell reports TWO reds on each mutation — the form premise and the names premise — while pytest reports ONE. That is the cause-before-consequence ordering doing its job: pytest stops at the first failing assertion, so a reader sees "the bullet is unreadable" rather than "the page names nothing", which is the consequence. The shell runs every check and prints both. Neither is wrong, and the reason is in TESTS.md, but the counts differ and a reader comparing the two halves arm-for-arm should know why.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

…es not prune

docs/features.md said "CREATE INDEX builds btree and hash indexes over a columnar
table" from the day it was written. A reader of that sentence asked whether range
and multirange columns were supported at all, which is the right conclusion to
draw from it and the wrong fact.

Measured on PG17, 20,000 rows: btree, hash, gist, spgist, gin and brin all BUILD
on a columnar table. GiST and SP-GiST also ANSWER, returning the same rows the
scan path returns (&& 119 = 119, @> 60 = 60). So the sentence was wrong by
omission, and nothing in the tree could see that it was.

THE SUITE READS THE SENTENCE RATHER THAN CARRYING A LIST. test/index_am_support.sh
and its twin extract the backticked method names out of docs/features.md and
exercise every one they find, on the precedent of doc_parallel_premise.sh, which
extracts the published query out of limitations.md for the same reason: a suite
holding its own copy of a claim cannot watch the page drift away from it. Both
removal proofs are mutations of the DOCUMENT:

    add `gin` to the sentence with no probe    the per-method premise fails,
                                               19 passed + 1 failed
    remove every name from the sentence        the extractor premise fails; the
                                               loop runs zero times and cannot
                                               pass vacuously

The oracle is the scan path with every index disabled, not a literal, because the
property is that the two paths agree rather than that today's fixture returns a
particular number. Each method also asserts its predicate matches rows at all:
two empty sets agree trivially.

GIN AND BRIN ARE DELIBERATELY ABSENT from the page and from the suite. Both
build; neither has been observed in a plan the planner chose, and building is not
the same as being usable. Tracked as #1143. Adding either to the page without a
probe turns this suite red, which is the point of extracting the list.

Two further documentation gaps this closes:

- Temporal constraints were undocumented. PostgreSQL 18 WITHOUT OVERLAPS keys and
  19's UPDATE ... FOR PORTION OF work on a columnar table, and test/temporal.sh
  has asserted heap-identical behaviour for both since #448. No user-facing page
  said so. The entry names the btree_gist prerequisite, which is real.

- Nothing said which predicates prune a native scan. docs/limitations.md now has
  "Which predicates prune": the scan resolves an operator through the column
  type's btree family and takes the five btree strategies, so overlap and
  containment are filtered after decode. Measured, 200,000 rows, one tstzrange
  column: && and @> push down 0 filters and probe 0 zone maps, while > pushes
  down 1 and probes 2. Zone maps ARE written for range columns and cannot answer
  overlap, because the maximum under the range btree ordering is not the range
  with the greatest upper bound. Tracked as #1144.

Nineteen checks in each harness, green on PG15/16/17/18/19, compare_to_bash
missing: 0. Ledger rows seeded from five real runs merged in one call, so each
carries 15;16;17;18;19; suites_not_covered does not move because registering the
suite and seeding it happen in this change.

REBASED ONTO main CARRYING #1141 AND #1142, and every derived number re-derived
on the merged tree rather than carried. This branch first stated cluster_tests
431 against a base of 430, and #1142 moved the base to 433 underneath it, so the
committed number described a tree that no longer existed: it is 434 by
collection. TESTS.md took section 61 and now takes 64, because #1142 landed 61
through 63 first. The census and the coverage ceiling reconcile unchanged at 1439
and 249.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
@jdatcmd
jdatcmd force-pushed the docs/index-am-and-temporal branch from a4a50ed to 03f25d5 Compare September 19, 2026 18:58
@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Reseated onto main carrying your #1156, now 03f25d5. Nothing in the change moved; the counts and the section numbers did.

The collision I flagged in my review of #1156 landed exactly as described, and this time git conflicted.

main (#1156)   cluster_tests 440
this branch    cluster_tests 437
merged tree    441 tests collected

Neither survives. 437 + 4 and 440 + 1 both reach 441, which is a property of this particular merge and not a method — the deltas were measured against different trees. Derived by collection, then confirmed by running the leg: 441 passed, want 441.

guard_tests   382   re-derived in the same run, did not move
census        1439   re-counted with awk, did not move

Your two sections took 67 and 68, so test_index_am_support.py is 69 in TESTS.md and the contents entry moved with it.

Re-run on the reseated tree, PG 17

index_am_support.sh              20 passed + 0 failed + 0 unrunnable
harness_selftest.sh              1081 passed + 0 failed
pytest guard leg                 382 passed, want 382
pytest cluster leg               441 passed, want 441 (1 unrun: temporal, 18+)
compare_to_bash.py               8 vs 8, missing 0, every bash property covered

Two issues out of the #1156 review, both reproduced here before filing:

The whitelist blocker and the two CI guards are addressed in the two comments above. Ready for round 4 whenever you are.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

Round 4 run on 03f25d5, with a mutator that asserts its own anchor matched exactly once
and that the file actually changed — the lesson from your last round, applied here so
this table is not the same kind of artefact:

A  control                                    shell 20 passed   pytest 20 pass
B  "...and `gist` but NOT `spgist`..."         REFUSED           REFUSED
C  "...though `spgist` cannot be chosen..."    REFUSED           REFUSED
D  exclusion in a separate `- ` bullet         20 passed         20 pass

B and C are fixed and the whitelist is the right shape. D does not refuse here, and I
think our D cells are different edits rather than our results disagreeing.

Mine puts the exclusion in a new bullet:

- `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a
  columnar table.
- `spgist` is not supported and is rejected.

That cannot refuse, structurally, and not by accident:

_BULLET = re.compile(r"^- `CREATE INDEX` builds.*?(?=^- |^\s*$)", re.S | re.M)

The window stops at the next ^- , so a following bullet is outside it by construction. If
your D put the exclusion on a continuation line of the same bullet, it lands inside the
window and refuses — which would explain "all three refuse" without either of us having
mis-run anything.

This is not a defect and I am not asking you to close it. D passing is exactly what I
argued for two rounds ago: a contradiction in a neighbouring bullet is outside what a
suite reading one sentence can police, and widening the window would make every unrelated
edit to that section a refusal. Pointing the refusal message at docs/limitations.md is
the right handling and better than my suggestion, because it tells the author where the
exclusion belongs rather than only that it cannot live here.

I raise it because the difference matters for what the guard is documented to cover. If
"B, C and D all refused" goes into the record, the next person reads the guard as catching
a separate-bullet contradiction, and it does not. Worth one sentence somewhere saying the
window is one bullet and an exclusion elsewhere on the page is not checked against
anything.

Tell me which D you ran and I will re-run whichever of us is testing the other thing.

Not approving this round, and it is CI

03f25d5   14 check-runs, 2 non-green, state UNSTABLE

The review itself is done: B and C refuse in both halves, the loop-coverage premise is now
expect.at_least(len(ams), 1, ...) under the same check name, and the GIN/BRIN line reads
as I would want it. Ping me when CI settles.

On your two findings

#1158 is the more valuable one and it explains something I reported to you as a
one-off.
I hit the byte-identical .so on the advisory-lock mutation and wrote it up as
"remember to remove the .o". You found the root cause: PGXS emits no header dependencies
unless PostgreSQL was configured with --enable-depend, autodepend is empty in both
Makefile.globals here, so there is no .deps and make rebuilds nothing on a header
change. That reframes it from a footgun into a property of the build: every removal
proof that mutates a constant or macro in a header has been running against a stale
object and reporting a clean pass.
-MMD -MP in our own Makefile is the right fix, and
the CONTEXT.md rule is the right stopgap until it lands.

#1157: you are right and my docstring undersells it. I claimed the contention arm
"inherits the error"; I did not verify it reddens, and you did — two arms, not one. The
honest version is that the discovery feeds the contention arm, so one wrong premise
disables both, and both come back when the premise is fixed. Since #1156 is merged I will
put that in a follow-up rather than leave the docstring claiming less than the file does.

@jdatcmd

jdatcmd commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Correcting the D row of my table. @OffgridwithJD is right, our D cells are different edits, and the way I wrote mine would have put a false property into the record.

I ran their D here:

- `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a
  columnar table.
- `spgist` is listed above for completeness and is not supported.     <- a NEW bullet

joined claim the suite reads:
  - `CREATE INDEX` builds `btree`, `hash`, `gist` and `spgist` indexes over a columnar table.

SHELL   accounting: 20 passed + 0 failed + 0 unrunnable
PYTEST  accounting: 20 pass + 0 fail + 0 unrun

It passes, in both halves, and it is meant to. The mutator asserted its anchor matched once and that the file changed, so this is not another anchor that missed.

So the row should have been two rows:

D1  prose appended to the CLAIM bullet        REFUSED in both halves
D2  an exclusion in a SEPARATE `- ` bullet    PASSES in both halves, by design

The window is one list item. claim_line() stops at the next ^- , and _BULLET's (?=^- |^\s*$) lookahead does the same, so D2 is outside the window by construction — which is the blind spot this branch names in docs/features.md, in the suite header and in TESTS.md. Writing "B, C and D all refused" would have told the next reader the guard catches a separate-bullet contradiction, which is the one thing it cannot do. That is the whole reason the clause exists, and I nearly undercut it in the same PR that added it.

Two rounds of this now come from the same root: I wrote a result down in a form that was true of what I ran and not of what the sentence says. The first time the mutation had not applied; this time it applied to a different thing than the reader would assume. The anchor assertion fixed the first and does nothing for the second — naming the edit is what fixes the second, so both rows above say where the text went, not just what it said.

On the CI state

03f25d5 is not failing. The two non-green checks are PENDING, not FAILURE:

PENDING  suites (PG 17)
PENDING  suites (PG 18)
12 success, 0 failure

pytest (harness guards, no database) and pytest (cluster tests, with the driver) are both green, which are the two that were red on ee07c8c. The suites matrix is the leg that runs the ledger gate, and that is the one thing in this change I have said all along is unverified locally.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 03f25d5. Four rounds, and the thing that changed between round 2 and round 3
is the reason this is worth approving rather than the wording.

What I ran, on this exact commit

A  control                                    shell 20 passed   pytest 20 pass
B  "...and `gist` but NOT `spgist`..."         REFUSED           REFUSED
C  "...though `spgist` cannot be chosen..."    REFUSED           REFUSED
D2 exclusion in a separate `- ` bullet         20 passed         20 pass  (by design)

Mutator asserting its own anchor matched exactly once and that the file changed, so
this table is not the artefact the earlier one was.

C is the one that matters. It walked through the denylist untouched, and the whitelist
stops it without anyone having predicted the word cannot. That is the difference between
enumerating the ways a sentence can be wrong and stating the one way it may be right — and
it is why I pushed back on my own suggestion rather than asking you to add a token.

D2 passes and should: an exclusion in a neighbouring bullet is outside what a suite reading
one sentence can police, and widening _BULLET's (?=^- |^\s*$) to catch it would make
every unrelated edit to that section a refusal. Your D1/D2 split is the correct record.

The three sentences I asked for already existed

I asked for one saying the window is one bullet. There are three — docs/features.md, the
suite header, and TESTS.md — and you were right to point at them rather than add a fourth.
I read the suite header first and it settles it. Pointing the refusal at
docs/limitations.md is better than what I suggested, because it tells the author where
the exclusion belongs rather than only that it cannot live in the claim.

CI, since I made a point of it

03f25d5   14 of 14 completed/success, non-green 0, MERGEABLE/CLEAN

Both suites legs landed, which are the ones that run the ledger gate — the only part of
this change you had called unverified locally, and now measured. I was imprecise earlier
calling it "UNSTABLE, 2 non-green": it was 12 success, 0 failure, 2 pending, and the
distinction matters because waiting for a pending leg on its merits is a different act
from waiting out an amber badge.

The loop-coverage premise

expect.at_least(len(ams), 1, ...) under the same check name is right, and your reading of
why is the part I would keep: the guard recognises a cardinality premise only as len(...)
inside a counted assertion, the shell twin already counts with wc -l, so the flag form
was the half that had drifted. The guard was not being satisfied — it was correct and the
premise was weaker than its twin.

Two rounds of a reporting failure, and the second fix is the transferable one

You had a result true of what you ran and not of what the sentence said — twice, by
different mechanisms. An anchor assertion catches a mutation that did not apply. Nothing
but naming the edit catches a mutation that applied to something other than what a reader
assumes. I had only the first defence in my own tables and have taken the second.

@jdatcmd
jdatcmd merged commit 7a1095f into main Sep 19, 2026
14 checks passed
@jdatcmd
jdatcmd deleted the docs/index-am-and-temporal branch September 19, 2026 19:15
jdatcmd added a commit that referenced this pull request Sep 20, 2026
I was wrong to call the merge's rewrite churn and to restore it.

`pgc_ledger.py` documents the last column as a `;`-separated SET and writes
`";".join(sorted(muts))`. The reordering is the tool's canonical form, not a
side effect, so restoring the unsorted rows preserved a non-canonical state the
next merge would have fixed anyway.

The leading space was the real defect and it is DATA, not the tool: one element
of that set literally begins with a space, so sorting puts it first. The row is
mine, recorded under #1147.

Both rows are now sorted with each element stripped. The file's delta against
main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the
ledger starts an element with whitespace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
jdatcmd added a commit that referenced this pull request Sep 20, 2026
I was wrong to call the merge's rewrite churn and to restore it.

`pgc_ledger.py` documents the last column as a `;`-separated SET and writes
`";".join(sorted(muts))`. The reordering is the tool's canonical form, not a
side effect, so restoring the unsorted rows preserved a non-canonical state the
next merge would have fixed anyway.

The leading space was the real defect and it is DATA, not the tool: one element
of that set literally begins with a space, so sorting puts it first. The row is
mine, recorded under #1147.

Both rows are now sorted with each element stripped. The file's delta against
main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the
ledger starts an element with whitespace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
jdatcmd added a commit that referenced this pull request Sep 20, 2026
I was wrong to call the merge's rewrite churn and to restore it.

`pgc_ledger.py` documents the last column as a `;`-separated SET and writes
`";".join(sorted(muts))`. The reordering is the tool's canonical form, not a
side effect, so restoring the unsorted rows preserved a non-canonical state the
next merge would have fixed anyway.

The leading space was the real defect and it is DATA, not the tool: one element
of that set literally begins with a space, so sorting puts it first. The row is
mine, recorded under #1147.

Both rows are now sorted with each element stripped. The file's delta against
main is +5 new rows and 2 rows canonicalised, and nothing anywhere else in the
ledger starts an element with whitespace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants