ADFA-4739: Kotlin docs DB pipeline + Build Kotlin Docs GitHub Action

[alexmmiller]

Summary

An end-to-end pipeline for loading Kotlin website content and
kotlin-stdlib/-reflect/-test JSON docs into documentation.db, plus a
GitHub Action (Build Kotlin Docs) that runs it against a Drive-hosted copy
of the database.

Scope

Split out of #21
into two ticket-scoped PRs:

• ADFA-5039 - producing the raw JSON data for the Kotlin website (md_to_json.py).
• ADFA-4739 (this PR) - the database-manipulation side and the GitHub Action.

This PR depends on ADFA-5039 merging first. populate_db.py,
build_nav.py, and find_missing_assets.py all import md_to_json.py
directly, and that file isn't included here. Please review this PR's
content, but hold off merging until #23 lands - README.md in particular
will very likely conflict with #23's own new README.md at the same path
(both add a new file there) and need a quick manual merge to combine the two
once both are in.

Changes

• ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/: build_nav.py
  (sidebar nav from kr.tree), find_missing_assets.py (source QA),
  populate_db.py (inserts pages/nav/media into documentation.db,
  supports pruning via --blacklisted-element-titles), optimize_media.py
  / insert_optimized_media.py (media optimization + DB update),
  templates/, assets/.
• scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py - syncs
  Dokka-generated kotlin-stdlib/-reflect/-test JSON into the database.
• ProcessDocs/ProcessKotlinDocs/run_e2e_pipeline_test.sh - local e2e test
  of the whole pipeline against a scratch copy of the database.
• .github/workflows/build-kotlin-docs.yaml - CI counterpart: pulls
  documentation.db (and Writerside's webHelpImages.zip) from Google
  Drive, runs the same pipeline, uploads the result back.
• Dokka-plugin-kdoc2json/scripts/kotlin/build-stdlib-json-docs.sh +
  build.gradle.kts tweak - builds the kdoc-to-json plugin and generates
  the stdlib JSON docs the sync script consumes.
• CLAUDE.md - repo-level orientation doc (schema drift between this repo's
  tooling and the live production database, repo tour, decisions log).

Test plan

• Once ADFA-5039 is merged and this branch is rebased, re-run
  run_e2e_pipeline_test.sh locally against a scratch database.
• Trigger Build Kotlin Docs with dry_run: true and confirm the
  summary/blacklist-verification steps pass.
• Reviewer confirms the scripts match their originals in ADFA-4739: Pipeline for producing template-based Kotlin documentation #21 (no edits
  made during the split, aside from README.md being trimmed to drop the
  md_to_json.py-only section now covered by ADFA-5039).

[hal-eisen-adfa]

Review scoped to the files that are NEW in this PR. I deliberately skipped populate_db.py, build_nav.py, insert_optimized_media.py, sync_kdoc_json_to_db.py and build-kotlin-docs.yaml — I verified by blob hash that those are byte-identical to #21, where I've already left 11 findings. Everything below is specific to this PR.

Blocking (2):

1. CLAUDE.md makes a factual claim about templateId that the same commit falsifies — and reasons from it to a scope conclusion.
2. nav.peb emits a nav-hidden class that nothing consumes, so entries meant to be hidden render visible.

Non-blocking (3) — real, but lower severity or arguably intentional; flagged for your judgment rather than as merge blockers.

Context worth stating plainly: this PR is a clean split from #21, and the split itself is well executed — README.md correctly documents the ADFA-5039 dependency, and I checked that this PR is not shipping a stale pre-fix version of anything. A number of things I went looking for came back clean: build-stdlib-json-docs.sh's restore trap genuinely fires on both failure and SIGINT (verified with timeout -s INT) and is registered before the file swap, so there's no race window; optimize_media.py handles pngquant failures and animated images with explicit fallback-and-log rather than silent data loss, and returns a non-zero exit on errors; docs.css/sidebar.js/tabs.js/nav.peb agree on breakpoints and class names; and the README's documented CLI invocations match the argparse definitions exactly.

I also considered and did not file the page.peb "lone <tab>" workaround here — the root cause is md_to_json.py's tag regex, which isn't in this PR, and I've already filed it as blocking on #23.

[hal-eisen-adfa]

Blocking — this claim is false as of the commit that adds it, and it's load-bearing.

The parenthetical asserts a repo-wide search for templateId "turns up zero matches outside WebServer.kt", and the surrounding sentence uses that to conclude the concern "is not a gap to fill; it's out of scope for this repo."

But this same commit adds populate_db.py, which references templateId at lines 87, 98, 145, 261, 266, 276 and 286 — including the actual insert:

INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (...)

and insert_optimized_media.py, which filters on it at lines 214-215 and 256 (... WHERE ... AND templateId != 0). sync_kdoc_json_to_db.py:13 mentions it too.

So the grep this sentence invites the reader to trust returns numerous in-repo hits the moment this PR lands.

What makes this worth blocking rather than a doc nit: CLAUDE.md exists to orient future readers and agents, and this isn't a stale aside — the false premise is used to justify a scope boundary. An agent reading this will conclude the repo doesn't touch templateId and may "helpfully" strip it from exactly the INSERT that needs it.

Suggest either dropping the parenthetical or narrowing it to the Templates/Bookshelf/BookCategories/PUCC names, which I did not find outside WebServer.kt.

[alexmmiller]

Fixed in 66da59d - dropped templateId from the "zero matches outside WebServer.kt" list, keeping only Templates, Bookshelf, BookCategories, PUCC (the names that actually are absent), and added a clarifying note that populate_db.py/insert_optimized_media.py do read/write templateId directly since it's a plain column on Content they populate.

[hal-eisen-adfa]

Blocking — the nav-hidden class is inert, so hidden nav entries render fully visible.

This line emits nav-hidden for nodes the tree marks hidden, and the header comment at line 10 documents hidden as "true for entries Writerside hides from the primary tree." But nothing anywhere consumes the class. Verified:

grep -c 'nav-hidden' assets/docs.css            -> 0
grep -n  'hidden'     assets/sidebar.js          -> (no matches)
grep -c 'nav-hidden' templates/nav.html          -> 29

I read docs.css in full (227 lines): the only display: none rules are for .nav-item > .nav-subtree, .nav-toggle and .nav-backdrop — none for .nav-hidden. And page.peb links only /assets/docs.css, so there's no other stylesheet that could supply the rule, and nothing injects CSS elsewhere.

Concrete effect: the committed nav.html already carries 29 nav-hidden entries — the Kotlin tour steps, e.g. "Hello world" — and every one of them shows in the sidebar identically to a normal entry. Writerside hides those individual tour steps from the primary nav by design, so the shipped sidebar gets cluttered with dozens of entries that were never meant to be listed.

One .nav-hidden { display: none; } rule in docs.css closes it. Worth deciding deliberately though — if these should actually be visible, the class and its comment are the thing to remove.

[alexmmiller]

Fixed in 66da59d - added .nav-hidden { display: none; } to assets/docs.css, matching the documented intent ("entries Writerside hides from the primary tree"). Went with hiding them rather than removing the class.

[hal-eisen-adfa]

Non-blocking — this section describes a branch state that is already out of date.

The text says Dokka-plugin-kdoc2json/ on main is "just a README.md describing the intended design plus a flowchart image; there is no code here yet", that the implementation "exists only on the unmerged branch fix/ADFA-4514", and flags "rebase fix/ADFA-4514 onto current main before merging". Line 183 repeats the rebase item in the decisions log.

fix/ADFA-4514 is already merged — 4c6b8aef ("Merge pull request #18 from appdevforall/fix/ADFA-4514") is on main, and git ls-tree -r main shows JsonOutputPlugin.kt, JsonRenderer.kt, ModelMapper.kt, LinkPostProcessor.kt, the test suite and the kotlin-stdlib-docs build scripts all present on main today.

So a reader is told a whole plugin implementation is missing from main and that a rebase is still outstanding, when both are resolved.

I've marked this non-blocking because it's a point-in-time note that was presumably true when drafted, and unlike the templateId claim above nothing reasons from it. But it's the kind of staleness that a repo-orientation doc is specifically supposed to avoid, and it'll mislead the next agent that reads it.

[alexmmiller]

Fixed in 66da59d - rewrote the Dokka-plugin-kdoc2json/ bullet to describe it as merged (fix/ADFA-4514, 4c6b8aef) with the actual files present on main, and removed the now-resolved rebase item from the decisions log (was line 183).

[hal-eisen-adfa]

Non-blocking — bare pip install / python3, against the repo's own established pattern.

This is the same objection I left as blocking on #23's review_build_json.sh, so flagging it here for consistency rather than re-litigating it.

Our convention (~/.claude/CLAUDE.md) is explicit: "Always use uv for Python project management. Never use pip, pip3, or the uv pip compatibility shim", and "never invoke python/python3 directly outside a uv run context."

In this PR that shows up in several places:

• these two lines (pip install markdown-it-py Pillow scour brotli, pip install cairosvg)
• README lines 53, 56, 59, 72, 90, 109, 115 — every usage example is python3 <script>.py ...
• optimize_media.py:47-48 (docstring) and :302 (a runtime RuntimeError telling the user to pip install cairosvg)
• run_e2e_pipeline_test.sh lines 100, 105, 125, 141, 145, 175 — six bare python3 invocations

This isn't only a global-preference thing: docdb-studio/ and check-tools/ in this same repo each ship a pyproject.toml + uv.lock and document uv sync / uv run, and docdb-studio/CLAUDE.md says "Dependencies are managed with uv. Always use uv run."

Related and independently worth fixing: scour and cairosvg are not in requirements.txt (unchanged by this PR), so they exist only in these prose instructions.

I've left this non-blocking because these are developer-facing helper scripts and you may have deliberately kept them dependency-light — but if so, that's worth a line in the README saying so.

[alexmmiller]

Non-blocking, addressed in 66da59d rather than just documented as intentional - converted every pip install/bare python3 reference (README usage examples, optimize_media.py's docstring + RuntimeError message, and all 6 invocations in run_e2e_pipeline_test.sh) to uv run --with-requirements <repo-root>/requirements.txt, matching the review_build_json.sh fix on #23. Also added scour/cairosvg to requirements.txt since they were only ever in prose.

[hal-eisen-adfa]

Non-blocking — per-file scan failures are swallowed and the script still exits 0.

The except catches everything from converter.convert_file(...), prints to stderr, and continues — but nothing counts the failures, and main() has no sys.exit(1) path except the directory-validation check at line 82. So if every file in the corpus raised, this still exits 0 and prints a report claiming nothing is broken.

That matters because of how the script is positioned: it's step 1/5 in run_e2e_pipeline_test.sh:100, and the README describes it as the pre-flight gate where you "fix anything broken in the source before converting it." set -euo pipefail in the shell script can't help — exit 0 is exit 0.

Same shape I flagged as blocking on populate_db.py:483 (#21) and md_to_json.py:607 (#23). I've scored it lower here because this tool only writes an advisory markdown report and doesn't mutate the database, so a bad run misleads rather than corrupts.

Counting failures and exiting non-zero — or at minimum printing a N files failed to scan line into the report itself — would make a broken run distinguishable from a clean one.

[alexmmiller]

Fixed in 66da59d - added a failure counter and --allow-failures flag mirroring md_to_json.py's pattern from #23: sys.exit(1) if any file failed to scan (unless the flag is passed), and a "N file(s) failed to scan" line (with an explicit "this report is incomplete" marker when non-zero) in the emitted report itself. Covered by new tests in tests/test_find_missing_assets.py, which stub the not-yet-merged md_to_json import via PYTHONPATH since that module isn't on this branch until #23 lands.

[alexmmiller]

Pushed 66da59d addressing your review. Replied inline to each of the 5 line comments individually; summary here.

Blocking (2)

1. CLAUDE.md's templateId "out of scope" claim was false as of this same PR (line 88) - dropped templateId from the "zero matches outside WebServer.kt" parenthetical, keeping only Templates/Bookshelf/BookCategories/PUCC (the names that actually are absent), and added a note that populate_db.py/insert_optimized_media.py do read/write templateId directly.
2. nav-hidden was inert (templates/nav.peb:38) - added .nav-hidden { display: none; } to assets/docs.css, so Writerside-hidden nav entries (the 29 tour-step cases you found in the committed nav.html) are actually hidden.

Non-blocking (3)

3. Stale Dokka-plugin-kdoc2json branch state (CLAUDE.md:150-156, decisions log line 183) - rewrote the bullet to describe the plugin as merged (fix/ADFA-4514, 4c6b8aef) with the actual files present on main, and removed the resolved rebase item from the decisions log.
4. pip/bare python3 vs. the repo's uv convention - converted every usage example in README.md, optimize_media.py's docstring + RuntimeError message, and all 6 invocations in run_e2e_pipeline_test.sh to uv run --with-requirements <repo-root>/requirements.txt. Also added scour/cairosvg to requirements.txt, since they'd only ever existed in prose.
5. find_missing_assets.py swallowed per-file scan failures and exited 0 regardless (line 98) - added a failure counter, a --allow-failures opt-out flag, sys.exit(1) on any scan failure, and a "N file(s) failed to scan" line in the report itself (with an explicit "this report is incomplete" marker when non-zero) - mirroring the pattern md_to_json.py uses on ADFA-5039: Convert kotlin-web-site docs to JSON #23.

Tests

Added tests/test_find_missing_assets.py covering item 5's exit-code/report behavior. Since md_to_json.py isn't on this branch yet (lands with #23), the tests stub that import via PYTHONPATH and run the script as a subprocess rather than importing it directly. 3/3 pass.

Verified the uv run --with-requirements invocations actually work (installed uv locally and ran optimize_media.py --help plus the heredoc pattern used in the e2e script through it) and that run_e2e_pipeline_test.sh is still syntactically valid (bash -n). Full end-to-end execution of find_missing_assets.py/populate_db.py still isn't possible here until #23 merges, per the PR description.

Sent prematurely, apologies

[alexmmiller]

Review: database-insertion path

Focused pass over the files that write to documentation.db. Four things on this branch do:

File	What it writes
ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/populate_db.py	wipes and rebuilds k/html/* + assets/*
ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/insert_optimized_media.py	replaces k/html/images/*, rewrites pages, deletes unreferenced media
scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py	overwrites/deletes k/kotlin-{stdlib,reflect,test}/* + orphaned tooltips
.github/workflows/build-kotlin-docs.yaml:244	one inline INSERT OR IGNORE INTO ContentTypes

optimize_media.py, build_nav.py, and find_missing_assets.py never open a connection.

Two things that are right, up front

Content types are handled correctly throughout. Every type is resolved by value through get_content_type/get_id, which raise if the row is missing; extension mapping is an explicit allowlist (EXTENSION_TO_CONTENT_TYPE) that skips-and-warns on an unknown extension rather than falling back to text/plain; compression is read from ContentTypes.compression rather than assumed. There is not one hardcoded contentTypeID in the PR. This matters because ~34% of the .html rows in the current production DB are typed text/plain, caused by exactly the two anti-patterns this PR avoids (a text/plain fallback for unrecognized extensions, and hardcoded positional type IDs). The text/html-for-JSON-storage choice at populate_db.py:149 is deliberate and correct — templateId points the server at page.peb, so the served type is HTML.

The big DELETE is correctly scoped. populate_db.py:530's DELETE FROM Content WHERE path LIKE 'k/html/%' OR path LIKE 'assets/%' hits 489 rows against the current production DB (267 pages + nav, 222 images) plus the 3 assets — exactly what the script reinserts. The 3,264 k/ rows outside k/html/ (stdlib, disambiguation) are untouched.

Findings

1. HIGH — a conversion failure silently drops a live page and leaves a normal-looking nav link to it. populate_db.py:483

convert_file failures are caught, logged to stderr, and skipped, so the page never reaches pages and no Content row is inserted — but line 530 already deleted the old, working row in the same transaction, which then commits. Worse, topic_index_db still contains the stem (only blacklisted stems get popped at line 460), so build_node resolves it, sets no_link = False, and emits a normally-styled <a href="/k/html/<stem>.html"> with a humanize()-derived title. Net effect of one bad .md: a page vanishes from production and the sidebar links to a 404 in the same styling as every working page.

This is a regression against the standalone path — build_nav.py's load_page_index builds stem_to_id from generated JSON, so a failed page drops out of nav automatically. The in-process variant loses that property.

Minimum fix: topic_index_db.pop(md_path.stem, None) in the except, matching the blacklist path so links render styled-broken. Better: exit non-zero on any conversion failure, since CI uploads this DB to Drive and nothing currently checks that Converted {len(pages)}/{len(md_files)} is a clean ratio.

2. MEDIUM-HIGH — duplicate .md stems abort the whole run. populate_db.py:475-479

md_files iterates every *.md and derives db_id = f"k/html/{md_path.stem}". Content.path is UNIQUE (UNIQUE('path') in the live schema does enforce — SQLite resolves the quoted string as an identifier; verified), so two same-stem files in different topics/ subdirectories mean two inserts at one path → IntegrityError → rollback → raw traceback, after the full conversion pass has already run. md_to_json.build_topic_index handles exactly this case with keep-first-plus-warning, and topic_index_db inherits that dedupe — but the conversion loop doesn't. It fails safe (nothing uploaded), but it should dedupe by stem the same way rather than crash on an upstream rename.

3. MEDIUM — delete_unreferenced_media is an unguarded mass delete with no dry-run. insert_optimized_media.py:290

It deletes every k/html/images/* row not referenced by a k/html/%.html page with templateId != 0. In CI that's safe — steps 2 and 3 share if: !inputs.skip_website_docs, so pages always exist. Run standalone against a DB without those page rows and collect_referenced_media returns an empty set, so it deletes every image, including the ones inserted seconds earlier in the same transaction. There's no floor check (if not referenced: abort) and, unlike sync_kdoc_json_to_db.py, this script has no --dry-run — the most destructive of the three is the one without a preview mode.

Related blind spot: only page/nav rows are scanned, so an image referenced from assets/docs.css or a .peb template would be collected as garbage. Latent today (no url() in docs.css, no image refs in either template — checked), but it's a silent-data-loss trap for whoever adds a CSS background later.

4. MEDIUM — sync_kdoc_json_to_db.py ignores the chunking contract the rest of the PR treats as mandatory. sync_kdoc_json_to_db.py:164

It does UPDATE Content SET content = ? with the entire new blob and never chunks. Meanwhile any existing <path>-N fragment matches the prefix LIKE, fails relative_target_path's file lookup, and gets deleted. So a stdlib page crossing 1 MiB produces one oversized row that no longer matches WebServer.kt's reassembly protocol — the same protocol populate_db.py:145 argues has to be byte-exact on both sides. Latent right now (largest stdlib blob is 172 KB, and none of the 44 chunked rows in the DB are under these prefixes), but the two scripts disagree on an invariant the PR itself documents as non-negotiable.

5. MEDIUM — unknown contentTypeID silently writes uncompressed bytes and commits. sync_kdoc_json_to_db.py:154-157

compression_by_type.get(...) missing → compression = "none" → raw bytes stored in a row whose declared type says brotli. The warning goes to stderr, the transaction commits, and the server gets undecompressable content. Given that a mistyped content column already shipped to production undetected for months, this should be fatal, not a warning.

6. LOW-MEDIUM — delete-after-insert in the rename loop can remove a just-inserted row. insert_optimized_media.py:415

Inserts run first, then for old_name in rename_map: delete_content(...). If one rename's new name equals another's old name (a.png → b.webp alongside b.webp → c.webp), the delete pass removes the row the insert pass just wrote from a.png. This is precisely the chain-rename hazard rewrite_pages' docstring reasons about carefully for text substitution, and the same reasoning wasn't applied here. Unlikely with Writerside sources; the fix is just moving the delete loop above the insert loop.

7. LOW — unescaped LIKE wildcards. insert_optimized_media.py:103

delete_content does path LIKE ? with f"{path}-%" built from a filename. _ and % are wildcards. No current image name contains either, but NAV_CONTENT_PATH is k/html/_nav.html, so the pattern is already k/html/_nav.html-% with a live wildcard in it. Add ESCAPE '\' and escape the parameter. Every other LIKE in the PR uses literal prefixes and is fine.

8. LOW — image_index_db assumes a flat zip and would break silently otherwise. populate_db.py:467

{name: name for name in image_names} keys on the full zip entry, but Converter.resolve_image_src looks up src.rsplit("/", 1)[-1], and md_to_json.build_image_index maps bare-name → rel-path. A nested zip entry gives a key nothing can match, so images resolve as missing while the row lands at k/html/images/sub/foo.png. {Path(name).name: name} makes it match the documented contract at no cost.

9. LOW — the three scripts back up two different ways. populate_db.py:212 and insert_optimized_media.py use VACUUM INTO (documented as WAL-safe); sync_kdoc_json_to_db.py:39 uses shutil.copy2. The live DB is journal_mode=delete, so this isn't currently biting, but the PR's own docstring makes the argument for VACUUM INTO and then one script doesn't follow it. Also worth noting all three run in CI against the same ~300 MB file, so a run keeps three full backup copies on the runner.

Nits

• build-kotlin-docs.yaml:151 says scour/cairosvg are "not in the root requirements.txt" — this same PR adds them to requirements.txt, so they're installed twice and the comment is stale.
• .gitignore's *.db doesn't match documentation.db.backup-<ts> or documentation.db.bak.<ts>, so every local run leaves untracked noise in git status.

Merge order

The PR body's caveat holds — populate_db.py, build_nav.py, and find_missing_assets.py all import md_to_json, which isn't in this diff, so nothing here is runnable until #23 lands. This review was done against #23's copy of md_to_json.py.

[alexmmiller]

Build Kotlin Docs now runs end-to-end — 06a43f5

Pushed 06a43f5. Context for reviewers: this workflow had never executed anywhere. gh api .../workflows/341694427/runs reports total_count: 0, and both boxes in this PR's own test plan are still unchecked. I ran it locally under act against a copy of the real 248 MB documentation.db, which turned up two blockers that stopped step 4/5 dead before it generated anything.

Step 4/5 could not build — two independent causes

1. It needs a JDK 8 toolchain that nothing installs. kotlin-stdlib-docs's own plugins:dokka-samples-transformer-plugin and plugins:dokka-version-filter-plugin each hardcode kotlin { jvmToolchain(8) }, and dokkaGenerateModuleJson depends on dokkaGeneratePublicationHtml — so their dependencies have to resolve even though we only ever want JSON out. With just the JDK 17 the workflow installs, Gradle dies at task-graph resolution:

> Failed to calculate the value of task ':plugins:dokka-samples-transformer-plugin:compileJava'
  property 'javaCompiler'.
   > Cannot find a Java installation on your machine (Linux ... aarch64) matching:
     {languageVersion=8, ...}. Toolchain download repositories have not been configured.

This only survives on a runner that happens to have an EOL JDK 8 lying around for Gradle's auto-detection — which is an undeclared dependency on the runner image, and is exactly what a self-hosted runner or an act container won't have. The injected build.gradle.kts now retargets those two subprojects at the running JVM (not a hardcoded 17, so it can never request a toolchain that isn't installed). The pipeline needs exactly one JDK.

2. It resolved stdlib artifacts that are published nowhere. kotlin_big pulls the real kotlin-stdlib/-reflect/-test binaries from <kotlin-root>/build/repo at the checkout's own defaultSnapshotVersion (2.5.255-SNAPSHOT) — artifacts that exist only after a full local build of the kotlin repo. Against git clone --depth 1 that resolves to nothing:

> Could not find org.jetbrains.kotlin:kotlin-stdlib:2.5.255-SNAPSHOT

Upstream honours -PkotlinLibsRepo/-PdeployVersion only under TeamCity. They're now ordinary Gradle properties, plumbed through build-stdlib-json-docs.sh, both workflows, run_e2e_pipeline_test.sh and the act runner as kotlin_libs_version (default 2.4.10) / kotlin_libs_repo. kotlin_big already declares mavenCentral(), so a released version needs no repo override at all — kotlin_libs_repo is there for a private/snapshot repo. Documenting the stdlib now takes ~3 minutes instead of a Kotlin rebuild.

Also in this commit

• run-build-kotlin-docs-with-act.sh retargeted. It drove build-kotlin-docs.yaml, whose WIF auth can never succeed under act, and hard-required all five Drive/Slack secrets. It now drives build-kotlin-docs-local.yaml, needs no secrets, takes host paths and bind-mounts them, and passes --container-daemon-socket - (colima can't bind-mount its socket).
• New skip_stdlib_docs input, the mirror of skip_website_docs; setting both is rejected. Skips the slowest half when iterating on website content.
• Blacklist de-duplicated. The ADFA-4737 trio was spelled out twice per workflow — once applied in step 2/5, once restated in the verification step — so the verification could drift onto a different list and still report PASS. Now one BLACKLISTED_ELEMENT_TITLES job env var read with mapfile -t at both sites, matching what run_e2e_pipeline_test.sh already did correctly.
• populate_db.py backup moved. Its ~250 MB VACUUM INTO backup ran before conversion — i.e. before the last step that can still refuse to write — so every run that bailed on a conversion failure left a full copy behind.
• populate_db.py no longer runs pngquant. It quantized every PNG on insert, all of which insert_optimized_media.py replaces moments later. Measured on the live corpus: 0 PNG rows survive step 3 (161 become .webp; the remaining 51 are .svg/.gif, which pngquant can't touch anyway). Step 2/5 went 35.4s → 14.5s.
• Documented that act does not apply workflow_dispatch input defaults — which inverts dry_run and would write the rebuilt database back over db_path. The runner script always passes every input explicitly.

Verification

Both halves run green under act against a copy of the real production database:

Website half	267 pages, 217 image rows (161 .webp), 3 assets, page.peb/nav.peb upserted, PASS: all 38 blacklisted topic page(s) confirmed absent
Stdlib half	Step 4/5 3m15s, Step 5/5 37s, 3132 k/kotlin-stdlib/* rows
Compression	sampled page, nav and stdlib rows all decode with brotli -D against the stored 256 KiB dictionary and fail under plain brotli — i.e. genuinely dictionary-compressed, and the existing dictionary was reused, never retrained
Tests	158 (ProcessKotlinWebsiteJSON + sync_kotlin_stdlib_docs) + 189 (docdb-studio) pass

Workflow parity is unchanged: 14 byte-identical shared steps, 2 differing only in Drive-specific packages (zip, google-api-python-client & friends), and 14 one-side-only steps confined to the Drive boundary. kotlin_libs_version/kotlin_libs_repo are shared with identical defaults, and Step 4/5's run block is identical in both files.

Worth a reviewer's eye

• kotlin_libs_version defaults to 2.4.10 while kotlin_ref defaults to master, so the out-of-the-box pairing documents 2.4.10 binaries against master's build scripts. It resolves and builds — that's what the run above did — but the coherent production setting pins both together. Deliberate default, flagging it so it's a decision rather than a surprise.
• The Java-8 fix compiles those two Dokka plugins for the running JVM instead of 8. They're only ever loaded in-process by Dokka under that same JVM, so this should be inert, but it is a change to how upstream's subprojects get built.
• All five findings from @hal-eisen-adfa's earlier review remain fixed; I re-verified each against the current head while working through this.

[hal-eisen-adfa]

Code review: 15 findings — 2 critical, 3 high, 10 medium

Reviewed the full main...HEAD diff (38 files, ~10,150 insertions). Inline comments follow, tagged F01–F15. Where a comment says "Reproduced" or "Verified", the failure was actually run against a scratch database or temp directory and the numbers are from that run — not inferred from reading.

The two criticals are the ones I'd want fixed before this merges. Both destroy content and then exit 0 with truthful-looking statistics:

• F01 — two images that collide on a rewritten extension (logo.png + logo.jpg → logo.webp) silently become one; the loser is gone and its pages render the survivor. On the live path via --webp.
• F02 — migrating page X DELETEs an unrelated page that merely happens to be named X-1, then reports errors 0. 122 rows in, 121 out, no diagnostic.

---

Test coverage for these 15 findings: zero

All 158 tests on this branch pass — 131 in ProcessKotlinWebsiteJSON, 27 in sync_kotlin_stdlib_docs — and not one of the 15 findings above is caught by any of them. That's a deduction, not a guess: each bug reproduces on demand, so if any test constructed the triggering input, that test would be red.

What the gaps have in common is that nothing in the suite feeds these functions data they didn't expect:

Finding	Input no test constructs
F01	Two source images in one directory that collide on a rewritten extension
F02	An unrelated X-1 page sitting alongside X in the migration input
F03	A second run of build_nav.py over a directory containing its own nav.json
F04	Any assertion on final file size or freelist pages, so a missing VACUUM is invisible
F06 / F08	A fragment chain starting at -2, or one with an interior gap (p-1, p-2, p-4)
F07	An invocation where the input and output directories are the same path
F09	Dokka output containing a page with no pre-existing Content row
F10	A tooltip URI carrying ?query rather than #fragment

F06 and F08 stand out: renumber_misnumbered_fragments.py was added in this PR specifically to repair chains that start at -2 or have gaps, yet no test builds either shape. The repair tool and the bug it should have caught arrived in the same changeset.

For the two criticals especially, the regression tests are cheap — both reproduce in a handful of lines against a tmp SQLite DB or tmp dir — and both failure modes are permanent, silent data loss, which is exactly where a test earns the most.

Separately: no workflow in this repo runs pytest at all

This PR adds roughly 1,300 lines of regression tests that cover the data-loss paths, and none of them execute in CI. Whatever coverage exists today holds only as long as each person remembers to run it locally. Worth adding a pytest job before the suite grows further — otherwise the tests this PR contributes will quietly rot.

---

Lower-severity observations (not commented inline)

• build_nav.py:64 resolves duplicate stems last-wins over an unsorted rglob, while md_to_json.build_topic_index does first-sorted-wins with a warning — nav output isn't reproducible across machines.
• build_nav.py:140 render_node does no HTML escaping despite claiming to be "byte-identical to nav.peb's renderNavNode" (Pebble autoescapes); a toc-title containing " breaks out of the aria-label attribute.
• populate_db.py:694 pops topic_index_db mid-loop while Converter holds the same dict by reference, so under --allow-conversion-failures only pages converted after the failure get broken-link styling.
• sync_kdoc_json_to_db.py:276 takes the VACUUM INTO backup before the MAX_DELETE_FRACTION precheck can abort, so runs that refuse to proceed still write a full-size copy.
• optimize_media.py:267 defaults loop to 0, turning a play-once GIF into an infinite loop.
• docdb_studio.py:1317's dictionary cache is never invalidated, which remint_dictionary.py (added in this same PR) can stale out.
• CLAUDE.md ships stale: says "three workflows" while adding two, points at a scripts/kotlin/build-stdlib-json-docs.sh path that doesn't exist, and claims nothing here writes to Templates while populate_db.py:482 does.

Checked and cleared

Recording these so nobody re-treads them — all were plausible suspects that held up:

• renumber_chain's two-pass parking rename is genuinely collision-free against UNIQUE(path) in both shift directions.
• fragment_chain's LIKE over-match plus regex re-check is exact (the _/% tests pass).
• upsert_template's ON CONFLICT(name) works against the real UNIQUE('name') quirk — confirmed in sqlite3; SQLite parses it as a column constraint.
• insert_chunked_content and sync_kdoc's write_content chunk identically and are idempotent across repeated runs.
• The two workflow files haven't diverged in any way that weakens the local one (full diff -u).
• The blacklist survives YAML → mapfile → parse_blacklist_path intact.
• The build.gradle.kts DSL changes compile and were exercised on Gradle 8.14.4.
• PAGE_PEB_STATIC_ASIDE still matches page.peb byte-for-byte, and the DB template rewrite produces the right data-nav-src.
• Nav ids, data-nav-id, and sidebar.js's pathname matching line up correctly on the database-served path.

[hal-eisen-adfa]

F01 · critical · silent data loss

encode_raster rewrites the destination to dst.with_suffix(".webp") (and optimize_svg does the same to .png at L350) with no detection that two different sources now map to one output. logo.png and logo.jpg in the same directory both become logo.webp; whichever one the rglob loop at L438 reaches second silently clobbers the first.

Reproduced end-to-end: exit 0, no warning, both DB rows marked [REMOVED], a single logo.webp inserted, and both pages rewritten to point at it. The JPEG is permanently gone and its pages render the PNG. The same collapse happens when an oversized diagram.svg rasterizes onto an existing diagram.png.

This is on the live path — --webp is passed at .github/workflows/build-kotlin-docs.yaml:301 and run_e2e_pipeline_test.sh:145. The seen_names guard in insert_optimized_media.py:469 cannot help, because the collision already happened in the work dir before that code runs.

Suggested fix: track claimed output paths inside the loop and either disambiguate (logo-jpg.webp) or hard-fail on collision. Silently dropping a source image shouldn't be reachable.

[alexmmiller]

Fixed in d9df0f0. Reproduced first (errors 0, both DB rows [REMOVED], one logo.webp, the JPEG gone) — thank you for the precise repro.

Went with disambiguation rather than hard-fail: a future upstream image pair shouldn't be able to break the whole pipeline, and the rename flows out through optimize_directory's existing renamed map, so rewrite_pages repoints the stored URLs automatically. Since optimize_raster/optimize_svg own the final extension, I de-conflict the stem up front instead of predicting the suffix: first in sorted order keeps it, later ones fold their original extension in (logo.jpg -> logo-jpg.webp), with a numeric suffix if that still collides. A warning is logged either way.

Checked the real webHelpImages.zip: 299 entries, 0 colliding stems — so this was latent, not live. Covered by test_sources_differing_only_by_extension_both_survive, a 3-way variant, and test_non_colliding_names_keep_their_own_stems so the de-confliction can't start renaming things that didn't collide.

[hal-eisen-adfa]

F02 · critical · silent data loss

write_item deletes everything in existing - wanted, and wanted comes from load_base_rows (L301), which classifies any path matching X-<digits> as a continuation chunk of X whenever X also exists — with no chunk-size test.

So two independent pages that merely happen to be named X and X-1 (neither one chunked) get misread as a single chunked item. Migrating X produces wanted = {}, and this line DELETEs X-1 outright. Worse, the victim was already excluded from base_rows, so it is never scanned, never counted, and never reported.

Reproduced: 122 rows in, 121 rows out, and the run printed scanned 121, migrated 121, errors 0.

Two things widen the blast radius: remint_dictionary.py imports this same write_item, and verify_remint_dictionary.py can't catch it because verify() enumerates from load_base_rows too — same blind spot.

The correct test already exists in this PR: renumber_misnumbered_fragments.py:19 keys on "content length is exactly CHUNK_SIZE". And load_base_rows already SELECTs LENGTH(C.content) without using it. Gating the continuation classification on that length should be a small change.

[alexmmiller]

Fixed in d9df0f0. Reproduced exactly as you described — 122 rows in, 121 out, scanned 121, migrated 121, errors 0, and the victim never even appeared in load_base_rows.

Both sites needed it, not just the classifier: write_item's existing came from fragment_chain, which matches every X-<digits>, so the DELETE would still have hit the row even with load_base_rows fixed. Both now gate on is_chunked_base — the base row actually being CHUNK_SIZE bytes.

One correction worth recording: my first attempt also required a contiguous walk from -1, which broke test_chain_numbered_from_minus_two_is_migrated_not_miscounted. An ADFA-5171 chain is still a real chain, so the gate is the base length only, exactly as you suggested. Chain membership stays with fragment_chain so -2 still resolves. write_item now resolves existing before the UPDATE, since it's the base row's current length that marks it chunked.

Covered by test_independent_page_named_like_a_fragment_is_not_a_continuation, test_write_item_does_not_delete_an_unrelated_lookalike_page, and test_genuinely_chunked_base_still_owns_its_continuations (the -2 case).

[hal-eisen-adfa]

F03 · high

load_page_index runs json.loads(...) then page.get("id") over every *.json under docs_json_dir. But main() writes nav.json — a top-level array — into output_dir, and the README documents the invocation (L82) with the output dir as the scan dir:

build_nav.py <docs-root> <output-dir> <output-dir>

So the second run reads its own previous output. Reproduced: AttributeError: 'list' object has no attribute 'get'. Either skip the generated nav.json by name, or guard on isinstance(page, dict).

[alexmmiller]

Fixed in d9df0f0 — isinstance(page, dict) guard, so anything that isn't a page object is skipped rather than crashing. Covered by test_load_page_index_skips_the_generated_nav_json, which writes both a page and a nav.json array into one directory.

Also took the build_nav.py:64 point from your lower-severity list while I was here: the rglob is now sorted() and duplicate stems resolve first-wins with a warning, matching md_to_json.build_topic_index — which is what the pages themselves were converted against, so the two now agree and the output is reproducible.

[hal-eisen-adfa]

F04 · high

This script rewrites every k/kotlin-stdlib* blob, deletes Content rows plus their fragment chains, and deletes Tooltips/TooltipButtons rows — then just commits. There's no final VACUUM, which every other writer in this repo does: populate_db.py:851, insert_optimized_media.py:507, renumber_misnumbered_fragments.py:166, docdb_studio.py:694.

Verified: a run that shrank content down to 27 bytes left the file at 3,178,496 bytes with 768 of 776 pages sitting on the freelist.

This is step 5/5 of the pipeline, so nothing downstream reclaims it — the bloat ships in the on-device database.

[alexmmiller]

Fixed in d9df0f0 — trailing VACUUM on its own connection after the transaction closes, same shape as populate_db.py and insert_optimized_media.py, skipped on --dry-run (nothing was committed, so there's nothing to reclaim). Module docstring updated to say so.

[hal-eisen-adfa]

F05 · high · regression

_find_brotli_cli() is called here while building the argv — i.e. before the plain-Brotli fallback at L1457 can run — and the BrotliCliMissing it raises isn't caught by the surrounding except OSError.

Result: on any database that has a CompressionDictionary, plain-Brotli rows become unreadable without the CLI installed. Those are exactly the rows the docstring calls the "load-bearing, not defensive" fallback: plugin-contributed rows and partially-migrated DBs. brotli.decompress reads them fine, and did read them before this PR. The user gets told to install brotli, which won't fix that row.

Resolve the CLI lazily (or catch BrotliCliMissing alongside OSError) so the fallback stays reachable.

[alexmmiller]

Fixed in d9df0f0. Confirmed the mechanism: BrotliCliMissing subclasses brotli.error, not OSError, so it went straight past the except OSError and the fallback below was unreachable.

Narrower fix than "catch it alongside OSError", though — that would have made a genuinely dictionary-compressed row fall back to plain decoding and fail with a confusing error, and it breaks test_missing_brotli_cli_is_reported_not_swallowed, which deliberately asserts the missing CLI is reported. So: resolve the CLI first; on BrotliCliMissing, try plain decoding and return it if it works (your case — the plain row), and re-raise the original BrotliCliMissing if it doesn't (a real dictionary row, where "install brotli" is the right advice). Verified both directions by hand.

[hal-eisen-adfa]

F11 · medium

Under set -euo pipefail (L49), this command substitution kills the script when grep matches nothing — so the friendly if [ -z "$DOKKA_VERSION" ] message at L107-110 is dead code and can never print.

A kotlin ref that renames the dokka catalog key fails "Step 4/5" with exit 1 and zero diagnostic output. Appending || true, or splitting the grep and sed into separate steps, restores the intended message.

[alexmmiller]

Fixed in d9df0f0 — || true on the substitution, so the if [ -z "$DOKKA_VERSION" ] message below is now reachable instead of the script dying with exit 1 and no output.

[hal-eisen-adfa]

F12 · medium

trap ... EXIT doesn't fire on an untrapped fatal signal, which contradicts the header's promise at L15-18 that "the kotlin checkout is left exactly as it was found."

Ctrl-C during the multi-hour Gradle build leaves the swapped-in build.gradle.kts sitting in the developer's kotlin clone and orphans the mktemp original. trap restore_build_gradle EXIT INT TERM HUP covers it.

[alexmmiller]

Fixed in d9df0f0 — trap restore_build_gradle EXIT INT TERM HUP, so the header's promise that the checkout is left exactly as found now holds for Ctrl-C during the Gradle build.

[hal-eisen-adfa]

F13 · medium

flat_nav / id_to_index include nav nodes whose id was synthesized by build_node for an unconverted *.topic (build_nav.py:122), and which therefore have no Content row.

This is already visible in the committed output — templates/nav.html:1285 carries data-nav-id="api-references". Nav rendering colours that node as a non-link, but the pager doesn't: its two neighbours in document order get prev/next rendered by page.peb:41-42 as ordinary pager links to a hard 404.

Filtering synthesized ids out of flat_nav before building the pager would make the two agree.

[alexmmiller]

Fixed in d9df0f0 — flat_nav is filtered to ids that actually have a page before the pager is built, so prev/next and the sidebar agree on what's reachable. Confirmed your example: api-references is in the committed nav.html as data-nav-id, and its neighbours were getting pager links to a 404.

[hal-eisen-adfa]

F14 · medium

The lock signalling is asymmetric. This "Grabbing baton" notification is ungated and fires on every run, while 'Notify Slack: build complete' at L427 carries if: ${{ !inputs.dry_run }} plus the implicit success().

Since dry_run defaults to true (L119), an ordinary run grabs the baton and never drops it — and so does any run that fails somewhere in the 3-hour build. The channel ends up showing a baton held forever by a dead run, which is the exact failure the convention exists to prevent.

Gate the two notifications identically, and consider if: always() on the release so a failed build still drops the baton.

[alexmmiller]

Fixed in d9df0f0 in both workflows. You're right that the asymmetry was the bug and that dry_run defaulting to true made it the normal case.

Went with if: always() on the release rather than matching the grab's gate: a failed build has to drop the baton too, which a plain !dry_run wouldn't do. Since it now always fires, the message reports the actual outcome — failed / dry run / updated — instead of always claiming an update.

[hal-eisen-adfa]

F15 · medium · silent corruption

get_compression_dictionary catches sqlite3.OperationalError — which covers "database is locked" — and returns None. compress_for_storage then reads that None as "this database has no dictionary" and writes a plain Brotli row into a dictionary database. The only signal is a stderr line no GUI user will see. (Not caching the failure limits later calls, but does nothing for the call that's writing right now.)

This is reachable from import_content_files itself: phase 1's orphan DELETE holds a write transaction on connection #1 while the dictionary read opens connection #2 against the same file.

Either propagate the error instead of returning None, or warm the dictionary before opening the write transaction.

[alexmmiller]

Fixed in d9df0f0. get_compression_dictionary takes strict=, and compress_for_storage passes it: an indeterminate answer now raises instead of being read as "no dictionary" and writing a plain row into a dictionary database.

Kept the lenient default for the read paths deliberately — the existing docstring's reasoning still holds there, and decoding a dictionary row without the dictionary raises loudly rather than returning wrong bytes, so a retry is safe. It's only the write path that can't tell the two apart.

[alexmmiller]

All 15 findings fixed, with regression tests and a CI job — d9df0f0

Thanks for this — the two criticals were both real and both reproduced before I touched anything. Replies are on each thread; this covers the two structural points and what I did not do.

On "test coverage for these 15 findings: zero"

Fair, and the deduction was right. Added tests/test_review_findings.py — 11 tests over the inputs you listed as unconstructed.

I checked the claim rather than assuming it. Stashing the source fixes and running the new tests against the pre-fix code:

8 failed, 3 passed
FAILED test_sources_differing_only_by_extension_both_survive        (F01)
FAILED test_three_way_extension_collision_all_survive               (F01)
FAILED test_optimizing_into_the_input_directory_is_refused          (F07)
FAILED test_independent_page_named_like_a_fragment_is_not_a_continuation  (F02)
FAILED test_write_item_does_not_delete_an_unrelated_lookalike_page  (F02)
FAILED test_reassemble_content_handles_a_chain_numbered_from_two    (F06)
FAILED test_chain_with_interior_gap_is_reported_as_gapped           (F08)
FAILED test_load_page_index_skips_the_generated_nav_json            (F03)

The 3 that pass both ways are deliberate must-not-regress guards: non-colliding names keep their stems, a contiguous chain from -1 is still left alone, and a genuinely chunked base still owns its continuations.

That last one earned its place immediately. My first cut at F02 gated the continuation test on a contiguous walk from -1, which is stricter than what you suggested and broke test_chain_numbered_from_minus_two_is_migrated_not_miscounted — an ADFA-5171 chain is still a real chain. The gate is now the base row's length alone, exactly as you proposed.

On "no workflow in this repo runs pytest at all"

Added .github/workflows/python-tests.yaml — push, PR and manual dispatch. Three suites, because their dependency setups differ: ProcessKotlinWebsiteJSON + sync_kotlin_stdlib_docs against the root requirements.txt, and docdb-studio through its own uv.lock. Installs pngquant, brotli and zstd, since the tests exercise the real binaries rather than mocking them.

Verified under act rather than assuming it works:

✅ Install system dependencies · Install Python dependencies
✅ ProcessKotlinWebsiteJSON + sync_kotlin_stdlib_docs   169 passed
✅ docdb-studio                                          189 passed
🏁 Job succeeded

From the lower-severity list

Fixed: build_nav.py:64 (sorted rglob, first-wins with a warning, agreeing with build_topic_index); build_nav.py:140 (render_node now escapes — which is what actually makes it byte-identical to autoescaping nav.peb; I checked the committed nav.html and no current toc-title contains &, " or <, so it isn't stale); sync_kdoc_json_to_db.py:276 (backup moved below the MAX_DELETE_FRACTION precheck); and the three CLAUDE.md inaccuracies.

Not fixed, deliberately, so they don't look handled:

• populate_db.py:694 — the mid-loop topic_index_db.pop() under --allow-conversion-failures. Fixing it properly means knowing which pages fail before converting any, which needs a second pass over the corpus; I didn't want to make that trade quietly. Only reachable with that flag.
• optimize_media.py:267 — GIF loop defaulting to 0. Small, but it needs the source's own loop count read back through Pillow, and I'd rather do it with a test than by inspection.
• docdb_studio.py:1317 — the never-invalidated dictionary cache that remint_dictionary.py can stale out. Needs a deliberate invalidation key (mtime? explicit clear on re-mint?), which is a design call rather than a patch.

Happy to take any of the three in a follow-up if you'd rather they land here.

State

358 tests pass (169 + 189, up from 347). Both Kotlin-docs workflows still parse and plan under act, and the parity between them is unchanged: 14 byte-identical shared steps, 2 differing only in Drive-specific packages, 14 one-side-only steps at the Drive boundary.

Still outstanding and not from this review: the PR is CONFLICTING against main and needs a rebase — build-kotlin-docs.yaml landed separately via e83bb9dc (#30) and #23 has since merged.

[hal-eisen-adfa]

Automated review pass (/code-review xhigh) over the full PR diff vs c627c1b.

14 inline findings below: 1 critical, 5 high, 8 medium. Two follow-up comments on the thread group them by shared root cause — F01/F02/F08/F13 (one chunking protocol, four implementations) and F04/F05/F10 (optimize_media name-planning pre-pass).

A 15th, low-severity finding is not worth an inline comment: renumber_misnumbered_fragments.is_contiguous_from_one is dead code after find_chains was rewritten to check contiguity inline — one grep hit repo-wide (its own definition), no test.

[hal-eisen-adfa]

F01 · High — unmatched_source_pages() is handed row IDs, not paths

all_rows is SELECT id, path, contentTypeID, languageID, templateId, so {row[0] for row in all_rows} is a set of integer ids. Inside unmatched_source_pages, candidate is a string like k/kotlin-stdlib/kotlin.text/index.html, which can never be in a set of ints — the membership test always fails and every source page is reported unmatched.

Run against normal Dokka output, this step prints warning: 38412 page(s) ... have no Content row and will NOT be inserted, listing pages that are in fact present and being updated. A genuinely new page — the exact condition this check was added to surface — is indistinguishable in that noise.

all_paths ({row[1] for row in all_rows}) is already computed 38 lines above and is what should be passed here.

[hal-eisen-adfa]

F02 · High — unindexed full-table LIKE scan for every row updated

write_content() is called once per matched Content row (tens of thousands for kotlin-stdlib), and stale = fragment_paths(cur, path) issues SELECT path FROM Content WHERE path LIKE '<path>-%' every single time. SQLite's default LIKE is case-insensitive, so the UNIQUE(path) index cannot serve it — each call scans the whole Content table. Roughly 30k updated rows over a ~40k-row table is >1e9 row comparisons added by this PR, where the previous code was a single indexed UPDATE ... WHERE id = ?.

The scan is also unconditional: stale can only be non-empty when the existing base row was exactly CHUNK_SIZE bytes, which is essentially never for these pages. Gate it the way migrate_content_to_dictionary_brotli.is_chunked_base does, or reuse the all_paths set main() has already built.

See the standalone comment on F01/F02/F08/F13 — this is one of four divergent copies of the chunking rule.

[hal-eisen-adfa]

F03 · Medium — the justifying comment is wrong about nav.peb, so the 404 link is only half removed

build_node synthesizes page_id = 'k/html/api-references' for api-references.topic and sets noLinkColor. But nav.peb's renderNavNode branches on {% if node.id %} and emits <a class="nav-link" style="color: ..." href="/k/html/api-references.html"> — a coloured link, not a non-link. build_nav.render_node does the identical thing.

So "nav.peb colours those as non-links" is not accurate, and this fix only removes such nodes from prev/next. A reader who clicks "API reference" in the sidebar still gets a 404 from WebServer.kt.

Either drop the synthesized id for nodes with no Content row (the pager filter right below already computes exactly that set), or make nav.peb render a .nav-group-title when noLinkColor is set.

[hal-eisen-adfa]

F04 · Critical — de-confliction is scoped per source subdirectory, but the output namespace is flat

claimed is keyed (rel.parent, candidate), so this guard only de-conflicts within one source subdirectory. insert_optimized_media.py addresses every image by bare basename.

Concrete failure: media_dir contains sub-a/logo.png and sub-b/logo.jpg. Both keep the stem logo and, with --webp (what both workflows pass), become sub-a/logo.webp and sub-b/logo.webp. insert_optimized_media's seen_names guard then skips the second with a warning and writes a single row at k/html/images/logo.webp — while build_rename_map maps both logo.png and logo.jpg to logo.webp, and rewrite_pages repoints both pages at the survivor. One image permanently gone, one page silently rendering the wrong picture, exit 0.

populate_db.py was updated in this same PR to flatten and warn on exactly this nested-zip case; optimize_media was not. Key claimed on the predicted output basename, not (parent, stem).

See the standalone comment on F04/F05/F10 for the shared root cause.

[hal-eisen-adfa]

F05 · Medium — de-confliction fires for collisions that cannot occur

The rename is unconditional, including in runs and for file types where no extension rewrite is possible.

Without --webp (the default), over a directory holding logo.png and logo.jpg: neither encoder changes the extension, so the outputs would be logo.png and logo.jpg with no collision at all — but the stem de-confliction fires anyway and writes logo-jpg.jpg. That lands in renamed, so build_rename_map records logo.jpg -> logo-jpg.jpg, rewrite_pages rewrites every stored /k/html/images/logo.jpg reference, the old row is deleted and a new one inserted. All for a non-collision. Same for two passthrough non-images sharing a stem (notes.txt + notes.md), which are only ever copy2'd.

Condition the guard on whether the extension can actually change (cfg["webp"], or the svg-rasterize path), or predict the output name per file rather than blanket-applying the rename.

[hal-eisen-adfa]

F10 · Medium — src.stat() sits outside the per-file try, so one bad entry kills the whole run

optimize_directory collects sources from rglob("*") filtered only by not p.is_dir(), which keeps broken symlinks (is_dir() is False for a dangling link). The first thing process_file does is original_size = src.stat().st_size, outside the try/except that exists precisely to increment stats["errors"] and keep going.

FileNotFoundError propagates out of optimize_directory, and insert_optimized_media's caller only catches ValueError — so the run dies with an unhandled traceback instead of the intended "N file(s) failed to optimize; aborting before touching the database".

Move the stat() inside the try. See the standalone comment on F04/F05/F10.

[hal-eisen-adfa]

F11 · Medium — this script's docs contradict the workflow change in the same PR

build-kotlin-docs-local.yaml:401 is now if: always(), carrying an explicit comment that it is deliberately not gated on dry_run. This script still tells the user — here in the header, and again in the runtime dry-run notice — that "build complete" "is gated on dry_run=false. Pass --live to write back and see it."

A user who exports SLACK_WEBHOOK_URL and does a dry run will fire a real Slack message they were told would not fire, on a channel where the baton messages are a shared lock convention. Update both strings, or re-gate the workflow step.

[hal-eisen-adfa]

F12 · Medium — pip in CI against the repo's uv convention, plus a dead install

Two things:

1. pip usage. The convention for this project is uv for Python dependency management — never pip, pip3, or the uv pip shim. This PR touches three such steps and leaves them all on pip: build-kotlin-docs.yaml:203/208, build-kotlin-docs-local.yaml:218/221, and python-tests.yaml:56. uv is already wired up in this repo — python-tests.yaml installs it for the docdb-studio job — and the ProcessKotlinWebsiteJSON README documents uv run --with-requirements ... as the supported invocation.

2. Dead install + wrong comment. The edited comment says "markdown-it-py: ProcessKotlinWebsiteJSON's own requirement (see its README); scour/cairosvg are in requirements.txt already" — but markdown-it-py>=2.0 is in requirements.txt on main and was installed by the preceding line all along. The extra pip install markdown-it-py is dead work and the comment's premise is wrong.

[hal-eisen-adfa]

F13 · Medium — reassemble_content no longer mirrors WebServer.kt's reassembly protocol

The old loop stopped as soon as a fragment came back shorter than CHUNK_SIZE, exactly as WebServer.kt does — and the docstring still claims that mirroring. The new loop concatenates every member of fragment_chain unconditionally, and the if row is None: break is dead code, since fragment_chain only returns paths that exist.

For the "gapped" chain shape that renumber_misnumbered_fragments explicitly refuses to repair — e.g. p-1 (CHUNK_SIZE), p-2 (short), p-4 — this tool now reassembles p + p-1 + p-2 + p-4 while the server serves p + p-1 + p-2. rewrite_pages would then re-compress and store a blob the server never had, or fail to decompress with a message that points nowhere.

Keep the short-fragment terminator and use fragment_chain only for suffix discovery. See the standalone comment on F01/F02/F08/F13.

[hal-eisen-adfa]

F14 · Medium — this header explains a three-way split, but only two suites actually run

The comment reads "docdb-studio and check-tools each own a pyproject.toml + uv.lock, while ProcessKotlinWebsiteJSON runs against the root requirements.txt", and the workflow's stated purpose is "Runs the repo's pytest suites on every push and PR". There are three suites' worth of tests in the repo and only two are executed.

check-tools/tests/test_db_health_checker.py — covering the health checker wired into the daily docdb-regression-test.yaml cron — continues to run nowhere. Add a uv run --frozen -- python -m pytest -q step with working-directory: check-tools, matching the docdb-studio step.

[hal-eisen-adfa]

F01 / F02 / F08 / F13 — one chunking protocol, four implementations

These four inline findings look unrelated (a wrong variable, a slow query, a missed repair, a dropped terminator) but they are the same defect: the Content chunking protocol is re-derived from scratch at every call site, and each derivation is wrong in a different way.

The protocol

WebServer.kt reassembles a page as: read path; if that blob is exactly CHUNK_SIZE, keep appending path-1, path-2, … in order; stop at the first fragment shorter than CHUNK_SIZE. Two rules follow from that, and both matter:

1. <base>-<N> is a continuation only if the base row is exactly CHUNK_SIZE long. The base merely existing proves nothing — guide.html and guide.html-1 are perfectly legal as two unrelated pages.
2. A short fragment terminates the chain. Anything after it is not part of the page the server serves, whatever is sitting in the table.

The correct implementation

migrate_content_to_dictionary_brotli.py already has it right, and this PR is where it got right:

is_chunked_base(lengths, prefix)   # gates on length == CHUNK_SIZE
load_base_rows(...)                # uses that gate

That is the reference. It encodes rule 1 explicitly and is the only copy in the repo that does.

How each site deviates

ID	Site	Deviation	Consequence
F01	sync_kdoc_json_to_db.py:390	Doesn't reach the protocol at all — passes row ids where paths are compared	Every page reported unmatched; the real signal is buried under ~38k false warnings
F02	sync_kdoc_json_to_db.py:136	Discovers fragments with an unindexed LIKE '<path>-%' scan, run unconditionally	~1e9 row comparisons; the scan can only ever return rows when rule 1 holds, which it almost never does
F08	renumber_misnumbered_fragments.py:59	Applies rule 1 without the length gate — "base exists" is treated as sufficient	Genuinely misnumbered chains are silently skipped; reports "Renumbered 0 chain(s)" on a DB that still serves the page truncated
F13	insert_optimized_media.py:194	Drops rule 2 — concatenates the whole fragment_chain	Reassembles a blob the server never serves on gapped chains, then re-compresses and stores it

F02 and F08 are the same misunderstanding pointing opposite directions: F02 does the expensive lookup when rule 1 says it cannot match, F08 skips the repair when rule 1 says it should have matched.

Recommended approach

Extract the protocol into one module — is_chunked_base(lengths, prefix) and a fragment_chain(...) that honours the short-fragment terminator — and have all four sites call it:

• F01 — pass all_paths (already computed 38 lines above), not {row[0] ...}.
• F02 — gate on is_chunked_base before any fragment lookup, or reuse the all_paths set main() already holds in memory. Either removes the LIKE scan entirely in the common case.
• F08 — replace the "base exists" test with is_chunked_base. Same for sync_kdoc_json_to_db.py:85 (is_fragment_path), which has the identical bug.
• F13 — restore the short-fragment terminator; use fragment_chain for suffix discovery only, not as the definition of the chain.

The shared module should be the one place that knows CHUNK_SIZE and the terminator rule, so the next tool that touches Content inherits the server's behaviour instead of guessing at it. A single round-trip test — chunk a blob, reassemble it, assert it matches what WebServer.kt's algorithm produces, including a gapped-chain case — would have caught F08 and F13 together.

[hal-eisen-adfa]

F04 / F05 / F10 — one root cause in optimize_media.py's name-planning pre-pass

All three findings live in the new pre-pass that walks sources and builds dst_rel_for. They share a single root cause:

The pre-pass plans entirely from source-side attributes — rel.parent, rel.stem, the extension as written on disk — and never reconciles that plan against the two things that actually determine the outcome: the predicted output name, and whether the source is a real readable file.

Each finding is one facet of that.

ID	Facet	What is assumed	What is true
F04	Namespace	Collisions are per-directory, so (rel.parent, stem) is a sufficient key	The consumer namespace is flat — insert_optimized_media addresses every image by bare basename, so sub-a/logo.png and sub-b/logo.jpg both land on logo.webp
F05	Extension	Two files sharing a stem will collide	They collide only if the encoder rewrites both extensions to the same thing. Without --webp, logo.png/logo.jpg never collide, yet logo.jpg is still renamed and every stored URL rewritten
F10	Inventory	Every entry from rglob("*") that is not p.is_dir() is a statable file	Broken symlinks pass that filter (is_dir() is False for a dangling link), and process_file stats outside the try, so one of them takes down the whole run

F04 and F05 are exact mirror images, which is the tell that the keying is wrong rather than merely too narrow: F04 misses collisions that will happen (different directories, same output name), F05 invents collisions that cannot happen (same stem, different output names). A key computed from the source path cannot get both right, because it is not the thing that collides.

Recommended approach

Compute the predicted output name once, per source, using the same logic process_file will use to decide the destination extension (cfg["webp"], the svg-rasterize path, passthrough copy), then:

1. De-conflict on that predicted name, keyed by basename alone — matching the flat namespace insert_optimized_media actually consumes. This fixes F04 and F05 in one change: names that will genuinely collide get de-conflicted regardless of directory, and names that cannot collide are left alone, so no gratuitous renamed entries and no needless stored-URL rewrites.
2. Validate the inventory at collection time — drop and report entries that are not readable regular files — and move the src.stat() inside process_file's try so a mid-run disappearance is still counted in stats["errors"] rather than escaping as an unhandled traceback. That fixes F10.

Worth noting: populate_db.py was changed in this same PR to flatten nested media and warn on basename collisions — i.e. the flat-namespace reality was already recognised on one side of the pipeline. optimize_media needs to model the same namespace, or the two halves will keep disagreeing.