Generate sitetree.json, llms.txt, llms-full.txt, and a tutorials catalog at build time - #5312
Conversation
…arkdown.html partial
…format Prevents a repeat of the list.<format> collision found in the previous round: any future .md-suffixed list.* template (e.g. an upcoming tutorials catalog) needs every candidate sharing that suffix to be explicitly name-qualified to its own format's Name, including this one, which was previously left as a bare, unqualified fallback.
Ported from the previous round's cherry-picked recursion, but generated via resources.Get | ExecuteAsTemplate instead of a custom output format -- avoids the list.<format> collision class entirely by construction. Schema change: a node's path is present only when it has real content of its own; a pure-signpost section carries redirect instead, so an agent can tell 'real page, safe to fetch' from 'stub, follow redirect' by field presence rather than comparing paths against children. Found and fixed during verification: ExecuteAsTemplate's context must be explicitly site.Home, not the ambient page context head.html happens to be rendering when the Pipes resource first executes -- otherwise the tree's root is whatever page triggers it first, not the home page.
…text template Reuses render-page-markdown.html unchanged, same as every page's own .md mirror. Real finding during verification: a Hugo Pipes resource's shortcode-template resolution inherits whichever output format's render pass is actually executing the resources.Get | ExecuteAsTemplate call -- not the resource's own naming, not the page context passed to it. Triggering from head.html (always HTML) resolved shortcodes to their raw HTML variants; triggering from within list.markdown.md (always MARKDOWN) resolves them to the clean .md variants instead. Verified empirically, including that nesting the trigger inside llms.txt's own template (itself reached via head.html) does not help -- it inherits the same HTML-rooted ambient context regardless of nesting depth. No <head> link for this file, per design -- llms.txt's own body link is its sole, properly-contextualized discovery path.
New TUTORIALCATALOG output format, scoped only to /tutorials/ via that section's own outputs: frontmatter override (matching how sitetree.json/ llms.txt/llms-full.txt are scoped, just via Hugo Pipes instead). Reuses list.typesense.json's existing filter and canonical-URL-fallback logic verbatim -- verified matching entry count (37) against that trusted, already-live index. Skips all of typesense.json's card-rendering fields (images, video, wordcount, cost, dates), keeping only what's useful to an agent: title, description, canonical-aware URL, and a compact facet line. llms.txt gets a new Tutorials section linking it, and its sitetree.json caveat now points here instead of dead-ending on 'not included.' Known risk, loudly noted in the template's own header comment: this file's existence depends entirely on docs/tutorials/_index.md's outputs: override. If /tutorials/ is ever removed from the site, this stops generating silently, and llms.txt's link to it 404s.
✅ Deploy Preview for viam-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
viambot
added
the
safe to build
…y-files.html CLAUDE.md's Agent discoverability files section is now the single place these explanations live; the code just points at it instead of duplicating the full story.
Both stub pages' HTML layouts (tutorials.html, tutorials-all.html) never reference .Content, so adding this pointer has no visible effect on the human-facing pages -- confirmed by grep. Real side effect found and fixed: schema.html's JSON-LD description uses .Summary (auto-derived from body content), not the frontmatter description -- adding body text overwrote it with the pointer instead of the real topical description. Added explicit summary: frontmatter matching description: on both pages to keep that metadata correct.
|
Reviewed 92ff9be: two clean prod builds on the CI-pinned Hugo 0.152.2, diffed against a baseline build of Confirmed working. Only 2 of 1282 generated 1.
|
| path | HTML | .md mirror |
|---|---|---|
/build-apps/ |
70,849 B real page | redirect stub |
/build-modules/ |
70,965 B real page | redirect stub |
/set-up-a-machine/ |
71,048 B real page | redirect stub |
/reference/apis/components/ |
235 B meta-refresh | redirect stub |
/reference/apis/services/ |
233 B meta-refresh | redirect stub |
/reference/components/ |
260 B meta-refresh | redirect stub |
The first three are the sharp ones: rich in a browser, empty for the Markdown consumer llms.txt is written for, in the same file that tells agents to append .md to any URL.
These paths are pre-existing — identical in static/llms.txt on main — so this isn't a regression. But this PR is the one that turns the list into data/llms_pages.yaml, and render-page-markdown.html already computes exactly the "is this a stub" signal via resolve-chain.html + .Params.canonical. Either point the YAML at the resolved targets, or resolve at generation time.
The same gap makes the sitetree schema's stated contract not quite true: $isRedirect keys only on manualLink/manualLinkRelref, so 12 nodes carry path while their mirror is a stub — the five canonical SDK pages, six empty_node sections, and the tree root / itself. / is harmless in practice; it's just the cleanest demonstration that field presence doesn't mean what the docs say it means.
2. single.md — and the rule I gave you on #5311 was wrong
I told you on #5311 to name-qualify every list.*/single.* template. That advice was imprecise, and acting on it naively breaks the site. Both corrections are mine to make.
The real rule is lexical sort order. A list.<fmt>.md / single.<fmt>.md shadows the unqualified template if and only if <fmt> sorts before md. Verified across controlled builds: llmsfulltxt, acat → shadowed; tutorialcatalog, zcat → clean. So #5311 broke because l < m, and #5312 is safe because t > m — not because TUTORIALCATALOG is scoped through frontmatter. Format assignment is irrelevant to the lookup.
And the obvious fix has a trap. Renaming single.md → single.markdown.md on its own silently guts /reference/glossary.md: 466 lines → 6, every term body gone, and it's the only file that changes so nothing else signals it. A qualified _default/single.markdown.md outranks the unqualified layout-specific layouts/docs/glossary.md. Renaming that to glossary.markdown.md in the same change restores byte-identical output.
Leaving single.md unqualified is defensible — nothing currently threatens it. What isn't defensible is leaving the wrong rule in CLAUDE.md for whoever adds the next .md output format.
3. The documented llms-full.txt publishing trigger doesn't exist
The PR body and CLAUDE.md's "Two more Pipes gotchas" both say head.html carries {{- $_ := $agentFiles.llmsFullTXT.RelPermalink -}} to force publishing. head.html:59 is a comment saying the opposite ("deliberately not linked or triggered here"), and agent-discoverability-files.html returns a dict with only sitetreeJSON and llmsTXT — the documented expression would fail outright. The real trigger is layouts/_default/list.markdown.md:20.
For a mechanism whose documented failure mode is "remove this and the file silently stops generating, no build error," both docs pointing at the wrong file is the expensive kind of wrong.
4. The Pipes files never regenerate under hugo server
Tested directly: started hugo server --disableFastRender, edited a section title in data/llms_pages.yaml, watched the server log Data changed /llms_pages.yaml and rebuild — and /llms.txt still served the old title. Same execute-once cache the site.Home gotcha describes. Production is unaffected, but the "live hugo server spot check" in the test plan only holds for the initial build, and CLAUDE.md's step 5 will mislead anyone iterating on the YAML. Worth a line next to the other Pipes gotchas: only editing the asset template itself busts the cache.
Nits
CLAUDE.mdlines 71, 75, 76 still referencelayouts/_default/list.md, which this PR renames away.assets/llms.txt:18still keys the sitetree link oneq .title "Reference"— renaming that YAML section silently drops the link. Raised on Generate sitetree.json, llms.txt, and llms-full.txt at build time #5311, still open.llms.txt's link text changed against production, not just against Generate sitetree.json, llms.txt, and llms-full.txt at build time #5311: the hand-written agent-oriented blurbs are now page.DescriptionSEO text, and some titles moved ("Viam CLI reference" → "CLI reference"). Deliberate?- Three
rel="describedby"links in every page's head now; a tutorials catalog isn't really a description of/reference/components/motor/. Taste, take it or leave it.
Not a finding, for the record: I measured 54.6s vs a 41.9s baseline and nearly reported a build-time regression. A second build came in at 41.2s. Noise — there's no performance cost here.
Generated by Claude Code
|
Pushed 6 commits addressing the review. Finding 1 (llms.txt stub-mirror links): fixed for
Finding 2 (collision rule): corrected — the real mechanism is lexical sort order ( Finding 3 (llms-full.txt trigger docs): fixed — CLAUDE.md now correctly points at Finding 4 (hugo server caching): documented as a Pipes gotcha. Nits: stale Link text vs. production — deliberate, and more than that: production's hand-written Also simplified |
|
One more commit (f2cb6cc): added an optional per-section |
|
Re-reviewed at f2cb6cc (prod builds of head vs
Nits:
|
…tion's broken example
|
Pushed 4 commits addressing this review. 1. 2. 3. Collision rule reasoning is wrong — corrected. Independently re-verified rather than taken on faith: built Nits: PR description corrected (no longer claims Thanks for the thorough re-review -- the |
|
It looks like the following files may have been renamed. Please ensure you set all needed aliases: |
|
Re-reviewed at 4fe00c8. Everything from my last pass is addressed, and I found nothing new.
|
d14d6df
into
viamrobotics:main
|
🔎💬 Inkeep AI search and chat service is syncing content for source 'Viam Docs' |
3 participants
Summary
Adds four build-time-generated, agent-facing discoverability files:
/sitetree.json,/llms.txt,/llms-full.txt, and/tutorials/catalog.md. This is a redesign of #5311 (closing that PR in favor of this one) — that approach used custom Hugo output formats for all three original files and hit three real bugs along the way (a template naming collision that silently replaced every section's real Markdown mirror content sitewide, a shortcode-rendering quirk that needed amediaTypetrick plus a Netlify redirect, and a misdiagnosed root cause for an HTML-escaping bug). Rather than patch forward, the mechanism was reconsidered from scratch against a stricter bar: minimum blast radius on the existing Markdown mirror (#5307), Hugo/Docsy-native where possible, no Netlify hacks if avoidable, and "set and forget" maintainability.Mechanism:
sitetree.json/llms.txt/llms-full.txtare generated via Hugo Pipes (resources.Get | resources.ExecuteAsTemplate— the same mechanism the Docsy theme itself uses for its ownoffline-search-index.json), not custom output formats. Pipes assets never touch Hugo'slist.<format>template lookup at all, so the collision class that broke #5311 is structurally impossible here, not just avoided by discipline./tutorials/catalog.mdis the one exception, generated as a Hugo output format instead, since its natural URL is already.mdand it directly reuseslist.typesense.json's existing filter/canonical-URL logic — a genuine, well-precedented fit rather than a default choice./sitetree.json— full site IA as JSON, mirroringlayouts/partials/sidebar-tree.html's (this repo's own override, not the Docsy vendor copy) child-selection logic. Schema change from Generate sitetree.json, llms.txt, and llms-full.txt at build time #5311: a node'spathis present only when it has real content of its own; a pure-signpost section carriesredirectinstead, so an agent can't accidentally fetch an empty stub, and no URL appears twice in the tree./llms.txt— curated entry point fromdata/llms_pages.yaml, unchanged curation approach from Generate sitetree.json, llms.txt, and llms-full.txt at build time #5311./llms-full.txt— full text of curated "orientation" pages, reusingrender-page-markdown.htmlunchanged. NomediaTypetrick or Netlify redirect needed this time — real win of the Pipes approach. Deliberately not linked from every page's<head>(only fromllms.txt's own body, which can carry the context that "-full" doesn't mean "the whole site")./tutorials/catalog.md— new. A flat, non-hierarchical index of all 37 currently-listed tutorials (verified matching count against the existingtypesense.jsonindex), since 21 of 48 tutorial pages point off-site and don't fitsitetree.json's single-parent hierarchy.CLAUDE.md's "Agent discoverability files" section documents the whole system, including two genuinely non-obvious Hugo Pipes behaviors found during implementation:.mdvariant vs. raw.html) follows whichever output format's render pass is actually executing theExecuteAsTemplatecall — not the resource's name, not the target-path string.llms-full.txt's trigger had to move fromhead.html(always HTML) intolist.markdown.md(alwaysMARKDOWN) to get clean shortcode rendering..Permalink, etc.) — executing the template isn't enough. Sincellms-full.txtis deliberately unlinked from<head>, its trigger inlist.markdown.mdcarries a one-line, no-visible-output reference purely to force publishing.Also carries forward the
render-page-markdown.htmlextraction (cherry-picked from #5311's own already-fixed commits) and thelist.md→list.markdown.mdrename — both land onmainfor the first time here, since #5311 was never merged.Test plan
make build-prodcompletes with no errorsvale(pinned 3.12.0, matching CI) clean on every.mdfile this PR touches or creates/sitetree.jsonvalid JSON,path/redirectschema verified against/hardware/- and/try/-style manualLink sections, zero/tags/*leakage/llms-full.txtverified byte-identical to the real.mdmirror for its curated pages, zero HTML/shortcode leakage, zero escaping artifacts/tutorials/catalog.mdentry count (37) matches the existing, trustedtypesense.jsontutorials index; off-site canonical URLs (codelabs.viam.com,viam.com/post/*) resolve correctlywarnf) both tested firing and silent in the correct conditionshugo serverspot check: all four routes return 200 with correct content-types;<head>links present/absent exactly as designed (llms.txt/sitetree.jsonlinked from every page;llms-full.txtand the tutorials catalog deliberately not — the tutorials catalog is linked only fromllms.txtand its own two stub pages)llms-full.txt's generation has no hidden dependency on it (list.markdown.md, its trigger point, is the generic Markdown-mirror template shared by ~90 sections, not tutorials-specific)Known, accepted risks (documented in
CLAUDE.md)/tutorials/catalog.mddepends entirely ondocs/tutorials/_index.md'soutputs:override existing. If/tutorials/is ever removed from the site, this stops generating silently, andllms.txt's link to it needs removing in the same change.🤖 Generated with Claude Code