Generate sitetree.json, llms.txt, and llms-full.txt at build time

[Jeremy Rose (jeremyrose-viam)]

Summary

Adds three build-time-generated, agent-facing discoverability files to the Hugo site, building on the just-merged native Markdown mirror (#5307):

• /sitetree.json — the full site IA as JSON, mirroring the core child-selection and manualLink logic of the real sidebar nav (layouts/partials/sidebar-tree.html, this repo's own override — not the vendored themes/docsy/ copy). It's a deliberate simplification, not a byte-for-byte mirror: it hard-excludes toc_hide pages (e.g. individual tutorials) entirely, and doesn't replicate the sidebar's truncation/hide_children/active-path-dependent display logic, which don't have a meaningful analog in a static JSON tree.
• /llms.txt — a curated, auto-generated entry point for agents, replacing the hand-maintained static/llms.txt. Driven entirely by data/llms_pages.yaml; titles/URLs/descriptions are always pulled live from the real pages via site.GetPage, so they can't drift out of sync.
• /llms-full.txt — full text of a small, curated set of "orientation" pages (currently 2), reusing the exact same Markdown-rendering partial (render-page-markdown.html) every page's own .md mirror already uses.

Also extracts a render-page-markdown.html partial out of single.md/list.md. Both templates were created by #5307 and have been byte-identical since then; this just centralizes their shared body into one partial instead of two copies of the same logic. Sensible refactor, no change in function — verified with a byte-for-byte diff of two representative pages' generated output (one single.md-path leaf page, one list.md-path section) before and after. It's what lets llms-full.txt reuse the exact same page-rendering instead of needing a third copy.

data/llms_pages.yaml's header comment documents the curation strategy: orientation/workflow pages get full: true and are inlined into llms-full.txt; reference/lookup pages stay link-only in llms.txt, since an agent fetching one resource's own .md mirror on demand beats a monolithic dump of every reference page.

CLAUDE.md has a new "Agent discoverability files" section documenting this system, including several non-obvious Hugo behaviors this PR ran into:

• Hugo's list.<format> page-template lookup keys off the output format's Name field, not its baseName — llms.txt/llms-full.txt initially had a silent naming collision (both rendered identical content) until their template files were renamed to match their format names exactly.
• Hugo's shortcode template lookup keys off the output format's file suffix instead — a different rule from the one above. llms-full.txt's output format needed mediaType = "text/markdown" (not text/plain) so shortcodes like alert/expand resolve to their existing clean .md variants instead of falling back to raw HTML. It publishes as /llms-full.md internally, with a Netlify rewrite (matching the existing section/home Markdown-mirror trick) exposing it at the conventional /llms-full.txt path.
• Fixed after independent review: giving llms-full.txt's output format the .md suffix (previous bullet) introduced a second list.*.md template alongside the pre-existing list.md. Hugo's list.<format> lookup turned out not to be a strict name-match — with two .md-suffixed candidates present, it silently preferred the new one for every section's and the home page's own real Markdown mirror output, replacing their content sitewide. Fixed by renaming list.md to list.markdown.md, name-qualifying it the same way every other output format's template already was. This is the kind of regression that a scoped, per-file verification pass doesn't catch — worth a standing full-site check (e.g. grepping every generated mirror for content that shouldn't be there) rather than relying on spot-checks of the specific new files.
• The HTML-escaping risk this PR ran into lives in printing partials with an .html/.txt file suffix, not in top-level page templates — bare interpolation directly inside a list.<format>/single.<format> template never escapes. render-page-markdown.html avoids it by composing a string and using {{ return }} instead of printing.

Scope note: this covers items 1–2 of a larger agent-experience handoff. Accept-header content negotiation via a Netlify Edge Function (item 3) is intentionally out of scope here — it's differently-shaped work (new deploy infrastructure) that a separate branch is already touching.

Test plan

• make build-prod completes with no errors
• vale (pinned 3.12.0, matching CI) is clean on every .md file this PR touches or creates
• /sitetree.json is valid JSON; /hardware/-style manualLink shims resolve to their real target; no /tags/* taxonomy leakage
• /llms.txt and /llms-full.txt are distinct, correctly generated, and contain no HTML-escaping artifacts
• /llms-full.txt's inlined pages render as clean Markdown (no raw HTML/SVG shortcode leakage), verified against their own .md mirrors
• Size guardrail (warnf over ~200KB) and missing-page guardrail (warnf on an unresolvable data/llms_pages.yaml entry) both tested in both directions
• Spot-checked live via hugo server: all three routes return 200 with correct content-types, and the new <link> tags render in a real page's <head>
• Full-site regression check after the list.md shadowing fix: every section's and home's .md mirror across all 1677 pages verified to no longer contain llms-full.txt's content

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for viam-docs ready!

Name	Link
🔨 Latest commit	15b3fb8
🔍 Latest deploy log	https://app.netlify.com/projects/viam-docs/deploys/6aa46870f44f2a0008bdeb54
😎 Deploy Preview	https://deploy-preview-5311--viam-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 38 (🔴 down 3 from production) Accessibility: 99 (no change from production) Best Practices: 100 (no change from production) SEO: 92 (no change from production) PWA: 60 (no change from production) View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[btshrewsbury-viam]

Independent review of 0522c7a against base c4c03b3, both built with hugo -e production --config config.toml,config_prod.toml --minify on the CI-pinned Hugo 0.152.2. Everything below I reproduced myself; where I disagree with the PR's own narrative I say so.

1. Blocking: list.llmsfulltxt.md shadows list.md for every section and home Markdown mirror

LLMSFULLTXT has mediaType = text/markdown, so its template carries the same .md suffix as the MARKDOWN format. Hugo's list lookup then prefers list.llmsfulltxt.md over the generic list.md for the MARKDOWN output too. Result in the prod build: all 90 index.md mirrors (every section plus the home page) now contain the llms-full text instead of their own page.

$ diff public-before/cli/index.md public-after/cli/index.md | head -3
< # Viam CLI
---
> # Viam — agent-facing pages, full text
$ find public-after -name index.md -not -path '*/include/*' | wc -l          # 90
$ ... | xargs -I{} head -c 40 {} | grep -c '# Viam — agent-facing pages'      # 90
$ cmp public-after/llms-full.md public-after/index.md && echo IDENTICAL       # IDENTICAL

Minimal repro (three-page site, section = ["HTML","MARKDOWN"] only, home additionally lists LLMSFULLTXT): public/sec/index.md renders list.llmsfulltxt.md even though sec never opted into that format. Build is clean, no warning. This undoes the section/home half of #5307.

Fixes I tested in the repro:

• Rename layouts/_default/list.md to layouts/_default/list.markdown.md. Sections, home /index.md, /llms-full.md, and /llms.txt all render the right template. This is the one I'd take.
• Renaming the new template to home.llmsfulltxt.md (leaving list.md alone) fixes the 89 sections but not the home page: home.llmsfulltxt.md still wins for home's own MARKDOWN output, so /index.md stays shadowed. Not sufficient on its own.

single.md is unaffected (single lookup never considers list.*). Worth adding a line to CLAUDE.md's "Markdown mirror" section: any future output format sharing the .md suffix must use format-qualified template names, or it silently shadows the mirror.

2. "Byte-for-byte diff" claim

diff -rq public-before public-after reports 634 differing files: 90 .md (item 1), 540 .html (spot-checked one: only the new <link rel=alternate ... sitetree.json> in <head>), plus _redirects, llms.txt, sitemap.xml (lastmod), and the two new files. The .md mirrors are the ones the claim was about, and they are not identical.

One-liner, latent: for a page with no description, the old inline template emitted # Title\n> Source: and render-page-markdown.html emits # Title\n\n> Source:. Every page under docs/ currently has a description, so nothing shows today.

3. /sitetree.json is not "every page on the site"

llms.txt line 19 advertises "every page on the site, with hierarchy, always current". The JSON has 415 nodes / 390 unique paths; the site serves 478 pages with .md mirrors (388 pages + 90 sections). All 57 tutorial pages are absent because docs/tutorials/_index.md sets toc_hide: true and the template filters children on toc_hide. That also blocks the plan already in CLAUDE.md to point /tutorials/ and /tutorials/all/ at sitetree.json. Either soften the wording or don't prune on toc_hide (a sidebar-visibility flag, not an "exists" flag).

4. Section landing URLs vanish; 25 paths appear twice

path is the resolved manualLink target, so /hardware/, /cli/, /data/, /build-apps/, /tutorials/ appear nowhere in the JSON; each section node instead carries its "Overview" child's URL, which then repeats as that child (25 duplicates, e.g. /cli/overview/, /data/overview/, /set-up-a-machine/first-machine/). Fine for a sidebar label, wrong for a machine-readable IA where path is the node's identity. There is also no field distinguishing section from leaf. Suggest path = the page's own permalink, with manualLink as a separate optional field.

5. The escaping rule added to CLAUDE.md is misdiagnosed

The new bullet says Hugo's html/template escaper is active for a top-level list.<format> template's own bare {{ }} output regardless of isPlainText/isHTML. I could not reproduce that. In the repro I gave pages titles like Sec's & co and a data value data's & val, then interpolated them bare in list.llmsfulltxt.md, list.llmstxt.txt, list.md, and single.md:

expression	output
{{ .Title }}, {{ site.Data.x.v }}, {{ printf ... }}, {{ "it's & lit" }} bare	raw, no escaping, in all four templates
{{ partial "p.html" . }} where the partial prints {{ .Title }}	Sec's & co
{{ partial "p.txt" . }} (same body)	Sec's & co
{{ partial "p.md" . }} (same body)	raw
{{ partial "p.html" . }} where the partial ends in {{ return ... }}	raw
{{ partial "p.html" . | safeHTML }}	still ' / & (escaping already happened inside the partial)

So what escapes is a printing partial with an .html (or .txt) suffix, not top-level interpolation. That is exactly why 8d5f257's {{ return }} change fixed render-page-markdown.html. Consequences: the | safeHTML calls in list.llmstxt.txt and list.llmsfulltxt.md are no-ops (harmless, but the CLAUDE.md text will send the next person after the wrong thing), and the isHTML = false line on LLMSFULLTXT is doing nothing. Suggest rewriting the bullet to: "a partial whose file suffix is .html/.txt HTML-escapes anything it prints; return a string via {{ return }} (or use a .md partial) instead."

Smaller things

• list.sitetree.json's header and the new CLAUDE.md section both cite themes/docsy/layouts/partials/sidebar-tree.html. Nothing exists there (docsy is a Hugo module via go.mod); the file in force is the repo's own layouts/partials/sidebar-tree.html.
• "Mirrors the sidebar" is approximate: the sitetree drops hide_children, sidebar_menu_truncate (default 50), the canonical link override, and the sidebar's show-toc_hide-when-on-active-path behavior. Four entries (/reference/sdks/{go,cpp,flutter,typescript}/) point at pages whose .md mirror is a "This page redirects to ..." stub.
• data/llms_pages.yaml's header says it's read by list.llmsfulltxt.txt; the file is list.llmsfulltxt.md.
• The sitetree link in llms.txt is injected by {{ if eq .title "Reference" }}; renaming that section in the YAML silently drops the link. Put it in its own unconditional block.
• The new <link rel="alternate" type="application/json" href=".../sitetree.json"> asserts the sitetree is an alternate representation of the current page, sitting next to the genuine rel=alternate type=text/markdown mirror link. rel="index" or a describedby sibling would be more honest. It also never renders on the home page (layout: "empty" uses layouts/docs/empty.html, which has its own <head>), and neither does the pre-existing rel="describedby" llms.txt link, so that's an inherited gap rather than a new regression, but home is the page an agent hits first.

Things I checked that hold up: vale 3.12.0 clean on CLAUDE.md and list.llmsfulltxt.md; markdownlint/prettier genuinely don't cover layouts/; make build-prod exits 0; the two pages inlined into llms-full.md are byte-identical to their own .md mirrors with no HTML/SVG/entity leakage; sitetree.json is valid JSON with no /tags/* leakage and a description on every node; the /llms-full.txt /llms-full.md 200 rule lands last in _redirects with no wildcard ahead of it; list.sitetree.json does not collide with list.typesense.json (typesense output is byte-identical before/after).

---

Generated by Claude Code

[viambot]

It looks like the following files may have been renamed. Please ensure you set all needed aliases:

 rename layouts/{_default/list.md => partials/render-page-markdown.html} (56%)

[Jeremy Rose (jeremyrose-viam)]

Thanks for this — the list.md shadowing finding is real and serious, and I completely missed it. Confirmed and fixed in 15b3fb8:

1. Finding 1 (blocking): reproduced exactly as described — every section's and home's .md mirror was getting llms-full.txt's content. Took your tested fix (list.md → list.markdown.md); verified across all 1677 pages that zero mirrors still show the shadow content.
2. Finding 2: fair — "byte-for-byte" in the PR description was only ever about the two representative pages I diffed for the Task 1 refactor, not a full-site claim. Reworded, and added your suggested full-site regression check to the test plan going forward.
3. Finding 3: fixed — llms.txt's sitetree.json line no longer claims "every page"; now notes the toc_hide exclusion explicitly.
4. Finding 4: didn't change the JSON schema (the recursion logic is a deliberate, unmodified cherry-pick from prior work), but documented the path non-uniqueness and the section-vs-leaf ambiguity in CLAUDE.md so it's not a surprise for the next reader.
5. Finding 5: you're right and I was wrong — re-tested it myself and reproduced your result exactly. The escaping risk is in a printing partial with an .html/.txt suffix, not top-level list.<format> interpolation. My original "confirmation" test was confounded (I changed isHTML and removed safeHTML in the same test and blamed the wrong variable). Removed the now-proven-unnecessary safeHTML calls and corrected CLAUDE.md's explanation.

Smaller things, all applied: fixed the sidebar-tree.html citation (repo's own override, not the shadowed themes/docsy/ copy) and softened the "mirrors the sidebar" claim to name the real divergences; changed the sitetree.json <link> to rel="describedby"; fixed the stale list.llmsfulltxt.txt reference in data/llms_pages.yaml.

Left alone deliberately: the four /reference/sdks/* manualLink-to-stub entries and the sidebar's fuller truncation/hide_children logic — both inherent to the cherry-picked recursion I intentionally didn't modify, now documented rather than fixed, since changing that logic is a bigger design call than this PR should make unilaterally.

Appreciate the depth here — this caught something my own verification process structurally couldn't have (I was diffing the new files against themselves, never re-checked the pre-existing mirror output after the change).

[btshrewsbury-viam]

Re-reviewed 15b3fb8: rebuilt it and diffed against a fresh baseline build of c4c03b3, same Hugo 0.152.2 and prod flags as before.

Confirmed fixed

• Finding 1. Zero of the 1282 generated .md files differ from the baseline. All 90 section and home mirrors are byte-for-byte back to their pre-PR content.

  One note on method rather than result: diff -rq against a baseline build is a stronger standing check than grepping every mirror for llms-full.txt's content. The grep confirms the known wrong template is gone, but wouldn't catch a different wrong template taking over. Worth making the baseline diff the test-plan line.

• Finding 5. With every safeHTML removed, llms.txt and llms-full.md contain zero '/&-class artifacts in the real build, so the corrected diagnosis holds outside the repro too.

• No collateral: llms-full.md and sitetree.json are byte-identical to the previous build, llms.txt differs only by the reworded sitetree line, vale 3.12.0 is clean on all three changed .md files (including the new list.markdown.md), and both rel="describedby" links render.

One thing I'd still want before merge

single.md is still unqualified, so the trap is only half-closed. The fix renamed list.md → list.markdown.md but left its twin layouts/_default/single.md as is. Same mechanism, still armed — verified in the repro:

# layouts/_default/single.md present, no single.*.md sibling
public/sec/p.md -> SINGLE-MD: Page

# add layouts/_default/single.llmsfulltxt.md
public/sec/p.md -> SINGLE-LLMSFULL

Every leaf page's .md mirror, silently replaced, exactly as happened to the sections. Renaming it to single.markdown.md closes it and makes the pair consistent. Since the whole point of the CLAUDE.md note is to stop the next person walking into this, leaving one of the two templates unqualified undercuts it.

Smaller

• CLAUDE.md now contradicts itself two bullets apart. The older bullet still says "Template filenames must match the output format's Name field exactly ... Hugo's own template lookup keys off the format name"; the new one says the lookup "isn't a strict 'must match this format's Name' rule." That stale first claim is the mental model that produced the bug.
• The new rule is overbroad as stated: "ambiguous whenever multiple list.*.<suffix> candidates share that suffix" isn't what happens. list.typesense.json and list.sitetree.json share .json and resolve correctly (typesense output is byte-identical before/after), and list.markdown.md + list.llmsfulltxt.md now coexist correctly. The actual rule is narrower: an unqualified list.<suffix> loses to any qualified list.<name>.<suffix> sharing that suffix, whether or not <name> matches the format being rendered. The prescription is right; the mechanism as written would mislead someone judging whether two qualified templates are safe.
• The finding-4 note documents path non-uniqueness but not the sharper half: each section's own permalink is absent from the JSON entirely. /hardware/, /cli/, /data/, /build-apps/, /tutorials/ appear nowhere, so an agent resolving /cli/ against the tree gets nothing back. Fine to defer the schema change, but worth a line in the same note.
• {{ if eq .title "Reference" }} in list.llmstxt.txt is untouched and wasn't mentioned — renaming that section in data/llms_pages.yaml still silently drops the sitetree link from llms.txt.

---

Generated by Claude Code

[Jeremy Rose (jeremyrose-viam)]

Closing in favor of #5312 — reworked to generate these files via Hugo Pipes instead of custom output formats, which structurally avoids the list.<format> collision bug found in review here, and adds a tutorials catalog. See #5312 for the full writeup.