From a5eac45719118ed01e702a0158f94a3230e8b8e3 Mon Sep 17 00:00:00 2001 From: growlf Date: Sun, 25 Oct 2026 14:40:11 -0700 Subject: [PATCH] session-notes: 2026-10-25 research skill, and why 'opencode run' cannot test a command Records the wrong turn that consumed the session: opencode run does not expand slash commands, so the command template never reached the model and the format failures were an artifact of the harness. Settled by capturing the provider request; commands are server-side (POST /api/session/{id}/command). --- docs/session-notes/2026-10-25.md | 85 ++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/session-notes/2026-10-25.md diff --git a/docs/session-notes/2026-10-25.md b/docs/session-notes/2026-10-25.md new file mode 100644 index 0000000..8558170 --- /dev/null +++ b/docs/session-notes/2026-10-25.md @@ -0,0 +1,85 @@ +# Session Notes — 2026-10-25 + +## Work Done + +### 1. `research` skill + `/research` command (#452 → PR #453, squash `b7444b3`) + +Operator wanted a general procedure that answers factual questions from sources +instead of memory: "use all available and appropriate skills, tools, web +research, and other resources". + +Shipped: + +- `.opencode/skills/research/SKILL.md` — the discipline: no claim reaches the + operator unless a source was read this session; an evidence ladder (repo/code + → live system state → vendor docs/API → secondary); cross-check load-bearing + claims; date anything that drifts. +- `.opencode/command/research.md` — `/research `, self-contained: carries + the mandate and the mandatory three-part answer (answer, evidence table, + confidence + unverified) so compliance does not depend on the model electing + to load the skill. +- Registered in `AGENTS.md`; `.claude/skills/research` symlink (the repo tracks + these 37 siblings the same way). +- `tests/test_research_skill.py` — six guards: canonical frontmatter, 60-line + ceiling, command names the skill, command keeps the three-part contract, + AGENTS.md registration, no-guess rule present. + +`websearch` stays the tool doc for the web rung; `research` is the procedure +that decides when to reach for it. Not merged into one. + +Global install: `~/.config/opencode/skills/research` and +`~/.config/opencode/commands/research.md`, both aimed at the **main checkout** +so removing a worktree cannot break them (see #454 for why that matters). + +## The expensive wrong turn, and what settled it + +Initial testing used `opencode run "/research …"`. Every run returned prose and +ignored the required format. Three rounds of prompt rewording later, the +conclusion "the local model won't follow the format" looked solid — and was +wrong. + +What settled it: capturing the actual provider request. + +- `opencode run` does **not** expand slash commands. It sends the literal text + `"/research "`. Captured user message confirmed it; marker commands + in both `.opencode/command/` and `.opencode/commands/` also arrived verbatim. +- Commands are a **server** concept: `GET /api/command?location[directory]=` + listed 34 commands including `research`, and + `POST /api/session/{id}/command` expanded the template to 1,066 chars with the + topic substituted. +- Feeding that exact template to the local model produced the answer, a 7-row + evidence table, a confidence line and `Unverified: none` — the format works. + +Method, for reuse: point the provider's `baseURL` at a local logging proxy, +run one command through the API, inspect `messages[]`, restore the config +(trap-protected so a dead proxy cannot be left in place). Read the system +prompt too — it carries the skills catalog and no command catalog, which is why +the model could load the skill from a hint but never saw the template. + +Cost: most of the session, plus one wrong commit message (amended in `4be10e5` +so history records the real cause). + +**Rule for next time:** slash commands are TUI/API-only. `opencode run` sends +prose, so it cannot test a command. Test commands through the TUI, or through +`POST /api/session/{id}/command`. + +## Also found + +- **#454** — 33 of 45 global skill symlinks in `~/.config/opencode/skills/` are + dead; they point into a deleted worktree. Silent, because a missing skill + looks like an irrelevant one. +- **`make test` on the main checkout is still red**, and the earlier diagnosis + was imprecise: the shadowing top-level `tests/` package belongs to + `routeros_api` 0.21.0 (a mikromcp dependency), not mikromcp itself. A fresh + worktree venv is unaffected — that is how the full suite ran green + (1713 passed, 1 skipped). Fix is a clean venv, not a code change. + +## Review outcome + +Self-review found one real gap: the command's format contract was unguarded, so +deleting the table or confidence line would leave every test green while the +deliverable disappeared. Added `test_command_carries_the_output_contract` +(`d18d9a2`) before merge. + +No infrastructure changes. Provider config edits were temporary and reverted; +the proxy was stopped and its port freed before the session ended.