Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ conformance/node_modules/
*.tgz
.DS_Store
.playwright-mcp/
trigger-results.json
138 changes: 128 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import {
createProgram,
insertDefaultVerb,
mutating,
SHARED_NOUNS,
type NounSpec,
} from "@bunizao/cli-kit";

Expand All @@ -35,17 +36,19 @@ const program = createProgram({
description: "Example CLI",
});

const units = program.command("units").aliases(["courses", "projects"]);
units.command("list").action(listUnits);
units.command("show <unit>").action(showUnit);
mutating(units.command("set <unit> <state>")).action(setUnitState);
const courses = program.command("courses").aliases(["units", "projects"]);
courses.command("list").action(listCourses);
courses.command("show <course>").action(showCourse);
const tasks = program.command("tasks");
tasks.command("list <course>").action(listTasks);
mutating(tasks.command("set <course> <task> <state>")).action(setTaskState);

const nouns: readonly NounSpec[] = [
SHARED_NOUNS.courses,
{
name: "units",
aliases: ["courses", "projects"],
verbs: ["list", "show", "set"],
defaultByArity: { 0: "list", 1: "show" },
name: "tasks",
verbs: ["list", "set"],
defaultByArity: { 1: "list" },
valueFlags: [],
},
];
Expand Down Expand Up @@ -163,13 +166,128 @@ aliases, positional arguments, options, enum values, nested commands, and `mutat
commands at noun depth must use a verb exported in `VERBS`; unsupported verbs throw. `auth` and
`skills` are action groups rather than domain nouns and are not assigned a `verb` field.

The exit-code table, error vocabulary, and verb set are public versioned API. Changing one requires
a major package release.
A top-level command named after a shared noun's alias throws, and so does a verb that shared noun
does not offer.

The exit-code table, error vocabulary, verb set, and shared nouns are public versioned API. Changing
one requires a major package release.

The conformance suite is isolated under `conformance/`. Until all three CLIs have published their
normalized releases, missing binaries are reported as skipped tests. Its CI workflow installs the
published packages before running the suite.

## Shared nouns

`SHARED_NOUNS` holds the nouns every CLI names the same way. A CLI that offers one uses its spec
as is, or with fewer verbs and its own `valueFlags`; domain nouns such as `tasks` or `lessons`
stay with each CLI.

| Noun | Aliases | Verbs | Default by arity |
|---|---|---|---|
| `courses` | `units`, `projects` | list, show, get | 0 list, 1 show |
| `grades` | | list | 0 list, 1 list |

`courses` is canonical because Ed and Moodle both call them courses; `units` is an Australian usage.

Each `grades` row carries `grade` and `max` as numbers or `null`, and a `status` from
`GRADE_STATUSES`: `graded`, `ungraded`, `unreleased` (marked but not released to the student),
`not_submitted`, or `unavailable` (the platform refused or failed for that row). Display strings
such as a letter grade or a range are optional extra fields.

## Intents, skills and plugins

An `Intent` maps what a person says to the command and the MCP tool that answer it. One list of
intents is the single source for the skill, the MCP tool descriptions and the agent packaging:

```ts
const INTENTS: readonly Intent[] = [
{
when: "my grades / what did I get",
what: "Graded items per course with grade, max and status.",
command: "example grades [COURSE]",
tool: "list_grades",
refs: "course code, name or id",
then: "show the item for feedback",
cost: "small",
},
];

server.registerTool("list_grades", { description: toolDescription(INTENTS[0]) /* , schema */ }, handler);

const files = agentFiles({
skill: {
name: "example-cli",
description: "Reads Example courses, grades and deadlines",
useWhen: 'the user names Example or asks "what\'s due", "my grades"',
title: "Example CLI",
intents: INTENTS,
mcpServer: "example",
workflows: [{ title: "Submit an assignment", steps: ["...", "Check the submission status"] }],
program: commandsJson(program),
},
version,
repository: "https://github.com/example/example-cli",
author: { name: "example" },
mcpServers: { example: { command: "npx", args: ["-y", "example-cli", "mcp"] } },
});
writeAgentFiles(process.cwd(), files); // { check: true } returns stale paths for CI
```

`agentFiles` returns, by path from the repository root:

- `skills/<name>/SKILL.md`, which `npx skills add owner/repo` and Claude Code both find. An agent
loads it whole, so it stays short: one table of what the user says against the CLI command and
the `server:tool` MCP tool, the workflows as copyable checklists, the rules and the setup. A CLI
with a mutating command gets the shared dry-run, confirm, apply, read-back workflow first. Each
tool's full description stays with the MCP server.
- The frontmatter description is `description` then "Use when `useWhen`". `renderSkill` throws
when the name or description breaks the Agent Skills limits (lowercase hyphenated name, 1024
characters, no `<` or `>`) or uses "I" or "you" outside quotes.
- `skills/<name>/references/commands.md`: every command with its arguments, options and whether
it mutates, plus the exit codes. Pages in `references` are written next to it.
- `skills/<name>/agents/openai.yaml`, when `openai` metadata is given.
- `.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json`: the repository is a Claude
Code plugin, which loads `skills/` by default, and its own marketplace. Install with
`claude plugin marketplace add owner/repo`, then `claude plugin install example-cli@example-cli`.
A repository with a root `CLAUDE.md` gets one validator warning, since that file is not plugin
context; `claude plugin validate .` passes, `--strict` does not.

`writeAgentFiles` never deletes, so a CLI moving to this layout removes its old root `SKILL.md`,
`references/` and `agents/` itself.

An MCP server that runs on Cloudflare Workers imports `toolDescription` from
`@bunizao/cli-kit/intents`, which pulls in neither commander nor Node builtins.

`checkIntentCommands(program, nouns, intents)` throws when an intent's command no longer parses:
a renamed noun or verb, a removed option, or the wrong binary name. Call it from the CLI's tests
so the skill and the MCP descriptions cannot drift from the command tree.

### Trigger evals

`evals/trigger.mjs` checks whether the descriptions route requests to the right skill. It puts
every sibling skill in one Codex workspace, so they compete as they do for a student who has all
three installed. A skill that steals its sibling's requests looks like the sibling undertriggering
when either is tested alone. For each query it records which `skills/<name>/` Codex opens first.

```bash
node evals/trigger.mjs \
--skill ../moodle-cli/skills/moodle-cli \
--skill ../ontrack-cli/skills/ontrack-cli \
--skill ../edstem-cli/skills/edstem-cli \
--runs 3
```

- The folder name is the skill name. `--pool` defaults to `evals/trigger-pool.json`: 29
requests, each with the skill it should reach or `none`. `--model` defaults to Codex's configured
model. `--out` keeps every run's pick and first command.
- Each Codex run is killed at its first command, and each skill's binary (`moodle`, `ontrack`,
`edstem`) is a failing stub first on `PATH`, so nothing reaches a real account. `HOME` is a
temporary folder that hides the user's own skills; `CODEX_HOME` keeps the sign-in.
- A run costs about 16k input tokens, so the default 29 × 3 is about 1.4M. Results move by about
one run per query between passes, so compare changes across three runs, not one.
- This measures Codex, not Claude. Claude Code reads the same folder layout and frontmatter, but
its routing can differ.

## Prompts and progress

`createUi` gives every CLI the same interactive surface, drawn with `@clack/prompts` on
Expand Down
2 changes: 2 additions & 0 deletions dist/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ export * from "./color.js";
export * from "./confirm.js";
export * from "./errors.js";
export * from "./help.js";
export * from "./intents.js";
export * from "./nouns.js";
export * from "./output.js";
export * from "./program.js";
export * from "./report.js";
export * from "./skills.js";
export * from "./ui.js";
export * from "./theme.js";
2 changes: 2 additions & 0 deletions dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ export * from "./color.js";
export * from "./confirm.js";
export * from "./errors.js";
export * from "./help.js";
export * from "./intents.js";
export * from "./nouns.js";
export * from "./output.js";
export * from "./program.js";
export * from "./report.js";
export * from "./skills.js";
export * from "./ui.js";
export * from "./theme.js";
31 changes: 31 additions & 0 deletions dist/intents.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import type { Command } from "commander";
import { type NounSpec } from "./nouns.js";
/**
* One thing a person asks for, mapped to the CLI command and the MCP tool that answer it.
* The skill, the MCP tool descriptions and the help text all render from these fields.
*/
export interface Intent {
/** What the person says, e.g. "my grades / what did I get". */
readonly when: string;
/** What comes back, as a sentence. */
readonly what: string;
/** The command as typed, with placeholders such as UNIT; omitted for an MCP-only tool. */
readonly command?: string;
/** The MCP tool for the same request; omitted for a CLI-only command. */
readonly tool?: string;
/** Where nearby requests go instead, e.g. "get_thread for thread bodies". */
readonly instead?: string;
/** The references and arguments it accepts. */
readonly refs?: string;
/** The usual next step. */
readonly then?: string;
/** Rough response size. */
readonly cost?: string;
}
/** The MCP tool description for an intent: prose, because a model reads it. */
export declare function toolDescription(intent: Intent): string;
/**
* Throw when an intent's command no longer parses against the program: a renamed noun or verb,
* a wrong binary name, or an option that was removed. Run it in the CLI's tests.
*/
export declare function checkIntentCommands(program: Command, nouns: readonly NounSpec[], intents: readonly Intent[]): void;
70 changes: 70 additions & 0 deletions dist/intents.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import { insertDefaultVerb } from "./nouns.js";
/** The MCP tool description for an intent: prose, because a model reads it. */
export function toolDescription(intent) {
return [
sentence(intent.what),
sentence(`Use when the user says "${intent.when}"`),
intent.instead && sentence(`Instead: ${intent.instead}`),
intent.refs && sentence(`Input: ${intent.refs}`),
intent.then && sentence(`Then: ${intent.then}`),
intent.cost && sentence(`Cost: ${intent.cost}`),
].filter(Boolean).join(" ");
}
/**
* Throw when an intent's command no longer parses against the program: a renamed noun or verb,
* a wrong binary name, or an option that was removed. Run it in the CLI's tests.
*/
export function checkIntentCommands(program, nouns, intents) {
const problems = intents.flatMap((intent) => (intent.command ? commandProblems(program, nouns, intent.command) : []));
if (problems.length)
throw new Error(`Intent commands out of date:\n${problems.join("\n")}`);
}
const ALWAYS_ACCEPTED = new Set(["-h", "--help", "-V", "--version"]);
function commandProblems(program, nouns, text) {
const [binary, ...rest] = words(text);
if (binary !== program.name())
return [`${text}: starts with "${binary}", not "${program.name()}"`];
const argv = insertDefaultVerb(rest.map(unwrapOptionalFlag), nouns);
const path = [program];
let unmatched;
for (const token of argv) {
if (token.startsWith("-"))
continue;
const current = path.at(-1);
const child = current.commands.find((command) => command.name() === token || command.aliases().includes(token));
if (!child) {
unmatched = token;
break;
}
path.push(child);
}
const problems = [];
const resolved = path.at(-1);
// A group that takes no arguments of its own must be followed by one of its commands.
if (unmatched !== undefined && resolved.commands.length && !resolved.registeredArguments.length) {
problems.push(`${text}: "${unmatched}" is not a command of "${path.map((command) => command.name()).join(" ")}"`);
}
const options = new Set(path.flatMap((command) => command.options.flatMap(optionNames)));
for (const token of argv) {
const flag = token.split("=")[0] ?? token;
if (flag.startsWith("-") && !ALWAYS_ACCEPTED.has(flag) && !options.has(flag)) {
problems.push(`${text}: unknown option "${flag}"`);
}
}
return problems;
}
// Quoted strings and bracketed placeholders such as <lesson id> stay one word.
function words(text) {
return text.replace(/^`|`$/g, "").match(/"[^"]*"|'[^']*'|<[^>]*>|\[[^\]]*\]|\S+/g) ?? [];
}
// `[--final]` documents an optional flag; check it like `--final`.
function unwrapOptionalFlag(token) {
const inner = /^\[(-[^\s\]]+)/.exec(token);
return inner?.[1] ?? token;
}
function optionNames(option) {
return [option.long, option.short].filter((name) => Boolean(name));
}
function sentence(text) {
return /[.!?]$/.test(text) ? text : `${text}.`;
}
26 changes: 26 additions & 0 deletions dist/nouns.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,30 @@ export interface NounSpec {
/** Option flags whose following token is a value, not a positional argument. */
readonly valueFlags?: readonly string[];
}
/**
* Nouns every CLI names the same way. A CLI that offers one uses its spec as is, or with fewer
* verbs and its own valueFlags, so `moodle grades` and `edstem grades` read alike.
*/
export declare const SHARED_NOUNS: {
readonly courses: {
readonly name: "courses";
readonly aliases: readonly ["units", "projects"];
readonly verbs: readonly ["list", "show", "get"];
readonly defaultByArity: {
readonly 0: "list";
readonly 1: "show";
};
};
readonly grades: {
readonly name: "grades";
readonly verbs: readonly ["list"];
readonly defaultByArity: {
readonly 0: "list";
readonly 1: "list";
};
};
};
/** The `status` of one row in `grades` output. */
export declare const GRADE_STATUSES: readonly ["graded", "ungraded", "unreleased", "not_submitted", "unavailable"];
export type GradeStatus = (typeof GRADE_STATUSES)[number];
export declare function insertDefaultVerb(argv: readonly string[], nouns: readonly NounSpec[]): string[];
19 changes: 19 additions & 0 deletions dist/nouns.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
/**
* Nouns every CLI names the same way. A CLI that offers one uses its spec as is, or with fewer
* verbs and its own valueFlags, so `moodle grades` and `edstem grades` read alike.
*/
export const SHARED_NOUNS = {
courses: {
name: "courses",
aliases: ["units", "projects"],
verbs: ["list", "show", "get"],
defaultByArity: { 0: "list", 1: "show" },
},
grades: {
name: "grades",
verbs: ["list"],
defaultByArity: { 0: "list", 1: "list" },
},
};
/** The `status` of one row in `grades` output. */
export const GRADE_STATUSES = ["graded", "ungraded", "unreleased", "not_submitted", "unavailable"];
const GLOBAL_VALUE_FLAGS = new Set(["--fields", "-o", "--output"]);
export function insertDefaultVerb(argv, nouns) {
const result = [...argv];
Expand Down
13 changes: 13 additions & 0 deletions dist/program.js
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
import { Command } from "commander";
import { colorEnabled } from "./color.js";
import { styledHelp } from "./help.js";
import { SHARED_NOUNS } from "./nouns.js";
export const VERBS = ["list", "show", "read", "get", "search", "send", "submit", "set", "mark-read"];
const VERB_SET = new Set(VERBS);
const SHARED = Object.values(SHARED_NOUNS);
const mutations = new WeakSet();
const ACTION_GROUPS = new Set(["auth", "skills"]);
const ACTIONS = {
Expand Down Expand Up @@ -53,9 +55,20 @@ function describe(command, noun) {
if (noun && ACTION_GROUPS.has(noun) && !ACTIONS[noun]?.has(command.name())) {
throw new Error(`Unsupported action "${command.name()}" on group "${noun}".`);
}
// A shared noun's name and aliases belong to that noun alone, whether used as a name or an alias.
for (const word of noun ? [] : [currentNoun, ...command.aliases()]) {
const owner = SHARED.find((spec) => spec.name === word || spec.aliases?.includes(word));
if (owner && owner.name !== currentNoun) {
throw new Error(`"${word}" belongs to the shared noun "${owner.name}"; name that command "${owner.name}".`);
}
}
if (verb && !VERB_SET.has(verb)) {
throw new Error(`Unsupported verb "${verb}" on noun "${noun}".`);
}
const shared = SHARED.find((spec) => spec.name === noun);
if (verb && shared && !shared.verbs.includes(verb)) {
throw new Error(`Unsupported verb "${verb}" on shared noun "${noun}".`);
}
return {
name: command.name(),
noun: currentNoun,
Expand Down
Loading
Loading