Repository navigation
feat(kb-engineering): add engineering checklists and project verification setup - #6
Merged
Merged
Conversation
…t helper Agents verified their work with unit tests alone and had no guidance for what to check per kind of system. Nine checklists (testing, reliability, performance, security, web, backend, Java, distributed, desktop) list what to look out for with a concrete check per item and sources per file; procbudget.py measures a process tree's memory, growth, CPU and VRAM for projects that opt in; references/setup.md describes the project verify recipe.
…y-setup Agents need only the checklists that apply to a change. The router picks them from project files and the task, and lists the always-on rules and the evidence block. /devline:verify-setup inventories a project, offers a menu and writes only the picked parts as Claude Code's native verify skill.
…idence Implementers and reviewers judged runtime changes from unit tests alone. Both now load the kb-engineering checklists that apply, always run typecheck and build, follow the voice comment rules, and run the project's verify recipe for runtime-facing changes; the reviewer blocks on missing evidence. CI runs the kb-engineering structure and helper tests with the same command as locally.
…ndings procbudget polled the process group while the leader was an unreaped zombie, so every command run waited the full grace period before SIGKILL. It now reaps the leader; tests drop from about a minute to 14 s. Also: security's always-on group is routed on every runtime-facing change, credits list the right licences, VRAM is labelled MiB, and verify-setup keeps the fast path under about two minutes.
…ists disable-model-invocation: true kept the skill out of the agents' skills frontmatter, so implementer and reviewer never saw it. With injection on, a dry run showed agents still picked checklists without reading them; both now read the selected files and report which. The recipe guidance now stops the app before scanning logs, runs it unbuffered, proves the scan with a planted warning, never allowlists real defects, and works without CI.
…checklists Deep review found that procbudget left its target running when it was itself sent SIGTERM or SIGHUP, because the target lives in its own session. main() now turns those signals into SystemExit so the finally block stops the process group; a lifecycle test covers it. The agents pointed at a comment-rules file that does not exist yet, so the rule is now inline, and a structure test checks that every plugin skill path named in agents/*.md exists. The reviewer must now Read every checklist the tables select, cite the checklist rule per finding, and list what it read. The docs now say Claude Code prompts Claude to run the verify skill before commits rather than enforcing it as a hook.
…docs A second SIGINT/SIGTERM during stop_group's grace period raised inside the finally block and leaked a TERM-ignoring target. The finally now ignores SIGINT/SIGTERM/SIGHUP first, and Popen moved inside the try so a signal right after spawn still reaches cleanup. Added double-SIGTERM and double-SIGINT tests (failed before the fix). Docs: scan_logs pattern matches plural errors/warnings/exceptions; reword the verify-skill claims to say Claude is prompted, not hook-enforced.
… items Several checklists were written mostly from model knowledge, and nothing covered how agents themselves go wrong. They now draw on fetched sources (OWASP, AIP, Postgres, SRE book, AWS Builders' Library, Azure patterns, Qt/GTK/Electron/Tauri docs), on peer-reviewed findings about LLM coding failures, and on lessons from real agent mistakes. agent-process.md is selected for every task: ground claims in the installed version or a live run, prove a check can fail, keep the diff to the task, stop after two failed attempts, re-run gates on the merged tree. testing.md now forbids editing or loosening tests to make them pass and takes expected values from the spec; security adds registry lookups for new packages and an analyser loop; reliability adds per-call-site checks when moving to shared helpers. Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
The double-SIGINT procbudget test hung when the suite ran as a background job: a non-interactive shell starts such jobs with SIGINT ignored and Python keeps the inherited SIG_IGN. The test now restores SIGINT's default in the child; procbudget itself is unchanged. Review follow-ups: - devline Stage 0 excludes .devline/ through .git/info/exclude. - The parallel-agents group is marked orchestrator only, since implementers commit in their own worktrees; the router skips it for everyone else. - Duplicated rules now live in one checklist and the others point to it (spec/test conflicts and CI gates in testing, reuse and API versions in agent-process, timeouts in distributed, malicious-input tests in security); the tenancy group keeps only what extends the reviewer list. - The implementer's retry limit follows the two-attempt re-diagnose rule. - The local-reference grep is identical in both places and allows justified hits; test volume follows the plan's Test Depth. - Credits add MDN, React docs, Google AIP, PostgreSQL docs and Electron fuses; web.dev is classified as CC-BY-4.0 throughout. Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
Only Java had a language checklist; the others were a few lines inside backend.md, so agents had no guidance on language-specific traps. Each new checklist is built from the language's official docs and its standard tools: context cancellation, goroutine leaks and the race detector for Go; panics, async cancellation safety and unsafe review for Rust; strict compiler options, floating promises and runtime validation at trust boundaries for TypeScript; asyncio pitfalls, strict typing and dependency audits for Python. The router selects them from go.mod, Cargo.toml, tsconfig.json or package.json, and pyproject or requirements files; backend.md is now language-neutral and points to them. Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
Go 1.27 renamed the waitgroup analyzer, so filtering vet output by the old name finds nothing on current toolchains. Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Gives devline's implementer and reviewer engineering knowledge beyond unit tests: checklists for what to look out for per kind of system, and a way to set up real verification for a project. Until now a change was judged mostly by its tests, with nothing checking that the real app still runs, stays within its resource limits and logs no errors.
kb-engineeringholds fourteen checklists: testing, reliability, performance, security, web, backend, distributed systems, desktop; one per language for Java, Go, Rust, TypeScript and Python; and agent-process (how agents themselves go wrong, selected for every task). A short router picks the others by the project's files (go.mod,Cargo.toml,tsconfig.json,pyproject.toml, …) and the change. Each item ends with a concrete check that names a tool or command, and each file lists its sources and licences.The content comes from fetched sources (OWASP, Google AIP and SRE, AWS Builders' Library, Azure patterns, PostgreSQL, Qt, GTK, Electron and Tauri docs, web.dev), from peer-reviewed studies of LLM coding failures, and from lessons recorded after real agent mistakes. Examples: never edit or loosen a test to make it pass (ImpossibleBench: frontier models do it on 49-76% of tasks with contradictory tests); look up every new package on its registry before adding it (models invent 5-22% of package names); prove a check can fail before trusting its pass; when moving call sites onto a shared helper, compare old and new behaviour per site.
/devline:verify-setupinventories a project and offers a menu (gate command, sandboxed real-app run, fault injection, log scan, resource budgets). It writes only what you pick, as Claude Code's native.claude/skills/verify/SKILL.md, which Claude Code 2.1.286+ prompts Claude to run before commits.procbudget.pyis an opt-in helper for memory, growth, idle CPU and VRAM budgets. A budget it can't measure exits with its own code instead of passing, and it stops the measured process cleanly when interrupted.python3 -m unittest discover -s skills/kb-engineering/tests), the same command as locally.https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr