Skip to content

feat(kb-engineering): add engineering checklists and project verification setup - #6

Merged
Conava merged 11 commits into
mainfrom
feat/quality-verification
Nov 16, 2026
Merged

Conava merged 11 commits into
mainfrom
feat/quality-verification

Conversation

@Conava

@Conava Conava commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Description

Gives devline's implementer and reviewer engineering knowledge beyond unit tests: checklists for what to look out for per kind of system, and a way to set up real verification for a project. Until now a change was judged mostly by its tests, with nothing checking that the real app still runs, stays within its resource limits and logs no errors.

kb-engineering holds fourteen checklists: testing, reliability, performance, security, web, backend, distributed systems, desktop; one per language for Java, Go, Rust, TypeScript and Python; and agent-process (how agents themselves go wrong, selected for every task). A short router picks the others by the project's files (go.mod, Cargo.toml, tsconfig.json, pyproject.toml, …) and the change. Each item ends with a concrete check that names a tool or command, and each file lists its sources and licences.

The content comes from fetched sources (OWASP, Google AIP and SRE, AWS Builders' Library, Azure patterns, PostgreSQL, Qt, GTK, Electron and Tauri docs, web.dev), from peer-reviewed studies of LLM coding failures, and from lessons recorded after real agent mistakes. Examples: never edit or loosen a test to make it pass (ImpossibleBench: frontier models do it on 49-76% of tasks with contradictory tests); look up every new package on its registry before adding it (models invent 5-22% of package names); prove a check can fail before trusting its pass; when moving call sites onto a shared helper, compare old and new behaviour per site.

  • The implementer and reviewer read the selected checklists, always run typecheck and build, and for changes a user or another process can observe, run the project's verify recipe and report a Verification Evidence block. Missing evidence blocks the review.
  • /devline:verify-setup inventories a project and offers a menu (gate command, sandboxed real-app run, fault injection, log scan, resource budgets). It writes only what you pick, as Claude Code's native .claude/skills/verify/SKILL.md, which Claude Code 2.1.286+ prompts Claude to run before commits.
  • procbudget.py is an opt-in helper for memory, growth, idle CPU and VRAM budgets. A budget it can't measure exits with its own code instead of passing, and it stops the measured process cleanly when interrupted.
  • A dry run on a sample project found that agents named checklists without reading them; they now read them and list which.
  • CI runs the structure and helper tests (python3 -m unittest discover -s skills/kb-engineering/tests), the same command as locally.

https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr

…t helper

Agents verified their work with unit tests alone and had no guidance for
what to check per kind of system. Nine checklists (testing, reliability,
performance, security, web, backend, Java, distributed, desktop) list what to
look out for with a concrete check per item and sources per file;
procbudget.py measures a process tree's memory, growth, CPU and VRAM for
projects that opt in; references/setup.md describes the project verify
recipe.
…y-setup

Agents need only the checklists that apply to a change. The router picks
them from project files and the task, and lists the always-on rules and the
evidence block. /devline:verify-setup inventories a project, offers a menu
and writes only the picked parts as Claude Code's native verify skill.
…idence

Implementers and reviewers judged runtime changes from unit tests alone.
Both now load the kb-engineering checklists that apply, always run typecheck
and build, follow the voice comment rules, and run the project's verify
recipe for runtime-facing changes; the reviewer blocks on missing evidence.
CI runs the kb-engineering structure and helper tests with the same command
as locally.
…ndings

procbudget polled the process group while the leader was an unreaped zombie,
so every command run waited the full grace period before SIGKILL. It now
reaps the leader; tests drop from about a minute to 14 s. Also: security's
always-on group is routed on every runtime-facing change, credits list the
right licences, VRAM is labelled MiB, and verify-setup keeps the fast path
under about two minutes.
…ists

disable-model-invocation: true kept the skill out of the agents' skills
frontmatter, so implementer and reviewer never saw it. With injection on, a
dry run showed agents still picked checklists without reading them; both now
read the selected files and report which. The recipe guidance now stops the
app before scanning logs, runs it unbuffered, proves the scan with a planted
warning, never allowlists real defects, and works without CI.
…checklists

Deep review found that procbudget left its target running when it was
itself sent SIGTERM or SIGHUP, because the target lives in its own
session. main() now turns those signals into SystemExit so the finally
block stops the process group; a lifecycle test covers it.

The agents pointed at a comment-rules file that does not exist yet, so
the rule is now inline, and a structure test checks that every plugin
skill path named in agents/*.md exists. The reviewer must now Read every
checklist the tables select, cite the checklist rule per finding, and
list what it read. The docs now say Claude Code prompts Claude to run
the verify skill before commits rather than enforcing it as a hook.
…docs

A second SIGINT/SIGTERM during stop_group's grace period raised inside
the finally block and leaked a TERM-ignoring target. The finally now
ignores SIGINT/SIGTERM/SIGHUP first, and Popen moved inside the try so a
signal right after spawn still reaches cleanup. Added double-SIGTERM and
double-SIGINT tests (failed before the fix).

Docs: scan_logs pattern matches plural errors/warnings/exceptions; reword
the verify-skill claims to say Claude is prompted, not hook-enforced.
… items

Several checklists were written mostly from model knowledge, and nothing
covered how agents themselves go wrong. They now draw on fetched sources
(OWASP, AIP, Postgres, SRE book, AWS Builders' Library, Azure patterns,
Qt/GTK/Electron/Tauri docs), on peer-reviewed findings about LLM coding
failures, and on lessons from real agent mistakes.

agent-process.md is selected for every task: ground claims in the
installed version or a live run, prove a check can fail, keep the diff to
the task, stop after two failed attempts, re-run gates on the merged tree.
testing.md now forbids editing or loosening tests to make them pass and
takes expected values from the spec; security adds registry lookups for
new packages and an analyser loop; reliability adds per-call-site checks
when moving to shared helpers.

Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
The double-SIGINT procbudget test hung when the suite ran as a background
job: a non-interactive shell starts such jobs with SIGINT ignored and
Python keeps the inherited SIG_IGN. The test now restores SIGINT's default
in the child; procbudget itself is unchanged.

Review follow-ups:
- devline Stage 0 excludes .devline/ through .git/info/exclude.
- The parallel-agents group is marked orchestrator only, since implementers
  commit in their own worktrees; the router skips it for everyone else.
- Duplicated rules now live in one checklist and the others point to it
  (spec/test conflicts and CI gates in testing, reuse and API versions in
  agent-process, timeouts in distributed, malicious-input tests in
  security); the tenancy group keeps only what extends the reviewer list.
- The implementer's retry limit follows the two-attempt re-diagnose rule.
- The local-reference grep is identical in both places and allows
  justified hits; test volume follows the plan's Test Depth.
- Credits add MDN, React docs, Google AIP, PostgreSQL docs and Electron
  fuses; web.dev is classified as CC-BY-4.0 throughout.

Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
Only Java had a language checklist; the others were a few lines inside
backend.md, so agents had no guidance on language-specific traps.

Each new checklist is built from the language's official docs and its
standard tools: context cancellation, goroutine leaks and the race detector
for Go; panics, async cancellation safety and unsafe review for Rust;
strict compiler options, floating promises and runtime validation at trust
boundaries for TypeScript; asyncio pitfalls, strict typing and dependency
audits for Python. The router selects them from go.mod, Cargo.toml,
tsconfig.json or package.json, and pyproject or requirements files;
backend.md is now language-neutral and points to them.

Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
Go 1.27 renamed the waitgroup analyzer, so filtering vet output by the old
name finds nothing on current toolchains.

Claude-Session: https://claude.ai/code/session_01DQu2E2rLeQNoCjGF639ENr
@Conava
Conava merged commit 747d1d4 into main Nov 16, 2026
1 check passed
@Conava
Conava deleted the feat/quality-verification branch November 16, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant