Skip to content

feat(agent)!: align skill install with the Agent Skills standard - #271

Open
powxenv wants to merge 13 commits into
TestSprite:mainfrom
powxenv:chore/agentskills-standard
Open

feat(agent)!: align skill install with the Agent Skills standard#271
powxenv wants to merge 13 commits into
TestSprite:mainfrom
powxenv:chore/agentskills-standard

Conversation

@powxenv

@powxenv powxenv commented Jul 22, 2026

Copy link
Copy Markdown

What does this PR do?

Aligns agent install with the Agent Skills open standard. Instead of hand-wiring a few agents with bespoke formats, the CLI now writes one canonical skill per project at .agents/skills/<skill>/SKILL.md (the standard's shared directory), and each agent reads it directly (universal agents) or via a symlink back to it. The bundled registry grows from 8 targets to the full standard set (72 agent ids), so any skills-compatible agent works out of the box.

This broadens adoption (works with 72 agents instead of 8), simplifies maintenance (one uniform code path instead of per-agent wrap()/mode/compact-body machinery), and follows a vendor-neutral standard already adopted by the agents users run.

Related issue

Closes #270

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only
  • Build / CI / chore

Checklist

  • PR targets the main branch.
  • Commits follow Conventional Commits
    (feat(...), fix(...), docs(...), …).
  • npm run lint and npm run format:check pass.
  • npm run typecheck passes.
  • npm test passes and coverage stays at or above the 80% gate.
  • New behavior is covered by unit tests (mock-based; no network or
    credentials required).
  • No secrets, API keys, internal endpoints, or personal data are included.
  • User-facing changes are reflected in README.md / DOCUMENTATION.md where
    relevant.

Notes for reviewers

Breaking changes

This restructures the install model, so existing behavior changes for current users:

  • Per-agent bespoke formats removed. Codex no longer receives a managed section in AGENTS.md; Windsurf no longer receives a size-capped compact body. Both now read the canonical .agents/skills/ (codex directly, windsurf via symlink), like every other agent. The wrap() / own-file / managed-section machinery is gone in favor of one standard SKILL.md.
  • Install output schema changed. agent install results carry a new mode (canonical | symlink) field, and path now reflects the canonical-or-symlink model rather than each agent's bespoke location.
  • Existing installs go stale. Skills written in the old format will surface as stale/modified under agent status until refreshed — re-running agent install (with --force where needed) brings them in line.

What's preserved: the claude / kiro / copilot aliases still resolve, and agent install with no --target still defaults to claude-code — so the commands existing scripts use keep working; only the on-disk layout and output change.

Implementation notes

  • Model: canonical .agents/skills/<skill>/SKILL.md is the single source of truth. Universal agents (Codex, Cursor, Cline, Gemini CLI, Copilot, …) read it directly; every other agent gets a relative symlink from its own skills folder back to it — no drift between agents.
  • Path safety: both the canonical write and the symlink landing walk each path component with lstat and refuse to traverse a planted symlink (exit 5), so a malicious symlink can't redirect a write outside --dir.
  • Tests: the install/status/conflict/force/dry-run/path-safety paths are covered exhaustively by unit tests using an in-memory filesystem (deterministic, cross-platform, no real-symlink fragility); coverage on agent.ts is ~92% lines, with the remainder being defensive error handling and real-fs wrappers exercised by the e2e suite.

Summary by CodeRabbit

  • New Features
    • Added supported-agent aliases and universal or symlinked skill installations, with copy fallback when symlinks are unavailable.
    • agent install now supports canonical skill storage, multi-target installs, dry-run previews, conflict handling, backups, and legacy migration.
    • Improved list and status reporting with installation health details.
    • Setup now defaults to claude-code, with clearer skill metadata.
    • Doctor reports organization membership, key binding, and workspace scope.
  • Documentation
    • Updated onboarding, contribution, and supported-agent guidance.
  • Bug Fixes
    • Improved skill-installation detection using installed landing paths.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 14494e3e-cde7-4193-a5a8-809ffab0f97e

📥 Commits

Reviewing files that changed from the base of the PR and between 90b7034 and 1afd744.

📒 Files selected for processing (1)
  • DOCUMENTATION.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • DOCUMENTATION.md

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


Walkthrough

The PR adopts canonical Agent Skills files under .agents/skills, classifies targets as universal or symlinked, adds aliases and migration handling, updates setup defaults to claude-code, and revises detection, tests, and documentation.

Changes

Canonical skills and target registry

Layer / File(s) Summary
Canonical skills and target registry
src/lib/agent-targets.ts, src/lib/agent-targets.test.ts, skills/*
Defines skill metadata, target classifications, aliases, canonical paths, frontmatter parsing, rendering, markers, and legacy layout metadata.

Canonical installation and status flow

Layer / File(s) Summary
Canonical installation and status flow
src/commands/agent.ts, src/commands/agent.test.ts, test/e2e/agent-install.e2e.test.ts
Writes canonical skill files, creates symlink or copy-fallback landings, handles force backups and migration, and reports install, list, and status results.

Setup defaults and command integration

Layer / File(s) Summary
Setup defaults and command integration
src/commands/init.ts, src/commands/init.test.ts, src/index.ts, test/e2e/setup.e2e.test.ts
Uses claude-code as the setup default and updates setup option forwarding and canonical-path expectations.

Skill detection and nudge integration

Layer / File(s) Summary
Skill detection and nudge integration
src/lib/skill-nudge.ts, src/lib/skill-nudge.test.ts, test/e2e/skill-nudge.e2e.test.ts
Detects verification skills through supported landing-path existence and removes managed-section file reading from the detection path.

Documentation and contribution policy

Layer / File(s) Summary
Documentation and contribution policy
README.md, DOCUMENTATION.md, CONTRIBUTING.md
Documents target IDs, installation modes, setup usage, status behavior, skill assets, and supported-agent maintenance requirements.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: 🔵 Low · up to 1afd7

The PR changes the skill installation layout and broadens supported agents, while two documentation details remain inaccurate: the status documentation omits the absent state, and examples use an alias instead of the canonical target name. The change is mergeable with explicit owner follow-up to correct these bounded usability issues.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant AgentInstall
  participant CanonicalSkills
  participant AgentLanding
  User->>AgentInstall: agent install --target id
  AgentInstall->>CanonicalSkills: write or validate canonical SKILL.md
  AgentInstall->>AgentLanding: create symlink or copy fallback
  AgentLanding-->>User: return install action and path
Loading

Possibly related PRs

Suggested reviewers: zeshi-du

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The implementation covers the main standard-install objectives but does not address the listed lexical containment and universal status reporting changes. Add containment checks before every filesystem mutation, report one canonical status row per installed skill, and update related tests and documentation.
Out of Scope Changes check ⚠️ Warning The organization and workspace reporting changes in src/commands/doctor.ts are unrelated to the linked Agent Skills installation objective. Move the doctor organization-reporting changes to a separate pull request or link an issue that requires them.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the breaking Agent Skills standard alignment, which is the primary change.
Docstring Coverage ✅ Passed Docstring coverage is 80.77% which is sufficient. The required threshold is 80.00%.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown

✅ This PR is linked to an issue assigned to @powxenv — thanks! The needs-issue label has been removed.

@github-actions github-actions Bot added the needs-issue PR not linked to an issue yet — please open one first and claim it (see CONTRIBUTING) label Jul 22, 2026
@github-actions github-actions Bot removed the needs-issue PR not linked to an issue yet — please open one first and claim it (see CONTRIBUTING) label Jul 24, 2026
@powxenv
powxenv marked this pull request as ready for review July 24, 2026 12:09

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
src/lib/agent-targets.test.ts (1)

131-165: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Coverage doesn't include adversarial alias tokens.

Tests confirm resolveTarget rejects a plain unknown token (line 156-158), but don't cover prototype-chain keys (constructor, __proto__, toString) that expose the bug flagged in agent-targets.ts (resolveTarget, lines 168-172). Once that's fixed, add a case here.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/agent-targets.test.ts` around lines 131 - 165, Add adversarial-token
coverage to the resolveTarget tests, specifically asserting that constructor,
__proto__, and toString are rejected as unknown tokens. Extend the existing
“resolveTarget rejects an unknown token” test in the TARGET_ALIASES +
resolveTarget suite, preserving the current null result expectation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@DOCUMENTATION.md`:
- Around line 111-118: Update the agent install examples in the documentation to
pass each agent name through the documented --target option instead of as a
positional argument. Preserve the existing install, list, and status examples
unchanged in purpose.

In `@src/commands/agent.ts`:
- Around line 714-728: Update runStatus around the TARGETS and DEFAULT_SKILLS
iteration to emit universal ok rows only for targets explicitly requested by the
user or targets with an installed actionable artifact. Preserve rows for
actionable states, while suppressing ok-only universal results for uninstalled,
unspecified targets.

In `@src/lib/agent-targets.ts`:
- Around line 168-172: Update resolveTarget’s alias lookup to verify raw is an
own key of TARGET_ALIASES before reading its value, returning null for inherited
keys such as constructor, __proto__, and toString. Preserve the existing TARGETS
lookup and canonical alias behavior, and add regression coverage in
agent-targets.test.ts for these tokens to ensure unknown-target validation is
reached instead of a crash.

---

Nitpick comments:
In `@src/lib/agent-targets.test.ts`:
- Around line 131-165: Add adversarial-token coverage to the resolveTarget
tests, specifically asserting that constructor, __proto__, and toString are
rejected as unknown tokens. Extend the existing “resolveTarget rejects an
unknown token” test in the TARGET_ALIASES + resolveTarget suite, preserving the
current null result expectation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 50a2a944-c182-480c-bf90-a91e921b1d6f

📥 Commits

Reviewing files that changed from the base of the PR and between fe07bc9 and 68f01d5.

⛔ Files ignored due to path filters (2)
  • package-lock.json is excluded by !**/package-lock.json, !package-lock.json
  • test/__snapshots__/help.snapshot.test.ts.snap is excluded by !**/*.snap, !**/*.snap
📒 Files selected for processing (18)
  • CONTRIBUTING.md
  • DOCUMENTATION.md
  • README.md
  • skills/testsprite-onboard.skill.md
  • skills/testsprite-verify.codex.md
  • skills/testsprite-verify.skill.md
  • src/commands/agent.test.ts
  • src/commands/agent.ts
  • src/commands/doctor.ts
  • src/commands/init.test.ts
  • src/commands/init.ts
  • src/lib/agent-targets.test.ts
  • src/lib/agent-targets.ts
  • src/lib/skill-nudge.test.ts
  • src/lib/skill-nudge.ts
  • test/e2e/agent-install.e2e.test.ts
  • test/e2e/setup.e2e.test.ts
  • test/e2e/skill-nudge.e2e.test.ts
💤 Files with no reviewable changes (2)
  • skills/testsprite-verify.codex.md
  • src/commands/doctor.ts

Comment thread DOCUMENTATION.md
Comment thread src/commands/agent.ts
Comment thread src/lib/agent-targets.ts
@zeshi-du

Copy link
Copy Markdown
Contributor

Status update, so this stops being ambiguous — and an apology first: gate (4) was "CI actually green," and CI had never been allowed to run. Your PR was sitting on an unapproved fork workflow this whole time, which is a maintainer failure, not yours. I've approved it.

Now that it has run, here is where the four gates actually stand:

  1. Per-agent verification citations — closed. Your follow-up on Windsurf → Devin Desktop and Gemini CLI → Antigravity did what I asked. Thank you for citing sources rather than asserting.
  2. Migration story for existing installs — still open. --force isn't a migration; someone with the current layout installed needs a path that doesn't silently orphan their files.
  3. CONTRIBUTING.md rewritten for the universal/symlink model — still open.
  4. CI green — now measured, and it is not. Unit Tests (Windows) fails, Unit Tests (Node 22) didn't complete, and the branch has gone CONFLICTING against main (two releases have shipped since you opened it — v0.5.0 on 08-06 and v0.6.0 on 08-12). A -5325-line change to the install layer needs a green Windows leg specifically, since symlink behaviour is where Windows diverges most.

What I'm committing to: once this PR is rebased and shows a green run, I'll give a merge-or-defer decision within one week. No more open-ended pause. If you'd rather not carry a rebase of this size, say so and I'll take the branch over rather than let it rot.

One more thing you should know, because it affects the stakes: this PR is currently freezing #236 (the bespoke Gemini target), which is green and waiting on this outcome. I've told that author the same timeline.

@zeshi-du

Copy link
Copy Markdown
Contributor

Two corrections to my comment above before the substance: Unit Tests (Node 22) did in fact complete, and it passed — it finished nine minutes before I wrote that, and I read it stale. And Dependency Review / ESLint Security have never run on this branch at all; when they do appear on your next push, treat both as noise (they fail on every PR here for infrastructural reasons and are fixed in our source repo, shipping on the next release).

So the only real gate is the Windows leg, and it's mechanical.

Root cause: init.test.tsrunInit — default claude-code target installs the canonical skill files > writes both testsprite-verify and testsprite-onboard into .agents/skills (currently lines 461-462) —

expect(symlinkCalls.some(s => s.link.endsWith('.claude/skills/testsprite-verify'))).toBe(true);
expect(symlinkCalls.some(s => s.link.endsWith('.claude/skills/testsprite-onboard'))).toBe(true);

s.link is linkAbs, which traces back to path.resolve(root, landingRel) in agent.ts. path.resolve produces backslash-separated paths on Windows, so a forward-slash literal can never satisfy endsWith there — that's the whole failure.

You already have the fix for this pattern in the same PR: skill-nudge.test.ts:18 defines const toPosix = (p: string) => p.replaceAll('\\', '/') and uses it for exactly this kind of path assertion, in five places. init.test.ts just doesn't have it. Normalizing those two lines the same way should take the Windows leg green — everything else on that run already passed.

Note that the branch is also CONFLICTING against main and will need a rebase regardless of the test fix.

One process note: the fork-workflow approval gate re-arms on every push from a first-time contributor, so CI goes dark again after you push. I'll re-approve promptly this time rather than leaving you waiting on it.

powxenv and others added 9 commits August 19, 2026 08:48
Replace the per-agent bespoke install formats with the Agent Skills open
standard: one canonical .agents/skills/<skill>/SKILL.md per project, read
directly by universal agents (Codex, Cursor, Cline, Gemini CLI, Copilot,
…) and via a relative symlink by every other agent. The bundled registry
grows from 8 hand-wired targets to the full standard set (72 agent ids).

BREAKING CHANGE: agent install no longer writes per-agent bespoke
formats. Codex no longer receives a managed section in AGENTS.md and
Windsurf no longer receives a size-capped compact body — both now read
the canonical .agents/skills/. The install result schema gains a `mode`
(canonical | symlink) field and `path` reflects the new model. Skills
written in the old format report stale/modified under `agent status`
until refreshed (re-run `agent install`, add --force where needed). The
legacy aliases (claude, kiro, copilot) and the claude-code default are
preserved, so existing commands keep working.
…ehavior

- Revise CONTRIBUTING.md to describe universal vs symlinked agent targets and requirements for adding new targets
- Update DOCUMENTATION.md to clarify agent install command usage and differentiate universal and symlinked agents
- Expand README.md with a detailed supported agents table, including canonical ids, aliases, and skills folder locations
- Update src/lib/agent-targets.ts with added agents, corrections to skillsDir and universal flags, and alias mappings
- Modify unit tests and e2e tests to reflect renamed and updated agent targets, replacing deprecated aliases like gemini-cli with antigravity-cli and windsurf with devin-desktop
- Ensure single source of truth for skills at `.agents/skills`, symlink strategy for non-universal agents, and consistency across docs and code
- Detect legacy own-file skill artifacts and AGENTS.md managed sections during install
- Back up legacy files and directories as *.bak before migrating or removing them
- Convert legacy skill folders (e.g., claude) to symlink landings pointing to canonical skills
- Remove managed section from AGENTS.md and back it up during migration
- Implement dry-run mode that plans migration without file changes
- Refuse plain installs when legacy artifacts block new symlink target paths
- Surface a status advisory listing legacy targets and how to migrate them
- Introduce findManagedSectionBounds utility to locate legacy sentinel blocks
- Update install action aggregation to consider migrated steps as updated
- Add comprehensive tests covering migration scenarios, error cases, and idempotency
- Extend filesystem mocks with readdir support for migration operations
- Update CLI help text for --force to mention legacy artifact migration and backups
- Update resolveTarget to use hasOwnProperty to exclude inherited prototype keys
- Add test to verify resolveTarget returns null for inherited keys like constructor, __proto__, toString, hasOwnProperty
- Ensure inherited prototype-chain keys are recognized as unknown targets and rejected explicitly
- Removed multiline conditional and combined it into a single line
- Maintained logic to check TARGET_ALIASES property existence before return
- Improved code readability and compactness without changing behavior
- Change `testsprite agent install` commands to use `--target` flag for specifying agents
- Update DOCUMENTATION.md examples to reflect new command syntax
- Modify README.md to replace old command usage with the new format
- Clarify onboarding instructions for different agents with consistent command style
…ked agents

- Update DOCUMENTATION.md to explain that 'agent status' checks each installed skill against CLI version
- Clarify that universal agents share one canonical skill file serving all agents
- Explain symlinked agents appear only when their own landing exists
- Update CLI help text to reflect these details for 'agent status' command
- Adjust snapshot test output to match the revised command description and behavior
path.resolve yields backslashes on Windows, so the forward-slash
endsWith assertions on symlink landing paths could never pass there.
Apply the same toPosix normalization pattern used in skill-nudge.test.ts.
@powxenv
powxenv force-pushed the chore/agentskills-standard branch from f3fb429 to 26cb673 Compare August 19, 2026 02:07

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
test/e2e/skill-nudge.e2e.test.ts (1)

146-153: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Make the plan-template regression test offline and observable.

This test starts the CLI with a real http://127.0.0.1:9 endpoint. It cannot prove that the update check and command path made zero network calls. Use injected fetch and sleep dependencies, then assert that neither is called for test create --plan-template.

As per path instructions, “Tests must be deterministic and offline: No real network and no real timers — inject fetch/sleep via the test deps.”

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@test/e2e/skill-nudge.e2e.test.ts` around lines 146 - 153, Update the
plan-template regression test around the CLI invocation to inject fetch and
sleep dependencies, using mocks that remain offline and deterministic. Run test
create --plan-template with those dependencies and assert neither the injected
fetch nor sleep function is called, while preserving the existing success,
warning-free stderr, and planSteps assertions.

Source: Path instructions

src/commands/init.ts (1)

638-650: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add direct auth configure coverage.

The existing test named auth configure --dry-run invokes setup, and runConfigure tests bypass the alias. Add direct alias tests for --api-key, --agent, --no-agent, --force, --dir, --yes, and --skip-if-configured. Add a missing-key test that asserts exit code 5.

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@src/commands/init.ts` around lines 638 - 650, Add direct tests for the auth
configure alias, invoking it rather than setup or runConfigure, covering
--api-key, --agent, --no-agent, --force, --dir, --yes, and --skip-if-configured.
Also add a missing-API-key case and assert that it exits with code 5, while
preserving the existing dry-run coverage.

Source: Path instructions

🧹 Nitpick comments (1)
test/e2e/agent-install.e2e.test.ts (1)

496-527: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Two full-registry loops spawn the binary once per target.

Each loop runs runCli for all 72 targets, so the suite adds about 144 process spawns plus 144 temp directories. Consider covering the full registry once and keeping the positional-form check on a representative subset.

Also applies to: 751-761

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@test/e2e/agent-install.e2e.test.ts` around lines 496 - 527, Reduce the e2e
process and temporary-directory overhead in the full-registry smoke tests by
consolidating coverage so the registry-wide installation runs only once per
target, while retaining the positional-form validation in a representative
subset rather than every target. Update the two loops containing the
full-registry and positional checks, preserving each test’s existing assertions
and target coverage.
🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

Inline comments:
In `@DOCUMENTATION.md`:
- Line 132: Update the agent status documentation to define the absent state
alongside ok, stale, modified, and unmarked, explaining how missing canonical
skill files and landings are reported. Ensure the documented exit-1 rule
includes absent artifacts as needing attention while preserving the existing
status behavior.
- Around line 130-134: Update the legacy-install migration documentation around
the --force behavior to state that it backs up and removes legacy artifacts for
the requested targets before installing canonical files and symlinks. Document
the resulting .bak locations and explain how the Codex AGENTS.md section is
handled.

In `@README.md`:
- Line 147: Update the README Agent row to make its listed coding-agent IDs
explicitly non-exhaustive by adding an “including” qualifier or linking directly
to the complete supported-agents registry, while preserving the existing command
descriptions.

In `@src/commands/agent.ts`:
- Around line 80-89: Update symlink to resolve Windows junction targets relative
to path.dirname(linkPath), rather than process.cwd(), while preserving the
existing relative target behavior on non-Windows platforms.

In `@test/e2e/agent-install.e2e.test.ts`:
- Around line 517-523: Update the landing-path check around lstatSync so
assertion failures from expect are not caught; perform lstatSync separately and
only handle errors from that call, preserving the existing
symbolic-link-or-directory validation and fallback behavior.

---

Outside diff comments:
In `@src/commands/init.ts`:
- Around line 638-650: Add direct tests for the auth configure alias, invoking
it rather than setup or runConfigure, covering --api-key, --agent, --no-agent,
--force, --dir, --yes, and --skip-if-configured. Also add a missing-API-key case
and assert that it exits with code 5, while preserving the existing dry-run
coverage.

In `@test/e2e/skill-nudge.e2e.test.ts`:
- Around line 146-153: Update the plan-template regression test around the CLI
invocation to inject fetch and sleep dependencies, using mocks that remain
offline and deterministic. Run test create --plan-template with those
dependencies and assert neither the injected fetch nor sleep function is called,
while preserving the existing success, warning-free stderr, and planSteps
assertions.

---

Nitpick comments:
In `@test/e2e/agent-install.e2e.test.ts`:
- Around line 496-527: Reduce the e2e process and temporary-directory overhead
in the full-registry smoke tests by consolidating coverage so the registry-wide
installation runs only once per target, while retaining the positional-form
validation in a representative subset rather than every target. Update the two
loops containing the full-registry and positional checks, preserving each test’s
existing assertions and target coverage.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: bcefc0cc-0038-4e86-981b-7cbe10539c76

📥 Commits

Reviewing files that changed from the base of the PR and between f3fb429 and 26cb673.

⛔ Files ignored due to path filters (1)
  • test/__snapshots__/help.snapshot.test.ts.snap is excluded by !**/*.snap, !**/*.snap
📒 Files selected for processing (13)
  • DOCUMENTATION.md
  • README.md
  • src/commands/agent.test.ts
  • src/commands/agent.ts
  • src/commands/doctor.ts
  • src/commands/init.test.ts
  • src/commands/init.ts
  • src/index.ts
  • src/lib/skill-nudge.test.ts
  • src/lib/skill-nudge.ts
  • test/e2e/agent-install.e2e.test.ts
  • test/e2e/setup.e2e.test.ts
  • test/e2e/skill-nudge.e2e.test.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread DOCUMENTATION.md
Comment thread DOCUMENTATION.md Outdated
`.agents/skills/` is the **single source of truth**: it is written on every install, even when you target a symlinked agent — `agent install --target claude-code` lands the skill in `.agents/skills/` (covering every universal agent) **and** links it into `.claude/skills/`. Because each symlink points _into_ `.agents/skills/`, you only ever edit a skill there and every symlinked agent reflects the change automatically (on systems where symlinks are unavailable — e.g. Windows without Developer Mode — a plain copy is written instead, which won't auto-update).

Re-running with `--force` on **own-file targets** (claude, cursor, cline, antigravity, kiro, windsurf, copilot) backs up the existing file to `<path>.bak` first.
`agent status` checks each installed skill against this CLI version and reports `ok`, `stale`, `modified`, or `unmarked`. Symlinked agents appear only when their own landing exists; universal agents share one canonical skill file (`.agents/skills/`), so installing for any one of them serves every universal agent and they all report together. Agents with nothing installed are omitted. It exits `1` when anything needs attention, so `testsprite agent status && …` can gate a CI step; `--dir <path>` inspects a different project root.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Document the absent status.

agent status lists only ok, stale, modified, and unmarked, then says agents with no installation are omitted. The required contract also includes absent. Document how absent canonical files and landings are reported, and keep the exit-1 attention rule aligned with that state.

As per path instructions, the agent-install documentation must define ok, stale, modified, unmarked, and absent states, with exit 1 for attention-needed artifacts.

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@DOCUMENTATION.md` at line 132, Update the agent status documentation to
define the absent state alongside ok, stale, modified, and unmarked, explaining
how missing canonical skill files and landings are reported. Ensure the
documented exit-1 rule includes absent artifacts as needing attention while
preserving the existing status behavior.

Source: Path instructions

Comment thread README.md Outdated
Comment thread src/commands/agent.ts
Comment thread test/e2e/agent-install.e2e.test.ts Outdated
@powxenv

powxenv commented Aug 19, 2026

Copy link
Copy Markdown
Author

Hi, thanks for the detailed root-cause on the Windows leg, that made it a mechanical fix indeed.

I've rebased the branch onto the latest main (c255d1d), all conflicts resolved, and addressed the remaining gates:

  • Gate 2 (migration): install --force now detects legacy own-file artifacts and legacy AGENTS.md managed sections, and migrates them. Legacy skill folders are converted to symlink landings, managed sections are removed, and everything touched is backed up as *.bak first. It's scoped (--target only migrates that agent), idempotent (a second run is a no-op), and has a --dry-run mode that plans the migration without writing anything. A plain install that would be blocked by legacy artifacts exits with a migration hint, and agent status surfaces an advisory listing which targets are legacy and the exact command to migrate them.
  • Gate 3 (CONTRIBUTING): rewritten for the universal/symlink model, covering how targets are classified, what a new agent target needs, and the update checklist.
  • Gate 4 (CI): applied the toPosix normalization you pointed out in init.test.ts. Locally everything is green: typecheck, ESLint, 2259 unit tests, 122 e2e tests.

Pushing will re-arm the fork-workflow approval per your process note, so CI will go dark briefly. Whenever you re-approve, the Windows leg should now pass.

If any of this doesn't match what you had in mind, especially the migration UX (say, you'd prefer an explicit agent migrate command instead of --force, or different backup semantics), happy to implement a more specific approach.

…ctory

The symlink target passed by linkOrCopy is relative to the link's own
directory, but path.resolve(target) resolves it against process.cwd().
With --dir pointing outside the cwd, the junction landed on the wrong
absolute path. Resolve against path.dirname(linkPath) instead.

Found by CodeRabbit (PR TestSprite#271, Aug 19 review).
The try/catch around lstatSync also caught the expect() throw, so the
symlink-or-directory assertion could never fail the test. Run lstatSync
outside the assertion; a missing landing now fails loudly.

Found by CodeRabbit (PR TestSprite#271, Aug 19 review).
…ent list

- agent status: define the fifth state 'absent' (artifact file missing)
  and that absent artifacts are omitted rather than reported
- agent install --force: document the scoped legacy migration — .bak
  backups for folders and AGENTS.md, exit-6 refusal with the exact
  migrate command, and --dry-run planning
- README: mark the agent-row id list as non-exhaustive

Found by CodeRabbit (PR TestSprite#271, Aug 19 review).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
DOCUMENTATION.md (1)

92-92: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use claude-code in both install examples.

claude remains a supported alias, but the documentation examples should use the canonical target ID.

  • DOCUMENTATION.md#L92-L92: replace --target claude with --target claude-code.
  • README.md#L160-L160: replace --target claude with --target claude-code.

As per path instructions, examples must use canonical target IDs.

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@DOCUMENTATION.md` at line 92, Update the install examples to use the
canonical claude-code target: replace --target claude with --target claude-code
in DOCUMENTATION.md lines 92-92 and README.md lines 160-160.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

Inline comments:
In `@DOCUMENTATION.md`:
- Line 136: Update the documentation sentence covering --force legacy migration
to include both legacy skill folders and legacy skill files, with each backed up
using the corresponding .bak path. Also document that installation falls back to
copying the canonical skill when symlinks are unavailable, while preserving the
existing migration, dry-run, and backup guidance.

---

Outside diff comments:
In `@DOCUMENTATION.md`:
- Line 92: Update the install examples to use the canonical claude-code target:
replace --target claude with --target claude-code in DOCUMENTATION.md lines
92-92 and README.md lines 160-160.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 235d3f18-5388-4b93-a2ea-4979cfbc76ad

📥 Commits

Reviewing files that changed from the base of the PR and between 26cb673 and 90b7034.

📒 Files selected for processing (4)
  • DOCUMENTATION.md
  • README.md
  • src/commands/agent.ts
  • test/e2e/agent-install.e2e.test.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

Comment thread DOCUMENTATION.md Outdated
…graph

The --force migration handles two artifact shapes (skill files backed up
to <path>.bak, and folders to <folder>.bak/) and may land a plain copy
where symlinks are unavailable; the sentence now states both.

Found by CodeRabbit (PR TestSprite#271, follow-up review).
@zeshi-du

Copy link
Copy Markdown
Contributor

CI ran today on 1afd744 and it's green where it counts: Unit Tests (Windows), Unit Tests (Node 20/22), Coverage (>= 80%), Typecheck, Lint & Format, Build, Local E2E Tests, Secret scan all pass. Dependency Review and ESLint Security (changed files) are red — that's our org-level dependency graph being off and a pre-existing lint backlog fixed in our source repo, not this PR; ignore both. Your push re-armed the fork-approval gate again, which is why the "green run" I owed you couldn't exist until I clicked approve just now — that's on our infrastructure, not you. The clock on merge-or-defer starts from today's result: I'll give you that decision by 2026-08-26.

Where the gates actually stand:

Gate 1 (citations) — still closed, per my 08-13 comment.

Gate 2 (migration) — closed. Verified against agent.ts at 1afd744: migrateLegacyArtifacts detects both legacy own-file artifacts and legacy AGENTS.md managed sections, backs each up as *.bak before touching it, and is scoped to --target/--skill (only the requested targets/skills are migrated). --dry-run reports the plan without writing (writeBackup is skipped, only the would-be rows print). One correction to your own summary comment: a plain (non---force) install blocked by an existing file exits 6 with a generic re-run with --force to overwrite (a .bak is kept) message — it doesn't name the migrate command. The specific Run `testsprite agent install --force --target <ids>` to back them up and migrate them (*.bak kept). line only appears in agent status's stderr advisory when it detects legacy targets outstanding. Not a gate issue, just don't rely on the install-side message to point people at the fix — agent status is what does that. One thing I verified via CI rather than by tracing the code myself: idempotency (a second --force run reporting no migration) — that's asserted directly in runInstall --force — idempotency... in both the unit and e2e suites, and it's part of what's green above.

Gate 3 (CONTRIBUTING) — closed. Read the diff: the per-target policy is now "cite the agent's current skills docs, register it in agent-targets.ts, update the unit-test target list, add the README row" instead of the old policy of documenting one bespoke editor rules/skill-file mechanism per proposal. That's the right replacement for a 70+-target registry.

Gate 4 (CI) — green, as above.

One code item, separate from the four gates. Pre-PR main had two lexical containment checks of the shape if (abs !== root && !abs.startsWith(root + path.sep)) throw ...refusing to write outside --dir.... Both are gone in this diff — they lived in the old codex-managed-section and own-file code that this PR replaces wholesale, and nothing took their place at the write sites. What's left is inspectTargetPath (pre-existing) and the new inspectAncestors, both of which walk path segments via lstat and reject symlinks — genuinely stronger against a planted symlink than a string compare ever was. But that's a different threat model: path.join/path.resolve honor a literal .. segment silently, and neither function rejects one. canonicalSkillDir, targetLandingDir, and legacyOwnFilePath all interpolate skill: string (not a closed type) straight into a path with no containment check of their own. Today, containment holds only because runInstall's eager target/skill registry validation (steps 1-2, before anything touches disk) runs before any of those functions are ever called with a value that reaches the filesystem — I traced that ordering and it's solid, so this isn't reachable through the shipped CLI right now. But that's caller discipline, not a backstop: one new call site or one validation refactor removes the only thing standing between an untrusted string and a write outside --dir, where before there was an explicit guard at the write site itself regardless of how the string got there. Cheap to close either way — restore the containment check in ensureCanonical/ensureAgentLink at the point of write, or push it into the path-builder functions so every caller gets it for free.

CodeRabbit: seven of its nine actionable comments on this PR carry the auto "✅ Addressed" annotation; two don't. One is a false negative — the DOCUMENTATION.md absent-status gap it flagged is already covered at 1afd744 (line 132 now documents the absent state and that universal agents "report together," landed in commit 90b7034), it just never got tagged. The other is live: runStatus loops over every target in TARGETS and, for universal targets, classifies the same canonical file every time — after a single --target=codex install it prints an ok row for every other universal target too (cursor, cline, copilot, …), none of which were installed. I won't lean on "contract violation" here — DOCUMENTATION.md:132 now explicitly documents this as intended, so the docs and the code agree with each other. But I still want agent status to report only targets that were installed or explicitly requested, not every universal target by inference: please suppress the ok-only rows for universal targets nobody asked about, and adjust that DOCUMENTATION.md sentence to match once you do. On review coverage: CodeRabbit's last actionable pass is on 90b7034; a follow-up ran through 1afd744 (the docs-only final commit), found nothing new, and correctly skipped it as similar to what it'd already reviewed — so the diff has been seen, there's just nothing further flagged. One process note: CodeRabbit's auto-review is paused on this branch (too many commits in a row) — a push won't get automatic coverage back on its own; post @coderabbitai resume (or @coderabbitai review for a one-off pass on the next commit) to restore it.

On the line count: -5414/+3095 isn't a silent functionality cut. The two largest deletions are test consolidation — agent.test.ts (-2468/+998) and agent-targets.test.ts (-811/+229) rewritten against the new model, not dropped — plus one genuinely obsolete asset, skills/testsprite-verify.codex.md (-127/+0), retired because codex now reads the canonical SKILL.md directly instead of a separate per-target variant. What I did not independently verify is scenario-for-scenario parity between the old and new test suites — that's what the passing coverage gate is standing in for, so if anything got quietly dropped, watch that number on the next run, not this comment.

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

ast-grep outline src/commands/agent.ts --items all --type function --match 'runStatus|ensureCanonical|ensureAgentLink|inspectAncestors|inspectTargetPath|migrateLegacyArtifacts'
ast-grep outline src/lib/agent-targets.ts --items all --type function --match 'canonicalSkillDir|canonicalSkillFile|targetLandingDir|legacyOwnFilePath|pathFor'
ast-grep outline src/commands/agent.test.ts --items all --type function --match 'runStatus'
rg -n -C 4 'universal|report together|agent status|absent' DOCUMENTATION.md
rg -n -C 3 'runStatus|ensureCanonical|ensureAgentLink|inspectAncestors|canonicalSkillDir|targetLandingDir|legacyOwnFilePath' \
  src/commands/agent.ts src/lib/agent-targets.ts src/commands/agent.test.ts

Length of output: 29044


🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '100,180p' src/commands/agent.ts
sed -n '230,420p' src/commands/agent.ts
sed -n '470,630p' src/commands/agent.ts
sed -n '680,835p' src/commands/agent.ts
sed -n '700,875p' src/commands/agent.test.ts
sed -n '1,220p' src/commands/agent.ts
sed -n '150,215p' src/lib/agent-targets.ts
sed -n '800,880p' src/lib/agent-targets.ts
rg -n -C 3 'writeFile|mkdir|symlink|unlink|rm|copyFile' src/commands/agent.ts

Length of output: 55491


@zeshi-du, the inspection confirms both findings.

ensureCanonical and ensureAgentLink resolve paths without a lexical containment check. The symlink checks do not reject a .. path segment.

The requested universal-target behavior needs one clarification. The shared canonical SKILL.md has no record of the universal target that started an earlier install. agent status cannot truthfully identify codex versus cursor after that install without new persistent metadata.

The smallest correct change is to emit one canonical status row for each installed shared skill. It will not emit inferred ok rows for every universal target. Symlinked targets will continue to report only when their own landing exists. This keeps agent status as artifact health reporting and avoids adding an installation manifest.

  • Add a lexical containment backstop for every agent-install filesystem mutation.
  • Replace inferred per-universal-target status rows with canonical artifact rows.
  • Update status tests and DOCUMENTATION.md.
  • Resume CodeRabbit automatic reviews for the next push.

✏️ Learnings added
Learnt from: zeshi-du
URL: https://github.com/TestSprite/testsprite-cli/pull/271

Timestamp: 2026-08-19T21:33:09.266Z
Learning: In the TestSprite CLI repository, `Dependency Review` and `ESLint Security (changed files)` can fail on pull requests because of known organization-level infrastructure and pre-existing lint backlog issues. For PR `#271`, do not treat those two checks as regressions in the change under review; evaluate the relevant CI checks instead.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


Failed to handle agent chat message. Please try again.

✅ Action performed

Reviews resumed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Hackathon] Support Agent Skills standard for install/setup

2 participants