Skip to content

EBP-2889: Address session-analysis feedback on solace-application-development - #4

Draft
aelsammak wants to merge 3 commits into
devfrom
EBP-2889
Draft

EBP-2889: Address session-analysis feedback on solace-application-development#4
aelsammak wants to merge 3 commits into
devfrom
EBP-2889

Conversation

@aelsammak

@aelsammak aelsammak commented Aug 19, 2026

Copy link
Copy Markdown

What is the purpose of this change?

An analysis of 10 real developer sessions that used the solace-application-development skill found 10 recurring violations of the skill's own instructions (the door question skipped, the disclaimer header dropped, verify.sh never run, the checklist never written, live brokers hit without warning, unfetched citations, a dropped logging pin, a stale version source). The root causes were structural: mandatory rules buried in one lazy-loaded file too large for a single read, a verification contract that only fits the two-CLI shape, rules with no execution slot, and self-contradictions. This change restructures the skill so the rules sit where every session reads them and become mechanically checkable.

How is this accomplished?

  1. Promoted the buried mandates to SKILL.md invariants (4-6): the AI-assisted disclaimer header on every generated source file, the always-emitted verification artifacts, and the live-broker heads-up; extended Invariant 2 to ban substitute grounding channels and Invariant 3 to name the authoritative lookup and ban the solrsearch index

  2. Surfaced the Implement Step 0 door-question gate and the environment-discovery rule (a running broker or existing config.json never answers a question) at the jcsmp.md front door, in design-mode's handoff, and as explicit Step 0 counters

  3. Split implement-mode.md into a core file (~12k tokens, fits one Read; the old file exceeded the 25k-token Read cap so its later steps were unreachable) plus three per-leaf wiring files; stated the generation file-output contract up front and moved the solace-verification-checklist.md write into Step 4 generation output

  4. Made verification shape-agnostic: verify.sh gains an app stage driven by a generated verify-hooks.sh (START_CMD / TRIGGER_CMD / READY_MARKER / PASS_MARKER), a warn-only conformance preflight (checklist present, disclaimer headers, log4j floor, sol-jcsmp freshness), and a live-run heads-up; curl may trigger a publish but never renders the verdict. The six existing stages are unchanged

  5. Replaced the log4j2 2.26.0 pin with outcome rules: resolve the highest stable 2.x from authoritative metadata, a framework-owned SLF4J backend wins (never two backends), a config file is required, com.solacesystems loggers are never silenced, and the 2.17.1 Log4Shell floor is enforced as a check, removing the contradiction with Invariant 3

  6. Bound doc grounding to each design question and redefined the Grounding docs field to fetched-pages-only; added a Generation conformance group to the checklist template

  7. Removed the mandated Core Concepts intro (0/10 compliance) from SKILL.md and prerequisites.md; prerequisites is now broker acquisition only

  8. Solace Suggested decoupling keeps the two-project default but requires a surfaced conflict with three resolutions on single-deliverable asks; silent merges are forbidden

  9. Renamed the "Java API Release Notes" link to "JCSMP API Release Notes" (verified: the JAVA_API_RN download serves the JCSMP release-notes file) and pointed the samples' logging comments at the Step 3 rule instead of granting a free swap

  10. Routed every build request through Design mode first: mode routing now keys on the design contract, not request phrasing. jcsmp.md gains a design-contract gate (three valid sources: a same-session confirmed summary, an explicit summary supplied in chat, a saved solace-design.md) and a topology rule for edits to existing apps (topology-changing edits re-enter Design; mechanical edits go straight to Implement). Implement Step 2 makes the eight-field summary a hard precondition. Design mode gains a fully-specified path that resolves a fully stated build prompt in ONE merged confirm at close-out. Implement Step 1 no longer re-asks whether a broker exists when Design Step 0 confirmed it in the same session. The door question stays an isolated Implement Step 0

  11. Bound the samples' best-practice comments to generation: implement-mode Step 4 gains the comments-follow-their-code contract (a construct carried from a sample keeps its comment with names adapted; dropped demo harness takes its comments with it; fresh messaging code that applies a documented practice gets a short comment naming it; the mandated one-line comments are unaffected). The checklist's Generation conformance group gains a matching binary item, placed outside the preflight's first-four mechanical checks

Anything reviews should focus on/be aware of?

  • The implement-mode.md split moves the per-leaf wiring verbatim into implement-guaranteed-pubsub.md, implement-direct-pubsub.md, and implement-request-reply.md; the VERIFY: marker strings and exec ids are unchanged character-for-character
  • verify.sh changes are additive (new app stage, preflight warnings, heads-up echo); the six existing stage bodies are untouched and bash -n plus shellcheck pass
  • The sample .java edits are comment-only, so compile-check semantics are unchanged
  • Credential-handling hardening was deliberately deferred (tracked separately under EBP-2889's analysis notes)
  • The design-first routing (item 10) adds one confirm to fully specified build prompts by design; web and embedded shapes route the same way. Mode-routing evals are deferred to future work; the current eval corpus tests skill triggering only
  • No plugin.json version bump, per the release rules
  • Local trigger evals: PENDING - will be run and recorded here before this PR is marked ready

…elopment

- Promote buried mandates to SKILL.md invariants: AI-assisted disclaimer header, verification artifacts (checklist + verify.sh + hooks), and the live-broker heads-up; extend the doc-grounding invariant to ban substitute channels and the version invariant to ban the solrsearch index
- Surface the Step 0 door-question gate and the environment-discovery rule (a found broker/config never answers a question) in jcsmp.md, design-mode.md, and implement-mode Step 0
- Split implement-mode.md into a core file plus three per-leaf wiring files so the core fits a single Read; state the generation file-output contract up front and move the checklist write into Step 4 generation output
- Redesign verification to be shape-agnostic: verify.sh gains an app stage driven by a generated verify-hooks.sh (START_CMD/TRIGGER_CMD/READY_MARKER/PASS_MARKER), a warn-only conformance preflight, and a live-run heads-up line; curl may trigger but never render the verdict
- Replace the log4j2 2.26.0 pin with outcome rules: resolve the highest stable 2.x from authoritative metadata, framework-owned SLF4J backends win, config file required, com.solacesystems loggers never silenced, and the 2.17.1 Log4Shell floor enforced as a check
- Bind doc grounding to each design question and redefine the Grounding docs field to fetched-pages-only; add a Generation conformance group to the verification checklist
- Remove the mandated Core Concepts intro from SKILL.md and prerequisites.md; prerequisites is now broker acquisition only
- Solace Suggested decoupling: keep the two-project default but require a surfaced conflict with three resolutions on single-deliverable asks; never merge silently
- Rename the Java API Release Notes link to JCSMP API Release Notes (the JAVA_API_RN download serves the JCSMP release notes) and point sample logging comments at the Step 3 rule
- Root-caused from 10 analyzed sessions; violations and fixes tracked in EBP-2889
- Key mode routing on the design contract instead of request phrasing; a build prompt without a contract enters Design first, including web and embedded shapes
- Add the design-contract gate to jcsmp.md and make the summary a hard precondition in Implement Step 2 (valid sources: same-session summary, explicit summary in chat, saved solace-design.md)
- Add the topology rule for edits to existing apps: topology-changing edits re-enter Design, mechanical edits go straight to Implement
- Add Design mode's fully-specified path: a prompt that answers every tree question gets one merged confirm at close-out
- Dedupe the broker-access question in Implement Step 1 when Design Step 0 already confirmed it this session
- Keep the door question an isolated Implement Step 0; a confirmed design never answers it
- Add the comments-follow-their-code contract to implement-mode Step 4: a construct carried from a sample keeps that construct's comment (names adapted), dropped demo harness takes its comments with it, and fresh messaging code that applies a documented practice gets a short comment naming it
- Add a binary item to the checklist's Generation conformance group so a run that strips the comments fails its own checklist; keep it outside the preflight's first-four mechanical checks
- Align the Step 6 group description with the new non-preflight item
- Evidence: a generated embedded Direct Pub/Sub project carried sample constructs without their attached best-practice comments, while every explicitly mandated one-line comment survived (slot-bound rules get followed; unbound ones do not)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant