spec.txt

Product spec — Research Agent v0 (Carter)
1) Goal

Validate an auditable, bounded, citation-grounded research agent that produces a 1–2 page research brief from a user question, with traceability from every claim to evidence.

2) Target users

Engineers / PMs needing quick technical landscape scans

Internal platform/infra teams producing “decision briefs”

Researchers creating fast lit-review scaffolds

3) User experience

Input

Research question (required)

Optional: constraints

time budget (minutes)

source constraints (domain allowlist/denylist)

recency window (e.g., last 2 years)

number of sources (e.g., 5–10)

Output

Research brief (markdown)

Executive summary

Key findings (bullet points)

Contradictions / disagreements (if found)

Gaps / open questions

References

Claims table (machine-readable + human readable)

claim → evidence snippets → source URLs

Run report

cost/time/tool-call counts

stopping reason

audit trail pointer (event log)

4) Success criteria (v0)

Grounding: ≥ 90% of non-trivial claims have at least one evidence snippet + source.

No hallucinated references: every cited source must be retrievable and appear in the final reference list.

Bounded: run completes within configured budgets (time + tool calls).

Auditable: can reconstruct “why” a claim exists via event log + evidence IDs.

5) Non-goals (v0)

Paywalled / proprietary retrieval solving

Fully automated credibility scoring

Domain-specific deep reading (PDF parsing/OCR heavy)

Multi-agent debate

Fine-tuning or learning across runs (save this for v1)

6) Key product decisions

Evidence-first writing: the agent must collect evidence snippets before drafting conclusions.

Strict citation policy: claims without evidence are either removed or placed under “Speculation / Hypotheses” with explicit labeling.

Stop rules are explicit: completion is a function of coverage score + diminishing returns + budget.

7) Core metrics

Time to draft (p50/p90)

Sources used (count + diversity)

Claim grounding rate

Reviewer acceptance rate (manual rubric)

Cost per brief (tokens + tool calls)

8) MVP rollout plan

Week 1: CLI-only internal prototype (markdown output + JSON event log)

Week 2: Simple web UI (optional), add replay + regression suite

Engineering spec — Research Agent v0 (Carter)
1) Architecture (kernel + plugins + policy)

Kernel (stable)

Orchestrator (control loop)

Event Log (append-only)

Artifact Store (sources, snippets, drafts)

Policy Engine (gates + redaction rules)

Evaluator (coverage/grounding/cost + stop)

Plugins (replaceable)

Retriever: web search + fetch (later: internal docs)

Extractor: snippet extraction + metadata

Synthesizer: outline → brief

Verifier: claim→evidence mapping checks

2) Data model (minimal)

TaskSpec (input)

{
  "question": "string",
  "constraints": {
    "max_tool_calls": 25,
    "max_minutes": 10,
    "max_sources": 10,
    "recency_days": 3650,
    "allow_domains": [],
    "deny_domains": []
  }
}


Event Log (append-only)
Each entry is immutable and timestamped.

{
  "event_id": "ulid",
  "ts": "iso8601",
  "type": "TASK_CREATED | SEARCHED | SOURCE_SELECTED | SNIPPET_EXTRACTED | OUTLINE_DRAFTED | CLAIMS_BUILT | BRIEF_RENDERED | STOPPED | ERROR",
  "payload": {}
}


Source record

{
  "source_id": "ulid",
  "title": "string",
  "url": "string",
  "published_date": "string|nullable",
  "retrieved_at": "iso8601",
  "content_hash": "sha256",
  "notes": "string|nullable"
}


Evidence snippet

{
  "evidence_id": "ulid",
  "source_id": "ulid",
  "quote": "string",
  "char_start": 123,
  "char_end": 234,
  "relevance": 0.0
}


Claim

{
  "claim_id": "ulid",
  "text": "string",
  "evidence_ids": ["ulid"],
  "confidence": 0.0,
  "tags": ["finding|contradiction|gap|definition"]
}

3) Control loop (v0)

Normalize TaskSpec (defaults + policy)

Retrieve candidates (search)

Select sources (diversity + relevance)

Extract snippets (evidence-first)

Build claims table (claim drafting grounded in evidence)

Draft brief from claims + outline

Verify:

every claim has ≥1 evidence snippet (unless explicitly labeled as speculation)

every evidence_id resolves

every source_id resolves + URL present

Evaluate & stop:

stop if: coverage score ≥ threshold AND diminishing returns hit

or budgets exhausted → “partial brief” with gaps + explicit stop reason

4) Stopping rules (concrete)

Hard stops

tool calls >= max_tool_calls

time >= max_minutes

sources >= max_sources

Soft stop: diminishing returns

run one more retrieval round only if it improves:

coverage score by ≥ Δ (e.g., 0.05), or

adds new contradiction/gap category

5) Evaluator (v0 scoring)

Coverage: fraction of brief sections populated with grounded claims

Grounding: % claims with evidence_ids length ≥ 1

Diversity: # unique domains / source types

Cost: tool calls + tokens

Return:

{
  "coverage": 0.82,
  "grounding": 0.94,
  "diversity": 0.6,
  "stop_reason": "DIMINISHING_RETURNS"
}

6) Security & privacy (v0)

No secrets ingestion: redact tokens/keys if detected in fetched text (basic regex)

Domain allow/deny enforcement

Store only what’s needed:

source URL + hash + snippets used (avoid saving full page where possible)

7) Repo structure
carter/
  README.md
  docs/
    product-spec.md
    engineering-spec.md
    schemas.md
  src/
    main.py (or main.ts)
    kernel/
      orchestrator.*
      event_log.*
      policy.*
      evaluator.*
      artifacts.*
    plugins/
      retriever_web.*
      extractor_basic.*
      synthesizer_llm.*
      verifier_basic.*
  tests/
    fixtures/
    regression_cases.json
    test_replay.*

8) Minimal deliverables checklist (v0)

 CLI: carter run --question "..." --max-sources 8 --max-minutes 8

 Outputs:

brief.md

claims.json

run_report.json

event_log.jsonl

 Replay: carter replay --run-id ... reproduces outputs with cached artifacts

 Regression: 5 canned questions with expected minimum grounding score

9) Tech stack suggestion

Python (fast prototype, good text tooling)

pydantic for schemas

sqlite or plain jsonl for event log (v0: jsonl is fine)

pluggable retriever interface

(If you prefer Node/TS, same architecture maps cleanly.)

10) Milestones (tight)

M0 (day 1–2): schemas + event log + CLI skeleton

M1 (day 3–4): web retriever + snippet extractor + claims table

M2 (day 5–6): brief synthesis + verifier + evaluator + stop rules

M3 (day 7): replay + regression harness