Spec: Bot Marketplace Phase 1 — GitHub-indexed catalog #913

Parent: #18 · Decision record: ADR-0131 · Milestone: v1.1

Problem Statement

A Human who wants a ready-made PersonaBot has no way to find one. Bots shared by other people already live as public GitHub Memory repositories, and BotHarness can already create a Bot from a Git URL (#298), but the Human has to know the exact URL in advance. An author who wants their Bot to be found has nowhere to put it. The previously planned path (#17: accounts, upload, hosted artifacts, Git remote) must ship in full before anyone can browse a single Bot.

Solution

A Bot Marketplace opened from the harness as a modal. It lists Indexed Repositories: public GitHub repositories whose authors added the botharness-bot topic. A dedicated Cloudflare Worker discovers and refreshes these repositories on a schedule and stores them in D1 with an FTS5 index. The Human browses by stars or recent activity, searches by name, description, topics and README, filters by topic, and reads the rendered README. Install asks for confirmation, showing the latest commit and a third-party risk notice, then creates a fresh PersonaBot from the repository's Git URL through the existing #298 path. An author who just added the topic can paste their repository URL to be listed immediately. Anyone can report an entry in one click. The same catalog is browsable without the harness on the product site (deepseekbot.botharness.ai), which reads the Worker's public read API. Phase 1 has no accounts and no download counts.

User Stories

1. As a Human, I want to open a Bot Marketplace from the harness, so that I can discover Bots without leaving my workspace.
2. As a Human, I want the Marketplace to open as a modal over my current view, so that I don't lose my place.
3. As a Human, I want to see a list of shareable Bots with name, description, tags, stars and last update, so that I can judge them at a glance.
4. As a Human, I want to sort the list by stars or by most recently updated, so that I can find popular or active Bots.
5. As a Human, I want to load more results as I scroll, so that long lists stay fast.
6. As a Human, I want to search by keywords that appear in a Bot's name, description, topics or README, so that I can find Bots for a specific job.
7. As a Human, I want search results ranked by relevance, so that the best matches come first.
8. As a Human, I want to filter by topic tags, so that I can narrow the list to a domain.
9. As a Human, I want to open a Bot's detail view with its rendered README, so that I understand what it does before installing.
10. As a Human, I want README images to display correctly, so that the detail view looks like it does on GitHub.
11. As a Human, I want a link to the GitHub repository from the detail view, so that I can inspect its files and history.
12. As a Human, I want an Install button on each Bot, so that I can adopt it in one action.
13. As a Human, I want the install confirmation to show the latest commit time and short SHA, so that I know which version I'm getting.
14. As a Human, I want the install confirmation to warn that third-party repositories may contain harmful files or instructions that my Bot will read as Memory, so that I install deliberately.
15. As a Human, I want the installed Bot to use the shared name, description, badges and avatar when the author provided them, so that it looks as intended.
16. As a Human, I want the installed Bot to fall back to a normal generated avatar when none is shared, so that every Bot still looks complete.
17. As a Human, I want the installed Bot's Memory to keep GitHub as its Git origin, so that I or my Bot can git pull the author's later changes.
18. As a Human, I want a failed install to leave no half-created Bot, so that I can simply retry.
19. As a Human, I want installing the same repository twice to create two independent Bots, so that I can experiment without affecting my first copy.
20. As a Human in mainland China, I want the Marketplace, its challenge and reporting to work without Google, Cloudflare Turnstile or GitHub web access, so that I'm not locked out.
21. As a Human, I want a one-click Report action with an optional reason, so that I can flag harmful or spam entries without an account.
22. As a Human, I want reported entries that are clearly abusive to disappear quickly, so that the catalog stays safe.
23. As an author, I want my Bot listed just by adding the botharness-bot topic to my public repository, so that I don't have to fill in a submission form.
24. As an author, I want to paste my repository URL into the Marketplace to be listed immediately, so that I don't wait for the next scheduled crawl.
25. As an author, I want a clear message when my pasted repository can't be listed (private, missing topic, archived, blocked, not found), so that I know what to fix.
26. As an author, I want my README, description and topics to be what the Marketplace shows, so that I maintain one source of truth.
27. As an author, I want to optionally commit a .botharness/bot.json to override display name, avatar and role badges, so that my Bot looks right without cluttering the repository root.
28. As an author, I want removing the topic, archiving, deleting or making the repository private to remove my listing, so that I stay in control.
29. As an author, I want renaming or transferring my repository to keep the same listing, so that I don't lose its history.
30. As an author, I want star and README changes to appear within about an hour, so that the listing stays current.
31. As an administrator, I want to blocklist a repository so that it can't be listed again, so that repeat abuse stops.
32. As an administrator, I want to restore an entry that was hidden by reports, so that false reports don't remove legitimate Bots.
33. As an administrator, I want discovery and refresh to stay within GitHub API limits as the catalog grows, so that the service keeps working.
34. As an administrator, I want URL paste and reporting protected by proof-of-work and rate limits, so that bots can't flood the crawler or mass-hide entries.
35. As an administrator, I want crawler failures to keep the last good data, so that a GitHub outage doesn't empty the catalog.
36. As a visitor to the product site, I want a Marketplace route where I can browse, search, filter and read Bot details without installing anything, so that I can see what Bots exist before trying DeepSeekBot.
37. As a visitor to the product site, I want each Bot's detail page to show how to install it in DeepSeekBot and its Git URL to copy, so that I can adopt it once I have the harness.
38. As a visitor, I want Marketplace pages to be shareable links in Chinese and English, so that I can send a Bot to someone else.

Implementation Decisions

• Marketplace Worker. A new Cloudflare Worker package in this monorepo, deployed separately from the docs and product sites (for example at market.botharness.ai) with its own D1 database. It exposes a small read API (list, search, detail), a paste-to-crawl endpoint, a report endpoint, an ALTCHA challenge endpoint, and an authenticated admin endpoint for blocklist and restore. It owns the scheduled discovery and refresh jobs.
• D1 schema (shape, not final DDL). repositories keyed by GitHub repository node ID, holding owner/name, URL, description, topics, stars, pushed_at, default branch, latest commit SHA and time, README text, optional descriptor fields, visibility state (listed, hidden_missing, hidden_reported, blocked), first/last seen and last refreshed times. An FTS5 virtual table over name, description, topics and README, kept in sync on every write. reports holding repository ID, reason, hashed source and time. blocklist holding repository ID or owner/name plus note. A crawl-state row for discovery progress.
• Discovery. Daily full scan of topic:botharness-bot using GitHub search, sliced by repository creation date so every slice returns fewer than 1,000 results; slices are subdivided automatically when a slice reaches the cap. Discovery adds new repositories and marks listed ones that no longer match as hidden_missing. It must be a full scan, because adding a topic does not change pushed_at.
• Refresh. Hourly authenticated GraphQL nodes queries in batches of 100 repository IDs for stars, pushed_at, topics, description, visibility, archived and default-branch head commit. README and .botharness/bot.json are refetched only when pushed_at changed. Losing the topic, going private, being archived or deleted hides the entry but keeps its row. A failed batch leaves existing rows untouched.
• Scale path. One cron invocation handles thousands of repositories. Batches are independent units so the same work can later be enqueued on Cloudflare Queues when Worker subrequest or CPU limits become the bound; no Queues in this slice.
• Paste to crawl. Accepts a GitHub repository URL, requires a solved ALTCHA challenge, applies per-IP and per-repository rate limits (a repository is crawled at most once per five minutes), fetches the repository immediately, and lists it only if it is public, not archived, carries the topic and is not blocklisted. It returns a typed reason when it refuses.
• Presentation rules. Display name is the repository name, summary is the GitHub description, tags are topics minus botharness-bot, detail is the README. A valid .botharness/bot.json may override display name, avatar (generated-avatar settings or a committed image path) and role badges; an invalid one is ignored with defaults kept.
• Browse and search. Browse sorts by stars or pushed_at, descending, with an opaque keyset cursor over (sort key, node ID). Search uses FTS5 bm25 ranking and returns at most 200 results with offset-based paging inside that cap. Topic filter applies to both modes. Only listed entries are returned.
• README rendering. The Client renders Markdown with HTML sanitized and rewrites relative image and link URLs to raw.githubusercontent.com and github.com at the entry's commit.
• Reporting. One-click, optional reason, requires a solved ALTCHA challenge and per-IP limits. When reports from a configured number of distinct hashed sources accumulate, the entry becomes hidden_reported until an administrator restores or blocks it. Raw IPs are never stored.
• Anti-bot. ALTCHA self-hosted proof-of-work with an HMAC key secret and escalating cost tiers (default, elevated, high) driven by recent request volume per source, as in Vain's anysoul project. No Cloudflare Turnstile.
• Public read API for the product site. List, search and detail are a stable, versioned, unauthenticated JSON API (for example under /v1/) shared by the harness and the product site. Responses carry CORS headers allowing the product site origins (https://deepseekbot.botharness.ai and its local dev origin) and short public cache headers. Detail returns README Markdown plus the commit SHA for image rewriting, so both surfaces render it the same way. Paste-to-crawl, report and admin endpoints are not part of this public contract for the site in this slice.
• Product site route. deepseekbot.botharness.ai gets a Marketplace route (/market for Chinese, /en/market for English) for browse, search, topic filter and detail, implemented in BotHarness/deepseekbot-site against the public read API. Detail shows the Git URL to copy and a short guide to install it through DeepSeekBot's Git import; the site never installs anything itself. Rendering strategy (server-side fetch for shareable, indexable pages, or client-side) is decided in the site repository.
• Host↔Client. The Client modal talks to the Marketplace Worker through new Host bridge methods, so network access, timeouts and errors follow the Host's existing outbound path rather than the Client webview's network policy. The ALTCHA challenge is solved in the Client and its payload is forwarded through the Host.
• Install. Install calls the existing createFromGit path with the repository's HTTPS URL, display name, description and roles from the catalog entry. After a successful clone, the Host reads .botharness/bot.json from the cloned tree, not from the catalog, and applies a shared avatar when present. createFromGit therefore gains an optional post-clone presentation step. The cloned commit may be newer than the one shown in the confirmation; the Bot keeps whatever commit was cloned.
• Secrets and config. GitHub token, ALTCHA HMAC key and admin token are Worker secrets. Report threshold, rate limits and cost tiers are Worker variables.
• Vocabulary. Use Bot Marketplace and Indexed Repository from CONTEXT. An Indexed Repository is not a SoulSnapshot, Listing or Version, and installing one is Git-URL creation, not Import.

Testing Decisions

• Tests exercise external behavior at the highest seams, never internal helpers. Proposed seams:
  1. Marketplace Worker boundary: its HTTP handlers and scheduled handler, run in the Workers runtime against a local D1 database, with GitHub REST/GraphQL replaced by a deterministic fake server. This one seam covers discovery slicing at the 1,000 cap, refresh batching, README refetch only on push change, hiding/restoring states, rename via node ID, keyset and FTS5 paging, topic filters, paste-to-crawl refusals, ALTCHA verification, rate limits, report threshold, blocklist and admin auth, and keeping last good data on GitHub failure.
  2. Host bridge boundary: the new Marketplace bridge methods and the extended createFromGit with a local bare Git fixture containing optional .botharness/bot.json and avatar, following the existing bridge RPC and Memory clone tests. It verifies the shared avatar is applied, the default avatar is used when absent, origin points at the given URL, and a failed clone leaves no Bot.
  3. Public read API contract: at the Worker boundary, assert response shapes, CORS for allowed and disallowed origins, and cache headers, so the product site can depend on them.
  4. Client modal: list, search, detail, install confirmation (commit time, short SHA, risk notice), report and paste flows against a mocked bridge, following the existing PersonaBot creation Client tests.
• Prior art: the core bridge RPC tests, Memory clone tests and registry tests, plus the Client PersonaBot creation and bridge tests.
• Runnable Human acceptance: a local Worker with local D1 and a real GitHub token crawls a real test repository with the topic. The Human pastes its URL, finds it by search in the modal inside an isolated local DSH instance, installs it, and inspects the new Bot's Memory, origin and avatar. Removing the topic and refreshing hides it. Reporting it past the threshold hides it.

Out of Scope

• Accounts, Better Auth, Email OTP, Passkey, Google sign-in and terms acceptance (Phase 2).
• Uploading Bots from the local harness, hosted Git artifacts, the read-only platform Git remote and withdrawal (Phase 2, Tracer: publish a Memory Git repo and import through a Bot link #17).
• Favorites, download or import counts and completed-import receipts (Phase 2).
• Installing, pasting URLs or reporting from the product site; the site is browse-only in this slice.
• Non-GitHub sources (GitLab, Gitea, raw Git URLs) in the catalog.
• Cloudflare Queues fan-out, until scale requires it.
• Automatic updating of installed Bots; updates stay native git pull.
• Production deployment, which is separately authorized.

Further Notes

• The seams above are proposed; adjust them on review before implementation starts.
• The product site route is built in BotHarness/deepseekbot-site by its own thread against the public read API defined here; it ships after the Worker's read API is deployed.
• GitHub search caps and GraphQL point costs were reasoned from GitHub's documented rules; verify them against the current official docs at implementation time.
• D1 supports FTS5 per Cloudflare's SQL statements documentation; also confirm current limits on virtual tables, for example around wrangler d1 export.

Closeout — Human accepted 2026-10-08

The Human confirmed acceptance and directly closed #913. Confirmation record. This supersedes the pending-acceptance status below.

Implementation children #916–#922 and guide #958 are closed. PR #943 records deployment to market.botharness.ai; site Marketplace implementation is present. Keep this specification open for final acceptance evidence:

• Verify a real opted-in GitHub repository through discovery/search, harness install and fresh Bot/Memory/origin inspection.
• Record real topic-removal refresh and report-hide behavior without treating fixture-modified topic metadata as production consent evidence.
• Confirm current production GitHub credential availability and a successful scheduled refresh. PR chore(market): point wrangler at the production D1 and market.botharness.ai #943 recorded GITHUB_TOKEN unset at deployment; no current-state claim is inferred from that older record.

Platform-account/upload work #17 was cancelled and is not a future Phase 2 gate. The historical Phase 2 references in the original scope exclusions do not create remaining work; current sharing scope is reconciled in #18.