Add Dataset Library backend: curate, upload, train custom AI art models

[guillaumelauzier]

What

Item 1 of the platform build backlog (spec: Step 8 of the published design brief). Creators can now upload or import their own images/video, curate them into a named dataset with a rights declaration, and train a custom AI art model against the existing fal_custom render-model lane — closing the gap between "the model registry supports creator-trained models" and "a creator can actually produce one here."

Changes

• migrations/0020_dataset_library.sql: datasets, dataset_items (content-hash deduped — exact-match only; perceptual near-dup detection is an explicit v2 cut, not a silent gap), training_jobs (queued → training → succeeded/failed, mirroring the render_jobs lifecycle already in the token ledger).
• src/db/datasets.ts: CRUD + the item_count denormalisation kept in sync via the same D1 batch as each insert/delete, not a trigger.
• src/ai/training.ts: fal.ai queue-API dispatch/poll for real training (submit → request_id → status → result), wired into the existing per-minute cron tick. Only one base model is verified end-to-end today (FLUX.1 [dev] via flux-lora-fast-training / flux-lora) — requesting any other base_model is rejected at the handler with a clear error rather than silently hitting a guessed endpoint slug. TRAINING_MOCK (opt-in, same convention as RENDER_MOCK/PINNING_MOCK/GITHUB_MOCK; production must never set it) completes a job synchronously in-request instead of simulating cron timing.
• src/datasets/handlers.ts: dataset CRUD, item upload (data URL, R2-backed, exact-dedup) and URL-list import (per-item success/fail reporting, never all-or-nothing), and the train endpoint (debit-before-enqueue, insufficient-balance guard, an eager private render_models shell so the job has somewhere to publish a version onto the moment training succeeds).
• LoRA training is priced at 150 tokens — deliberately inside the 200-token signup grant, so a new creator's free grant covers one training run, not just renders of others' models.

Testing

• 6 new unit tests (rights/visibility/kind validators, the trainable-base-model allowlist, tiered pricing) — 33/33 pass.
• 24 new smoke checks (create/list/patch, upload + exact-dedup, unsupported-mime rejection, import-urls validation, item delete + count integrity, the full train-to-published-model path under TRAINING_MOCK, private-dataset isolation) — 136/136 pass.
• Typecheck clean; wrangler deploy --dry-run clean on both environments.

Known scope cuts (deliberate, called out in code comments)

• Only FLUX.1 [dev] is a verified trainable base model; adding another means verifying its real fal.ai trainer + inference endpoint pair, not guessing a slug.
• Dataset visibility (private/public) is stored but there's no cross-owner public dataset browsing surface yet — GET /v1/datasets/:slug is owner-only for now (slugs are unique per-owner, not globally).
• Near-duplicate (perceptually similar, not byte-identical) item detection is out of scope — dedup is exact content-hash only.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01QuSa99HABSrWRarT9BQ5Tz

---

Generated by Claude Code