Add cross-provider cloud transfer and vp serve (web UI + REST API) - #10
Merged
CaioWing merged 4 commits intoJul 11, 2026
Merged
Conversation
Cloud/multisource layer: - Cross-provider 'copy' targets no longer error: when source and target live on different providers (local->S3, S3->GCS, ...), sync relays the bytes it already read for the sha256 - one read + one upload, never a second download. Same-provider transfers stay server-side. - New Resolver.write_bytes (local + fsspec) backs the relay path. - CloudTarget gains server_side and accepts the already-read bytes; ensure_object stays idempotent and metadata-only on re-sync. Frontend integration: - vp serve (optional [server] extra: fastapi + uvicorn) hosts a REST API over the same core functions the CLI calls: project/sources/stats, dry-run sync plans, background jobs with progress (sync/validate/ export), paginated assets with provider-agnostic image streaming, splits and snapshots. Credentials are stripped from every response; mutating jobs take the project lock and run one at a time. - Single-file, dependency-free web dashboard (light/dark): pipeline board, source table with dry-run preview, class/split charts, thumbnail gallery, versions, and a live job bar. Tests: 131 passing (cross-provider relay + 11 API tests); docs updated (README, ARCHITECTURE, cloud-sync, new web-ui page). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGXiC5XYqEsQis1cpimXz8
The frontend will live in a separate product; VisionPack stays a CLI-first DatasetOps core. This removes vp serve, the FastAPI app, the dashboard, the [server] extra and its dev deps. The cloud-sync improvement stays: cross-provider 'copy' targets relay the already-read bytes (one read + one upload) instead of erroring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGXiC5XYqEsQis1cpimXz8
… COCO/ImageFolder Closes the gaps found in the cloud-sync review: - Retries with exponential backoff (sources/retry.py) on every remote call. Transient provider errors (throttling, dropped connections, 5xx) are retried; exhaustion surfaces as a per-object IngestFailure instead of aborting the sync on an unknown exception type. - Target CAS membership now comes from one prefix listing per run (rclone-style fast-list) instead of a per-object existence check: ~100 paginated LISTs instead of 100k HEADs at 100k objects. - Cross-provider relay uploads are size-verified against the landed object before the index points at them. - vp sync --jobs controls ingest concurrency; remote sources default to a floor of 16 workers (object-store throughput is latency-bound, the CPU-derived executor default undersizes it). - Remote labels are fetched once: class inference reads them in parallel and caches the texts for ingest to replay. - COCO and ImageFolder sources now work remotely through the same resolver + blob pipeline as YOLO (probe cache, copy modes, cloud target included); the shared ingest core lives in _ingest_asset and the pool-drain loop is shared by all three formats. 134 tests passing (14 new covering retry semantics, fast-list, relay verification, single-read labels, remote ImageFolder/COCO, pool sizing). Docs updated (cloud-sync, usage, README, ARCHITECTURE). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGXiC5XYqEsQis1cpimXz8
Machine-readable output as the stable integration surface for external
programs (services, UIs, CI):
- New cli/output.py: schema-versioned envelope. Success carries data;
failure carries a structured error (main() emits it in JSON mode) —
consumers check 'error' + exit code, never stderr. schema bumps only
on breaking shape changes; additive fields don't.
- --json on sync (run + --dry-run), import, validate, stats, split
(create/lock/list/show), snapshot (create/list/show), diff, export,
pack, fsck, eval, autolabel, queue. Progress bars are suppressed so
stdout is always exactly one parseable document. The pre-existing raw
--json of stats/diff/eval/queue moved into the same envelope.
- tests/test_json_contract.py drives the real CLI entry point and pins
the envelope + load-bearing fields per command (9 tests; 143 total).
Docs site fixes and updates:
- index.md and installation.md used bare {% link %} tags mid-sentence,
which render as a raw URL path instead of a link — now proper
markdown links.
- index.md was stale: task table missed YOLO-seg and semantic-mask
export; feature list missed eval/autolabel/queue and multi-provider
sync. Brought in line with the README.
- New docs page: json-output.md (envelope, per-command payloads,
script/Python examples). Linked from index, usage, README.
- README test badge and status count were stale at 118 -> 143.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGXiC5XYqEsQis1cpimXz8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloud/multisource layer:
live on different providers (local->S3, S3->GCS, ...), sync relays the
bytes it already read for the sha256 - one read + one upload, never a
second download. Same-provider transfers stay server-side.
ensure_object stays idempotent and metadata-only on re-sync.
Frontend integration:
over the same core functions the CLI calls: project/sources/stats,
dry-run sync plans, background jobs with progress (sync/validate/
export), paginated assets with provider-agnostic image streaming,
splits and snapshots. Credentials are stripped from every response;
mutating jobs take the project lock and run one at a time.
board, source table with dry-run preview, class/split charts,
thumbnail gallery, versions, and a live job bar.
Tests: 131 passing (cross-provider relay + 11 API tests); docs updated
(README, ARCHITECTURE, cloud-sync, new web-ui page).
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01CGXiC5XYqEsQis1cpimXz8