Skip to content

Fix remote-asset crashes and parallelize hot paths (validate, pack, dedup, sync) - #12

Merged
CaioWing merged 1 commit into
mainfrom
claude/cv-framework-optimization-7twrhy
Jul 12, 2026
Merged

CaioWing merged 1 commit into
mainfrom
claude/cv-framework-optimization-7twrhy

Conversation

@CaioWing

Copy link
Copy Markdown
Owner

Correctness:

  • vp fsck no longer aborts on cloud-backed assets: resolved_path raises for
    remote URIs, so byte checks are skipped for them (referential checks still
    apply); regression tests added.
  • vp validate no longer floods cloud-referenced datasets with false
    image.unreadable errors: remote assets skip the local header probe.

Performance:

  • validate: image header probes now fan out across a thread pool (order
    preserved, report deterministic) instead of decoding sequentially.
  • near-duplicate LSH: identical hashes are collapsed before bucketing, so
    batches of re-encoded near-uniform images no longer trigger the O(n^2)
    bucket pathology; over-threshold pairs are marked seen so shared bands
    never re-XOR them. phash backfill is parallel.
  • vp eval: each prediction's IoUs are computed once and replayed across all
    10 COCO thresholds (and the precision/recall pass) instead of once per
    threshold.
  • WebDataset pack: shards are written in parallel (they're independent tar
    streams), the asset->annotation map is materialized once up front, and
    sample labels serialize via orjson.
  • Archive pack: zstd multi-threaded compression (threads=-1) for the
    single-stream tar.zst.
  • SQLite index: WAL + synchronous=NORMAL on the live db; snapshot freeze and
    archive pack run a truncating checkpoint first so the copied/hashed
    index.db stays self-contained. Frozen snapshot dbs are left untouched.
  • sync ingest pool: results are consumed as they complete (no head-of-line
    blocking on a slow object) and folded in input order, keeping the index
    and summaries deterministic.
  • local listing: os.scandir walk captures stat in the same pass as
    discovery (one syscall per file instead of two).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01WtJWqB5e95s9FgEZF2VccN

…edup, sync)

Correctness:
- vp fsck no longer aborts on cloud-backed assets: resolved_path raises for
  remote URIs, so byte checks are skipped for them (referential checks still
  apply); regression tests added.
- vp validate no longer floods cloud-referenced datasets with false
  image.unreadable errors: remote assets skip the local header probe.

Performance:
- validate: image header probes now fan out across a thread pool (order
  preserved, report deterministic) instead of decoding sequentially.
- near-duplicate LSH: identical hashes are collapsed before bucketing, so
  batches of re-encoded near-uniform images no longer trigger the O(n^2)
  bucket pathology; over-threshold pairs are marked seen so shared bands
  never re-XOR them. phash backfill is parallel.
- vp eval: each prediction's IoUs are computed once and replayed across all
  10 COCO thresholds (and the precision/recall pass) instead of once per
  threshold.
- WebDataset pack: shards are written in parallel (they're independent tar
  streams), the asset->annotation map is materialized once up front, and
  sample labels serialize via orjson.
- Archive pack: zstd multi-threaded compression (threads=-1) for the
  single-stream tar.zst.
- SQLite index: WAL + synchronous=NORMAL on the live db; snapshot freeze and
  archive pack run a truncating checkpoint first so the copied/hashed
  index.db stays self-contained. Frozen snapshot dbs are left untouched.
- sync ingest pool: results are consumed as they complete (no head-of-line
  blocking on a slow object) and folded in input order, keeping the index
  and summaries deterministic.
- local listing: os.scandir walk captures stat in the same pass as
  discovery (one syscall per file instead of two).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WtJWqB5e95s9FgEZF2VccN
@CaioWing
CaioWing merged commit 1f8b482 into main Jul 12, 2026
8 checks passed
@CaioWing
CaioWing deleted the claude/cv-framework-optimization-7twrhy branch July 12, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants