Fix remote-asset crashes and parallelize hot paths (validate, pack, dedup, sync) - #12
Merged
Merged
Conversation
…edup, sync) Correctness: - vp fsck no longer aborts on cloud-backed assets: resolved_path raises for remote URIs, so byte checks are skipped for them (referential checks still apply); regression tests added. - vp validate no longer floods cloud-referenced datasets with false image.unreadable errors: remote assets skip the local header probe. Performance: - validate: image header probes now fan out across a thread pool (order preserved, report deterministic) instead of decoding sequentially. - near-duplicate LSH: identical hashes are collapsed before bucketing, so batches of re-encoded near-uniform images no longer trigger the O(n^2) bucket pathology; over-threshold pairs are marked seen so shared bands never re-XOR them. phash backfill is parallel. - vp eval: each prediction's IoUs are computed once and replayed across all 10 COCO thresholds (and the precision/recall pass) instead of once per threshold. - WebDataset pack: shards are written in parallel (they're independent tar streams), the asset->annotation map is materialized once up front, and sample labels serialize via orjson. - Archive pack: zstd multi-threaded compression (threads=-1) for the single-stream tar.zst. - SQLite index: WAL + synchronous=NORMAL on the live db; snapshot freeze and archive pack run a truncating checkpoint first so the copied/hashed index.db stays self-contained. Frozen snapshot dbs are left untouched. - sync ingest pool: results are consumed as they complete (no head-of-line blocking on a slow object) and folded in input order, keeping the index and summaries deterministic. - local listing: os.scandir walk captures stat in the same pass as discovery (one syscall per file instead of two). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtJWqB5e95s9FgEZF2VccN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correctness:
remote URIs, so byte checks are skipped for them (referential checks still
apply); regression tests added.
image.unreadable errors: remote assets skip the local header probe.
Performance:
preserved, report deterministic) instead of decoding sequentially.
batches of re-encoded near-uniform images no longer trigger the O(n^2)
bucket pathology; over-threshold pairs are marked seen so shared bands
never re-XOR them. phash backfill is parallel.
10 COCO thresholds (and the precision/recall pass) instead of once per
threshold.
streams), the asset->annotation map is materialized once up front, and
sample labels serialize via orjson.
single-stream tar.zst.
archive pack run a truncating checkpoint first so the copied/hashed
index.db stays self-contained. Frozen snapshot dbs are left untouched.
blocking on a slow object) and folded in input order, keeping the index
and summaries deterministic.
discovery (one syscall per file instead of two).
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01WtJWqB5e95s9FgEZF2VccN