In one sentence: PHerc0343P/volumes/20250521134555-8.640um-1.2m-116keV-masked.zarr/0/ contains 555 objects that are stale chunks from earlier export runs at 256³ and 512³ chunk shapes; they occupy 14.80 GiB and are incompatible with the array's declared chunk shape; zarr 3.3.0 fails on them with the reproducer below.
What was observed. The array declares chunks: [128,128,128], |u1, no codec — 2,097,152 bytes per chunk. A full listing of the level-0 prefix finds 8,543 present objects: 7,988 at 2,097,152 bytes, 499 at 16,777,216 bytes (256³) and 56 at 134,217,728 bytes (512³). All 555 sit inside the declared 43×40×40 grid and carry multipart ETags.
aws s3api list-objects-v2 --no-sign-request --bucket vesuvius-challenge-open-data \
--prefix 'PHerc0343P/volumes/20250521134555-8.640um-1.2m-116keV-masked.zarr/0/' \
--query 'Contents[?Size!=`2097152`].[Key,Size]' --output text | wc -l
# 555
Reader failure. One stale object placed in a minimal local store with the published .zarray, read with zarr 3.3.0:
ValueError: cannot reshape array of size 16777216 into shape (128,128,128)
Reproduction: https://github.com/dud8/fill-census/blob/master/tools/reader_test.py
What the evidence indicates. The 7,988 correctly-sized chunks are exactly the set of grid cells the final export wrote. An exporter that omits all-fill chunks writes only non-empty cells, so that set is its own record of which cells held data at full resolution; the volume's clean level-2 and level-4 pyramids corroborate it (set equality both directions; twelve random anomalous keys read all-zero in level 2, twelve random correct keys read non-zero). So the 555 survive precisely on keys the final run never wrote. Stride autocorrelation on the stale bytes peaks at 1 / 256 / 65,536 (C-order 256³) and 1 / 512 / 262,144 (512³). A dense 256³ block cross-correlates with the current volume at exactly key × 256 with r = 0.9993; for 0/6/8/11 all eight 128³ sub-cubes are byte-identical to the current chunks. All 10,549 objects have LastModified inside a 35-second window on 2026-03-04, interleaved, which is consistent with one bulk copy; whether the same objects exist in the source prefix (s3://scrollprize-volumes/esrf/20250506/...) has not been checked from here. The 8 MiB multipart parts are the CLI default; a consequence of size, not a cause.
Scope. Isolated to level 0 of this volume. Levels 1–5 are clean (1,614 / 286 / 66 / 19 / 6 objects, all 2,097,152 B). Fully enumerated, no chunk-size anomalies: the 2.215 µm sibling (285,735 objects), PHerc0343/20250521140437 (207,230), PHerc0175A/20250521115057 (166,316), PHerc0268/20251110183117 (406,415), PHerc0500P2/20250820143440. 63 of 71 catalogued volumes share this .zarray signature, so "every level-0 object is exactly 2,097,152 bytes" is a near-universal invariant.
Downstream. Mapping every defined cell of the eight published segments traced on this volume to level-0 chunk indices gives zero intersections with the 555 keys, so no published surface cell reads an affected chunk. Layer extraction samples some voxels along the normal beyond the surface; that neighbourhood was not separately checked. Exposure is any read that touches an affected key. This is PHerc0343P's only standard-resolution volume.
Proposed repair: delete the 555 objects. Because the final export's own sparse-write set says those cells are fill, deletion leaves readers with fill_value: 0, which both the export and the pyramid already assert. Re-splitting would instead write voxels from an older intensity window into regions the current mask clears. 14.80 GiB (48.7% of this volume's level-0 payload) freed, nothing re-derived. If the same objects exist in the upstream source prefix, it needs the same treatment or a re-sync restores them. If the bucket is versioned, the delete is reversible. The exact 555 keys with observed sizes are committed as a manifest in fill-census (reports/manifests/).
Prevention. A publish-time check for the 63 uncompressed volumes: every chunk object is prod(chunks) × itemsize. One ListObjectsV2 pass per volume. The existing live test (villa/volume-cartographer/core/test/test_pherc0172_live.cpp:77) checks only chunks[0] on level 5 and cannot see this.
Not established. The provenance of the 56 × 512³ objects (no cross-correlation lock found). The exporter mask_layers_zarr_export is not in any public repository, so the earlier-runs-never-removed explanation is inferred from the data, not read from source.
Method. Read-only, anonymous, no credentials. Full census across 71 volumes: https://github.com/dud8/fill-census — the 555-key manifest is reproducible from the listing command above.
In one sentence:
PHerc0343P/volumes/20250521134555-8.640um-1.2m-116keV-masked.zarr/0/contains 555 objects that are stale chunks from earlier export runs at 256³ and 512³ chunk shapes; they occupy 14.80 GiB and are incompatible with the array's declared chunk shape; zarr 3.3.0 fails on them with the reproducer below.What was observed. The array declares
chunks: [128,128,128],|u1, no codec — 2,097,152 bytes per chunk. A full listing of the level-0 prefix finds 8,543 present objects: 7,988 at 2,097,152 bytes, 499 at 16,777,216 bytes (256³) and 56 at 134,217,728 bytes (512³). All 555 sit inside the declared 43×40×40 grid and carry multipart ETags.Reader failure. One stale object placed in a minimal local store with the published
.zarray, read with zarr 3.3.0:Reproduction: https://github.com/dud8/fill-census/blob/master/tools/reader_test.py
What the evidence indicates. The 7,988 correctly-sized chunks are exactly the set of grid cells the final export wrote. An exporter that omits all-fill chunks writes only non-empty cells, so that set is its own record of which cells held data at full resolution; the volume's clean level-2 and level-4 pyramids corroborate it (set equality both directions; twelve random anomalous keys read all-zero in level 2, twelve random correct keys read non-zero). So the 555 survive precisely on keys the final run never wrote. Stride autocorrelation on the stale bytes peaks at 1 / 256 / 65,536 (C-order 256³) and 1 / 512 / 262,144 (512³). A dense 256³ block cross-correlates with the current volume at exactly
key × 256with r = 0.9993; for0/6/8/11all eight 128³ sub-cubes are byte-identical to the current chunks. All 10,549 objects haveLastModifiedinside a 35-second window on 2026-03-04, interleaved, which is consistent with one bulk copy; whether the same objects exist in the source prefix (s3://scrollprize-volumes/esrf/20250506/...) has not been checked from here. The 8 MiB multipart parts are the CLI default; a consequence of size, not a cause.Scope. Isolated to level 0 of this volume. Levels 1–5 are clean (1,614 / 286 / 66 / 19 / 6 objects, all 2,097,152 B). Fully enumerated, no chunk-size anomalies: the 2.215 µm sibling (285,735 objects), PHerc0343/20250521140437 (207,230), PHerc0175A/20250521115057 (166,316), PHerc0268/20251110183117 (406,415), PHerc0500P2/20250820143440. 63 of 71 catalogued volumes share this
.zarraysignature, so "every level-0 object is exactly 2,097,152 bytes" is a near-universal invariant.Downstream. Mapping every defined cell of the eight published segments traced on this volume to level-0 chunk indices gives zero intersections with the 555 keys, so no published surface cell reads an affected chunk. Layer extraction samples some voxels along the normal beyond the surface; that neighbourhood was not separately checked. Exposure is any read that touches an affected key. This is PHerc0343P's only standard-resolution volume.
Proposed repair: delete the 555 objects. Because the final export's own sparse-write set says those cells are fill, deletion leaves readers with
fill_value: 0, which both the export and the pyramid already assert. Re-splitting would instead write voxels from an older intensity window into regions the current mask clears. 14.80 GiB (48.7% of this volume's level-0 payload) freed, nothing re-derived. If the same objects exist in the upstream source prefix, it needs the same treatment or a re-sync restores them. If the bucket is versioned, the delete is reversible. The exact 555 keys with observed sizes are committed as a manifest in fill-census (reports/manifests/).Prevention. A publish-time check for the 63 uncompressed volumes: every chunk object is
prod(chunks) × itemsize. One ListObjectsV2 pass per volume. The existing live test (villa/volume-cartographer/core/test/test_pherc0172_live.cpp:77) checks onlychunks[0]on level 5 and cannot see this.Not established. The provenance of the 56 × 512³ objects (no cross-correlation lock found). The exporter
mask_layers_zarr_exportis not in any public repository, so the earlier-runs-never-removed explanation is inferred from the data, not read from source.Method. Read-only, anonymous, no credentials. Full census across 71 volumes: https://github.com/dud8/fill-census — the 555-key manifest is reproducible from the listing command above.