Repository navigation
Fix prepare(download=True) for custom download() datasets, add hf_is_prepared(), and optimize FlattenAndShuffleDataset - #43
Conversation
…plement is_prepared()
…nch multiprocess pools with lazy loading
bpiwowar
left a comment
There was a problem hiding this comment.
Thanks Victor, the prepare(download=True) fix is good – esp. the duplicated _find_cached_hashed_builder. A few small things:
- The description mentions
DatasetNotPreparedErrorfordownload=False, but it's not in the code. I'd rather leave it out, so could you just update the description? hf_is_prepared()sometimes says "cached" when it shouldn't. That matters now, because experimaestro relies on it to decide whether to download (details are inline)- In the shuffle fallback, catching every
OSErroris a bit wide.
| candidates = [ | ||
| d | ||
| for d in repo_dir.iterdir() | ||
| if d.is_dir() and (d.name == cfg_name or d.name.startswith(f"{cfg_name}-")) |
There was a problem hiding this comment.
startswith(f"{cfg_name}-") also matches another config like corpus-en when looking for corpus. Could you match on the hash suffix instead (-[0-9a-f]{16}$)? (Same pattern at line 82.)
There was a problem hiding this comment.
Done in e9e18c0: new _is_config_dir() helper matches the plain name or <name>-[0-9a-f]{16} exactly (verified the xxhash64 digest is 16 hex chars in datasets 5.0), used both in hf_is_prepared and _find_cached_hashed_builder. Tests cover corpus-en vs corpus.
| return builder | ||
|
|
||
|
|
||
| def hf_is_prepared( |
There was a problem hiding this comment.
revision is ignored here, so a pinned revision may never get downloaded. Maybe just return False when revision is set?
There was a problem hiding this comment.
Done in e9e18c0: hf_is_prepared takes revision and returns False (with a debug log) when it is set; HuggingFaceDataset.is_prepared() now forwards it.
| continue | ||
| if base is not None and base not in splits: | ||
| continue | ||
| if any(info_file.parent.glob("*.arrow")): |
There was a problem hiding this comment.
This accepts any .arrow file. Could you check for the requested split's shards instead (f"*-{base}*.arrow")?
There was a problem hiding this comment.
Done in e9e18c0, with a slightly stricter pattern than *-{base}*.arrow, which would still match e.g. fiqa_en_dev when looking for fiqa_en. Shards must match -{split}(-\d{5}-of-\d{5})?\.arrow$ — the two names datasets produces after renaming (<builder>-<split>.arrow / <builder>-<split>-SSSSS-of-NNNNN.arrow). When no single split is requested, every split in dataset_info.json must have its shards.
| ) | ||
| try: | ||
| shuffled.save_to_disk(self.output_dir, num_proc=num_proc) | ||
| except (ModuleNotFoundError, OSError) as e: |
There was a problem hiding this comment.
A full disk would also hit this and then retry single-process for a long time. Could you catch only the pool errors you actually saw?
There was a problem hiding this comment.
Done in e9e18c0: only ModuleNotFoundError (the multiprocess.queues failure we saw) triggers the fallback; OSError propagates (test with ENOSPC). The partially written output_dir is also removed before the single-process retry.
- hf_is_prepared: match hashed configs on the exact 16-hex suffix (corpus-en is not a variant of corpus); same in _find_cached_hashed_builder - hf_is_prepared: require Arrow shards of the requested split (exact name, single or -NNNNN-of-NNNNN), or of every split when none is requested - hf_is_prepared: return False when a revision is pinned; forward revision from HuggingFaceDataset.is_prepared - FlattenAndShuffleDataset: only fall back on ModuleNotFoundError (not OSError, e.g. ENOSPC) and wipe partial output before the retry - debug logging on every 'not prepared' path
This PR resolves three related issues when preparing and preprocessing HuggingFace-backed datasets on HPC clusters (where login nodes have internet access and compute nodes run with
HF_DATASETS_OFFLINE=1):AbstractDataset.prepare(download=True)bypassedBase.download(): Datasets that manage their own download viaBase.download()rather than__resources__(such asHuggingFaceDatasetand composite datasets likeEmbeddingsFineTuningSamples) were never downloaded duringprepare_dataset(..., download=True).HuggingFaceDataset.is_prepared():HuggingFaceDatasetpreviously inheritedBase.is_prepared()(which only checks__resources__and always returnedTrue), making it impossible to distinguish cached vs. uncached splits without triggering network calls.FlattenAndShuffleDatasetredundant I/O &multiprocesspool failures on shared filesystems:FlattenAndShuffleDataset.execute()previously wrote the entire shuffled dataset twice (once toHF_DATASETS_CACHEviaflatten_indices()and once tooutput_dirviasave_to_disk()), and could fail withModuleNotFoundError: No module named 'multiprocess.queues'if.venvmetadata changed on Lustre during the long table-loading phase.Detailed Changes
1. Call
ds.download()inAbstractDataset.prepare()(src/datamaestro/definitions.py)AbstractDataset.prepare(download=True), when the built config is aBaseinstance, callds.download()(which runs the resource downloads and anydownload()override, e.g.HuggingFaceDataset); otherwise keep the previousself._download_or_raise()path.download=Falsebehaviour is unchanged.2. Read-only
hf_is_prepared()&HuggingFaceDataset.is_prepared()(src/datamaestro/download/huggingface.py,src/datamaestro/data/huggingface.py)hf_is_prepared(source, name, data_files, split, revision), which inspectsdatasets.config.HF_DATASETS_CACHEread-only. It errs on the side ofFalse(a false "cached" means the data is never downloaded):<name>or split-restricted<name>-<16 hex digits>(socorpus-enis not mistaken forcorpus);dataset_info.jsonmust list the requested split, and Arrow shards of that split must exist (<builder>-<split>.arrowor<builder>-<split>-NNNNN-of-NNNNN.arrow); without a single split, every recorded split must have its shards;revisionis pinned, returnsFalse(the cache layout does not let us verify the revision read-only), sodownload()is always run — it is idempotent and works offline from the cache._find_cached_hashed_builder(): removed the duplicated definition, uses the same exact hashed-config matching, and takessplitto prefer the cached config containing the requested split.HuggingFaceDataset.is_prepared()(forwardingrevision) and wrappedHuggingFaceDataset.download()failures inDownloadError.3. Faster & More Resilient
FlattenAndShuffleDataset(src/datamaestro/data/huggingface.py)flatten_indices():Dataset.save_to_disk()already materializes and flattens_indicesinto contiguous Arrow shards inself.output_dir, so callingshuffled.flatten_indices()beforehand was writing the entire dataset to disk twice (and pollutingHF_DATASETS_CACHEwith multi-GB temporary cache files).multiprocesssubmodules at the start ofexecute(), before the long concatenation step, preventing lazy-import failures on shared cluster filesystems (e.g. Lustre).ModuleNotFoundErrorfromsave_to_disk(num_proc>1)triggers the fallback (other errors such asOSError/ENOSPC propagate). Before retrying, the partially writtenoutput_diris removed so the retry starts clean.Test Plan (
src/datamaestro/test/test_hf.py)TestHuggingFaceDatasetDownload:prepare_dataset(..., download=True)invokeshf_download_and_prepareonHuggingFaceDataset; download failures raiseDownloadError.HuggingFaceDataset.is_prepared()across cold cache, streaming mode, non-matching split cache, and matching split cache.TestHfIsPrepared: prefixed configs (corpus-en) ignored; shards of other/prefix-sharing splits (fiqa_en_dev) rejected; single- and multi-shard layouts and sliced splits accepted; no-split requires all splits; pinnedrevision→False(also viaHuggingFaceDataset).TestHuggingFaceOfflineHashedFallback: hashed-config fallback, and prefixed configs are not picked.TestFlattenAndShuffleDataset: noflatten_indices; fallback onModuleNotFoundErrorwith partial output wiped; no fallback onOSError(ENOSPC).