Skip to content

Fix prepare(download=True) for custom download() datasets, add hf_is_prepared(), and optimize FlattenAndShuffleDataset - #43

Merged
bpiwowar merged 4 commits into
experimaestro:masterfrom
VictorMorand:fix/dowload-or-raise
Nov 3, 2026
Merged

bpiwowar merged 4 commits into
experimaestro:masterfrom
VictorMorand:fix/dowload-or-raise

Conversation

@VictorMorand

@VictorMorand VictorMorand commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

This PR resolves three related issues when preparing and preprocessing HuggingFace-backed datasets on HPC clusters (where login nodes have internet access and compute nodes run with HF_DATASETS_OFFLINE=1):

  1. AbstractDataset.prepare(download=True) bypassed Base.download(): Datasets that manage their own download via Base.download() rather than __resources__ (such as HuggingFaceDataset and composite datasets like EmbeddingsFineTuningSamples) were never downloaded during prepare_dataset(..., download=True).
  2. Missing read-only HuggingFaceDataset.is_prepared(): HuggingFaceDataset previously inherited Base.is_prepared() (which only checks __resources__ and always returned True), making it impossible to distinguish cached vs. uncached splits without triggering network calls.
  3. FlattenAndShuffleDataset redundant I/O & multiprocess pool failures on shared filesystems: FlattenAndShuffleDataset.execute() previously wrote the entire shuffled dataset twice (once to HF_DATASETS_CACHE via flatten_indices() and once to output_dir via save_to_disk()), and could fail with ModuleNotFoundError: No module named 'multiprocess.queues' if .venv metadata changed on Lustre during the long table-loading phase.

Detailed Changes

1. Call ds.download() in AbstractDataset.prepare() (src/datamaestro/definitions.py)

  • In AbstractDataset.prepare(download=True), when the built config is a Base instance, call ds.download() (which runs the resource downloads and any download() override, e.g. HuggingFaceDataset); otherwise keep the previous self._download_or_raise() path.
  • download=False behaviour is unchanged.

2. Read-only hf_is_prepared() & HuggingFaceDataset.is_prepared() (src/datamaestro/download/huggingface.py, src/datamaestro/data/huggingface.py)

  • Added hf_is_prepared(source, name, data_files, split, revision), which inspects datasets.config.HF_DATASETS_CACHE read-only. It errs on the side of False (a false "cached" means the data is never downloaded):
    • config directories are matched exactly: plain <name> or split-restricted <name>-<16 hex digits> (so corpus-en is not mistaken for corpus);
    • dataset_info.json must list the requested split, and Arrow shards of that split must exist (<builder>-<split>.arrow or <builder>-<split>-NNNNN-of-NNNNN.arrow); without a single split, every recorded split must have its shards;
    • when revision is pinned, returns False (the cache layout does not let us verify the revision read-only), so download() is always run — it is idempotent and works offline from the cache.
  • _find_cached_hashed_builder(): removed the duplicated definition, uses the same exact hashed-config matching, and takes split to prefer the cached config containing the requested split.
  • Implemented HuggingFaceDataset.is_prepared() (forwarding revision) and wrapped HuggingFaceDataset.download() failures in DownloadError.

3. Faster & More Resilient FlattenAndShuffleDataset (src/datamaestro/data/huggingface.py)

  • Removed redundant flatten_indices(): Dataset.save_to_disk() already materializes and flattens _indices into contiguous Arrow shards in self.output_dir, so calling shuffled.flatten_indices() beforehand was writing the entire dataset to disk twice (and polluting HF_DATASETS_CACHE with multi-GB temporary cache files).
  • Pre-imported multiprocess submodules at the start of execute(), before the long concatenation step, preventing lazy-import failures on shared cluster filesystems (e.g. Lustre).
  • Single-process fallback, narrowly scoped: only a ModuleNotFoundError from save_to_disk(num_proc>1) triggers the fallback (other errors such as OSError/ENOSPC propagate). Before retrying, the partially written output_dir is removed so the retry starts clean.

Test Plan (src/datamaestro/test/test_hf.py)

  • TestHuggingFaceDatasetDownload:
    • prepare_dataset(..., download=True) invokes hf_download_and_prepare on HuggingFaceDataset; download failures raise DownloadError.
    • HuggingFaceDataset.is_prepared() across cold cache, streaming mode, non-matching split cache, and matching split cache.
  • TestHfIsPrepared: prefixed configs (corpus-en) ignored; shards of other/prefix-sharing splits (fiqa_en_dev) rejected; single- and multi-shard layouts and sliced splits accepted; no-split requires all splits; pinned revision → False (also via HuggingFaceDataset).
  • TestHuggingFaceOfflineHashedFallback: hashed-config fallback, and prefixed configs are not picked.
  • TestFlattenAndShuffleDataset: no flatten_indices; fallback on ModuleNotFoundError with partial output wiped; no fallback on OSError (ENOSPC).

@bpiwowar bpiwowar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Victor, the prepare(download=True) fix is good – esp. the duplicated _find_cached_hashed_builder. A few small things:

  • The description mentions DatasetNotPreparedError for download=False, but it's not in the code. I'd rather leave it out, so could you just update the description?
  • hf_is_prepared() sometimes says "cached" when it shouldn't. That matters now, because experimaestro relies on it to decide whether to download (details are inline)
  • In the shuffle fallback, catching every OSError is a bit wide.

Comment thread src/datamaestro/download/huggingface.py Outdated
candidates = [
d
for d in repo_dir.iterdir()
if d.is_dir() and (d.name == cfg_name or d.name.startswith(f"{cfg_name}-"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

startswith(f"{cfg_name}-") also matches another config like corpus-en when looking for corpus. Could you match on the hash suffix instead (-[0-9a-f]{16}$)? (Same pattern at line 82.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e9e18c0: new _is_config_dir() helper matches the plain name or <name>-[0-9a-f]{16} exactly (verified the xxhash64 digest is 16 hex chars in datasets 5.0), used both in hf_is_prepared and _find_cached_hashed_builder. Tests cover corpus-en vs corpus.

return builder


def hf_is_prepared(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revision is ignored here, so a pinned revision may never get downloaded. Maybe just return False when revision is set?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e9e18c0: hf_is_prepared takes revision and returns False (with a debug log) when it is set; HuggingFaceDataset.is_prepared() now forwards it.

Comment thread src/datamaestro/download/huggingface.py Outdated
continue
if base is not None and base not in splits:
continue
if any(info_file.parent.glob("*.arrow")):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This accepts any .arrow file. Could you check for the requested split's shards instead (f"*-{base}*.arrow")?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e9e18c0, with a slightly stricter pattern than *-{base}*.arrow, which would still match e.g. fiqa_en_dev when looking for fiqa_en. Shards must match -{split}(-\d{5}-of-\d{5})?\.arrow$ — the two names datasets produces after renaming (<builder>-<split>.arrow / <builder>-<split>-SSSSS-of-NNNNN.arrow). When no single split is requested, every split in dataset_info.json must have its shards.

Comment thread src/datamaestro/data/huggingface.py Outdated
)
try:
shuffled.save_to_disk(self.output_dir, num_proc=num_proc)
except (ModuleNotFoundError, OSError) as e:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A full disk would also hit this and then retry single-process for a long time. Could you catch only the pool errors you actually saw?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e9e18c0: only ModuleNotFoundError (the multiprocess.queues failure we saw) triggers the fallback; OSError propagates (test with ENOSPC). The partially written output_dir is also removed before the single-process retry.

- hf_is_prepared: match hashed configs on the exact 16-hex suffix
  (corpus-en is not a variant of corpus); same in _find_cached_hashed_builder
- hf_is_prepared: require Arrow shards of the requested split (exact name,
  single or -NNNNN-of-NNNNN), or of every split when none is requested
- hf_is_prepared: return False when a revision is pinned; forward revision
  from HuggingFaceDataset.is_prepared
- FlattenAndShuffleDataset: only fall back on ModuleNotFoundError (not
  OSError, e.g. ENOSPC) and wipe partial output before the retry
- debug logging on every 'not prepared' path
@bpiwowar
bpiwowar merged commit 0321ad7 into experimaestro:master Nov 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants