Skip to content

DiskDict: atomic writes and one-shot reads; ReusableOptimizer race fix (#89) - #90

Merged
jcmgray merged 5 commits into
jcmgray:mainfrom
atpneonex:atomic-cache-fix
Nov 6, 2026
Merged

jcmgray merged 5 commits into
jcmgray:mainfrom
atpneonex:atomic-cache-fix

Conversation

@atpneonex

Copy link
Copy Markdown
Contributor

Fixes #89.

Problem

DiskDict exposes partly written files when multiple processes share the same cache directory.

Changes

DiskDict (cotengra/utils.py)

  • Writes are now atomic replacements. This ensures no reader see a half-written file. On Windows the replacements may fail, so the OSError is caught and turned into a warning.

ReusableOptimizer (cotengra/reusable.py)

  • _fname helper
  • The original check-then-read behaviors (_maybe_run_optimizer, update_from_tree) now use a single .get.

Tests

  • New tests/test_diskcache.py: regular dict behavior checks and a concurrent read/write stress test.
  • New tests in tests/test_optimizers.py: cache_only with a missing entry should fail; corrupt entries are recomputed.

@jcmgray

jcmgray commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Thanks for this, I've pushed a few small followup changes:

  • fixed the windows CI (test only)
  • use threads to stress test so KeyError can be caught directly.
  • just removed max_retries and retry_delay since these are internal and no longer used.
  • some other stylistic changes to do with comments etc.

@jcmgray
jcmgray merged commit 153f232 into jcmgray:main Nov 6, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReusableHyperOptimizer crashes when several processes share the same on-disk cache

2 participants