Skip to content

Publish commits atomically, and run tests in CI - #2

Merged
SpookyJumpyBeans merged 2 commits into
mainfrom
fix/atomic-commit-publish
Oct 3, 2026
Merged

SpookyJumpyBeans merged 2 commits into
mainfrom
fix/atomic-commit-publish

Conversation

@SpookyJumpyBeans

Copy link
Copy Markdown
Owner

Two commits: a correctness fix for how commits are published, and CI.

The bug

A commit needs two properties at once:

  • Exclusivity. If two writers race for the same version, exactly one wins.
  • Atomic visibility. A reader that sees a version sees all of it.

The old writer created <version>.json with O_CREAT|O_EXCL and then wrote into it. That gave exclusivity but not atomic visibility. For a moment the version existed and was empty. A reader in that window reported it as the latest version, and once the writer finished, the same version number returned different data. A version that changes after it becomes visible breaks the guarantee the whole format rests on.

The fix

Write the commit to a uniquely named temp file, fsync it, then hard-link it to the version name. link() is atomic and fails if the target exists, so the losing writer still gets ConcurrentWriteError, and the version name can only ever point at complete, durable contents.

Also:

  • The log directory is fsynced where the platform allows it, so the new name survives a crash.
  • A buffered write replaces os.write, whose return value was ignored and can be short.
  • Temp names don't match the version pattern, so readers never see them, and vacuum removes any that a crashed writer left behind.

Tests

The old concurrency test simulated a stale writer one step at a time. The new ones use real threads:

  • Racing writers produce exactly one winner.
  • Writers that retry after losing lose nothing and duplicate nothing, with no gaps in the log.
  • A failed publish leaves no version and no temp file.
  • Temp files are invisible to readers and cleaned up by vacuum.

The regression test freezes a writer as it starts writing commit bytes, whether through os.write or a file object, and checks what a reader sees. It passes on this code and fails on the original. I checked a random stress test against the original too, and it passed 6 times out of 6: the window lasted microseconds, so random timing almost never hit it. That test is kept as a stress check, and its docstring says it can't catch this race on its own.

19 tests, up from 12.

CI

Python 3.11 to 3.13 on Linux, plus 3.13 on Windows and macOS. Since commits now depend on hard links, each OS runs against its native filesystem: ext4, NTFS and APFS. Each job also round-trips a table through the CLI. Adds a status badge to the README.

A commit needs exclusivity, so one of two racing writers wins, and
atomic visibility, so a reader that sees a version sees all of it.
Creating the version file with O_CREAT|O_EXCL and then writing into it
gave only the first. For a moment the version existed but was empty: a
reader in that window reported it as the latest version, and once the
writer finished the same version number returned different data.

The commit now goes to a uniquely named temp file, is fsynced, and is
hard-linked to the version name. link() is atomic and fails if the
target exists, so exclusivity is unchanged and the version name only
ever points at complete contents. The log directory is fsynced on
platforms that allow it, and a buffered write replaces os.write, whose
return value was ignored and can be short. Temp names do not match the
version pattern, so readers never see them; vacuum removes any left by
a crashed writer.

The old concurrency test simulated a stale writer one step at a time.
These use real threads: racing writers produce exactly one winner,
retrying writers lose and duplicate nothing with no gaps in the log,
and a failed publish leaves no version and no temp file behind.

The regression test freezes a writer as it starts writing commit bytes
and checks what a reader sees. It passes on this code and fails on the
original. A random stress test kept as well passed 6 of 6 times against
the original, because the window lasted microseconds, and says so.
Python 3.11 to 3.13 on Linux, plus 3.13 on Windows and macOS. Commits
are published with a hard link, so each OS runs against its native
filesystem: ext4, NTFS and APFS. Each job also round-trips a table
through the command line.
@SpookyJumpyBeans
SpookyJumpyBeans merged commit a461081 into main Oct 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant