Skip to content

automation: torn-tail repair of a JSONL store has no fallback when the atomic replace can't complete #748

Closed

Description

@jussielo-amd

append_jsonl_record in crates/rocm-core/src/automation.rs repairs a torn tail (crash debris, or a write that failed partway) by rewriting the whole store to a temp file and renaming it over the original (write_atomic), rather than truncating the live file in place. That avoids a lock-free reader ever observing the file shrink and regrow mid-scan.

The repair itself has no fallback when that rewrite can't complete:

  • Disk pressure. The usual cause of a partial write is a full disk. Recovering now needs free space roughly equal to the whole store (never rotated or capped), where the old in-place truncate needed only one line's worth. Until that space exists, every append to the affected store fails and re-reads/re-writes the whole store each time, which can make a disk-full condition worse rather than degrading gracefully.
  • Windows. The same repair can fail with a sharing violation if any handle on the store lacks delete-sharing, or on a volume without POSIX rename semantics, wedging the store the same way.

rocmd treats an append error as fatal in apps/rocmd/src/service.rs (record_event at daemon start and in the watcher loop), so a store stuck in this state can keep the daemon from starting back up until space frees.

Two directions were raised in review:

  • Fall back to truncating the debris in place (under the store's lock) when the atomic replace itself fails, accepting the narrower shrink-window risk only on that rare path.
  • Reseal a torn tail with a sentinel the readers already skip, so repair becomes an O(record) append instead of an O(store) rewrite — no rename needed at all, at the cost of a store-format change.

Raised in review of #738, where the rename-based repair was first introduced to fix record loss under concurrent writers (ROCMAI-721). Accepted as a known tradeoff for that PR; tracked here for a follow-up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions