Skip to content

A followed file with no timestamp yet is waited for, not held and not an error - #204

Open
torstei wants to merge 1 commit into
mainfrom
feature/follow-waits-for-a-stamp
Open

torstei wants to merge 1 commit into
mainfrom
feature/follow-waits-for-a-stamp

Conversation

@torstei

@torstei torstei commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

import --follow (and file-intake) used to bail when the first 1000 lines of a file held no timestamp. The unit restarted every two seconds, and the lines were held in memory meanwhile. Now:

  • A file with no stamp yet is waited for, holding nothing. When the first stamp arrives, the unstamped prefix is streamed from the file in 256 KiB blocks with that stamp's time.
  • A file rotated away without ever having had a stamp is imported with its modification time.
  • A note is printed at 100 MiB and at each doubling, since a never-stamped file looks like a quiet one.
  • Copytruncate re-arms the wait when the store is still empty.

Evidence

  • 286 MiB unstamped file, stamp appended 4 s after start: peak RSS 12.5 MB, prefix imported at the stamp's time (1145 chunks).
  • 3000-line unstamped file: before, error exit in 0.1 s and 0 of 3002 lines stored. After, 3002 of 3002.
  • 6 new unit tests; each part mutated and caught by its own test.

Known limits

  • A restart while waiting rescans from the start of the file (persisting the position needs the parked hint file).
  • A live file that never gets a stamp waits forever, by decision.

Docs: man page, deployment guide, plan status.

🤖 Generated with Claude Code

… an error

The live importer used to bail when the first 1000 lines held no stamp, which
the unit restarted every two seconds, and held those lines in memory. Now it
reads on holding nothing, and when the first stamp arrives streams the
unstamped prefix from the file with that stamp's time. A file rotated away
without ever having a stamp is imported with its modification time. Notes at
100 MiB and each doubling make the wait visible.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant