Repository navigation
A plan for resuming a source by its content - #199
Merged
Merged
Conversation
Where in a growing source does the store's data end? Today's two answers (an offset checked by decompressing the whole store, and a hash set of every overlapping line) both cost the store or the overlap on every resume, both anchor on a head that retention removes, and both run under a unit that restarts on any exit. The plan anchors on the store's tail, finds the end of it in the file with a byte match inside a region narrowed by bisection, keeps a derived hint that is verified on every use, and says what a resume does when it cannot decide: make progress, because an exit is a restart loop. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
No timestamp found was listed among the errors that stop the unit. For a followed file it is a state, not an error: the file can gain a stamp later, and a unit that has stopped never looks at it again. A followed file that does not exist yet is already waited for; a file with no stamp yet is the same kind of thing. Today the absence exits and systemd restarts it every two seconds, each time re-scanning the same lines, which recovers only by accident. The plan now holds unstamped lines (bounded), stamps them by arrival once with a note, and keeps reading, so a file that gains stamps is read with them. Only a regex that does not compile, or an unknown format, stops the unit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A file that starts with unstamped content, and gets its first stamp later, does not need that content held while it waits: it is a byte range of a file that already has it. The hint records "awaiting a stamp, scanned through X" with the file's identity, size and a hash of its first KiB, so a restart continues the scan instead of rereading it; when a stamp turns up at S the range [0, S) is streamed from the file with that stamp's time. Nothing is imported while it waits, and a file that never carries a stamp looks like a quiet one, so the wait is made visible. Arrival stamping becomes a declaration for sources that have no stamps, not a fallback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tamp For a live file with no stamp, waiting is free: the file is still there and may gain one. For a rotated file it is not. The tail holds the descriptor, so a rotation leaves it reading the rotated file to its end, and that is the last time anything reads it: logrotate removes it in time, and at startup only .1 and .0 are looked for. A wait that cannot end is a loss that has not happened yet. A file closed while awaiting a stamp is imported with its modification time, the end of its content's life and the reference the extractor already uses for a stamp with no year; one note says which file, how much, and what time it was given. The hint also records which timestamp declaration it scanned with, so fixing the declaration takes effect on a file that is still there. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… closed file While a live source awaits a stamp the follower holds the descriptor, the read buffer, the offset scanned through, the first 256 bytes of the line being read and the hint: constant, whatever the file holds. Today's Stamper.leading holds up to 1000 whole lines, bounded by their number and not their size. A file closed without ever carrying a stamp is imported at its modification time; that question is decided and leaves the list of open ones. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The aligned candidate holds only until the first rotation: after it the same log is a different file whose byte 0 sits at the tape end of the moment it was opened. The hint records each file's tape_start, so the candidate is the tape end minus it, and the search is left for a store with no hint yet. A store fed by a set of files (a glob, day files) keeps its existing handling; it is a bulk use, not the resume. A live source with no stamp waits, for ever if need be. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A set of files into one store is an initial or bulk import. Several live files feeding one store is not a use, and the writer lock already allows one writer per store, so the resume can assume a single live source and the files its rotations leave behind. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/plans/import-resume.md, plus its entry in the plans index. Docs only, no code.It settles how a growing source is resumed before any of it is built, because it replaces how every
importresumes:size()is relative to the current head, so today's offset is only right before the first head-drop.<name>.resume), verified against the bytes on every use, makes a repeat resume O(1).Restart=always,RestartSec=2, so a resume that exits is a loop that repeats the expensive part. An ambiguous resume makes progress and records which action it took; a permanent error exits withEX_CONFIGand the units gainRestartPreventExitStatus=78.It replaces
verify_prefix(and--quick), thefirst stamp == store's firstgate, and the regrown-file case ofoverlap_line_countsandcatch_up. The open questions are listed at the end.🤖 Generated with Claude Code