A followed file is resumed where the store ends in it, not by its first stamp

[torstei]

Fixes data loss in the live importer (import --follow, file-intake) on restart.

Problem

On start, catch_up compared the file's first timestamp with the store's first chunk. A file that began before the store was skipped entirely, and the tail started at its end:

timberfs: skipping app.log — it starts 12:00:00, before the oldest data in s.log (12:32:06); …

Once retention has dropped the store's head, a long-lived file always begins before it. So a restart (a package upgrade through --exit-on-upgrade, a reboot, a crash) of a follower on a store with retention loses every line written while it was down. A running follower is not affected, only a restarted one.

Reproduced end to end with the released binary: a 6,000-line file followed into a store, retention dropped the head while the follower was stopped, 100 lines written to the file meanwhile, follower restarted: 0 of the 100 reached the store.

Change

catch_up first asks where the store's data ends in the file, by bytes: the store's last 16 KiB against the file's bytes just before that offset (the byte-verified resume import already has since #201, now one function, import::resume_point). If it verifies, the tail carries on from there: no scan of the file and no hash set of the store. It uses the tape end (dropped + size), so it holds for a store whose head retention dropped, which is exactly what the first-stamp comparison got wrong.

• It never errors on an ambiguity (the live importer passes claims_same = false), because under Restart=always an exit is a restart loop. Where it cannot tell, the existing line-by-line re-sync runs as before.
• A rotated file found at startup (.1, .0) is finished the same way and its trailing fragment committed.
• import's behaviour is unchanged; the refactor only lifts its match into a function.

Checked with the real binaries

Same scenario, main against this branch:

	restart log	of the 100 lines written while down
main	skipping app.log — it starts 12:00:00, before the oldest data in s.log	0
this branch	s.log ends at byte 574890 of it (579890 bytes); resuming there	100

Tests

Five in follow.rs, each simulating a process start exactly as cmd_follow does it (open the store, catch_up the rotated files and then the live one, drain): a restart after retention dropped the head picks up what was written while down; a restart with nothing new changes nothing; a line still being written when it stopped arrives whole; a file rotated while down is finished and then the new one is read (with a trimmed store); a file that does not continue the store is re-synced line by line as before. With the new resume disabled, exactly the two data-loss tests fail and the other three still pass.

The man page, the deployment guide and the two unit files described the startup scan as the normal case; they now say it is the fallback.

Not in this PR, and a remaining hazard

• The fallback still has the old skip. If the byte comparison cannot tell (a store that lost its tail in a crash and so is ahead of anything it can verify, a file replaced by a different one), catch_up takes the old path, where t0 < first still skips the file and starts at its end. Stage 2 (the hint file, with anchors) narrows when that happens, and the plan's failure policy says what the fallback should do instead; I did not change it here.
• Rotation is not made cheap here. After a rotation the store ends with the old file's bytes, so the new file does not line up and takes the old path. That is what the recorded anchors in the resume plan are for.
• Rotated files are still found by the .1/.0 names only.

🤖 Generated with Claude Code