Repository navigation
Read every input to end of file, and accept - as standard input - #3
Merged
Merged
Conversation
The reported size was taken as the member's size. /proc files report 0 and
read as text, so /proc/cpuinfo was stored as an empty member with a valid
SHA-256 and exit 0 -- a backup that reports success and holds nothing. The
previous commit's regular-file check cannot catch this, because to stat these
are regular files.
The size is now a hint for choosing the segment length and nothing more. The
member is whatever reading to end of file returns, so size, segments and
SHA-256 always describe the same bytes: /proc/cpuinfo stores its 15,678 bytes,
a file that grows mid-run is stored as read rather than cut at its old length,
and a read error ends the member early with a message and exit 1 instead of
"short read" beside an index entry claiming the full size. When the seek says
0, the pre-pass count -- which already read the file through -- is the better
hint.
The probe that detected record geometry read the head of the file and then
seeked back. That seek is what made pipes impossible. The first segment is
now read before setup and geometry detected from it; it is exactly the bytes
the probe read, min(size, segment), so detection is unchanged and archives are
byte-identical.
With nothing seeking, "-" works as an input:
pg_dump mydb | gleipnir c -5 db.gl -
It is stored as "stdin", may appear once, is never hashed in advance so it
takes no part in deduplication, and is put in binary mode on Windows so CRLF
survives. A file literally named "-" is still reachable as "./-". The archive
itself must remain a real file, since segment offsets are taken from it.
progress() also computed total - done unsigned, which a grown file or stdin
can now overrun.
Archives byte-identical to main at -1 -5 -9 -f1 -f2, -t1 and -t4, -s1, and on
trees of empty files. On Linux, release and ASan+UBSan: /proc files, binary
and multi-segment stdin at -t4, empty stdin, a pipe that stalls mid-stream,
and cross-reading with main in both directions; sfuzz, tfuzz, gfuzz and fuzz
(648) pass. On Windows, binary stdin with CR, LF and ^Z round-trips exactly,
and sfuzz, tfuzz and gfuzz pass.
…raph The README said the whole input is buffered in RAM, that `readfile` reads the entire file, and that streaming would need the match model windowed. None of that has been true since the archive format: readfile no longer exists, input is read a segment at a time, and model_alloc sizes each worker's history -- the buffer the match model indexes into -- from the segment and frees it after. The paragraph was describing the v1 compressor. Checking that turned up the one place memory did follow the data. Each embedded DEFLATE stream is capped at ~128 MB of plaintext, but nothing capped their sum, and the model then sizes its history, arithmetic output and stored section from the expanded buffer. Measured on Linux, a 1.3 MB file of forty zlib streams of zeros peaked at 2,556 MB. A pg_dump -Fc of a repetitive table is that shape, and it is the example the previous commit leads with. Streams are now recovered only while the segment's plaintext stays within 4x the segment + 64 MB, 320 MB at the default -s64; past that a stream is left compressed, exactly as an unrecoverable one already is. The same file now peaks at 109 MB and round-trips exactly. Encoder-only: the decoder replays descriptors, so older builds read the output. Silesia is byte-identical before and after -- mozilla at -f2 and -5, ooffice and samba at -f2, and the whole corpus at -f2 -t4 -- so ordinary embedded DEFLATE sits well inside the budget. sfuzz, gfuzz and fuzz (648) pass. The paragraph now says what is true: memory follows -s, not input size, with the DEFLATE budget stated, and -s above the file size as the one way to hold a whole file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap left open by #2:
/procfiles report a size of 0, so they were stored as empty members with a valid SHA-256 and exit 0.The reported size is now only a hint for choosing the segment length. Each member is whatever reading to end of file actually returns, so its size, segments and SHA-256 always describe the same bytes.
/proc/cpuinfonow stores its 15,678 bytes. Before, it stored 0.-now reads standard input, for examplepg_dump db | gleipnir c -5 db.gl -.stdinand may appear once per command. It isn't deduplicated.progress()also had an unsignedtotal - done, which a growing file or stdin could overrun. That's clamped now.USAGE.md documents stdin and the read-to-EOF behaviour.
Verified
mainat-1 -5 -9 -f1 -f2,-t1/-t4,-s1, and on trees of empty files./procfiles (contents match exactly), binary and multi-segment stdin at-t4, empty stdin, a pipe that stalls mid-stream, and a file literally named-. sfuzz, tfuzz, gfuzz and fuzz (648) all pass.^Zand NUL bytes round-trips exactly. sfuzz, tfuzz and gfuzz pass.Second commit: cap on DEFLATE expansion, and the memory paragraph
pg_dump -Fcof a repetitive table produces.-s64). Past that, a stream stays compressed. The same file now peaks at 109 MB and round-trips exactly. Only the compressor changed, so older builds still read the archives.-f2and-5, ooffice and samba at-f2, and the whole corpus at-f2 -t4. sfuzz, gfuzz and fuzz (648) pass.