Repository navigation
feat(pdf-codec): list a PDF's images without decoding them, and decode one at a time - #1817
Merged
Merged
Conversation
…hout an interleaved copy encodePng built one interleaved copy of the whole image before filtering it, and the palette search allocated a full index plane before it knew a palette would fit. For a 2360x1309 picture with a soft mask that was 12 MB and 3 MB more than the encode needs, at the moment its filtered rows and deflate buffer are also live. filterScanlineRows filters rows read through a callback; encodePng now writes each row straight from its colour and alpha planes into the filter's own row buffer (a missing sample still reads as 0), and finds the palette before storing any index. The bytes are unchanged: the existing known answers still hold, and new ones recorded from the previous encoder pin a short raster with alpha, a short grey raster and a palette with alpha. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted size Inflating a Flate image stream grew fflate's output buffer by doubling and then copied it to its exact length, so a 9 MB raster briefly cost about 27 MB. An image dictionary states how long its samples are, so readImageXObject now passes that length to decodeStream, which inflates a final FlateDecode (through its predictor's row framing) into one buffer of that size plus a spare byte. The bytes are the same either way: a stream that does not fit (the spare byte fills), does not decode, or is expected to be longer than deflate could possibly expand its data is inflated exactly as before, so a dictionary claiming a huge image over a tiny stream never buys a huge buffer. The content interpreter also imports COLOR_BLACK from document- schema.js/color rather than the package barrel, which loaded every content and layout schema (about 7 MB of heap) for one constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e one at a time readPdf decodes every image a read page draws in one synchronous call and keeps each in its registry as a base64 string, so a picture-heavy file costs memory in proportion to its number of images as well as its largest: a deck shaped like the one that exceeded a 128 MB Worker isolate peaked at about 140 MB of heap and array buffers in 5.8, and 526 MB in 4.5.2. listPdfImages lists every image the pages draw by reference (an image XObject's object number, or an inline image's page and position) with its placements, exactly as readPdf's image items state them, and what its dictionary declares, without decoding any image stream. decodePdfImage decodes one image by that locator to exactly the bytes, format, size and imageId readPdf registers for it, without interpreting any page again. openPdfImages does both against one opened document. Listing and decoding that deck's 108 images one at a time peaks at about 56 MB, measured by scripts/measure-image-memory.mjs against a generated fixture of the same shape. Page geometry, the header check and page selection move out of read.ts into page-frame.ts and read-open.ts so the listing places items as readPdf does without loading readPdf's text pipeline; read.ts keeps exporting normalizeRotation and pageRotationTransform. Tests pin every decoded image of two generated fixtures to the digests pdf-codec 4.5.2 produced for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
encodePng now builds the palette and its index plane only after the truecolour pass has finished with its filtered rows and deflate buffer, so the two are never held at once. The result is unchanged: the same comparison picks the smaller encoding. Also states filterScanlineRows' contract plainly: a byte the callback leaves unwritten keeps the previous row's value, which is how filterScanlines treats a raw buffer shorter than its rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An image dictionary's declared size now sizes the inflate buffer only when readImageXObject would go on to decode that image: its width, height and pixel count within encodePng's limits, and a bit depth it reads. The size is worked out only when FlateDecode is the stream's last filter. A soft mask is checked against its image's size and depth before it is decoded, so a mask that cannot be used is never inflated. A stream shorter than its declared size is copied out, so the buffer sized for the declaration is not kept alive behind a view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pdf-codec/read-images is now an explicit package entry, with a typesVersions mapping, and the smoke and workerd suites read an image through it. PdfImageLocator moves to its own module with PdfImageLocatorSchema, so a stored locator can be validated. Per-image reading also changes: - A decoded image's bytes are its own, never a view onto the file. - Two references resolving to one stream are listed once. - An inline locator with a negative or fractional index throws PdfImageLocatorError. - Finding an inline image again no longer re-reports its page's diagnostics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README now leads its per-image example with openPdfImages from pdf-codec/read-images. It gives the measured peaks both with a collection forced before each image and left to the collector, which is what a Worker sees. It also states the failure policy for locators, that two entries can decode to one imageId, and that an open encrypted document keeps the streams it has decrypted. The measuring script says it reports heap in use, not committed heap, and that its timings are taken under sampling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dictionary can state any size up to its stream's deflate bound, and a Worker isolate ends rather than throws on an allocation that large, so an expected length past 32 MiB is inflated into a growable buffer instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he size hint Sizing an image's inflate buffer resolved its colour space a second time, reporting a missing reference twice and decoding an Indexed lookup table twice; it now resolves only the space's shape, through a resolver the decode shares. A soft mask of the wrong size is decoded again, so its filters still report what they find, but into a buffer that grows with its stream rather than one its dictionary sized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An opened PdfImages keeps the last page's inline images, so decoding them one after another no longer interprets the page once per image. Documents that an xobject locator is not checked against a listing and decodes any image stream it names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und's runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
🎉 This PR is included in version 5.9.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a caller list a PDF's images without decoding them, and decode them one at a time, so a picture-heavy PDF can be read inside a memory-limited runtime such as a 128 MB Worker isolate.
What changes
pdf-codec/read-images(new entry):openPdfImages(bytes)lists each image by a plain-dataPdfImageLocator(an XObject's object reference, or an inline image's page and position) with its placements and declared dimensions, without decoding anything;decodePdfImage/ the handle'sdecode(locator)decodes one image on request without re-running page layout.PdfImageLocatorSchemavalidates a stored locator.Array.from; an inflate buffer is sized from the image's declared size only when the stream can plausibly produce it and never above 32 MiB; a page's inline images are interpreted once per handle.byte-codec: PNG rows are filtered from a callback and encoded without an interleaved copy; the truecolour form is encoded before the palette so the two are never held together.src/read-images.test.tsagainst digests recorded from 4.5.2), andreadPdf's output and diagnostics are as before. All API changes are additive.Memory (generated image-heavy fixture, peak
heapUsed + arrayBuffers)openPdfImages)readPdf(this branch)readPdfin pdf-codec 5.8readPdfin pdf-codec 4.5.2Known limit, documented: an encrypted PDF keeps each decrypted image stream it resolves for the handle's lifetime.
Checks
pdf-codec 3105 and byte-codec 655 unit tests, workerd and smoke tests, lint, typecheck, attw and knip pass locally. Expected release on merge: pdf-codec 5.9.0 and byte-codec 2.1.0 (dependents patch-bumped).
🤖 Generated with Claude Code