Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion packages/documents.js/src/convert/composition.ts
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,10 @@ export interface UnifiedConversionOptions {
* The document-level surfaces of a pdf source to read, as pdf-codec's `ReadPdfOptions.surfaces` takes them: one boolean per surface (`outline`, `destinations`, `form`, `layers`, `structure`, `residue`, `images`), each read unless set to `false`, so omitting it reads everything as before. Independent of `pages`, and handed to the same hop: only the plan's first hop, and only when the conversion's source is pdf, so an intermediate pdf a route renders internally is always read whole. A skipped surface is never parsed and is absent from the read, so what the conversion builds from it is absent too: no `outline-N` entries in the tree's `destinations` table without `outline`, no destinations and no internal link constructs without `destinations` (a link to a URI is kept, and a read outline's `outline-N` entries stay, each without a `destination`), no form `contentControl` constructs without `form`, no layers table without `layers`, no structure entries in `definitions` without `structure`, no residue without `residue`, no images without `images` (no image stream is decoded, so a text-only conversion of an image-heavy page costs no image memory); skipping the outline also renumbers the reader-minted `destN` names internal link anchors carry, since the outline mints its direct destinations first. Two skips also change reconstruction: without `structure` a tagged PDF's headings, table cells and divisions are inferred from geometry as for an untagged file, and without `layers` content the default view hides cannot be told apart and is kept as if visible. See `ReadPdfOptions.surfaces` for exactly what each covers.
*/
readonly surfaces?: ReadPdfSurfaces;
/**
* The most bytes a pdf source's stream decoding may produce, as pdf-codec's `ReadPdfOptions.maxDecodedBytes` takes it: every content stream, font program, image, metadata packet and embedded file the read decodes is charged against it, and the first decode that would pass it throws `PdfDecodeBudgetError` (a `PdfParseError` with code `pdf/decode-budget-exceeded`, carrying the budget, the stream's object number and its filters) before materialising the rest, so a caller can tell a conversion that would cost too much from a malformed file. Handed to the same hop as `pages` and `surfaces`: only the plan's first hop, and only when the conversion's source is pdf, since an intermediate pdf a route renders internally is this package's own output rather than untrusted input. Absent, decoding is unbounded.
*/
readonly maxDecodedBytes?: number;
}

// What the fromPdf executor knows about a conversion's pages that a text build needs and the caller's options do not carry, passed to a text node's build alongside those options: the original one-based page number of each page the read returned, set only for a page-selected read (see UnifiedConversionOptions.pages), where a page's position in the output no longer says which page it is. Internal to the executors, never caller-supplied.
Expand Down Expand Up @@ -657,6 +661,7 @@ export function executeFromPdf(
sink: options?.sink,
pages: options?.pages,
surfaces: options?.surfaces,
maxDecodedBytes: options?.maxDecodedBytes,
});
const content = RECONSTRUCTORS[node.variant](layout, {
signal: options?.signal,
Expand Down Expand Up @@ -726,7 +731,7 @@ export function runCompositionPlan(
}
// onDocument fires on the last hop only: null out onDocument for every hop before the last, then pass the caller's options through unchanged on the final hop so the callback receives the package that actually produced the output bytes. This mirrors the convention the original hand-written composed functions (xlsxToPdf/pdfToXlsx/xlsxToMarkdown/markdownToXlsx) already followed — each forwarded onDocument to its own last hop, never the first — so a caller of xlsxToPdf still receives the odsToPdf hop's content+layout package rather than the intermediate xlsx->ods bridge's content-only one.
const isLastHop = i === plan.hops.length - 1;
// pages and surfaces select what to read of the caller's own pdf source, so only the hop that reads that source receives them: the first hop of a plan whose source is pdf. Every other hop's pdf (an intermediate rendering, as on a docx -> ods route through toPdf then fromPdf) is not the caller's document, and selecting its pages would silently drop content or throw for an index the caller never meant against it, and skipping its surfaces would drop what the caller's own source carried through it.
// pages, surfaces and maxDecodedBytes govern how the caller's own pdf source is read, so only the hop that reads that source receives them: the first hop of a plan whose source is pdf. Every other hop's pdf (an intermediate rendering, as on a docx -> ods route through toPdf then fromPdf) is not the caller's document, and selecting its pages would silently drop content or throw for an index the caller never meant against it, and skipping its surfaces would drop what the caller's own source carried through it.
const readsCallerPdf = i === 0 && hop.from === "pdf";
const hopOptions: UnifiedConversionOptions | undefined =
options === undefined
Expand All @@ -736,6 +741,9 @@ export function runCompositionPlan(
onDocument: isLastHop ? options.onDocument : undefined,
pages: readsCallerPdf ? options.pages : undefined,
surfaces: readsCallerPdf ? options.surfaces : undefined,
maxDecodedBytes: readsCallerPdf
? options.maxDecodedBytes
: undefined,
};
if (hop.executor === "toPdf") {
if (!isSourceContentFormat(hop.from)) {
Expand Down
131 changes: 131 additions & 0 deletions packages/documents.js/src/convert/from-pdf-decode-budget.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
import { DEFAULT_LAYOUT_FONT } from "document-schema.js";
import { bytesToBase64, encodePng } from "byte-codec";
import { LAYOUT_FORMAT_VERSION, writePdf } from "pdf-codec";
import { describe, expect, it } from "vitest";
import { PdfDecodeBudgetError, PdfParseError } from "../index";
import { convertDocument } from "./composition-to-pdf";
import {
pdfToDocx,
pdfToMarkdown,
readContentDocument,
readNativeDocumentTree,
} from "./from-pdf";
import { encodeMarkdownText } from "../markdown/text";

const IMAGE_SIDE_PX = 64;
const RGB_CHANNELS = 3;
// A budget far beyond what the fixture decodes (its image's samples, some kilobytes).
const AMPLE_BUDGET = 1 << 24;

// One page of text and one image, which writePdf stores FlateDecode-compressed: reading it decodes the image's samples. Built once at module level, so the writer's first-use cost falls outside every test's timeout.
function imagePdf(): Uint8Array<ArrayBuffer> {
const png = encodePng({
width: IMAGE_SIDE_PX,
height: IMAGE_SIDE_PX,
channels: RGB_CHANNELS,
data: new Uint8Array(IMAGE_SIDE_PX * IMAGE_SIDE_PX * RGB_CHANNELS),
});
return writePdf({
formatVersion: LAYOUT_FORMAT_VERSION,
metadata: {},
images: {
img1: {
format: "png",
base64: bytesToBase64(png),
widthPx: IMAGE_SIDE_PX,
heightPx: IMAGE_SIDE_PX,
},
},
pages: [
{
widthPt: 300,
heightPt: 300,
items: [
{
kind: "text",
text: "Caption text",
xPt: 20,
yPt: 250,
font: DEFAULT_LAYOUT_FONT,
sizePt: 12,
color: { r: 0, g: 0, b: 0 },
},
{
kind: "image",
imageId: "img1",
xPt: 20,
yPt: 100,
widthPt: 100,
heightPt: 100,
},
],
},
],
});
}

const IMAGE_PDF = imagePdf();

function refusal(run: () => unknown): unknown {
try {
run();
} catch (error) {
return error;
}
return undefined;
}

describe("the pdf conversions with maxDecodedBytes", () => {
const conversions: readonly {
readonly name: string;
readonly run: (maxDecodedBytes?: number) => unknown;
}[] = [
{
name: "pdfToMarkdown",
run: (maxDecodedBytes) => pdfToMarkdown(IMAGE_PDF, { maxDecodedBytes }),
},
{
name: "pdfToDocx",
run: (maxDecodedBytes) => pdfToDocx(IMAGE_PDF, { maxDecodedBytes }),
},
{
name: "convertDocument from pdf",
run: (maxDecodedBytes) =>
convertDocument("pdf", "ods", IMAGE_PDF, { maxDecodedBytes }),
},
{
name: "readNativeDocumentTree",
run: (maxDecodedBytes) =>
readNativeDocumentTree("pdf", IMAGE_PDF, { maxDecodedBytes }),
},
{
name: "readContentDocument",
run: (maxDecodedBytes) =>
readContentDocument("pdf", IMAGE_PDF, { maxDecodedBytes }),
},
];

for (const { name, run } of conversions) {
it(`${name} refuses a pdf that would decode past the budget with PdfDecodeBudgetError, and converts one within it`, () => {
const error = refusal(() => run(1));
expect(error).toBeInstanceOf(PdfDecodeBudgetError);
expect(error).toBeInstanceOf(PdfParseError);
expect(refusal(() => run(AMPLE_BUDGET))).toBeUndefined();
});
}

it("converts a pdf within the budget exactly as without one", () => {
expect(pdfToMarkdown(IMAGE_PDF, { maxDecodedBytes: AMPLE_BUDGET })).toEqual(
pdfToMarkdown(IMAGE_PDF),
);
});

it("budgets only the caller's own pdf: an intermediate pdf a route renders is not charged", () => {
const markdown = encodeMarkdownText("# Title\n\nBody.\n");
expect(
refusal(() =>
convertDocument("markdown", "ods", markdown, { maxDecodedBytes: 0 }),
),
).toBeUndefined();
});
});
12 changes: 12 additions & 0 deletions packages/documents.js/src/convert/from-pdf.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ const METADATA_SURFACES: Required<ReadPdfSurfaces> = {

export interface ReadDocumentMetadataOptions {
readonly signal?: AbortSignal;
// The decode budget for a pdf source, as UnifiedConversionOptions.maxDecodedBytes describes it; consulted only for a pdf source.
readonly maxDecodedBytes?: number;
}

// Pulls a LayoutMetadata out of any DocumentFormat's bytes. Both dispatch halves already carry each format's own cancellation policy (a one-time throwIfAborted for the synchronous single-pass readers, straight signal-forwarding for pdf, matching odtToDocx's own reasoning in convert.ts for the identical shape of read).
Expand All @@ -48,6 +50,7 @@ export function readDocumentMetadata(
signal: options?.signal,
pages: [],
surfaces: METADATA_SURFACES,
maxDecodedBytes: options?.maxDecodedBytes,
}).metadata;
}
return CONTENT_READERS[format](bytes, { signal: options?.signal }).metadata;
Expand All @@ -67,6 +70,10 @@ export interface ReadNativeDocumentTreeOptions {
* The document-level surfaces of a pdf source to read, exactly as UnifiedConversionOptions.surfaces describes them: one boolean per surface, each read unless set to `false`, independent of `pages`. Consulted only for a pdf source. A skipped surface is absent from the read, so the tree carries nothing built from it (see UnifiedConversionOptions.surfaces for exactly what: `outline-N` entries in `destinations`, destinations and internal link constructs, form `contentControl` constructs, the layers table, structure entries in `definitions`, residue, images), and reconstruction loses what it took from it: without `structure` a tagged PDF's headings, table cells and divisions are inferred from geometry, and without `layers` hidden-layer content is kept as if visible.
*/
readonly surfaces?: ReadPdfSurfaces;
/**
* The decode budget for a pdf source, exactly as UnifiedConversionOptions.maxDecodedBytes describes it: the first decode that would pass it throws `PdfDecodeBudgetError`. Consulted only for a pdf source.
*/
readonly maxDecodedBytes?: number;
}

// readContentDocument's own options are the identical shape — one options type for both, so a caller forwarding {signal, images, sink} through either entry point never needs to know which fields the OTHER one ignores.
Expand All @@ -84,6 +91,7 @@ function readNativeContent(
sink: options?.sink,
pages: options?.pages,
surfaces: options?.surfaces,
maxDecodedBytes: options?.maxDecodedBytes,
});
const content = reconstructWordprocessing(layout, {
signal: options?.signal,
Expand Down Expand Up @@ -142,6 +150,10 @@ export interface PdfToDocumentOptions {
* The document-level surfaces to read, exactly as UnifiedConversionOptions.surfaces describes them: one boolean per surface (`outline`, `destinations`, `form`, `layers`, `structure`, `residue`, `images`), each read unless set to `false`, independent of `pages`. A skipped surface is never parsed, and the conversion carries nothing built from it. Without `structure` a tagged PDF's headings, table cells and divisions are inferred from geometry as for an untagged file, and without `layers` content the default view hides is kept as if visible.
*/
readonly surfaces?: ReadPdfSurfaces;
/**
* The decode budget for the pdf source, exactly as UnifiedConversionOptions.maxDecodedBytes describes it: the first decode that would pass it throws `PdfDecodeBudgetError`.
*/
readonly maxDecodedBytes?: number;
// Called exactly once, synchronously, with the DocumentTree (the readPdf-produced layout, and the reconstructed content built from it) this conversion built internally, before the function returns its bytes — see convert.ts's DocumentToPdfOptions.onDocument for the mirrored X-to-PDF side of this same side channel.
readonly onDocument?: (pkg: DocumentTree) => void;
}
Expand Down
1 change: 1 addition & 0 deletions packages/documents.js/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ export type {
} from "pdf-codec";
export {
NOOP_DIAGNOSTIC_SINK,
PdfDecodeBudgetError,
PdfEncryptedError,
PdfPageSelectionError,
PdfParseError,
Expand Down
Loading
Loading