Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/documents.js/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -959,7 +959,7 @@ To run a single test file: `pnpm vitest run src/path/to/file.test.ts`.
- **Font resolution uses a real registry, standard 14 as last resort.** A family with no embedded/caller/vendored face (Aptos, third-party typefaces) renders through the nearest standard-14 face with a width-correction factor — expect a visual approximation, not line-identical output. MathML formula rendering is separate: it embeds STIX Two Math, not registry-resolvable.
- **Justified paragraphs stretch inter-word gaps** in all three layout engines (`engine.ts`, `slides.ts`, `sheets.ts`). `justifyLineGapsPt` divides slack evenly across detected word gaps; final lines stay left-aligned.
- **Encrypted PDFs and CCITT/JBIG2/JPX images are real capabilities** in pdf-codec — not scope boundaries. The permanent boundary is adversarial/malformed-input robustness.
- **PDF → docx/pptx/odt/odp table recovery requires drawn rules**: never text alignment alone (which would invent structure). Either a real gridline lattice, or a table ruled between its rows (`src/layout/ruled-rows.ts`): at least three horizontal rules of one span, with the columns taken from the gutters every row holding text in more than one place leaves, a heading row above the first rule read when its lines keep to the columns, a row crossing a gutter read as a spanning cell, a band whose every line fills its columns split into a row per line, and a band whose first line sets a group heading over sub-headings on the lines under it (two or more of them starting under the heading) read as two rows: the heading one cell from its own column up to where the first line's next run starts, over the sub-headings, and every other column one cell down both rows. Such a grid is refused when every column reads as running prose, when anything other than horizontal strokes and full-width shading is drawn inside it (a diagram), or when one row runs across a gap another row leaves between two places other than as such a group heading, since the columns would then read the text either side of that gap as one cell. A lattice with no text inside is rejected. A page that draws several tables reads each as a table of its own (`detectGridLattices`, `recoverTables` in `src/layout/reconstruct-tables.ts`): every ruled grid that closes into a rectangle, where a grid is the largest of them for the callers that read one grid per page (a worksheet, a slide: `detectGridLattice`).
- **PDF → docx/pptx/odt/odp table recovery requires drawn rules**: never text alignment alone (which would invent structure). Either a real gridline lattice, or a table ruled between its rows (`src/layout/ruled-rows.ts`): at least three horizontal rules of one span, with the columns taken from the gutters every row holding text in more than one place leaves, a heading row above the first rule read when its lines keep to the columns, a row crossing a gutter read as a spanning cell, a band whose every line fills its columns split into a row per line, and a band whose first line sets a group heading over sub-headings on the lines under it (two or more of them starting under the heading) read as two rows: the heading one cell from its own column up to where the first line's next run starts, over the sub-headings, and every other column one cell down both rows. Such a grid is refused when every column reads as running prose, when anything other than horizontal strokes and full-width shading is drawn inside it (a diagram), or when one row runs across a gap another row leaves between two places other than as such a group heading, since the columns would then read the text either side of that gap as one cell. A lattice with no text inside is rejected. A page that draws several tables reads each as a table of its own (`detectGridLattices`, `recoverTables` in `src/layout/reconstruct-tables.ts`): every ruled grid that closes into a rectangle, where a grid is the largest of them for the callers that read one grid per page (a worksheet, a slide: `detectGridLattice`). A table breaking across pages leaves a row open at the page's edge, its rule drawn under some columns or none and the side borders running on to it; a grid that does not close loses its top or bottom row boundary when that boundary does not span it, and is tested again, so the rows the page closes are still read as the table.
- **Merged table cells round-trip as merged.** docx: horizontal merge collapses to one `w:tc` with `w:gridSpan`; vertical merge needs one `w:tc` per covered row with `w:vMerge`. ODF: one entry per grid position, covered cells get `table:covered-table-cell`.
- **docx headers/footers/comments/footnotes/numbering are readable via `readDocxExtras`** — `readDocxContent` still drops them (`ContentDocument` has nowhere to put them). `PAGE`/`NUMPAGES` field substitution is never read (it's a render-time value).
- **A docx inline image reads as a real `ContentImageBlock`** — `buildDocxPackage` recognises the flat docx reader's two-block pattern (empty-text paragraph + image) and writes it back as one paragraph, avoiding spurious blank paragraphs on round trip.
Expand Down
69 changes: 69 additions & 0 deletions packages/documents.js/src/layout/lattice-multiple.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,75 @@ describe("detectGridLattices: a boundary drawn in pieces", () => {
});
});

describe("detectGridLattices: a table continuing onto the next page", () => {
// Three rows by three columns drawn cell by cell, the side borders one piece per row. The table's next row starts at the foot of the page: its first cell's top is the grid's bottom rule, its left border runs on below the grid as one more piece, and a rule under that first cell alone closes it there.
const table: LayoutItem[] = [100, 70, 40, 10].flatMap((y) => [
line(0, y, 100, y),
line(100, y, 200, y),
line(200, y, 300, y),
]);
for (const x of [0, 100, 200, 300]) {
for (const [low, high] of [
[70, 100],
[40, 70],
[10, 40],
] as const) {
table.push(line(x, low, x, high));
}
}
const stub = [line(0, -5, 0, 10), line(0, -5, 100, -5)];

it("reads the grid the table closes, leaving out the stub under one cell", () => {
const [lattice] = detectGridLattices([...table, ...stub]);

expect(lattice?.rowBoundariesDescPt).toEqual([100, 70, 40, 10]);
expect(lattice?.columnBoundariesAscPt).toEqual([0, 100, 200, 300]);
});

it("reads the stub's rule as no boundary of the table", () => {
const [lattice] = detectGridLattices([...table, ...stub]);

expect(lattice?.sourceItems.has(stub[1]!)).toBe(false);
});

it("reads the grid above an open last row whose rule is drawn under its last cell alone", () => {
// Every border runs on down to the foot of the page and a rule closes only the last column there, as a table whose last column ends on this page and whose other cells continue on the next.
const open = [
line(0, -5, 0, 10),
line(100, -5, 100, 10),
line(200, -5, 200, 10),
line(300, -5, 300, 10),
line(200, -5, 300, -5),
];

const [lattice] = detectGridLattices([...table, ...open]);

expect(lattice?.rowBoundariesDescPt).toEqual([100, 70, 40, 10]);
expect(lattice?.columnBoundariesAscPt).toEqual([0, 100, 200, 300]);
});

it("reads the grid below a first row the previous page opened, its rule drawn over one cell alone", () => {
const opened = [
line(0, 100, 0, 115),
line(200, 100, 200, 115),
line(0, 115, 100, 115),
];

const [lattice] = detectGridLattices([...table, ...opened]);

expect(lattice?.rowBoundariesDescPt).toEqual([100, 70, 40, 10]);
});

it("takes off no column boundary, so a side border that stops short leaves the grid unclosed", () => {
const shortRight = table.filter(
(item) =>
!(item.kind === "line" && item.x1Pt === 300 && item.y1Pt === 10),
);

expect(detectGridLattices(shortRight)).toHaveLength(0);
});
});

describe("a page drawing more strokes on one axis than clustering reads", () => {
it("is read as having no lattice, rather than spending a pair test on every pair of its strokes", () => {
const grid: LayoutItem[] = [
Expand Down
60 changes: 51 additions & 9 deletions packages/documents.js/src/layout/lattice.ts
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,51 @@ function hasClosedOuterRectangle(
);
}

interface ClosedGrid {
readonly rowLines: readonly AxisLine[];
readonly columnLines: readonly AxisLine[];
}

/**
* The closed grid a cluster of boundaries draws: the cluster itself when its outer rectangle closes, else the grid without its top or its bottom row boundary, or both, when that one closes, or undefined.
*
* A table breaking across pages leaves a row open at the page's edge: its last row at the foot of a page is closed by a rule drawn under some of its columns or none, the side borders running down to it, and its first row at the head of the next is opened the same way. Joined to the borders as one boundary (`segmentsTouchCollinearly`), that open row would otherwise leave the whole table unclosed and read as prose, every row of it. So an outer row boundary that does not span the grid is taken off, once at the top and once at the bottom, and the grid tested again; nothing else is, so a grid with a gap in a side border, or a drawing whose strokes close no rectangle, stays unclosed.
* @param rowLines - the cluster's row boundaries, top first
* @param columnLines - the cluster's column boundaries, left first
* @returns the boundaries of the closed grid, or undefined
*/
function closedGrid(
rowLines: readonly AxisLine[],
columnLines: readonly AxisLine[],
): ClosedGrid | undefined {
if (columnLines.length < MIN_GRIDLINE_COUNT_PER_AXIS) {
return undefined;
}
const leftXPt = columnLines[0]!.position;
const rightXPt = columnLines[columnLines.length - 1]!.position;
const spans = (line: AxisLine): boolean =>
unionCoverageRatio(line.ranges, leftXPt, rightXPt) >= EDGE_COVERAGE_RATIO;
const openTop = rowLines[0] !== undefined && !spans(rowLines[0]);
const openBottom =
rowLines[rowLines.length - 1] !== undefined &&
!spans(rowLines[rowLines.length - 1]!);
const candidates = [
rowLines,
...(openTop ? [rowLines.slice(1)] : []),
...(openBottom ? [rowLines.slice(0, -1)] : []),
...(openTop && openBottom ? [rowLines.slice(1, -1)] : []),
];
for (const rows of candidates) {
if (
rows.length >= MIN_GRIDLINE_COUNT_PER_AXIS &&
hasClosedOuterRectangle(rows, columnLines)
) {
return { rowLines: rows, columnLines };
}
}
return undefined;
}

// A rectangular region of the atomic grid (the finest grid the detected boundaries admit) that drawn strokes never actually divided — one real ContentTableCell, spanning more than one atomic row/column exactly when colSpan/rowSpan would (see reconstruct.ts's own recoverTable, which anchors one real cell at [rowStart, colStart] and a rowSpan continuation placeholder at the same column position in every row up to rowEnd, matching how ooxml.js's own docx/pptx table writers already expect a ContentTable to be shaped).
export interface TableRegion {
readonly rowStart: number;
Expand Down Expand Up @@ -488,17 +533,14 @@ function latticeCandidates(items: readonly LayoutItem[]): LatticeCandidate[] {
for (const cluster of clusterSegments(horizontal, vertical)) {
const clusterHorizontal = cluster.filter((s) => s.axis === "horizontal");
const clusterVertical = cluster.filter((s) => s.axis === "vertical");
const rowLines = dedupeAxisLines(clusterHorizontal, true);
const columnLines = dedupeAxisLines(clusterVertical, false);
if (
rowLines.length < MIN_GRIDLINE_COUNT_PER_AXIS ||
columnLines.length < MIN_GRIDLINE_COUNT_PER_AXIS
) {
continue;
}
if (!hasClosedOuterRectangle(rowLines, columnLines)) {
const grid = closedGrid(
dedupeAxisLines(clusterHorizontal, true),
dedupeAxisLines(clusterVertical, false),
);
if (grid === undefined) {
continue;
}
const { rowLines, columnLines } = grid;
const regions = findCellRegions(rowLines, columnLines);
if (regions === undefined) {
continue;
Expand Down
4 changes: 2 additions & 2 deletions packages/pdf-codec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,9 +334,9 @@ A TrueType subset can lose its glyph names and its `cmap` both and keep nothing

A `/ToUnicode` CMap that a producer writes for the Symbol font or one of the Windows dingbat fonts (Word does, for a bullet, a tick or an arrow it draws in them) gives each glyph the private-use code point its 8-bit code makes in the 0xF000 range, which identifies the glyph in that one font and nothing else: 0xFC is a check mark in Wingdings and a piece of a right brace in Symbol. `symbol-font-private-use.ts` reads such a code point as the character the named font draws at that code, applied to the text a `/ToUnicode` CMap gave for a Symbol, Wingdings, Wingdings 2, Wingdings 3 or Webdings font, by name (`Wingdings 2` and `Wingdings2` alike), and to no other font. Symbol is recognised as `Symbol`, `SymbolMT`, `SymbolOOEnc` and MathType's `EuclidSymbol` (read in Symbol's encoding on the evidence of three checked glyphs, the rest of its layout assumed), each also with an `-OV-` variant tag at the end (`SymbolMT-OV-JQGJHA`) or a mixed-case pseudo subset tag run straight onto the front with no `+` or separator (`SmmfqrSymbol`), which is looked past only when what follows it is one of those names, so `SymbolicSans`, `SegoeUISymbol` or `Lucida Symbol` is not taken for Symbol. A font whose `/BaseFont` names no symbol font is identified by its embedded program's own PostScript name instead, read only for text that holds a private-use code point. Every `/BaseFont` spelling of Symbol also gets Symbol's built-in encoding for a code no `/ToUnicode` covers; a font recognised only through its program's name does not, since that name is read only for `/ToUnicode` text. Symbol is read through its published encoding, and the four dingbat fonts through `wingdings-characters.ts` and `dingbat-characters.ts`, tables of Unicode's own Wingdings and Webdings document (WG2 N4384), whose source-reference list is vendored under `assets/unicode-wingdings/` and checked against every table entry. A code the font has no character at, and ZapfDingbats, stay as the producer wrote them. The pieces a tall bracket, parenthesis or brace is assembled from, the integral and arrow extensions and the serif and sans-serif copyright, registered and trade mark signs are glyphs the Adobe Glyph List gives private-use code points (`parenlefttp` is U+F8EB) although Unicode encodes them (U+239B): `glyph-names.ts` resolves those names to the Unicode characters on every route a glyph name takes (Symbol's built-in encoding, `/Differences`, an embedded program's glyph names), and a Symbol font's `/ToUnicode` value at one of the list's private-use code points is read as the same character. `radicalex`, which Unicode does not encode, keeps its private-use code point.

Two ways a producer names a core face other than the PostScript name the face's own tables carry are read through the same carried glyph order. A TrueType program Windows-era producers embed under the face's Windows name (`TimesNewRoman`, `Arial,Bold`) is the face `TimesNewRomanPSMT` or `Arial-BoldMT`, still accepted glyph by glyph against the program's own advance widths. And Adobe Acrobat's optimiser rewrites a TrueType face as a CFF subset that names each glyph `G` followed by its glyph id in the face (`G36` is the capital A of Times New Roman) under a font name carrying a size suffix (`Times.New.Roman092`, `Arial.Bold0133`): `coreFaceTextsOfNumberedGlyphs` reads those names through the face's order, but only for a font whose dictionary widths agree with the face's advances over at least five numbered glyphs and nine in ten of them, so a subset of another face, or one too small to tell, is read as before. The Symbol face and the Type 3 fonts such producers also write are not covered: the first has no carried order, and a Type 3 glyph is a drawing procedure with no character.
Two ways a producer names a core face other than the PostScript name the face's own tables carry are read through the same carried glyph order. A TrueType program Windows-era producers embed under the face's Windows name (`TimesNewRoman`, `Arial,Bold`) is the face `TimesNewRomanPSMT` or `Arial-BoldMT`, still accepted glyph by glyph against the program's own advance widths. And Adobe Acrobat's optimiser rewrites a TrueType face as a CFF subset that names each glyph `G` followed by its glyph id in the face (`G36` is the capital A of Times New Roman) under a font name carrying a size suffix (`Times.New.Roman092`, `Arial.Bold0133`), as Acrobat Distiller does with a lower-case `g` for Century Schoolbook, the face of many standards documents written in Word (`g36` under `CenturySchoolbook-Bold`), whose order is carried too: `coreFaceTextsOfNumberedGlyphs` reads those names through the face's order, but only for a font whose dictionary widths agree with the face's advances over at least five numbered glyphs and nine in ten of them, so a subset of another face, or one too small to tell, is read as before. The Symbol face and the Type 3 fonts such producers also write are not covered: the first has no carried order, and a Type 3 glyph is a drawing procedure with no character.

Some producers write the font dictionary's own tables wrong for a symbol subset: a typesetter's Type 1C math fonts give each glyph an arbitrary name, assign the names to codes with `/Differences`, and have a `/ToUnicode` CMap read each name as the character it spells, so an equals sign is read as `¼` and an omega as `!`, from tables that are valid and wrong. `verified-glyph-tables.ts` holds tables for the exact subsets whose glyphs were rendered from the embedded program and checked one by one (`docs/verified-glyph-tables/`), each keyed by the SHA-256 of the decoded font program; a glyph of such a subset is read as what it draws, before the font dictionary's tables, and every other subset, family and code is read as before.
Some producers write the font dictionary's own tables wrong for a symbol subset: a typesetter's Type 1C math fonts give each glyph an arbitrary name, assign the names to codes with `/Differences`, and have a `/ToUnicode` CMap read each name as the character it spells, so an equals sign is read as `¼` and an omega as `!`, from tables that are valid and wrong. `verified-glyph-tables.ts` holds tables for the exact subsets whose glyphs were rendered from the embedded program and checked one by one (`docs/verified-glyph-tables/`), each keyed by the SHA-256 of the decoded font program; a glyph of such a subset is read as what it draws, before the font dictionary's tables, and every other subset, family and code is read as before. The glyph a code selects is named by `/Differences`, else by the named base encoding, else, for a dictionary that names no base encoding, by the program's own built-in encoding, which is then the base encoding.

**Where nothing states an answer, the answer is the replacement character plus a `text/unmapped-encoding` diagnostic, never a guess.** Cases that reach it in practice: a symbolic font with no embedded program and no `/ToUnicode`; a subsetted font that both strips its glyph names and maps its glyphs only from private-use code points; a nameless, `cmap`-less subset of a face whose glyph order this package does not carry, or of a carried face at a glyph its recorded releases do not number; and a composite font whose `/Encoding` is a CMap other than Identity-H or Identity-V, whose codes reach CIDs this reader does not map. A private-use code point identifies a glyph inside one font and says nothing about the character it draws, so it is treated as no answer rather than a wrong one.

Expand Down
Loading
Loading