[Maya-Kid]

The chunker keeps a paragraph with the table below it when the paragraph is at most 200 bytes or ends in a colon, but it checked only :, so a long Chinese lead-in ending in ： was split from its table.

Problem

200 bytes is about 66 Chinese characters, so a longer lead-in is recognised only by its colon. On dev 15fd7d5, chunk_text on this input (first line 81 characters, 243 bytes):

根据公司第三届董事会……现金流量充足，主要会计数据如下：

| 项目 | 本期 |
| --- | --- |
| 科目0 | 10000 |
... 60 rows

returns four chunks: the sentence alone, then three that start at | 项目 | 本期 |. Chunks are extracted separately, so no table chunk says what the numbers are. The same sentence ending in : opens every chunk.

Change

• caption_like in chunker.rs accepts either colon, as table.rs already does.

Like :, ： now carries a paragraph of any length: a long analysis paragraph ending in ……如下： repeats in every table chunk. At the 300-token budget, a 271-character paragraph above 60 rows goes from 4 chunks to 15; a 399-character one above 20 rows, from 3 chunks to 20 one-row chunks of 381 tokens, over budget. English paragraphs ending in : already do this. A token cap for both colons could be a follow-up.

Verification

• a_caption_ending_in_a_full_width_colon_stays_with_its_table_at_any_length asserts that every chunk of the input above starts with the sentence and the header row. On dev 15fd7d5 with only this test added, it fails at chunk 0, the sentence alone.

Linux x86_64 container, PostgreSQL 16 + pgvector 0.8.6, Poppler 25.03, DB required (UTOPIA_TEST_REQUIRE_DB=1), commit 9fb292d on dev 15fd7d5: fmt and clippy clean on Rust 1.98.1 and on 1.99.1, the stable toolchain CI installs; cargo test --locked --workspace (1.98.1) 1,564 passed, 0 failed, 6 ignored.

[WaylandYang]

Thank you. Over 200 bytes, a Chinese lead-in can only be recognised by its colon, so accepting ： alongside : fixes the case where ……如下： was split from its table, and it matches what table.rs already accepts. You're right that a lead-in can now be repeated into pieces that go over budget; English already does this. A token cap that applies to both colons deserves its own issue if you'd like to open one. Merging.