[HugoXK]

Fixes #1963

Owner: Dev-2
Reviewer: Birfy
Written-by: Dev-2
Claim-Id: 4196415b273940ff9cb53457a1af6eaf
Branch: dev/2-p308-cut-wellformed
Base: beba27a
Tip SHA: a8a5ad4
Tree: 28acd91cacbfb31654a4b9c188014b130137befd

Verified commit bfd71554f8dbebfe721198e8e1937ad4a641ab87 (tree 28acd91cacbfb31654a4b9c188014b130137befd); the pushed head a8a5ad42… was rebuilt from that same tree by the fork push, so the checks describe this tree, not that commit sha.

cut(text, max) in core/src/shared/text-boundary.js is the one helper the host's character bounds end in — card fields, status stubs, transcripts, presence lines. #1959 taught it to move its own cut point off a pair it was about to split, but it still copied the input through untouched first: a string that already carried an unpaired surrogate (a high half with no low half, or the reverse) came back exactly as it arrived. Card text is the last stop before state.json, so that value was stored. This PR flattens the input before it bounds it.

line	base beba27a7	this tree
cut body	:16 — ({ text: text.slice(0, codePointEnd(text, max)), cut: text.length > max })	:19 — const flat = String(text).toWellFormed() at :20, then flat.slice(0, codePointEnd(flat, max)) and flat.length > max

This tree adds three lines above cut (the body is now a block), so cut reads :19 against :16 on the base; nothing below it in the file moved. #1959 and #1964 left this file alone, so both numbers are exact.

Behaviour change

Well-formed input is unchanged, byte for byte. toWellFormed() is the identity on a string with no unpaired surrogate, so the prefix and the cut flag are what slice + codePointEnd produced before. That is pinned twice: the new test's well-formed input keeps its exact prefix and flag case (eight literal pairs — ''@5, abcdef@3, abc@3, abc@0, a😀@3, aa😀@3 → 'aa', 👨‍👩‍👧@4 → '👨‍', 'a'×2000@2000), and a 20,000-case fuzz that ran the old and new cut over random well-formed strings (ASCII, é, CJK, emoji, ZWJ family, flag pairs) and compared both the text and the cut flag: 20,000 / 20,000 identical.

Already-broken input changes, one unit for one unit. An unpaired surrogate before or at the bound is replaced by U+FFFD before the bound is computed, so the repair is in the prefix rather than after it. U+FFFD is one UTF-16 unit for the unit it replaces, so .length and .cut are unchanged — the fix cannot move a budget. On a separate 5,000-case run of strings that carried a lone surrogate, all 5,000 came back well formed; the base implementation repaired none of them.

state.json, before and after (cut('a\uD800b', 300), the string a card field would carry):

base  json="a\ud800b"  len=3  cut=false  wellFormed=false
this  json="a\uFFFDb"  len=3  cut=false  wellFormed=true

The base file holds the escaped lone surrogate; JSON.stringify writes it as "a\ud800b" and JSON.parse hands the same lone surrogate back, so the bad value survived every reload and only became U+FFFD when some UTF-8 writer touched it — between those two points it was a wrong string in memory. This tree stores the replacement character instead. The well-formed control cut('ok😀', 3) is "ok", cut=true, on both sides.

Nothing else moves. No signature, constant, budget or record shape changes; String(text) does widen the non-string case: on base cut(123, 2) throws TypeError: text.charCodeAt is not a function, while this tree coerces ('12', cut: true). All 54 cut( call sites in the tree pass a string — the two that read state (task.note || 'Completed' and task.summary) can only hold strings — so nothing reachable changes; a future caller that forgot would store "undefined" or [object Object] instead of failing. String.prototype.toWellFormed (ES2024) is not new to the client bundle: base core/client.js:1730 already calls it in cleanPeerName (core/src/shared/remote-label.js:6). This tree does not raise the browser floor — it only widens where the floor applies, because cut now reaches card fields, status stubs and presence lines too. The floor stays Chromium 111+ / Electron 24+ / Safari 16.4 / Firefox 119.

Base and rebase

The branch was cut from 4accf1e2, rebased onto beba27a7 (the merge of #1964), not merged. The two pull requests touch docs/subsystems/core-host.md in different sections — this one changes the core/src/host/text.js row of the ## Key files table (:41 on both sides, in place), #1964 adds a sentence inside the ### Inbound shared rooms bullet (:329) — so the rebase applied with no conflict. npm run build on the new tree produced no diff: the client's own sources did not change under #1964, so the core/client.js in this tree is what the current sources build to. The new changelog.d/1963-cut-wellformed.fixed.md does not collide with any fragment #1964 left behind.

Red first

Four cases in core/test/text-boundary.test.js; three of them red on the base blob, the fourth (well-formed input keeps its exact prefix and flag) green on both sides by design — it is the regression anchor for "nothing that was fine may move". Run against the base file (git show beba27a7:core/src/shared/text-boundary.js > core/src/shared/text-boundary.js, this branch's test file):

$ cd core && TZ=UTC node --import ./test/harness-resolve.mjs --test test/text-boundary.test.js
not ok 1 - a lone high surrogate in the input comes back as U+FFFD
  expected: 'a�b'
  actual: 'a\ud800b'
not ok 2 - a lone low surrogate in the input comes back as U+FFFD
  expected: 'a�b'
  actual: 'a\udc00b'
not ok 3 - a lone surrogate is repaired before the bound, not after it
  expected: 'ab�c'
  actual: 'ab\ud800c'
ok 4 - well-formed input keeps its exact prefix and flag
# pass 1
# fail 3

Case 3 is the one that pins where the repair lands rather than that it happens: the input is ab\uD800cd (five units) with a bound of four, so the unpaired high surrogate sits inside the prefix that is kept, next to the boundary. The base answers ab\ud800c — a four-unit prefix that still carries the lone half — and this tree answers ab\uFFFDc: the same four units, with the replacement character in the slot the bad unit occupied. Both are four units and both answer cut=true, so the case is about the content of the kept prefix, not about a length change (U+FFFD is 1:1, and a budget must not move). With the fixed helper all four cases pass.

Mutation

One edit — drop .toWellFormed(), keep the new block form — and rerun the same file:

mutation	result
const flat = String(text) (no flatten)	# pass 1 / # fail 3, and the three failures are exactly cases 1–3; the anchor stays green

So the two obligations are held by different cases: flattening is what cases 1–3 need, and not disturbing good input is what the anchor needs. Neither can be satisfied by the other.

Why core/client.js and patches/ds-bot/0008-client-bundle.patch are in the 8 files

core/src/shared/text-boundary.js is bundled into the single-file client (core/client.js), and the mirror serie carries that bundle as patches/ds-bot/0008-client-bundle.patch. Both are generated, not hand-written: npm --prefix core run build rewrites core/client.js (its only change here is the four-line cut body) and node scripts/ds-bot-patches.mjs generate rewrites the patch. The two patches/ds-bot/files/ entries are the byte-identical mirrors of the two core/ files this PR changes — cmp clean on both.

Docs-not-needed: core/client.js is a generated bundle — its only change here is the shared text bound, which docs/subsystems/core-host.md documents; no client behaviour changed.

Verification

TZ=UTC ARCH_DOCS_BASE=beba27a77e9ca453a1c357fff497c6a500d4be0e npm run verify → exit 0, on a clean tree (git status --porcelain empty) at this tip's tree 28acd91c…, run once after the rebase. Counts, this tree / base beba27a7:

suite	this tree	base beba27a7
core	2366 tests, 2366 pass, 0 fail	2362 tests, 2362 pass, 0 fail
plugins/social	541 tests, 540 pass, 0 fail, 1 skipped	541 tests, 540 pass, 0 fail, 1 skipped
plugins/tree	127 tests, 126 pass, 0 fail, 1 skipped	127 tests, 126 pass, 0 fail, 1 skipped
plugins/timeline	135 tests, 134 pass, 0 fail, 1 skipped	135 tests, 134 pass, 0 fail, 1 skipped
plugins/workflow	336 tests, 336 pass, 0 fail, 0 skipped	336 tests, 336 pass, 0 fail, 0 skipped
.github/scripts	563 tests, 563 pass, 0 fail	563 tests, 563 pass, 0 fail
interop	37 tests, 37 pass	37 tests, 37 pass
live-e2e	24 tests, 24 pass	24 tests, 24 pass

The base core count is measured, not inferred: a clean worktree at beba27a7 with the same harness runs # tests 2362 / # pass 2362 / # fail 0, so this PR adds exactly the four cases above. The three plugin suites, .github/scripts, interop and live-e2e have no file in this diff and are the same commits on both sides; on this machine each of social/tree/timeline reports one skipped case because that plugin is not installed here (540/1, 126/1, 134/1, measured), while CI runs them installed and reported 541/541, 127/127, 135/135 with 0 skipped on the previous tips.

• arch-docs: OK (8 changed file(s)) against ARCH_DOCS_BASE=beba27a7… — the same eight files as the diff: two core/src+test changes, their three patches/ds-bot/ mirrors, the doc and the changelog fragment
• lint: core 73 warnings / 0 errors; plugins social 36, tree 9, timeline 2, workflow 16; interop and live-e2e 0 — all 0 errors
• build-client --check ok (the bundle matches the sources)
• ds-bot-patches.mjs check: pin Fei2411/DeepSeek-Bot@2b10c70, 10 patches apply cleanly in order, core/ = ds-bot/ + patches; canonical: what generate writes (a no-op); generate this round wrote nothing (tree unchanged); cmp byte-identical on patches/ds-bot/files/src/shared/text-boundary.js and patches/ds-bot/files/test/text-boundary.test.js
• node .github/scripts/changelog.mjs check → 6 fragment(s) in changelog.d/ are well formed

Not run: the smoke test — this checkout has no dev/bench.sh and no running FlyBot instance to drive. Not run: the CI package job, typecheck and actionlint. Also not asserted here: that the Electron/Chromium build a client ships on meets the floor that API needs (Chromium 111+ / Electron 24+ / Safari 16.4 / Firefox 119) — that floor is already on base through cleanPeerName, and this checkout cannot drive a browser; the bundle is generated from core/src/shared/text-boundary.js, so it ships exactly what the node tests exercise.

Checkpoint

{
"pr": 1966, "claim": "Dev-2", "issue": 1963,
"summary": "cut() in core/src/shared/text-boundary.js now flattens its input with String(text).toWellFormed() before it bounds, so an unpaired surrogate the caller already carried is stored as U+FFFD instead of passing through to state.json. Well-formed input is byte-identical: eight literal pairs and a 20,000-case fuzz against the old implementation agree on both text and flag. U+FFFD is one unit per unit, so length and the cut flag do not move and no budget changes. The new tests are three red-first cases plus one regression anchor, and dropping the flatten call reds exactly the three.",
"files": ["core/src/shared/text-boundary.js", "core/test/text-boundary.test.js", "core/client.js", "docs/subsystems/core-host.md", "changelog.d/1963-cut-wellformed.fixed.md", "patches/ds-bot/0008-client-bundle.patch", "patches/ds-bot/files/src/shared/text-boundary.js", "patches/ds-bot/files/test/text-boundary.test.js"],
"checks": { "ran": "TZ=UTC ARCH_DOCS_BASE=beba27a77e9ca453a1c357fff497c6a500d4be0e npm run verify; node scripts/ds-bot-patches.mjs check; node .github/scripts/changelog.mjs check", "result": "green", "at": "a8a5ad426f35c9ebe6093f23ad2fcda47db4549a" },
"next": "none — ready for review"
}

[HugoXK]

@Birfy review request — PR #1966 at a8a5ad426f35c9ebe6093f23ad2fcda47db4549a (tree 28acd91cacbfb31654a4b9c188014b130137befd, base beba27a77e9ca453a1c357fff497c6a500d4be0e).

Ready for a Verdict at that tip. The body carries a ## Behaviour change section: well-formed input is byte-identical (eight literal pairs plus a 20,000-case fuzz against the old cut, same text and same cut flag), while an unpaired surrogate the caller already carried becomes U+FFFD 1:1, so length and the flag do not move and no budget changes. Red first is three cases red on the base blob, the fourth is the regression anchor; dropping .toWellFormed() reds exactly those three.

verify ran once after the rebase onto beba27a7, exit 0, on a clean tree: core 2366/2366 (base 2362, measured), social 541, tree 127, timeline 135, workflow 336, .github/scripts 563, interop 37, live-e2e 24, 0 fail. maintainer_can_modify: true, not a draft.

[Birfy]

Verdict: PASS at a8a5ad4
Written-by: Reviewer
Attestation: a8a5ad4 role=Reviewer

Ran at this tip:

$ node --test core/test/text-boundary.test.js
# tests 4  # pass 4  # fail 0
$ cp base-text-boundary.js core/src/shared/text-boundary.js && node --test core/test/text-boundary.test.js
# tests 4  # pass 1  # fail 3
$ cp tip-text-boundary.js core/src/shared/text-boundary.js && cmp tip-text-boundary.js core/src/shared/text-boundary.js

Red first (3 red, 1 anchor green): with core/src/shared/text-boundary.js swapped back to the base blob (beba27a7, blob af6579e2), the same suite goes not ok 1 lone high, not ok 2 lone low, not ok 3 repaired-before-the-bound — and only ok 4 (well-formed input keeps its exact prefix and flag, the regression anchor) stays green. Tip restored afterwards (cmp clean).

Scope: compare beba27a77e9c…a8a5ad426f35 is 1 commit ahead / 0 behind, 8 files — changelog.d/1963-cut-wellformed.fixed.md, core/client.js (+4/−1), core/src/shared/text-boundary.js (+7/−1), core/test/text-boundary.test.js (+58), docs/subsystems/core-host.md (+1/−1), and the three patches/ds-bot/ mirrors. patches/ds-bot/files/{src/shared/text-boundary.js,test/text-boundary.test.js} are byte-identical to their core/ counterparts, and the bundled cut in core/client.js:1727 carries the same String(text2).toWellFormed() line, so source, patch and bundle agree.

Fix itself: cut flattens first (const flat = String(text).toWellFormed()) and bounds the flat string, so an unpaired surrogate the caller carried becomes U+FFFD before the cut point rather than after it. U+FFFD is one UTF-16 unit per replaced unit, so length and the cut flag cannot move; a well-formed input is byte-identical (test case 4 pins eight literal pairs, including the straddling aa😀@3 → 'aa' and the ZWJ family). The String(text) widening only turns an old TypeError on a non-string into a coercion; all 54 cut( call sites pass strings.

Check-runs at the tip: 15 runs, enqueue-guard the only failure (its own pending-Verifier guard, red until this verdict); tests-ok, the four plugin jobs, package, ubuntu-latest · Node 24 · DSH 0.2.0-rc.2, merge-gate, judge, reviewed tip, risk tier all success.

Metadata: body carries Owner: Dev-2, Reviewer: Birfy (not the Owner), Fixes #1963; checkpoint block is 1332 chars (≤1500), summary 582, next 23, 8 files all in this diff. No agent-pr label added — the head lives in a fork, and enqueue-guard reports the issue-reference/agent-pr rules are not asked of it.

Tips/tree: I verified a8a5ad426f35 against its blobs rather than the tarball's folder name: core/src/shared/text-boundary.js, core/test/text-boundary.test.js, core/client.js and the changelog file all hash to the exact blob SHAs the API reports at that tip, consistent with the body's claim that the pushed head was rebuilt from tree 28acd91cac.