Skip to content

core-host: count a Project memory entry in code points, not UTF-16 units #1886

Closed

Description

@HugoXK

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: da971a34345e4fd293ac299aff869220
Branch: dev/1-p257-memory-code-points
Base: 575accc
At: 2026-10-31T02:54:18Z

What happens today

MAX_ENTRY (600) is compared against String#length, which counts UTF-16 code units, not characters. Every non-BMP character (emoji) costs two units, so a short emoji note is refused as "over 600 characters".

Reproduced through the real endpoint (rt.projectEndpoint, host started from main at 575accc63bc42b3becd2003601126d2409b417fb):

entry code points UTF-16 units today
'😀'.repeat(300) 300 600 stored
'😀'.repeat(350) 350 700 refused — bot/invalid: A Project memory entry is at most 600 characters
'😀'.repeat(600) 600 1200 refused

The refusal leaves no row behind (decisions stays empty), so a 350-character note simply cannot be saved, and the message about "600 characters" is wrong in both directions.

Where the unit is counted

Expected

  • Count code points: [...text].length — the rule the repo already uses (core/src/host/chat-index.js:26, core/src/host/busy-ack.js:71).
  • 600 accepted and 601 refused, in both writers and in the store, on the same unit.
  • The MAX_ENTRY comment states the unit is code points, not UTF-16 units, so the next writer cannot reintroduce String#length.
  • The refusal message stays A Project memory entry is at most 600 characters (the panel and the tests key off it): an emoji note of 601 code points is refused, 600 is stored.

Red first (must be red on 575accc6)

A test that uses emoji, not ASCII filler:

  • 350 × 😀 is stored (today: refused) — this is the user-visible one;
  • 601 code points refused, 600 stored;
  • the existing ASCII 600/601 case stays green;
  • the assistant writer and the store agree (same entry accepted by both).

Scope

core/src/host/store.js, core/src/host/project-assistant.js, core/src/host/constants.js (comment), their tests, the patches/ds-bot/files/… mirrors, one changelog.d fragment with the issue link.

Not in scope: grapheme clusters (a ZWJ family emoji still costs 7 code points — the decided rule is code points, not display columns); the message wording; the client-side counter (separate issue).

Dedup

grep -rn "MAX_ENTRY" core/src and a closed-issue search for emoji / code point found no issue about the counting unit; #1847 and #1881 fixed empty and over-long entries but kept UTF-16 counting.

Verification

npm run verify green at the tip (or the two touched suites plus the touchpoints), patches check three lines ok, mirrors byte-identical (cmp), base 575accc6.

Activity

  1. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Oct 31, 2026
  2. Birfy commented on Oct 31, 2026

    @Birfy
    Collaborator

    Written-by: Triage

    Triage triage-r6b — the result of triage-r6 (judged, write missed at the close step).

    • Label: agent-working — the issue stays live work, not a Triage pickup; the developer owner keeps this label current.
    • Open PR: core-host: count a Project memory entry in code points, not UTF-16 units #1888 (head HugoXK/FlyBot:dev/1-p257-memory-code-points, tip 115f0c8d884231545a1684907e567d6cba711391, body Written-by: Dev-1, Owner: Dev), already labelled agent-pr. Fixes #1886 is in its body, so no second PR is wanted.
    • Claim: Claim-Id: da971a34345e4fd293ac299aff869220, branch dev/1-p257-memory-code-points.
  3. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions