Skip to content

change(web): check for empty final token in tokenization method - #16917

Open
jahorton wants to merge 2 commits into
betafrom
change/web/detect-empty-final-token
Open

jahorton wants to merge 2 commits into
betafrom
change/web/detect-empty-final-token

Conversation

@jahorton

Copy link
Copy Markdown
Contributor

Relates-to: #16587
Supersedes: #16611

This change ensures that model wordbreakers consistently present a context-final empty token when appropriate. Certain custom models and the ascii breaker do not currently properly do so otherwise, and there is a case in which even our default wordbreaker does not.

Build-bot: skip build:web
Test-bot: skip

Relates-to: #16587

Build-bot: skip build:web
Test-bot: skip
@keymanapp-test-bot

keymanapp-test-bot Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

  • Web (no artifacts on BuildLevel "build")
    • KeymanWeb Test Home - build : ✅ all tests passed

@keymanapp-test-bot keymanapp-test-bot Bot added this to the B19S5 milestone Oct 28, 2026
@github-actions github-actions Bot added web/ web/predictive-text/ change Minor change in functionality, but not new labels Oct 28, 2026
}

export function enforceFinalEmptySpan(spans: LexicalModelTypes.Span[], str: string) {
let emptyToken = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let emptyToken = {
const emptyToken = {

Do we need to define it as a variable at the top of the function? Could we just have it inline on line 46?


// If a new word should be starting at the current caret position,
// ensure an empty '' token indicates this.
let tailToken = spans[spans.length - 1];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let tailToken = spans[spans.length - 1];
const tailToken = spans[spans.length - 1];

What if spans.length == 0?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retrieval of an item at index -1 will always return undefined in JS; note the if(!tailToken conditional on the following line.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly true as it gets converted to a string and deref as a property, and this also smells buggy. We should test spans.length as an input condition as that is self-documenting. Don't rely on side-effects which look accidental.

let a = [];
a[-1] = 'ha';
console.log(a[-1]);     // ha
console.log(a.length);  // 0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You and I have different definitions of "looks accidental" here, but fine.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

change Minor change in functionality, but not new web/predictive-text/ web/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants