Skip to content

fix: improve sil.km.gcc wordbreaking #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jahorton wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix: improve sil.km.gcc wordbreaking #355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eced684
fix: improve sil.km.gcc wordbreaking
jahorton Oct 27, 2026
5d0a293
fix: update version in .kps
jahorton Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions release/sil/sil.km.ggc/HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
Google Crawler Change History
====================

2.1 (2026-Oct-27)
----------------
* update wordbreaking function to better handle edge cases

2.0 (2024-Oct-24)
----------------
* update wordlist and documentation
Expand Down
4 changes: 2 additions & 2 deletions release/sil/sil.km.ggc/sil.km.ggc.kpj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@
<ID>id_a02fd3894fae146aa99da3213c3509d4</ID>
<Filename>sil.km.ggc.model.kps</Filename>
<Filepath>source\sil.km.ggc.model.kps</Filepath>
<FileVersion>2.0</FileVersion>
<FileVersion>2.1</FileVersion>
<FileType>.kps</FileType>
<Details>
<Name>Khmer Google Crawler</Name>
<Copyright>© SIL International</Copyright>
<Version>2.0</Version>
<Version>2.1</Version>
</Details>
</File>
<File>
Expand Down
2 changes: 1 addition & 1 deletion release/sil/sil.km.ggc/source/sil.km.ggc.model.kps
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<Name URL="">Khmer Google Crawler</Name>
<Copyright URL="">© SIL International</Copyright>
<Author URL="">SIL International</Author>
<Version URL="">2.0</Version>
<Version URL="">2.1</Version>
<Description URL="">Google Crawler generated from template</Description>
</Info>
<Files>
Expand Down
28 changes: 27 additions & 1 deletion release/sil/sil.km.ggc/source/sil.km.ggc.model.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,25 @@ const source: LexicalModelSource = {
}
}

let emptyToken = {
left: str.length,
start: str.length,
right: str.length,
end: str.length,
length: 0,
text: ''
};

let latestIndex = 0;
return tokens.map(function(token) {
const finalizedTokens = tokens.map(function(token, index) {
// Make sure we properly place a new end-of-context token;
// indexOf doesn't do this well by default.
if(token == '' && index == tokens.length - 1) {
return emptyToken;
}

// Find the first occurrence of the word that appears after the end of
// the last word we checked.
const start = str.indexOf(token, latestIndex);
latestIndex = start + token.length;
return {
Expand All @@ -73,6 +90,15 @@ const source: LexicalModelSource = {
text: token
}
});

// If a new word should be starting at the current caret position,
// create an empty '' token to indicate this.
let tailToken = finalizedTokens[tokens.length - 1];
if(!tailToken || tailToken.end != str.length) {
finalizedTokens.push(emptyToken);
}

return finalizedTokens;
},
punctuation: {
insertAfterWord: "\u200B"
Expand Down