Skip to content

Improve tokenizer performance for large IMAP responses - #201

Merged
stevebauman merged 1 commit into
masterfrom
fix/tokenizer-performance
Nov 16, 2026
Merged

stevebauman merged 1 commit into
masterfrom
fix/tokenizer-performance

Conversation

@stevebauman

@stevebauman stevebauman commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Parsing large IMAP responses spends a significant amount of CPU time reading quoted strings, atoms, whitespace, and email addresses one byte at a time. Each byte requires several PHP method calls, even when the data is already buffered.

This updates those four readers to consume character runs using strcspn() and strspn(). The scanning stays directly in each reader, with explicit handling for escapes, delimiters, and incomplete input.

Benchmarking a synthetic FETCH response containing 35,798 messages on PHP 8.4.27 produced the following parser CPU timings.

Run Before After CPU time reduction
1 3.937 seconds 1.721 seconds 56.3%
2 3.889 seconds 1.795 seconds 53.8%

The reporter's Gmail response has not been benchmarked locally.

All 546 unit tests pass, along with 3,125 comparisons of token values, token types, and exception classes/messages against the original tokenizer.

Fixes #200.

@stevebauman
stevebauman merged commit f1091b3 into master Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ImapTokenizer spends most of a large FETCH reading one byte per method call

1 participant