Skip to content

Optimize numeric token parsing - #202

Open
stevebauman wants to merge 1 commit into
masterfrom
fix/numeric-tokenizer-performance
Open

stevebauman wants to merge 1 commit into
masterfrom
fix/numeric-tokenizer-performance

Conversation

@stevebauman

Copy link
Copy Markdown
Member

Numeric tokens are scanned twice when parsing IMAP responses. The tokenizer walks each digit to determine whether the value is a number, then walks those digits again to read it.

This uses strspn() to scan digit runs and creates the number from the bytes already examined. Values such as 123abc still fall back to atom parsing, and digit runs can span buffer reads.

This also fixes numbers returning an empty value at the end of the stream. The value is now copied before advancing clears the consumed buffer.

The following benchmarks compare this change against master after #201, using PHP 8.4.27 and 35,798 synthetic responses per workload. The numeric-heavy workload fetches message numbers, UIDs, sizes, and modification sequences.

Workload Before CPU time After CPU time CPU time reduction
Mixed FETCH, run 1 1.738 seconds 1.727 seconds 0.6%
Mixed FETCH, run 2 1.703 seconds 1.656 seconds 2.8%
Numeric-heavy FETCH, run 1 0.555 seconds 0.374 seconds 32.6%
Numeric-heavy FETCH, run 2 0.565 seconds 0.391 seconds 30.8%

The mixed-workload difference is small enough that more runs would be needed to distinguish it from measurement variation. The numeric-heavy workload shows a substantial improvement in both runs.

All 916 unit tests pass. Of 3,125 comparisons against the previous tokenizer, four differ because the numeric EOF bug is corrected. The remaining token and exception results match.

Refs #200.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant