Repository navigation
perf(state): read a whole pair's baseline as one pass over the table - #30
Merged
Merged
Conversation
The 5-minute server poll reloads the whole baseline, and on the desktop that took 3-4 s for 352,312 rows every time, against about 1 s for the scan itself. Measuring a copy of that database showed the time going on the primary-key index: the baseline is a rowid table, so each row found through the index needs a second lookup into the table, in path order, which is random across a 140 MB file with SQLite's 2 MB page cache. Reading the table straight through returns the same rows in 0.65 s rather than 2.5 s. A full load now reads the table directly when the pair holds at least a quarter of the table's rows (MAX(rowid) stands in for the table size), and stays on the index for a small pair in a big table. The pair's row count also sizes the map up front. On the desktop copy LoadBaseline went from 2.8 s to 0.9 s. Nothing is kept in memory, so low-memory mode keeps its footprint. Deck d1102.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 5-minute server poll reloads the whole baseline, and on a 352k-row account that took 3-4 s each time. The baseline is a rowid table, so reading a pair through the primary-key index costs a second, random lookup into the table for every row. A full load now reads the table straight through when the pair is at least a quarter of the table, and stays on the index otherwise. On a copy of that database LoadBaseline went from 2.8 s to 0.9 s, with nothing extra held in memory.