Repository navigation
Fix caching and skiptoken handling in paged fetches (fetchAll / fetchMany) - #109
Open
MrRoundRobin wants to merge 2 commits into
Open
MrRoundRobin wants to merge 2 commits into
MrRoundRobin wants to merge 2 commits into
Conversation
fetchAll/fetchMany paged with expand() over fetch(). With cache-and-network and a warm cache, every page emitted twice (cached, then network), so expand followed each next link twice: page N was requested 2^(N-1) times and reduce concatenated duplicated entities. All six fetchAll/fetchMany implementations now use a shared fetchPages helper that clones the resource per page. For cache-and-network it makes two full passes: a cache-only pass, emitted only when every page is cached and whose cache misses skip the api errorHandler, then a network-only pass. Other policies keep their per-page behaviour and emit one result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ODataEntitiesAnnotations.skiptoken matched /[\d\w\s'\-]+/ on the already decoded nextLink, so the token was cut at the first ',', ':', '.', '%' or '&'. Servers that order by several keys (composite keys, dates) send one Key-value pair per sort key, comma separated; the truncated token made fetchAll/fetchMany request a wrong page or fail with 400. Read the option from the raw next link and decode it once, falling back to the raw value when it is not valid percent-encoding. The next request encodes it again. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two bugs in
fetchAll/fetchManyon entity sets, navigation properties and properties.1.
cache-and-networkduplicates pages and multiplies requestsWith
fetchPolicy: 'cache-and-network'and a warm cache, every page emitted twice: once from the cache, then again from the network.expandfollowed the next link on both emissions, so page N was requested 2^(N-1) times andreduceconcatenated the duplicated entities. The first call (cold cache) looked fine; the bug appeared from the second call on.Fix: all six implementations now use one shared helper,
resources/types/paging.ts(fetchPages):Each page gets its own clone of the resource, so concurrent pages no longer mutate a shared query.
cache-and-networkmakes two full passes:cache-onlypass, emitted only when every page is cached. A cache miss ends this pass silently and does not reachapi.errorHandler; this uses an internal opt-out flag inODataApi.requesttogether with the existingisCacheMiss.network-onlypass, which requests every page once, stores it, and emits the aggregated result.The result is at most 2 emissions, with no duplicates.
Other policies keep their per-page behaviour and emit one aggregated result. When the call passes no policy, the API default policy is used.
ODataCollection.fetchAll/fetchManyare unchanged; they already reset the collection on every emission.2.
$skiptokenis cut shortODataEntitiesAnnotations.skiptokenmatched/[\d\w\s'\-]+/on the already decodednextLink, so the token was cut at the first,,:,.,%or&. Servers that order by several keys (composite keys, dates) send comma-separatedKey-valuepairs, so the truncated token requested the wrong page or failed with a 400.Fix: read
$skiptoken(or%24skiptoken) from the raw next link and decode it once. If the value isn't valid percent-encoding, use it as is. The next request encodes it again.Docs
handbook/caching.mdhas a new "Paged fetches" section.Tests
resources/types/paging.spec.tscovers:cache-and-networkwith a warm, cold or partial cache, including thaterrorHandleris not called.cache-first,cache-only, and the API default policy.fetchMany, the navigation-property path, collectionfetchAll, and composite skiptokens across pages.cache-and-networktests fail.annotations.spec.tshas new skiptoken cases.npm test: 874 passed.npm run build: OK.