feat(perf): server side paging - #464
Open
Zacgoose wants to merge 10 commits into
Open
Conversation
…plit entities Add-CIPPDbItem's orphan sweep projected OriginalEntityId without PartIndex and PartCount. Get-AzDataTableLargeEntity then recognised a split row but could not reassemble it, threw IncompleteEntity, and the wrapper skipped the whole entity - so the sweep never saw the stale split rows it exists to delete, and one uncollectable generation of rows accumulated per run. Project all three row-level split markers so split entities reassemble into logical rows that each carry their RunId, keeping the identity-based orphan check sound; Remove-CIPPAzDataTableEntity then deletes every part row. SplitOverProps and the Data chunk columns are deliberately left out: the sweep needs identity, not payload. Fixes #462
Add continuation-token paging for report DB-backed guest user, mailbox, MFA, log, and AllTenants Graph cache reads, including shared table paging helpers and frontend support for the { Results, Metadata.nextLink } contract. This keeps large cached datasets usable without loading everything at once, improves log ordering and single-entry lookup, and fixes excluded license defaults so dropdown visibility matches the intended behavior.
Replaces GUID-keyed cache rows with deterministic tenant+policy RowKeys so overlapping fan-outs upsert rather than append duplicates. Adds a manualPagination query mode to Invoke-ListConditionalAccessPolicies that returns one page per request with a continuation token in Metadata.nextLink, preventing full-table scans on large multi-tenant deployments. PageSize defaults to 5000 and is clamped to [250, 10000]. The cold-cache queue guard is skipped when a nextLink token is present so mid-walk empty pages do not re-trigger the fan-out. The frontend passes manualPagination=true to opt in. Pester tests cover the idempotent cache writes and the paged/unpaged endpoint branches.
The AllTenants Conditional Access list served cached rows by deserializing every policy blob into a PowerShell object graph and handing it back for the host to re-serialize. Each cached Policy blob is already the final flattened shape the frontend consumes, so stitch the allowed rows straight into the Results array with a raw application/json body instead, mirroring the raw passthrough Invoke-ListGraphRequest already uses. This covers both the paged and legacy unpaged AllTenants reads that share the tail. Measured on a 5000-policy page: ~1.4x faster and ~2x less allocation, and it also removes the host-side re-serialization of the whole graph. Tenant scope filtering still runs at the row level before the stitch; the response is byte-identical over the wire, so no frontend change is needed. Ordering is now table-walk order (client sorts), matching the paged groups list.
The paged AllTenants groups list deserialized every cached group blob into a PowerShell object graph just to derive membersCsv/ownersCsv, decorate it, and hand it back for the host to re-serialize. On tenants with 50k-member groups that means materializing tens of thousands of member objects per group on the read path, which is a large per-request memory spike under load. Precompute membersCsv/ownersCsv when the group is cached, then serve the page with a new Get-CIPPGroupsReport -AsRawJson mode that stitches the stored blobs into the Results array verbatim and splices the two row-level fields (CacheTimestamp, and Tenant for AllTenants) onto each blob without parsing it. The member array is never deserialized on the read path. Measured on a 5x50,000-member page: ~2x faster and ~4.5x less allocated than the previous read, and the host no longer re-serializes the whole graph. The response is byte-identical over the wire, so no frontend change is needed. membersCsv/ownersCsv appear once the groups cache is refreshed; until then the member/owner sub-tables fall back to their existing on-demand live fetch. Also fixes a stale test assertion (paged default size is 750, not 1000).
Contributor
✅ PR title follows Conventional CommitsReceived: Thanks for fixing it! |
Contributor
|
brother unit test is angry at us. |
Set-CIPPDBCacheGroups now precomputes membersCsv/ownersCsv next to their source arrays so the paged list read can stream the cached blob verbatim. Update the documented-order assertion to expect the two new fields, and give the fixture an owner and a member with a UPN so both CSVs are actually exercised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant enhancements to the API's pagination capabilities and improves data handling in several backend PowerShell modules. The main focus is on adding support for manual, token-based pagination across multiple endpoints, updating OpenAPI documentation, and making data storage more robust and deterministic.
API Pagination Enhancements:
manualPagination,nextLink,PageSize) to multiple API endpoints, including Conditional Access policies, Entra ID groups, guest accounts, Exchange mailboxes, MFA registration status, and CIPP platform audit logs. Descriptions were updated to clarify pagination behavior and response structure. (backend/Config/openapi.json: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]maxPageBytes(for controlling raw JSON page size) and improved descriptions and filtering options for audit log queries. (backend/Config/openapi.json: [1] [2]Backend Data Handling Improvements:
Get-CIPPDbItemPageto efficiently fetch one page of items from the reporting database, supporting continuation tokens and multi-tenant scenarios. (backend/Modules/CIPPCore/Public/Get-CIPPDbItemPage.ps1: backend/Modules/CIPPCore/Public/Get-CIPPDbItemPage.ps1R1-R61)Add-CIPPDbItemto ensure all row-level split markers are projected during queries, enabling correct reassembly of split entities and preventing orphaned rows. (backend/Modules/CIPPCore/Public/Add-CIPPDbItem.ps1: backend/Modules/CIPPCore/Public/Add-CIPPDbItem.ps1L133-R137)Deterministic and Robust Data Storage:
RowKeyinPush-ListConditionalAccessPoliciesAllTenants.ps1to use a deterministic, tenant-and-policy-based key instead of a GUID, preventing duplicate entries during concurrent runs and ensuring idempotency. Error rows now use a similarly deterministic key. (backend/Modules/CIPPActivityTriggers/Public/Entrypoints/Activity Triggers/Push-ListConditionalAccessPoliciesAllTenants.ps1: [1] [2] [3]