Skip to content

feat(inmemory): optional max_entries LRU capacity without changing defaults (#54) - #85

Merged
agustin18 merged 10 commits into
stablefrom
feat/54-bounded-inmemory-lru
Nov 21, 2026
Merged

agustin18 merged 10 commits into
stablefrom
feat/54-bounded-inmemory-lru

Conversation

@agustin18

@agustin18 agustin18 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

2026-11-21 integration gate (no merge): #89 and #90 are now merged into stable (b118890c, d36a2138). This #85 branch is 2 commits behind stable and must be synchronized and retested on the exact final head before approval; the historical “behind 0” snapshot below refers to the pre-merge baseline. The required independent expiry-heap/LRU/TTL/concurrency review remains open. No #54 production code has been merged; max_entries is not in published 0.5.0. Update 0.6.0 docs only after approval, and retain the public-API privacy/compatibility gates.


Purpose

Implement #54 as an opt-in feature for a future minor release (candidate 0.6.0). Published 0.5.0 remains unchanged.

Contract

  • InMemoryBackend(max_entries=None): historical per-instance, unbounded and lazy-expiration behavior unchanged.
  • InMemoryBackend(max_entries=N), with positive integer N: keep at most N entries in each backend instance. Track recency on successful get/get_with_ttl and set, evict the least recently used when full, and use an indexed min-heap of expirations to reclaim one expired key only when a new insertion needs space; updates do not scan the store.
  • All bookkeeping uses the same per-instance async lock; no background workers, sweeper task, max-bytes estimate, Redis changes, cross-worker capacity claims or new dependencies.
  • Invalid capacity (0, negatives, bool, non-integer) fails closed. Preserve expire=None, TTL and clear(namespace/key) semantics, private/auth safety and all legacy default-key vectors. Costs: O(log N) expiry-index insert/update/discard with an exact per-key heap (no stale tombstones, no periodic O(N) compaction); LRU is O(1). Explicit namespace clear scans N keys and removes k matching keys from the indexed heap: O(N + k log N), O(N log N) in the worst case, under the lock. This bulk operation is not part of ordinary writes.

Verification

  • New tests: constructor validation, legacy default >100 entries, LRU reads/writes, expired-first reclaim, namespace boundaries, explicit key clear, per-instance isolation, 180 concurrent writers/readers, FastAPI MISS/HIT/capacity eviction, regression for repeated TTL overwrites, and CI 10k/100k fill plus full-capacity live-LRU churn and expired-key reclaim on Python 3.8 and 3.12. A seeded state-machine fuzz test runs 20 seeds × 2,000 mixed mutations and validates the exact expiry heap, positions, store and recency.
  • Existing Python 3.8–3.13 CI, modern security, CodeQL, Redis compatibility and installable wheel/sdist must pass on this exact HEAD.
  • Standalone changelog fragment. No version bump, release tag, PyPI publish, or automatic merge.

Reviewer questions

Audit the indexed heap for reordering/removal and tie/expiry boundaries, verify Python 3.8/3.12 10k/100k full-capacity churn and TTL reclaim scaling, fairness under concurrent access, Pydantic/FastAPI integration, and the key/token privacy impact. This bounds entry count only, not object size. Refs #54 and future 0.6.0 tracking.

Performance blocker resolution (review required)

A reviewer reproduced O(N) per-write costs at capacities 1k–10k in the original implementation. The current code replaces that design, using a bounded indexed min-heap for expiration and evicting at most one entry per new full-capacity write. No stale expiry generations or growing heap tombstones. The scripted 10k/100k benchmark is a new Python 3.8/3.12 CI gate. Do not merge until the exact final HEAD benchmark, type/lint/security and migration checks pass and independent code review confirms the heap's index invariants.

Base synchronization and benchmark update (2026-11-21)

  • Squash-rebased onto the post-docs(site): polished 0.5.0 onboarding, safe migration and 0.6.0 roadmap (#48) #87 stable commit 8c079c8dbca645f3a2c87cf971c84a6192cbed41. All branch changes were preserved as one new commit using a guarded ref update; historical individual commit hashes were replaced. GitHub compare confirmed behind 0 / ahead 1, originally only the five functional/CI files, with no deleted documentation. Subsequent benchmark and fuzz additions remain scoped to feat(inmemory): optional bounded cache capacity and eviction #54.
  • Replaced the original fill-only benchmark with explicit fill, full_lru_churn (one entire capacity of new keys beyond a full cache), and full_expiry_reclaim (full cache of expired entries, then one new key per old entry), each at 10k and 100k on Python 3.8 and 3.12. Scaling fails closed if per-write cost grows more than 4.5x.
  • Added deterministic, dependency-free 20 seeds × 2,000 operations state-machine regressions covering all store/recency/heap-position invariants, TTL boundaries, indexed deletes and namespace clears.
  • The previous 10k/100k fill-only measurements and earlier fuzz evidence are historical and do not approve this latest exact HEAD. Require all new CI/security/Redis/migration lanes green and independent code review before ready/merge.

Counteraudit handoff — synced exact-head candidate

  • Candidate HEAD: c33aa6f571330299cf5606802e0db811836404b7 (production implementation unchanged after merge-sync; only adversarial tests added).
  • Updated base: stable at d36a21388f25ed9d9528d4ac8ed832a76b571eed (docs(pages): prepare manual hosting, public API and alternatives for 0.6.0 (#48) #89 + docs(migration): runnable published before-and-after demo for adoption #90); GitHub compare verified behind 0. The six existing feat(inmemory): optional bounded cache capacity and eviction #54 files are retained, plus tests/test_bounded_inmemory_adversarial.py.
  • Additional seeded oracle: 12 seeds × 1,800 directly mutated expiry-index states, verifying heap-parent order, exact key/position bijection and minimum expired selection, including ties, negative timestamps, overwrites and removals.
  • Additional regression: get_with_ttl refreshes LRU; a canceled waiter waiting for the async lock cannot alter the store/index; strict expire=0 boundary matches pre-existing semantics.
  • Reviewer challenge: independently replay heap mutators and expiry tie scenarios, cancellation under real contention, bulk clear under load, and changes of max_entries after construction (attribute is writable; not supported as runtime reconfiguration unless explicitly implemented/documented). Validate single-event-loop/per-process assumption; asyncio locks do not establish thread-safe cross-loop sharing. Confirm performance threshold measures per-write cost, not payload-byte memory or request latency.
  • Remaining gates: exact-head CI/CodeQL/Redis/security/migration and 10k/100k microbenchmarks on Python 3.8 + 3.12; independent sign-off. No merge, release, or claim that self-authored tests constitute the independent review.

…gates (#54)

Squash-rebase PR #85 onto post-#87 stable, preserve exactly five functional/CI files. Refs #54 #86.
@agustin18
agustin18 force-pushed the feat/54-bounded-inmemory-lru branch from 757a216 to dbd66b8 Compare November 21, 2026 03:37

Copy link
Copy Markdown
Owner Author

Exact-head CI complete — ready for independent counteraudit, NOT ready to merge (2026-11-21)

Audited candidate SHA: c33aa6f571330299cf5606802e0db811836404b7. Branch now 0 commits behind current stable; seven changed files, no new runtime dependency, no release/version bump. PR remains DRAFT.

All latest-HEAD workflows completed SUCCESS:

Reproducible scaling from the current exact-head CI:

Python 100k/10k average per-write cost: fill live LRU churn expired-key reclaim
3.8 0.986× 1.127× 1.007×
3.12 0.877× 1.210× 1.311×

100k writes on Python 3.8: 0.493 s fill; 0.686 s churn; 1.715 s expiry reclaim. Python 3.12: 0.331 s / 0.451 s / 1.099 s. Microbenchmarks only: do not infer production request latency or payload-byte memory limits. No claim of proven asymptotic complexity from one timing run.

Specific counteraudit questions: independently validate _ExpirationIndex.discard/update heap position repairs and expiry ties, get_with_ttl recency, TTL=0 boundary, stress under async cancellation/contention and clear(namespace) lock duration/complexity. Check constructor-only max_entries versus its currently writable public attribute, and unsupported cross-event-loop usage. Re-run a separate deterministic random-oracle test/negative tests rather than trusting only the self-authored suite. Check whether any visible/private cache paths or defaults change unintentionally; default mode must stay compatible.

Decision: technical CI preflight PASS; independent external code review and risk sign-off still OPEN. Do not ready/merge #85, amend 0.5.0, release 0.6.0, or count this test work as an independent consumer. Refs #54 #86 #88 #91.

Copy link
Copy Markdown
Owner Author

Counteraudit request — fix for mutable max_entries (2026-11-21)

Review candidate exact SHA: 098e37cb4530a3d4fd282080e2d0c3251e089f4a, based on stable d36a21388f25ed9d9528d4ac8ed832a76b571eed. Draft / not merged.

Confirmed previous defect at c33aa6f: after InMemoryBackend(max_entries=2) holds 2 entries, backend.max_entries = 1 changes the limit without evicting or rebuilding the expiry/LRU structures. A subsequent insert yields 2 entries while reporting a limit of 1. Assigning None, 0, negatives etc also bypasses construction validation.

Remediation (constructor-only fixed bound): fastapi_cache/backends/inmemory.py now validates once and stores self._max_entries, exposes @property def max_entries(self) -> Optional[int] without setter. No dynamic resize. The existing set code reads this immutable public property; no change to cache keys, external backends, LRU or expiry heap.

Regression: tests/test_bounded_inmemory.py tests failed assignment on both bounded and unbounded instances (including shrink, expand, None, zero, negative, boolean and string), confirms the old shrink-to-one reproduction now raises AttributeError, and then checks max_entries==2, exact LRU eviction, store/recency count and the empty heap in a no-TTL run.

Verifier tasks: On exact SHA, rerun the reproduction and independent negative mutations, confirm constructor validation + no accidental setter, and review behavior with inherited/custom subclasses. Check get_with_ttl, TTL, namespace clear, heap index invariants and concurrent insertion regressions did not change. _max_entries remains an internal (private-by-convention) field, not a supported resize API.

CI note: First revised SHA e1a4dc3 failed only Ruff B010 on deliberate setattr in the regression. Follow-up SHA 098e37cb4530a3d4fd282080e2d0c3251e089f4a adds only a per-line # noqa: B010 for this negative test. Current exact-head CI/CodeQL/migration/security results must complete before GO; previous c33aa6f 29/29 + 13/13 greens cannot be inherited. No release, merge or Pages activation authorized.

Other 0.6 gates: security #53 review, two independent consumer exercises #91 (currently 0/2), final artifacts/supply-chain review and explicit manual publication. #50 remains optional and should not delay 0.6.0; campaign plan #88.

Please respond with GO/NO-GO for merge on the exact SHA, identifying any remaining functional blocker and separating documentation/launch concerns from the #85 code change.

Copy link
Copy Markdown
Owner Author

Final counteraudit handoff — exact HEAD is CI-clean (2026-11-21)

Exact HEAD: 098e37cb4530a3d4fd282080e2d0c3251e089f4a; base stable: d36a21388f25ed9d9528d4ac8ed832a76b571eed; behind 0. PR is still DRAFT / unmerged. Defect reproduction, patch explanation and independent review questions.

New exact-head verified workflows — ALL SUCCESS:

10k → 100k capacity mean per-write cost ratio from exact-head CI (backend-only microbenchmark, not request latency):

Python Fill Full live LRU churn Full expired reclaim
3.8 0.939× 1.157× 1.263×
3.12 0.806× 1.314× 1.446×

Independent reviewer decision needed: Verify (1) that all public post-construction attempts to mutate max_entries raise AttributeError; (2) constructor validation and pre-existing unbounded behavior remain untouched; (3) the original capacity=2 → 1 reproduction is prevented, with LRU and expiry-index consistency; (4) any remaining heap, lock, TTL and namespace-clear hazard at this exact head. The read-only property intentionally does not implement dynamic resizing. Internal _max_entries is private by convention, not a supported user-facing setter.

Proposed disposition: CI preflight GO; await independent counteraudit GO/NO-GO before ready/merge. This success does not authorize a 0.6.0 tag/RC/PyPI, external marketing or 0.6.0 launch. Release checklist #92, security review #53, external consumers #91 (0/2) and Pages #48 remain separate gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant