[experiment] On-device decompressed AssemblyStore cache (CoreCLR) by simonrozsival · Pull Request #11967 · dotnet/android

simonrozsival · 2026-07-03T07:33:57Z

Warning

Experimental, opt-in prototype — not ready to merge. The cache is CoreCLR-only and defaults off. Its memory, storage, first-launch I/O, and energy trade-offs still need broader measurement.

Builds on the Zstd AssemblyStore compression introduced by #11730. This prototype caches decompressed assemblies on-device so subsequent launches can skip Zstd decompression and use clean, file-backed mappings instead of dirty anonymous memory.

Design

Enable with AndroidEnableAssemblyStoreDecompressionCache=true in a CoreCLR Release build. The default is false.
Cache entries live under <codeCacheDir>/decompressed-assembly-cache-v1/<store-content-id>/<descriptor-index>.bin.
Assembly-store format v4 carries a deterministic 64-bit content ID, so cache lookup does not need to hash compressed payloads on every launch.
Numeric descriptor keys avoid path traversal, filename limits, and satellite-assembly names such as fr/App.resources.dll.
Each entry contains the decompressed payload plus a footer with cache magic/version, store ID, descriptor index, payload size, and an XXH3 payload checksum.
Cache hits use mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE) so runtime writes become COW while untouched pages remain clean and file-backed.
Cache misses take an immutable snapshot before exposing the shared decompression buffer to the runtime.
A detached worker drains the queue and exits. Queued snapshots are capped at 32 MiB; additional entries are skipped rather than delaying startup or growing memory without bound.
Persistence uses a content-addressed per-entry temp file followed by atomic rename. The cache is intentionally disposable: it does not fsync; incomplete or power-loss-damaged entries fail checksum validation and fall back to decompression.
Storage failures disable further writes for that process, while reads and normal assembly loading continue.
debug.net.asmcache=0|1 remains available as an internal A/B override; XA_DISABLE_ASSEMBLY_CACHE forces it off.

Android deletes codeCacheDir contents on app and platform updates. The store content ID, cache format version, exact-size checks, and payload checksum provide additional invalidation and corruption protection.

Coverage

Assembly-store generation verifies the v4 header and content ID.
Application-config generation verifies the MSBuild opt-in is emitted for CoreCLR.
A device integration test launches a Release app, waits for persisted entries, corrupts one entry, relaunches successfully, and verifies cached assemblies are file-mapped.
The native Release solution builds the cache path for all supported Android ABIs.

Performance status

The earlier prototype measured a small warm-launch improvement on a Samsung Galaxy A16:

app	warm OFF	warm ON	delta
blank MAUI	1062.8 ms	1035.0 ms	-27.8 ms
MAUI sample content	2205.6 ms	2156.6 ms	-49.0 ms

Those numbers predate the payload-integrity scan, bounded writer, and new cache format, so they must be re-measured. The expected stronger justification remains memory composition rather than latency: dirty/private PSS should move toward clean file-backed pages, but that has not yet been quantified.

Before considering a default-on product feature, measure:

dirty, clean, and total PSS after first render and on subsequent launches;
first-launch peak RSS, bytes written, worker completion time, and energy;
cache storage footprint and launch break-even;
low-, mid-, and high-end devices, including 32-bit where supported;
direct LZ4 versus Zstd comparisons under identical packaging settings.

Prototype exploring caching decompressed assemblies on-device so that subsequent launches skip zstd decompression and load the data via a file-backed mmap instead of dirty anonymous memory. - assembly-store.cc: on a decompression cache miss, a single background thread atomically writes the decompressed bytes to <codeCacheDir>/decompressed-assembly-cache/<Assembly.dll> (temp -> fsync -> rename). On the next launch the file is mmap'd (MAP_PRIVATE, COW) and decompression is skipped. Per-assembly, only assemblies actually touched are cached. Staleness guarded by an 8-byte footer holding an xxhash of the compressed payload. - Plumb codeCacheDir (Context.getCodeCacheDir()) through Java initInternal -> appDirs[3] -> AndroidSystem, so a stale cache is auto-wiped by Android on app/platform update. - Runtime A/B toggle via `debug.net.asmcache` system property (and XA_DISABLE_ASSEMBLY_CACHE env var). Experimental only: no MSBuild opt-in, no assembly-store version stamp, CoreCLR only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The background writer thread read directly from the shared uncompressed_assemblies_data_buffer, but on a cache miss that same buffer is handed to the runtime once the decompress lock is released, and the runtime may write into the assembly image (the reason the cache-hit path maps the file MAP_PRIVATE / COW). Concurrent writes could persist a torn or post-mutation image; since the staleness footer only hashes the *compressed* payload, that corrupt image would then be reloaded from cache as if pristine on the next launch. Take a private snapshot of the decompressed bytes in enqueue_write, while the caller still holds assembly_decompress_mutex and before the buffer is exposed to the runtime, so the writer only ever touches immutable memory it owns. On allocation failure we skip caching that assembly rather than aborting. Trade-off: this adds one memcpy per newly-cached assembly on the first-launch (cache-miss) path and holds the queued snapshots (up to the touched working set) transiently until the writer drains them. Subsequent launches hit the mmap path and never enqueue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tidy up the file-writing path without pulling <fstream>/iostreams into the runtime .so (only the build-time pinvoke-table generator uses those; the runtime deliberately sticks to raw syscalls to keep the library small and startup cheap). - Lay out the full on-disk image ([payload][8-byte token footer]) in the snapshot buffer at enqueue time, so the writer emits it in a single contiguous write. This drops the separate footer write (and with it a bug: that write didn't handle EINTR/partial writes) and lets WriteRequest lose its token field. - Extract a write_fully() helper for the EINTR/partial-write retry loop, leaving writer_loop as open -> write_fully -> fsync -> close -> rename. No behavior change: the cache file format is identical, so existing cache files remain valid. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move the open/write/fsync/close/rename ceremony into a dedicated write_cache_file() method so writer_loop() only owns the concurrency concerns (waiting on the queue, dequeuing under the lock) and delegates the actual persistence. No behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

simonrozsival

⚠️ Needs Changes — 1 error, 2 warnings.

Strong foundation: the cache is lazy and per-assembly, cache misses fall back cleanly, MAP_PRIVATE preserves the runtime’s writable-image requirement, and the private snapshot fixes the writer/runtime data race. The A/B switch and benchmark write-up are unusually honest about the small latency effect.

Before this could move beyond a prototype, the durability path and cache key need correction, and the first-run memory/I/O cost needs a bounded design plus measurement. CI build 1492915 is green, but there is no targeted test proving miss → persist → hit, invalidation, corruption fallback, or satellite-assembly behavior.

Make the CoreCLR cache opt-in, content-addressed, checksum-validated, and memory-bounded. Add assembly-store v4 content IDs, reader support, documentation, and host/device regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7be7d34c-aff3-4338-b15b-6061bc7a3ba9

simonrozsival · 2026-07-14T10:29:24Z

@copilot resolve the merge conflicts in this pull request

…-cache Co-authored-by: simonrozsival <374616+simonrozsival@users.noreply.github.com>

Copilot · 2026-07-14T10:59:50Z

@copilot resolve the merge conflicts in this pull request

Resolved. There was one conflict in Xamarin.Android.Common.targets where this branch added AndroidEnableAssemblyStoreDecompressionCache and main added _AndroidAssemblyStoreCompressionLevel in the same location — kept both properties.

## Summary The CoreCLR host used to locate the assembly store by **parsing the APK ZIP central directory**, `mmap`-ing the wrapper `.so`, and **walking its ELF section headers by hand** to find the payload. Since the store is already a valid ELF shared library (#9154), this PR instead makes its payload a **loadable** section (`SHF_ALLOC`, in a `PT_LOAD` segment) exposed via an exported **dynamic symbol**, so the host just: ```c void *h = dlopen("libassembly-store.so", RTLD_NOW | RTLD_LOCAL); void *start = dlsym(h, "_assembly_store"); // -> AssemblyStoreHeader ``` …and lets the platform dynamic linker locate + map it out of the APK. The hash-index lookup (`AssemblyStore::open_assembly`) is unchanged. **This is CoreCLR-only. MonoVM is untouched** and keeps the classic `DSOWrapperGenerator` + ZIP-scan store. The mechanism is **hardcoded by runtime and cannot be overridden** (`_AndroidDlopenAssemblyStore` = true for CoreCLR, false for MonoVM), because after this change the CoreCLR host has no ZIP fallback and MonoVM's host doesn't understand the loadable-symbol layout. ## What's removed (CoreCLR) - `src/native/clr/startup/zip.cc` **in its entirety** (the ZIP central-directory parser) + the `xamarin-startup` static library. - `Host::zip_scan_callback`, `AssemblyStore::map(fd/offset)`, and `Util::get_wrapper_dso_payload_pointer_and_size` usage (the ELF section-header walk). - The `DSOApkEntry {fd,offset}` native-library fast path in `MonodroidDl` (it was populated by the same ZIP scan). Bundled native libraries now load via `dlopen`-by-name, which works because they're uncompressed + page-aligned in `lib/{ABI}/` (`extractNativeLibs=false`). The now-dead `DSOApkEntry` struct, its `dso_apk_entries[]` emission in `ApplicationConfigNativeAssemblyGeneratorCLR`, and the stub definition are removed too (MonoVM keeps its own copy). Net: **889 lines deleted, 197 added (−692 net)**. The arm64 CoreCLR host (`libnet-android.release.so`) shrinks **34,784 B (−2.73%)** in a clean same-tree isolation (referenced code removed; arm −3.05%, x64 −3.34%); the actual shipped `main`-vs-PR APK host delta is **−28,640 B** (see the Size table below). ## Why this hasn't been done before (history) `dlopen`-based loading was never used, and this is **not** a regression from a nicer past design — ZIP-scan + `mmap` is the original and only design, three layers accreted over the years: | When | PR | What changed | Loading | |------|----|--------------|---------| | 2019 | #2570 | assemblies `mmap`-ed in place from the APK (never extracted) | ZIP-scan + `mmap` | | 2021 | #6311 | assembly store introduced (`assemblies.blob`, O(n)→O(1)) | raw blob, ZIP-scan + `mmap` | | 2024 | #8478 | all assemblies RID-specific; store moved to `lib/{ABI}/`, renamed `lib*.so` (not real ELF) | ZIP-scan + `mmap` | | 2024 | #9154 | wrapped in a **real ELF** (`objcopy --add-section`, non-loadable, 16 KB-aligned) for the Android 16 KB page-size requirement | ZIP-scan + `mmap` + **ELF section-header walk** | | 2025 | #9778, #9572 | CoreCLR host re-implements the same approach | same | `dlopen` only became **possible** in #9154 (the store became real ELF), and even then the payload was deliberately kept **non-loadable** so the existing `mmap` loader could be reused — on the *unverified* assumption (see [`ApkSharedLibraries.md`](https://github.com/dotnet/android/blob/main/Documentation/project-docs/ApkSharedLibraries.md): *"…without having to load the ELF image into memory"*) that `mmap`-the-section beats loading the ELF. This PR measures that assumption. ## Benchmark results Measured on a physical **Samsung A16 (SM-A165F)**, `net11.0-android` **MAUI sample-content** app (`dotnet new maui --sample-content`), Release, arm64, CoreCLR, `extractNativeLibs=false`, 30 **interleaved** cold-starts (`am start -W` TotalTime). ### Apples-to-apples (both built from the same local P7 SDK — isolates the store mechanism) Three independent 30-interleaved rounds, `main` (classic ZIP-scan) vs this PR (dlopen). Rounds 1–2 reused a fixed dlopen APK; round 3 rebuilt `latest` **fresh from the committed code** (verified store exports `_assembly_store`, section `payload` is loadable, host has no `DT_NEEDED`): | Round | main mean | dlopen mean | Δ (dlopen − main) | paired test | |---|---|---|---|---| | 1 | 2059.7 ms | 2076.1 ms | **+16.4 ms (+0.80%)** | t=2.31, p≈0.03, CI [+1.9, +31.0] | | 2 | 2061.1 ms | 2054.4 ms | **−6.6 ms (−0.32%)** | t=−1.00, p≈0.32, CI [−19.7, +6.4] | | 3 (fresh) | 2056.0 ms | 2032.3 ms | **−23.6 ms (−1.15%)** | t=−3.95, p≈0.0004, CI [−35.4, −11.9] | The three rounds span **+16 → −24 ms** (a ~40 ms between-session spread that exceeds any single round's effect) ⇒ the cross-session noise floor dominates. **dlopen is startup-neutral — within noise, if anything trending faster, and never a regression.** ### Size (arm64, `extractNativeLibs=false`, all `Stored`/uncompressed in the APK; round-3 fresh build) | Component | main | dlopen | Δ | |---|---|---|---| | APK total | 29,670,166 | 29,637,398 | **−32,768 B** | | `libmonodroid.so` (host) | 1,264,464 | 1,235,824 | **−28,640 B (−2.27%)** | | `libassembly-store.so` | 7,674,048 | 7,674,784 | +736 B (negligible) | (The host shrinks because the ZIP parser + ELF section-header walk + `DSOApkEntry` path are gone. Clean referenced-code isolation on the same tree: arm64 −34,784 B, arm −3.05%, x64 −3.34%.) ### Is there a real difference? No — it's noise The `dlopen`+`dlsym` call itself was measured on-device against the **real 7.6 MB store**: ``` dlopen = 0.14–0.49 ms dlsym = ~0.006 ms (reads magic 0x41424158 "XABA") ``` That's **~0.02% of a ~2050 ms cold start** — far below the ±20–40 ms cross-session variance. Deleting the ZIP scan + ELF walk is provably not a startup regression. > ⚠️ An earlier naive comparison (stock **global .NET 11 preview 5** vs this local **preview 7** build) showed +72 ms — but that was **confounded** by the P5→P7 delta and official-vs-local-build differences, not the dlopen change. Building `main` with the *same* local P7 toolchain isolates it to the ~0 ms above. (The residual local-P7-vs-global-P5 gap is likely partly #11730, the LZ4→Zstd store-compression switch merged after P5.) ## Can we optimize the dlopen further? No — there's nothing to skip The store `.so` is inspected with `readelf`: **0 relocations, no `DT_NEEDED`, no `DT_INIT`/init-array**, two `PT_LOAD` segments. The linker therefore does **no** relocation, dependency resolution, or constructor work — it just `mmap`s the (demand-paged) segments + minor bookkeeping. Concretely: - `android_dlopen_ext` has **no** "skip relocations/init" flag — and there's nothing to skip anyway. - `RTLD_LAZY` vs `RTLD_NOW` is moot (no code → no PLT relocations). We pass `RTLD_LOCAL` (we only `dlsym` our own handle, so there's no need for `RTLD_GLOBAL`'s extra global-scope bookkeeping), though the difference is immaterial at <0.5 ms. - The bionic `apk!/lib/{abi}/lib*.so` zip-path syntax (how `extractNativeLibs=false` is implemented) works, but measured **slower** on-device (~0.52–0.98 ms — the linker still opens the zip + does a CD lookup for the entry) and would re-introduce runtime APK-path / split-config handling. The by-soname `dlopen` is simpler and already at the floor. ## Follow-ups (out of scope) - Pairs naturally with #11967 (on-device decompressed store cache): the decompressed cache can itself be the dlopen-able wrapped `.so`, so warm starts skip both Zstd decompression and any ZIP work; `android_dlopen_ext(USE_LIBRARY_FD[_OFFSET])` is the right tool to load it by fd. - The now-orphaned `Util::get_wrapper_dso_payload_pointer_and_size` definition can be deleted as pure dead-code cleanup. - The runtime-config blob uses the same wrapper mechanism and could adopt the same symbol-based loading. ## Validation - CoreCLR app built with **no flag** auto-produces a store exporting `_assembly_store` (hardcode works) and cold-launches cleanly (Status: ok). - Store is `Stored` (uncompressed) + 16 KB-aligned; `dlsym` returns a pointer to a valid `AssemblyStoreHeader` (magic `XABA`). - The wrapper's payload lives in a section named `payload` (matching the MonoVM `DSOWrapperGenerator` and `tools/assembly-store-reader-mk2`), so the `CoreClrTrimmableTypeMap*` build tests that inspect the APK through the reader keep working. - The whole read-only store pointer chain (`configure_from_payload(const void*)`, `data_start`, `image_data`/`debug_info_data`/`config_data`, `assemblies`, `descriptor`, `assembly_store_hashes`) is `const` — no `const_cast` on the store path. ## Reference PRs - #6311, #8478, #9154, #9382, #9778, #9572 - LZ4→Zstd store compression: #11730 · on-device store cache: #11967 - [`AssemblyStores.md`](https://github.com/dotnet/android/blob/main/Documentation/project-docs/AssemblyStores.md) · [`ApkSharedLibraries.md`](https://github.com/dotnet/android/blob/main/Documentation/project-docs/ApkSharedLibraries.md) · [Android 16 KB pages](https://developer.android.com/guide/practices/page-sizes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

simonrozsival force-pushed the dev/simonrozsival/assembly-store-decompression-cache branch from 7cac3de to 9d831a5 Compare July 3, 2026 08:08

simonrozsival force-pushed the dev/simonrozsival/assembly-store-decompression-cache branch from 9d831a5 to f1de798 Compare July 3, 2026 08:12

simonrozsival and others added 2 commits July 3, 2026 10:18

simonrozsival added the copilot `copilot-cli` or other AIs were used to author this label Jul 3, 2026

simonrozsival mentioned this pull request Jul 10, 2026

[CoreCLR] dlopen()-based assembly store loading #12033

Merged

simonrozsival added the Area: CoreCLR Issues that only occur when using CoreCLR. label Jul 10, 2026

simonrozsival commented Jul 10, 2026

View reviewed changes

Comment thread src/native/clr/host/assembly-store.cc Outdated

Comment thread src/native/clr/host/assembly-store.cc

Comment thread src/native/clr/host/assembly-store.cc

Copilot started work on behalf of simonrozsival July 14, 2026 10:29 View session

Copilot AI and others added 2 commits July 14, 2026 10:59

Merge origin/main into dev/simonrozsival/assembly-store-decompression…

6167cd5

…-cache Co-authored-by: simonrozsival <374616+simonrozsival@users.noreply.github.com>

Merge origin/main into dev/simonrozsival/assembly-store-decompression…

1ad2fd8

…-cache Co-authored-by: simonrozsival <374616+simonrozsival@users.noreply.github.com>

Copilot finished work on behalf of simonrozsival July 14, 2026 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[experiment] On-device decompressed AssemblyStore cache (CoreCLR)#11967

[experiment] On-device decompressed AssemblyStore cache (CoreCLR)#11967
simonrozsival wants to merge 7 commits into
mainfrom
dev/simonrozsival/assembly-store-decompression-cache

simonrozsival commented Jul 3, 2026 •

edited

Loading

Uh oh!

simonrozsival left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonrozsival commented Jul 14, 2026

Uh oh!

Copilot AI commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

simonrozsival commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design

Coverage

Performance status

Uh oh!

simonrozsival left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonrozsival commented Jul 14, 2026

Uh oh!

Copilot AI commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simonrozsival commented Jul 3, 2026 •

edited

Loading