Repository navigation
Make the getindex add! agree with its value - #872
CarloLucibello wants to merge 3 commits into
Conversation
The `add!` of the `InplaceableThunk` returned by the `getindex` and `view` rules added the raw cotangent, skipping the `ProjectTo(x)` that `val` applies, so it could differ from `unthunk` or throw. It now projects `dy` when `ProjectTo(x)` acts elementwise, staying O(length(dy)), and adds `val` otherwise. Scalar `∇getindex` also projects `dy` before building the `OneElement`, which avoids materializing it for mixed precision. Part 1 of #863. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #872 +/- ##
==========================================
- Coverage 96.65% 96.61% -0.05%
==========================================
Files 28 28
Lines 3615 3630 +15
==========================================
+ Hits 3494 3507 +13
- Misses 121 123 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
@devmotion could you take a look? |
|
I'm surprised by how many changes this PR makes, and the downstream Zygote tests fail. AFAICT the only problem is that The Zygote failures ( I opened #875 with a smaller alternative: |
Part 1 of #863. The
add!of theInplaceableThunkfrom thegetindex/viewrules added the rawdy, skipping theProjectTo(x)thatvalapplies. Sot.add!(dx)could differ fromdx + unthunk(t), or throw (e.g. a complexdyfor a realx), and an AD couldn't safely call it.ProjectTo(x)acts elementwise (arrays of numbers, including GPU arrays),add!projectsdyand adds it intodx[inds...], staying O(length(dy)). Arrays of arrays use only the projectors of the indexed elements. Other arrays (Diagonal,Symmetric, sparse, …) fall back toadd!!(dx, val).∇getindexprojectsdybefore building theOneElement, so aFloat64cotangent for aFloat32array no longer materializes a dense array.Benchmarks with Zygote 0.7.16 on Julia 1.13:
x[i]loop,x::Vector{Float32}of length 10⁴, summed inFloat64: 26.5 ms / 391 MiB → 1.80 ms / 8.3 MiB.X[:, j]loop over a 100×1000 matrix: unchanged at 36 ms / 767 MiB, since Zygote unthunks. With Zygote callingadd!when the inferred eltype ofvalfits its buffer, which this PR makes safe: 0.41 ms / 5.1 MiB.add!itself costs the same as before, on CPU and on GPU.🧔♂️🤖 Assisted by AI