Skip to content

Fix AD on optical axis - #148

Merged
StackEnjoyer merged 11 commits into
masterfrom
fix/ad-optical-axis
Oct 27, 2026
Merged

StackEnjoyer merged 11 commits into
masterfrom
fix/ad-optical-axis

Conversation

@TacHawkes

@TacHawkes TacHawkes commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator
  • _radial is exactly 0 on the axis. Surfaces that cross the axis are computed from smooth expressions: the 3D distance to the sphere center (CutSphereSDF, ConvexSphericalSurfaceSDF), and sag and slope of aspheres and conics as functions of r².
  • UnionSDF resolves ties between operands by the operand that is closest beside the point, independent of operand order. Within the aspheric SDFs, the surface wins ties with its closing segments.
  • Hits of _raymarch_outside that end inside have their normal evaluated outside.
  • refractive_index(optic, λ) returns the number type of the optic and the wavelength.

Tests: new test/TestAutoDiff.jl compares ForwardDiff with central finite differences (42 tests):

  • rays on and off the axis through spherical (bi-convex, bi-concave, meniscus), aspheric (plano-convex, plano-concave) and cylindrical lenses
  • spherical, parabolic and conic mirrors
  • the derivative w.r.t. a refractive index
  • a GaussianBeamlet

ForwardDiff is added to the test dependencies. SDF values, normals and hit lengths agree with master to ~1e-17 m. Merged with #142, the AD tests and the exit-point check pass as well.

Cost: no additional allocations. 40 lenses take ~11 % longer (more SDF evaluations for normals in unions), the aspheric lens slightly less time.

Agent skill: updated skills/beamletoptics/WORKFLOW.md (new section "Derivatives"); no other skill files affected.

Fixes #147.

TacHawkes and others added 2 commits October 10, 2026 09:30
The distance field of an aspheric or acylindric surface evaluated the
asphere at its edge (sag and gradient) on every call, although both are
constant for a surface, and summed the even terms with integer powers.
The SDFs now store the edge values (`edge`, see `aspheric_edge`) and the
sums use Horner's scheme in r². An empty coefficient vector now gives
the conic surface instead of an error.

One `sdf` call of an asphere drops from ~107 ns to ~29 ns; a beam group
of 20000 rays through an aspheric lens solves in 83 ms instead of 163 ms
(1 thread). Distances agree with the previous code to ~3e-15 relative,
ray intersections are bit-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ForwardDiff through `solve_system!` gave the correct derivative for
rays off the axis but 0 or a wrong value for rays on it, without a
warning, e.g. for the spot position behind a lens w.r.t. its lateral
shift: 0 instead of 1.90 (bi-convex), -0.97 instead of -1.98
(bi-concave), NaN for parabolic and conic mirrors. Rays on the axis,
e.g. the chief ray of a rotationally symmetric system, are the ones an
optimizer differentiates. The causes, all where the derivative of a
normal is taken, i.e. in nested automatic differentiation:

- The radius of revolution `norm(Point2(x, z))` is `NaN` on the axis in
  nested AD, which turned the comparisons of SDF branches, and the
  derivative of the radius is not defined there. `_radial` is exactly 0
  on the axis, and the surfaces that cross the axis are computed from
  smooth expressions: the 3D distance from the center of a sphere
  (`CutSphereSDF`, `ConvexSphericalSurfaceSDF`), the sag and slope of
  aspheres and conics as functions of r².
- Where a concave surface touches the face of the lens body at its
  vertex, both are equally close, and the flat face was taken, which
  has no curvature. `UnionSDF` now resolves such ties by the operand
  that is closest beside the point (`_closest_operand`), independent of
  the order of the operands; within the aspheric SDFs the surface wins
  ties with the segments that close the perimeter.
- A hit of `_raymarch_outside` that ends slightly within the shape has
  its normal evaluated outside, as the exits of the inside march.
- `refractive_index(optic, λ)` returned `Float64`, which dropped the
  derivative w.r.t. a refractive index or the wavelength; it returns the
  number type of the optic and the wavelength.

New `test/TestAutoDiff.jl`: ForwardDiff against central finite
differences for rays on and off the axis through spherical, meniscus,
aspheric and cylindrical lenses and spherical, parabolic and conic
mirrors, w.r.t. a refractive index and through a GaussianBeamlet
(ForwardDiff added to the test dependencies). SDF values, normals and
hit lengths agree with the previous code to ~1e-17 m. Built on the
aspheric changes of #137.
@TacHawkes TacHawkes added this to the v0.14.0 milestone Oct 11, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@TacHawkes TacHawkes added the bug Something isn't working label Oct 11, 2026
@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.80851% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.52%. Comparing base (30931ea) to head (341a97f).

Files with missing lines Patch % Lines
src/SDFs/AbstractSDF.jl 80.00% 2 Missing ⚠️
src/SDFs/AsphericalLensSDF.jl 97.77% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #148      +/-   ##
==========================================
- Coverage   90.85%   90.52%   -0.34%     
==========================================
  Files          86       86              
  Lines        4626     4610      -16     
==========================================
- Hits         4203     4173      -30     
- Misses        423      437      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A wider check of ForwardDiff against finite differences found further
failures, now fixed:

- Polarized rays and astigmatic beamlets at normal incidence threw an
  orthogonality error: `isparallel3d` took the `norm` of a cross product
  that vanishes but has a derivative, which is `NaN` (also its value)
  for dual numbers, hence the s-p basis was built from a zero vector. It
  now compares the squared norm. The Fresnel coefficients of lenses and
  polarizers are computed from the cosine of the angle of incidence
  (`_fresnel_coefficients`, `cos_angle3d`), since the derivative of the
  `acos` is not defined at normal incidence.
- `ThinBeamsplitter` (also within `CubeBeamsplitter`) required the
  reflectance in the number type of the shape; a constructor converts it.
- `CircularFlatMesh` built its vertices as `Float64`, which broke round
  plate beamsplitters, polarization filters and mirrors.

`test/TestAutoDiff.jl` now covers 68 cases: radius, thickness, tilt,
refractive index, polarized rays and astigmatic beamlets on the axis,
thin, plate and cube beamsplitters, total internal reflection, Malus'
law, the power on a detector and the power of a Michelson
interferometer vs. the position of a mirror.

New docs page "Automatic differentiation" (usage, covered components,
limitations: discrete events, edges, forward mode only, compile time,
`ObjectGroup(objects, T)`), and a section in the agent skill
(`WORKFLOW.md`). `ObjectGroup` still defaults to `Float64`; its
constructor is changed by #135.
# Conflicts:
#	src/SDFs/AsphericalLensSDF.jl
@StackEnjoyer

Copy link
Copy Markdown
Owner

Agent review, round 1: approve with changes · 0 blocker, 4 should, 1 nit, 2 notes · merges cleanly, 5425 tests pass

  • Asphere SDFs evaluate sag and slope twice, slower also without AD
  • Tie-break probes 1 µm beside the surface: too coarse for micro-optics
  • No AD test against a closed-form reference, only finite differences
  • PR description does not state which skill files changed (AGENTS.md)
Full review for agents (7 findings)

Scope

Verified

  • On the merge with master: TestAutoDiff, Geometry/SDFs/* (AbstractSDF, ConicSDF, UnionSDF, DifferenceSDF), Geometry/TestMesh, Lenses/* (Aspherical, Spherical, Cylindrical, Surfaces), Components/TestBeamsplitters, TestPolarizers, TestPrisms, ConicMirrors/TestConicMirror, TestUtils, TestPolarizedRays, TestAstigmaticGaussianBeamlet, TestBugFixes, E2E/TestAnalyticReferences, TestDoubleGaussLens, TestSonnarLens, TestMichelson: 5425/5425 pass (Julia 1.13.0)
  • Checked by hand: _aspheric_slope2_r2 equals (dz/dr)² with dz/dr = r·G(r²); cos_angle3d equals cos(angle3d(...)) (angle3d does not fold into the first quadrant); norm(Point3(p[1], p[3], q[2])) == norm(q) in CutSphereSDF and ConvexSphericalSurfaceSDF; the clamped r_c2 in ConicSDF equals the old clamp of r_c; ThinBeamsplitter outer constructor is not ambiguous with the default one
  • Not run: the full suite, the rendering tests, any benchmark

Findings

F1 · should · src/SDFs/AsphericalLensSDF.jl:251: sag and slope are computed twice per sdf call

  • Finding: convex_aspheric_surface_distance (l. 251–252) and concave_aspheric_surface_distance (l. 304–305) still evaluate aspheric_equation(r, …) and gradient_aspheric_equation(r, …), now used only for the NaN check and the branch conditions; l. 273 and l. 322 then evaluate the same sag and slope again via _aspheric_sag_r2 and _aspheric_slope2_r2. _aspheric_slope2_r2 (l. 195) repeats the formula of gradient_aspheric_equation (l. 178) and _even_aspheric_sum_derivative (l. 165).
  • Evidence: read at head. The PR description reports ~11 % more time for 40-lens systems; that figure is from the author, the reviewer did not measure it.
  • Expected: one evaluation of sag and slope per sdf call, as functions of r², and one copy of the gradient formula. The overhead for tracing without AD shrinks measurably; SDF values stay within ~1e-15 relative of head, intersections unchanged.
  • Suggestion: take the NaN guard from r² (1 - (1 + k) c² r² < 0) and let the r-based functions delegate to the r² forms, if anything outside these two functions still uses them (also check BeamletOpticsGUI.jl).

F2 · should · src/SDFs/UnionSDF.jl:92: fixed probe offset ρ = 1e-6 in _closest_operand

  • Finding: ties between operands are broken by probing 1 µm beside the surface. The offset is absolute and not configurable, and the docstring does not state a validity range.
  • Evidence: read at head. For mm-scale optics 1 µm is small against the features; for µm-scale features (micro-optics, thin coatings modelled as volumes) the probes can land beyond the feature.
  • Expected: either the docstring states the scale for which the tie-break is valid, or ρ comes from Config like get_sdf_raymarch_eps, so it can be changed without editing code.

F3 · should · test/TestAutoDiff.jl:130: no case against a closed-form reference

  • Finding: every case compares ForwardDiff with finite differences of the same BMO function. That checks the derivative, but not that the traced quantity itself is right; this project's physics tests also compare with closed-form references (raised in the review of PR Bugfix: GaussianBeamlet phase in media and behind lenses #133).
  • Evidence: read at head. A case with an exact reference exists: for a ray on the axis and a lens shifted laterally by dx, d x_spot / d dx at dx = 0 is the paraxial value 1 - A, with A the position element of the ABCD matrix from the front vertex to the detector. Computed by hand for the docs example (R = ±50 mm, t = 5 mm, n = 1.5, front vertex at y = 0, detector at y = 100 mm): 1.9017, consistent with "≈ 1.90" in docs/src/basics/autodiff.md.
  • Expected: at least one case whose expected value is built in the test from a formula (here the ABCD matrix of the parameters), with a tight tolerance (~1e-6 relative, the reference is exact). A mirror counterpart ([1 0; -2/R 1]) would be welcome but is optional.

F4 · should · PR description: skill files not named

  • Finding: AGENTS.md ("Agent skill") requires the PR description to state which skill files were updated. The PR changes skills/beamletoptics/WORKFLOW.md but the description does not say so.
  • Evidence: PR description at the time of review.
  • Expected: one line in the description naming WORKFLOW.md.

F5 · nit · docs/src/basics/autodiff.md:27: example uses internal names

  • Finding: the entry example of the page reads the result through BeamletOptics.hit_point(only(BeamletOptics.hits(pd))); neither name is exported or declared public.
  • Evidence: src/Exports.jl at head. docs/src/tutorials/openraman.md does the same, so this is not new.
  • Expected: fine as it is if there is no public accessor; no further internal names in the docs.

F6 · note · src/SDFs/AsphericalLensSDF.jl:213: _min_surface_first may overestimate the distance

  • Finding: within sdf_surface_threshold (1e-9 m) it returns the surface distance instead of d_closure / g, so the SDF is no longer a strict lower bound in that band, by at most the threshold.
  • Evidence: read at head; the docstring says so, the tests pass.
  • Expected: no change requested. Keep the statement in the docstring.

F7 · note · src/SDFs/AbstractSDF.jl:134: normal after a step that ends inside

  • Finding: the normal is evaluated after backing off eps - dist along dir. At grazing incidence that point can still lie slightly inside the shape.
  • Evidence: read at head, by reasoning; not observed in a test.
  • Expected: no change requested; a limitation of the approach, not a defect.

Hugo Uittenbosch and others added 3 commits October 23, 2026 16:04
The surface distances computed the sag and the slope via the functions of r for the NaN guard and the branch conditions, then again via the functions of r^2 for the distance. They now evaluate both once, as functions of r^2, with the guard taken from the square root argument. gradient_aspheric_equation delegates to the same slope, so its formula exists once.

Review F1 on PR #148.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review F2 on PR #148.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For a ray on the axis and a laterally shifted lens or mirror, the derivative of the spot w.r.t. the shift is 1 - A of the paraxial ABCD matrix, built in the test from the parameters.

Review F3 on PR #148.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@StackEnjoyer

Copy link
Copy Markdown
Owner

Agent has addressed review comments

uitt_hu and others added 2 commits October 23, 2026 22:31
Since #171, optical_power takes the sampling as an argument instead of the keywords n, x_min, ...

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@StackEnjoyer

StackEnjoyer commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

This branch now conflicts with #174 (which is already merged into master) and is not mergable without failing master pre-CI due to #173.

@TacHawkes

Conflict with #174 in _raymarch_outside: the hit is the secant root of #174,
and its normal is evaluated outside as in this branch. _secant_root also
returns the SDF at the root and the slope along the ray; the point is moved
back until it lies about as far outside as the root lies inside, at least
eps/100 (ties of composite SDFs exactly on the surface) and at most eps.
Julia 1.14.0-alpha1 calls issubnormal in normalize, which ForwardDiff 1.4.6
does not define for Dual (JuliaDiff/ForwardDiff.jl#852, fix in #855). The
job keeps running and reporting, but no longer blocks merges.
@TacHawkes

Copy link
Copy Markdown
Collaborator Author

Merged master into the branch (010acd3). The conflict with #174 is in _raymarch_outside: the hit is now the secant root of #174, and its normal is still evaluated outside as in this PR, moved back along the ray by about the depth of the root, at least eps/100 (to break ties of composite SDFs exactly on the surface) and at most eps. AD (70/70), SDF and the full suite pass locally. For #173, the pre job now has continue-on-error (341a97f): it keeps reporting, but no longer blocks merges until ForwardDiff releases JuliaDiff/ForwardDiff.jl#855.

@StackEnjoyer
StackEnjoyer merged commit 8b85aa4 into master Oct 27, 2026
5 of 7 checks passed
@StackEnjoyer
StackEnjoyer deleted the fix/ad-optical-axis branch October 27, 2026 13:55
TacHawkes added a commit that referenced this pull request Oct 28, 2026
Conflicts with #148: the Hint glue in Lenses.jl and LinearPolarizer.jl stays
removed, the generic methods use the transition. Ported from #148 into the
transition code: Fresnel coefficients from the cosine of the angle of
incidence (cross_boundary, PolarizationFilter), smooth for AD at normal
incidence; the refractive indices of a transition keep the number type of
the ray (e.g. ForwardDiff.Dual) instead of Float64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ForwardDiff gives wrong derivatives for rays on the optical axis

2 participants