fix(graphical): EP builds strict NormalMessage from negative-variance cavity → sigma<0 crash

[Jammy2211]

Overview

The nightly PyAutoHeart workspace-validation.yml run is RED on a single cell, run_scripts (3.12, autolens, guides), because autolens_workspace/scripts/guides/modeling/advanced/expectation_propagation.py crashes during expectation propagation with autofit.exc.MessageException: NormalMessage sigma cannot be negative. It is stochastic/unseeded (sigma=-0.016 on 2026-07-13, sigma=-0.17 on 2026-07-11), so the negative width is produced inside the EP message-passing algebra, not by a fixed model mistake. The fix belongs in the PyAutoFit library — the user-facing guide must keep exercising the real EP path (no seeding / no autoimmunity).

Plan

• Reproduce the crash locally with a fixed seed that forces the negative-variance EP branch; capture the exact frame constructing a strict NormalMessage from a negative-precision/variance message.
• Determine whether (a) the EP path should route that construction through NaturalNormal / update_invalid so a transient negative-variance cavity is tolerated or damped, or (b) the projection/inversion feeding sigma has a sign/validity bug.
• Implement the minimal library fix in autofit/graphical/expectation_propagation and/or autofit/messages/normal.py; keep the strict assert_sigma_non_negative guard intact for the genuine prior-passing misuse it was added for.
• Correct the now-misleading RelativeWidthModifier hint in the exception text (that path already abs()-guards and cannot emit a negative sigma).
• Add a regression test under test_autofit/graphical/ driving an EP update through a negative-variance cavity without raising.
• Validate: full test_autofit/ green (3.12/3.13), smoke the EP guide script, Heart readiness. The guide script is not modified.

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean

Suggested branch: feature/ep-negative-sigma-crash

Root cause (grounded)

• The guard assert_sigma_non_negative (autofit/messages/normal.py:45) rejects sigma < 0 at strict NormalMessage.__init__ (normal.py:110). It was added deliberately for the prior-passing misuse case and is load-bearing there — do not delete it.
• Its error hint blames RelativeWidthModifier, but that is stale for this path: RelativeWidthModifier.__call__ already returns value * abs(mean) (autofit/mapper/prior/width_modifier.py:110, Priors & messages: 9 confirmed bugs — guidance wanted on 5 decisions #1331 D5), so it cannot emit a negative sigma.
• EP legitimately produces intermediate messages with negative/infinite variance (cavity division of Gaussians). The correct vehicle already exists: NaturalNormal (normal.py:508), which permits eta2 in (-inf, 0) and bypasses the guard via AbstractMessage.__init__. The bug is that somewhere on the EP/declarative path a strict NormalMessage is built from a negative-variance state where a NaturalNormal should be used (or the projection/inversion that yields the passed sigma has a sign/validity bug).

Implementation Steps

1. Reproduce: run the EP guide (or a minimal EP factor-graph) under a seed forcing a negative-variance cavity; capture the traceback frame that calls strict NormalMessage(mean, sigma).
2. Inspect autofit/graphical/expectation_propagation/ep_mean_field.py (cavity prod/division ~L196-222, rescale ~L447) and factor_optimiser.py (calc_exact_update/calc_exact_projection).
3. Route the offending construction through NaturalNormal / AbstractMessage.update_invalid (abstract.py:346), or correct the inversion sign — narrowest change that unblocks EP without weakening the prior-passing guard.
4. Fix the misleading RelativeWidthModifier hint in the exception at normal.py:48.
5. Add a regression test in test_autofit/graphical/ that drives an EP update through a negative-variance cavity without raising; keep an assertion that strict NormalMessage(sigma<0) still raises for the prior-passing case.
6. Run python -m pytest test_autofit/ (3.12 + 3.13); smoke the EP guide script; confirm Heart readiness.

Key Files

• autofit/messages/normal.py — strict NormalMessage guard (L45/L110) vs NaturalNormal (L508); exception hint (L48).
• autofit/graphical/expectation_propagation/ep_mean_field.py — cavity build / division / rescale.
• autofit/graphical/expectation_propagation/factor_optimiser.py — exact update/projection.
• autofit/messages/abstract.py — update_invalid (L346), from_natural_parameters (L173), project (L266).
• test_autofit/graphical/ — new regression test.

Trade-off

Preserve the strict sigma<0 guard for prior passing; only tolerate/damp negative-variance messages on the EP natural-parameter seam. Keep the change minimal.

Evidence

PyAutoHeart workspace-validation runs 29227574734 (2026-07-13) and 29153442364 (2026-07-11). Related history: #1331 / #1348 (sigma<=0 semantics, point-mass idiom).

Original Prompt

Click to expand starting prompt

Type: bug · Target: autofit · Repo: PyAutoFit · Difficulty: medium · Autonomy: safe

The nightly PyAutoHeart workspace-validation.yml run is RED on run_scripts (3.12, autolens, guides) because the EP guide script crashes with NormalMessage sigma cannot be negative (sigma=-0.016 on 2026-07-13, sigma=-0.17 on 2026-07-11 — stochastic/unseeded). Root cause: the sigma<0 guard (normal.py:45) is load-bearing for prior passing, but RelativeWidthModifier already abs()-guards, so the EP crash comes from the message-passing algebra producing a negative-variance message that is wrapped as a strict NormalMessage instead of NaturalNormal. Fix in the PyAutoFit library (graphical + messages/normal.py), not the workspace guide (no autoimmunity). Fix the stale RelativeWidthModifier hint; add an EP regression test.

[Jammy2211]

--auto run — parked at ship sign-off (supervised checkpoint)

The fix is implemented and gated, but the effective autonomy level is supervised (the bug work-type cap overrides the prompt's safe header: min(safe, supervised) = supervised), so the PR-open is a human checkpoint rather than an unattended ship.

What was done — root cause pinned by deterministic reproduction: the EP guide's HierarchicalFactor draws mass.slope from GaussianPrior(mean, sigma) whose sigma is a truncated hyper-prior (lower_limit=0). EP's factor optimiser evaluates at proposed scales that dip just below 0; Factor.__call__ then built GaussianPrior(sigma<0), tripping the strict NormalMessage guard and crashing. Fix (24 insertions, library-only): Factor.__call__ catches exc.MessageException and returns -inf (an out-of-support distribution has zero density — correct EP semantics); the strict guard is untouched; the stale RelativeWidthModifier hint is corrected; a deterministic regression test is added. Committed locally as a6aae649f on feature/ep-negative-sigma-crash (not pushed).

Four-leg gate:

• Tests — PASS: full test_autofit/ = 1473 passed, 1 skipped (in-worktree). test_full_hierachical fails only in isolation (order/RNG-sensitive, pre-existing on main — identical value with the fix stashed); passes in the in-order suite.
• Smoke — the EP guide is not in the curated smoke_tests.txt subset (stated fact, not grown). As targeted validation the actual failing guide (guides/modeling/advanced/expectation_propagation.py) is being run end-to-end against the branch; it is past the former ~4.7s crash point and into the Nautilus EP factor fits with zero MessageException markers.
• Review — CLEAN: narrowly-scoped except, correct zero-density semantics, no public-API change, guard intact, deterministic test.
• Heart — YELLOW score 50: test_run NOT ready (3 failed) dated 2026-07-09 (stale evidence) + 58 stale parked scripts. Both are organism-scope and unrelated to this branch. Not acknowledged at launch.

Fork — needs a human decision: Heart YELLOW's reason set was not acknowledged at launch, and the ship sign-off is a supervised checkpoint. To ship: (a) acknowledge the YELLOW reason set (stale test_run + stale parked scripts) as unrelated to this branch, and (b) authorize PR-open. On approval → push feature/ep-negative-sigma-crash and open the pending-release PR (merge still stays human). On hold → the branch stays committed-but-unpushed here.

Recommendation: ship — legs 1 & 3 are clean, the YELLOW reasons are stale organism-scope evidence with no bearing on a localized EP fix, and the guide is validating the branch live.

[Jammy2211]

Correct-mode validation (the fast smoke) + confirmation it's not a test-mode-only artifact

Good catch on test mode. The nightly runs the guide with PYAUTO_TEST_MODE=1 PYAUTO_SMALL_DATASETS=1 PYAUTO_FAST_PLOTS=1 (reduced real sampler, 15×15 datasets) — fast, seconds-to-a-minute. My earlier full-mode run (no flags, hours) was the wrong mode and is moot; the proper smoke is the guide in the nightly's mode:

• base (pre-fix) + test-mode → crashes on run 1: NormalMessage sigma cannot be negative, got sigma=-0.0705…, raised from autofit.graphical.laplace.newton — the Laplace/Newton optimiser for the hierarchical factor.
• fix + test-mode → completes clean (exit 0, full MeanField output).

Is it a test-mode trigger? Test-mode's reduced iterations make the Laplace optimiser's coarser steps overshoot the scale hyper-prior's lower_limit=0 more readily — so the crash is far more frequent there. But it is a genuine code path, not a test-mode-only artifact: the scale posterior sits at mean≈0.0005 (hard against the boundary), so the optimiser probes negative scales; a real run whose scale posterior is near zero can hit the same path. Returning -inf for an out-of-support distribution scale is the correct EP semantics in every mode, and it does not weaken the strict NormalMessage guard. The fix is validated end-to-end (both directions) in the exact mode the failure occurs.

[Jammy2211]

Library PR created — #1364 (pending-release)

Shipped behind the human ship sign-off at the supervised checkpoint. PR: #1364

Gate: tests 1473p/1s (full test_autofit/) · smoke base-crashes-vs-fix-clean in the nightly test-mode · review CLEAN · Heart YELLOW acknowledged as unrelated (stale test_run + stale parked scripts).

Workspace impact: none (option iii). HierarchicalFactor's public API is unchanged; the change is internal to Factor.__call__ plus an exception-message correction. No workspace migration needed.

Merge stays human. The next nightly workspace-validation will independently confirm the autolens/guides cell is green once this is on main.

[Jammy2211]

Shipped — merged in #1364

Merged to main (squash) 2026-07-13. The EP hierarchical-factor out-of-support scale now returns -inf instead of crashing; validated base-crash/fix-clean in the nightly test-mode. The next workspace-validation run will confirm the autolens/guides cell green. Closing.