Skip to content

[FIX] Carry process variance through Chainladder and Benktander predict - #1652

Open
Huyvux12 wants to merge 1 commit into
casact:mainfrom
Huyvux12:fix/issue-1639-predict-process-variance
Open

Huyvux12 wants to merge 1 commit into
casact:mainfrom
Huyvux12:fix/issue-1639-predict-process-variance

Conversation

@Huyvux12

Copy link
Copy Markdown
Contributor

[FIX] Carry process variance through Chainladder and Benktander predict

On a triangle from BootstrapODPSample, fit() adds the simulated process variance to ultimate_ through MethodBase._include_process_variance, but predict() did not call it, so predict(sims).ultimate_ was each draw's point estimate. VotingChainladder builds its ultimate from est.predict(...).ultimate_, so a voted bootstrap lost the variance too.

This makes Chainladder.predict and Benktander.predict apply _include_process_variance to the predicted triangle (a shared helper gained an optional triangle argument), so predict matches fit. VotingChainladder inherits the fix through predict with no change of its own.

Closes #1639

Repro (script from the issue, base a59efc9d)

Before the fix, total IBNR standard deviation:

fit       18939.9
predict   17337.3
voting    17337.3

After the fix:

fit       18939.9
predict   18939.9
voting    18939.9

predict and voting ultimates are exactly equal to the fit ultimate (same draws: _get_process_variance seeds from the triangle's random_state + 1).

The same bug existed in Benktander.predict (genins bootstrap, n_sims=2000, random_state=42, Benktander(apriori=1.0, n_iters=2)): fit std 967268.7 vs predict std 710740.2 before, 967268.7 for both after. Voting with Chainladder + Benktander members goes 1674616.4 -> 1787187.8; the remainder vs the Chainladder fit std (2926830.8) is the averaging effect from question 2 of the issue (average of two different draws narrows the spread), not missing variance.

New tests (chainladder/methods/tests/test_predict.py)

  • test_predict_carries_process_variance: predict std equals fit std, predict ultimate equals fit ultimate, predict does not mutate the fitted model, voting std equals fit std, and a plain triangle still predicts exactly as before (process_variance_ is None).
  • test_predict_subset_of_simulations_keeps_shape: predicting on a subset of simulations keeps the subset's shape and falls back to point estimates (see below).
  • test_benktander_predict_carries_process_variance: same check for Benktander.

Regression scope

  • pytest chainladder/methods/tests/test_predict.py: 38 passed (new tests run under both the numpy and sparse backends).
  • pytest chainladder/methods/tests/: 104 passed.
  • ruff check and ruff format --check clean on all touched files.
  • Docs doctests run in CI via the jb build doctest job; not run locally here. The predict docstring examples use plain (non-bootstrap) triangles, whose outputs are unchanged.

Risks checked and excluded

  • No mutation of fitted state or the caller's triangle: model.ultimate_, model.X_ and model.process_variance_ are bit-identical after predict; the caller's sims is unchanged; predict twice gives identical results; fit_predict matches fit.
  • sample_weight path: with a ones weight and with a latest_diagonal weight, predict std still equals fit std exactly.
  • Plain triangles cost nothing extra: when the predicted triangle carries no _get_process_variance, predict returns None before building the full triangle (an earlier version built it eagerly and OOMed test_misaligned_index[sparse_only_run] on a large sparse triangle).
  • Subset of simulations: _get_process_variance is bound to the full bootstrap triangle, so a variance draw computed for a subset would silently carry the full 2000-sim shape. The fix only applies the draw when its shape matches the predicted ultimate's shape and otherwise keeps the base point estimates with the correct shape.

Deliberately unchanged

  • Mixture vs average (issue question 2): out of scope. VotingChainladder still takes a weighted average of the member ultimates draw by draw; mixing draws per Shapland §6 would be a separate design change.
  • MackChainladder.predict: not fixed here. Its predict never assigns X_new.ultimate_, so the same one-liner does not apply, and it already raises ValueError: Index data must be 1-dimensional on bootstrap triangles in voting — pre-existing.
  • BornhuetterFerguson / CapeCod / ExpectedLoss: their fit() never adds process variance, so fit and predict are already consistent.
  • fit() on a simulation subset has the same shape defect as the subset-predict case above (pre-existing, also on main); this PR only guards the predict path, which would otherwise have regressed it.

predict() on a bootstrapped triangle returned point estimates while
fit() added simulated process variance, so predict(sims).ultimate_
had a smaller IBNR spread than fit (17337.3 vs 18939.9) and
VotingChainladder inherited the gap through predict(). Chainladder
and Benktander predict now apply _include_process_variance to the
predicted triangle, matching fit().

Closes casact#1639
@github-actions

Copy link
Copy Markdown

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 18.0% of exported symbols fully typed (303 / 1680)

Known Ambiguous Unknown Total
Project (head) 303 96 1281 1680

Other symbols referenced but not exported by chainladder: 24

Known Ambiguous Unknown Total
Other (head) 11 1 12 24

Symbols without documentation:

  • Functions without docstring: 333
  • Functions without default param: 0
  • Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 3)

Known Ambiguous Unknown Total
Patch 0 0 3 3
Patch symbol details
Symbol Status Change
chainladder.methods.tests.test_predict.test_benktander_predict_carries_process_variance ❌ unknown new
chainladder.methods.tests.test_predict.test_predict_carries_process_variance ❌ unknown new
chainladder.methods.tests.test_predict.test_predict_subset_of_simulations_keeps_shape ❌ unknown new

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.08%. Comparing base (a59efc9) to head (cdbd8e4).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1652      +/-   ##
==========================================
+ Coverage   95.03%   95.08%   +0.05%     
==========================================
  Files         100      100              
  Lines        6460     6470      +10     
  Branches      819      822       +3     
==========================================
+ Hits         6139     6152      +13     
+ Misses        203      201       -2     
+ Partials      118      117       -1     
Flag Coverage Δ
unittests 95.08% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@scks1

scks1 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

I ran BornhuetterFerguson and CapeCod through this branch too, since both reach Benktander.predict through super(). On cdbd8e4 all three now match fit:

import chainladder as cl
import numpy as np

raa = cl.load_sample("raa")
sims = cl.BootstrapODPSample(n_sims=2000, random_state=42).fit_transform(raa)
sd = lambda ult: float(np.std((ult - sims.latest_diagonal).sum("origin").values))
exposure = sims.latest_diagonal * 0 + 40000
for name, est in [
    ("BornhuetterFerguson", cl.BornhuetterFerguson(apriori=0.5)),
    ("Benktander", cl.Benktander(apriori=0.5, n_iters=2)),
    ("CapeCod", cl.CapeCod()),
]:
    model = est.fit(sims, sample_weight=exposure)
    pred = model.predict(sims, sample_weight=exposure)
    print(f"{name:20s} fit {sd(model.ultimate_):9.1f}  predict {sd(pred.ultimate_):9.1f}")
BornhuetterFerguson  fit    9313.2  predict    9313.2
Benktander           fit   10806.3  predict   10806.3
CapeCod              fit   12147.3  predict   12147.3

On main a59efc9 the predict column reads 5144.4, 7709.5 and 9024.6. The new tests cover Chainladder, VotingChainladder and Benktander, so a BornhuetterFerguson or CapeCod case might be worth adding.

(This comment was written with Árbol, an AI assistant, under my direction.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[QUESTION] predict() on a bootstrapped triangle leaves out process variance, and VotingChainladder inherits that

2 participants