Repository navigation
Conversation
Pyright Type CompletenessView the full Project (full
Other symbols referenced but not exported by
Symbols without documentation:
Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 3)
Patch symbol details
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1610 +/- ##
==========================================
+ Coverage 95.00% 95.04% +0.04%
==========================================
Files 100 100
Lines 6461 6481 +20
Branches 820 820
==========================================
+ Hits 6138 6160 +22
+ Misses 204 203 -1
+ Partials 119 118 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The (3.14, pandas3, ubuntu-latest) job failed with "The hosted runner lost communication with the server" partway through Run tests, and it left no log. Every other job passed, including 3.14 on pandas 2.3.3. Could you re-run that job when you get a chance? I don't have permission to re-run checks on this repo. (This comment was written with Árbol, an AI assistant, under my direction.) |
henrydingliu
left a comment
There was a problem hiding this comment.
responding to the questions in the PR summary
R also switches to Mack when the slope's p-value is above 0.05. I did not add that. The largest fill left on clrd is 3,341,035 (CumPaidLoss, 24-36, extrapolated back from four sigmas that fall from 14.7 to 0.00059). Its p-value is 0.048, so R would keep it too.
anyone using sigma_ should understand that smaller triangles or triangles with lots of missing data is going to cause issues. i don't think we need to nanny
R warns when it switches to Mack. This PR does not warn, since clrd would warn on many triangles at once. Should it?
since the sigma calculations are happening on every development even when people are just trying to get to a normal chainladder result, i don't want to warn on a corner case on an unused statistic. if we had a toggle to turn sigma calcs on or off (defaulting to off), then it becomes reasonable to warn this switching behavior when someone intentionaly turns on the sigma calcs.
mack_interpolation still returns 9.9999e-321 where a sigma is 0, because of the same 1e-320 stand-in. Turning those into 0 would make them NaN under "mack" and could change Mack results, so I left it out.
i'm not following. so is mack interpolation not doing anything?
|
|
||
| # One positive sigma: the fill falls back to Mack's approximation. | ||
| brotherhood = cl.Development().fit(tri.loc["Brotherhood Mut Ins Co", "prodliab"]) | ||
| assert np.nanmax(brotherhood.sigma_.set_backend("numpy").values) < 1e3 |
There was a problem hiding this comment.
please double check my logic flow here.
we are here to test mack's approximation for sigma_ when loglinear regression doesn't have enough valid data. have we tested mack's approximatio itself somewhere? i.e. are any of our hardcoded mack tests covering the approximation for sigma?
There was a problem hiding this comment.
Your logic is right, and the Brotherhood case didn't test it. Its raw sigmas are NaN, NaN, 7.67, 0, then NaN. Mack's approximation needs two positive sigmas just before a gap, so it fills nothing there; the case only checks that the 1.8e308 fills are gone. I fixed its comment and added Great Amer Grp / medmal, whose sigmas are 14.98, 52.90, then a gap: the fallback fills 36-48 with 14.978, and the test checks that against sigma_interpolation="mack" on the same triangle. On main that fill is 1.2e-142. The approximation itself is covered by printed values: test_mack1999_hardcode checks mortgage's filled last sigma (16.887 against 16.89 in Mack 1999), and test_cdr_matches_merz2008 checks mw2008's filled last sigma^2 (0.040 against 0.04). test_mack1994_hardcode only reaches it through the oldest origin's standard error.
(This comment was written with Árbol, an AI assistant, under my direction.)
| """ | ||
| Fill undefined sigmas by regressing log(sigma) on development period. | ||
|
|
||
| As in R's ChainLadder (``estimate.sigma``), only positive sigmas enter |
There was a problem hiding this comment.
since we are pivoting away from pinning any results to R chainladder implementation, we should also provide independent rationale for design choices, rather than referring back to R chainladder.
There was a problem hiding this comment.
Rewritten without R. A zero sigma has no logarithm, and the 1e-320 stand-in put a point near -737 on the log scale that outweighed every other point. A line through two points fits both exactly, so nothing is left to judge it by; on clrd IncurLoss one such line gave a fill of 1,921,341 from sigmas of 0.0268 and 11.12.
(This comment was written with Árbol, an AI assistant, under my direction.)
| fill_ = xp.nan_to_num(xp.exp(reg.x * slope + intercept) * (1 - estimated)) | ||
| use_mack = xp.sum(w, axis=self.axis, keepdims=True) < 3 | ||
| if use_mack.any(): | ||
| with warnings.catch_warnings(): |
There was a problem hiding this comment.
i didn't see a rationale for this warning suppression in the PR summary. any specific issue this is trying to solve?
There was a problem hiding this comment.
The default path now calls mack_interpolation, which divides by the square of the sigma two periods back. Where that sigma is 0, numpy warns "divide by zero" and "invalid value" in cells whose fill comes out as 0 anyway. Without a filter, a default fit would gain those two warnings whenever one of its triangles takes the fallback and any of them has a zero sigma. I replaced the blanket RuntimeWarning filter with np.errstate(divide="ignore", invalid="ignore") around that one call. On clrd the default path now gives the same warnings as with the old filter, under numpy and sparse.
(This comment was written with Árbol, an AI assistant, under my direction.)
|
Thanks. I'll leave out the p-value fallback and the warning. On "is mack interpolation not doing anything?": it does fill. It fills a missing sigma from the two sigmas just before it, which covers the usual missing last sigma. My bullet was only about one detail: where a sigma is exactly 0, (This comment was written with Árbol, an AI assistant, under my direction.) |
are you referring to this line? assuming yes, then i'm wondering if the brotherhood test is valid, i.e. if we plan to change that line, wouldn't that likely change brotherhood test? while reading through the mack interpolation, the docstring makes me wonder what usefulness may remain of this mack-interpolation fallback if we change that line to not creep 0 to 1e-320 |
loglinear_interpolation replaced a zero sigma with 1e-320 and fitted its log (-736.8) at full weight, so a single zero-variation period could push the extrapolated sigmas to 1.8e308. Only positive sigmas now enter the fit, as in R's ChainLadder estimate.sigma, and zero sigmas are not filled. Where fewer than three positive sigmas remain, the fill falls back to Mack's approximation, which R's Mack.S.E also does for one point and for two points (undefined slope p-value). Closes casact#1609. Developed with an AI assistant (Árbol, running Muse).
Give the log-linear fill's reasons in the docstring: a zero sigma has no logarithm, and a line through two points leaves no residual to judge it by. Scope the fallback's error suppression to numpy's divide and invalid flags, which mack_interpolation sets where an earlier sigma is zero. Add a case where the fallback fills a sigma and compare it with sigma_interpolation="mack", and correct the comment on the case where Mack has nothing to fill from. Developed with an AI assistant (Árbol, running Muse).
a274a90 to
eb71331
Compare
|
Yes, that's the line. The Brotherhood test passes with and without it. The fill only uses that value squared or to the fourth power, and
On the docstring: Mack's rule matches a two-point log-linear extrapolation only while the sigmas fall. Mack (1993, p. 217) offers log-linear regression or, more simply, keeping the ratio of successive sigmas constant as long as they decrease, which gives I can correct the (This comment was written with Árbol, an AI assistant, under my direction.) |
|
regrouping on this. i want to lay out all the branches, compare the before/after, and crystalize what the changes are
so right now the net effect of filling with mack_interpolation is (correct me if i'm wrong):
|
A line through rising sigmas grew the tail fill without bound. Marysville Mut Ins Co othliab IncurLoss has positive sigmas 1.228, 3.872 and 3.569, and the 108-120 fill came out at 107.6. Where the fitted slope is not negative, each fill after the last positive sigma now stays at or below that sigma, the bound Mack's approximation already keeps. Developed with an AI assistant (Árbol, running Muse).
|
I ran each case on this branch and on main, with hand-built sigma vectors and with every
So I'd add one item to your net effect. Zero sigmas no longer enter the log-linear fit, and that drives most of the changes. On decreasing sigmas, Mack's rule can't rise, because each fill is at most the smaller of the two sigmas before it. Log-linear has no such check on the current head. There, 17 CumPaidLoss, 20 IncurLoss and 37 BulkLoss cells after a triangle's last positive sigma get a fill above that triangle's largest positive sigma. Marysville Mut othliab IncurLoss has [1.228, 3.872, 3.569, 0, 0, 0, 0, 0, nan], and the head fills 107.6 at 108-120. I have a commit ready for the next push that caps each log-linear fill after the last positive sigma at that sigma, the bound Mack's rule keeps. With it, none of the 17, 20 and 37 cells stays above the largest positive sigma. The Marysville fill drops to 3.569, and Horace Mann wkcomp IncurLoss drops from 176.4 to 3.418. The hand-built vectors for your five cases give the same fills as before, and the commit adds a test for Marysville. The cap also reaches tails where the line falls but still sits above the last sigma. It changes 87 of the 275 CumPaidLoss triangles without zeros, so case 1 would no longer mean no change. A narrower version caps only where the fitted slope is not negative. It clears the same cells and changes none of those 275 triangles, and 4 of 336 for IncurLoss and 13 of 123 for BulkLoss. Which scope would you prefer? Neither version touches backward fills. IL State Bar othliab CumPaidLoss still gets 3,341,035 at 24-36 from three positive sigmas. A cap at the largest positive sigma for fills before the first one brings it to 14.71 and changes 11 CumPaidLoss cells beyond the tail cap. I left that out of the commit unless you want it. For the non-consecutive case, the branch leaves those cells NaN. I can keep that, or fill them from the line through the two points with the cap above. Which would you prefer? (This comment was written with Árbol, an AI assistant, under my direction.) |
to me, this is a deciding factor for not filling log-linear with mack_interpolation yet. let me know if you agree.
i like this.
agreed. it feels more arbitrary to cap early period sigma. is there another option for limit exponential growth on the early period sigmas? maybe something like a max coefficient of variation of 10? |
|
I agree on leaving Thanks for the go on the tail cap. Please confirm the scope before I push it. The full cap at the last positive sigma changes 87 of the 275 CumPaidLoss triangles that have three or more positive sigmas and no zeros. The rising-only version caps a fill only where the fitted slope is not negative. It clears the same runaway cells and changes 0 of those 275, 4 of 336 for IncurLoss and 13 of 123 for BulkLoss. I'd go with rising-only. I tried your coefficient of variation idea on the early fills. Under volume weights the variance of the next value is The snippets below run on the On Codeimport numpy as np
import chainladder as cl
from chainladder.utils.weighted_regression import WeightedRegression as WR
# Keep the raw sigmas that Development passes to sigma_fill.
raw = []
fill = WR.sigma_fill
def spy(self, interpolation):
raw.append(np.array(self.sigma_, dtype=float))
return fill(self, interpolation)
WR.sigma_fill = spy
WR.max_fill_cv = np.inf # rising-only cap, no CV cap
clrd = cl.load_sample("clrd")
for col in ["CumPaidLoss", "IncurLoss"]:
raw.clear()
tri = clrd[col]
d = cl.Development().fit(tri)
n = tri.shape[0]
r = raw[0].reshape(n, -1)
s = d.sigma_.values.reshape(n, -1)
f = d.ldf_.values.reshape(n, -1)
c = tri.values[:, 0]
link = ~np.isnan(c[:, :, :-1]) & ~np.isnan(c[:, :, 1:])
with np.errstate(all="ignore"):
cbar = np.nanmean(np.where(link, c[:, :, :-1], np.nan), axis=1)
cv = s / (f * np.sqrt(cbar))
filled = np.isnan(r) & ~np.isnan(s)
print(col, "largest CV of computed sigmas", round(np.nanmax(np.where(r > 0, cv, np.nan)), 2))
for cap in [1, 5, 10]:
hit = filled & (cv > cap)
print(f" CV {cap:>2} {hit.sum():>2} cells / {hit.any(axis=1).sum()} triangles")At a cap of 10 all three cells come before a triangle's first positive sigma. The columns show the sigma with the CV cap off and with it on. Codeimport numpy as np
import chainladder as cl
from chainladder.utils.weighted_regression import WeightedRegression as WR
tri = cl.load_sample("clrd")["CumPaidLoss"]
n = tri.shape[0]
WR.max_fill_cv = np.inf
before = cl.Development().fit(tri).sigma_
WR.max_fill_cv = 10.0
after = cl.Development().fit(tri).sigma_
b = before.values.reshape(n, -1)
a = after.values.reshape(n, -1)
for i, j in zip(*np.nonzero(~np.isclose(a, b, equal_nan=True))):
name, lob = tri.index.iloc[i]
print(f"{name:<38} {lob:<9} {before.development[j]:<6} {b[i, j]:>11.6g} {a[i, j]:>9.6g}")
idx = tri.index
il = np.flatnonzero(idx["GRNAME"].str.startswith("IL State Bar") & (idx["LOB"] == "othliab"))[0]
print("IL State Bar sigmas after the cap", np.round(a[il], 4))IL State Bar's other positive sigmas are 14.71, 9.45 and 0.0082, so 89.81 is still the largest sigma in that triangle. It sits far below 3,341,035, though. The code reads the exponent for each average, since the variance follows For the case with two non-consecutive positive sigmas, the branch leaves those cells NaN. I can keep that, or fill them from the line through the two points with the caps above. Which would you prefer? (This comment was written with Árbol, an AI assistant, under my direction.) |
|
thanks for looking into my wild ideas. think we have teased out all the nooks and crannies. going back to basic principles. our main goal here is to produce more reasonable interpolation results
so from an implementation standpoint, we just have
let me know if this all makes sense. |
…1e-320 The log-linear fill extended its line to ages before the first positive sigma. That is extrapolation, and Mack's approximation never fills those cells. The fill now leaves them unfilled, so they stay NaN under both sigma_interpolation settings. IL State Bar's CumPaidLoss 24-36 sigma goes from 3,341,035 to NaN. mack_interpolation no longer swaps zero sigmas for 1e-320. The fills it makes do not change, and zero sigmas now come out as NaN, as they already do under log-linear. Developed with an AI assistant (Árbol, running Muse).
|
That makes sense to me, and I've made the change locally on two commits on top of this PR. I haven't pushed them yet. The first is the rising-only cap. The second stops the log-linear fill before the first positive sigma, so those cells stay NaN the way they do under On Codeimport numpy as np
import chainladder as cl
clrd = cl.load_sample("clrd")
print("log-linear fills before the first positive sigma")
for col in ["CumPaidLoss", "IncurLoss", "BulkLoss"]:
tri = clrd[col]
ll = cl.Development().fit(tri).sigma_.set_backend("numpy").values[:, 0, 0]
mack = cl.Development(sigma_interpolation="mack").fit(tri)
mk = mack.sigma_.set_backend("numpy").values[:, 0, 0]
# Mack's fill needs two sigmas before a cell, so its first value above
# 1e-300 is the first positive sigma.
pos = np.nan_to_num(mk) > 1e-300
first = np.where(pos.any(1), pos.argmax(1), mk.shape[1])
before = np.arange(mk.shape[1]) < first[:, None]
print(col, int((before & (np.nan_to_num(ll) > 0)).sum()))The IL State Bar 24-36 sigma goes from 3,341,035 to NaN. A NaN sigma adds no variance at its age, so the total Mack standard error falls wherever origins still develop through that age. Buckeye goes from 540.8 to 208.3 and Fitchburg from 11569.2 to 2507.5. IL State Bar's total was NaN before and stays NaN. Codeimport chainladder as cl
tri = cl.load_sample("clrd")["CumPaidLoss"]
for name, lob, age in [
("IL State Bar Assoc Mut Ins Co", "othliab", 1),
("Buckeye Ins Grp", "othliab", 0),
("Fitchburg Mut Ins Co", "ppauto", 0),
]:
t = tri.loc[name, lob]
dev = cl.Development().fit(t)
sigma = dev.sigma_.set_backend("numpy").values.ravel()[age]
se = cl.MackChainladder().fit(dev.transform(t)).total_mack_std_err_.values.ravel()[0]
print(name, dev.sigma_.development[age], round(float(sigma), 4), round(float(se), 1))Two questions before I push.
(This comment was written with Árbol, an AI assistant, under my direction.) |
for triangles like IL State Bar Assoc Mut Ins Co, I don't think it ever made sense that the standard error was that high. but i want to close this PR out and live to fight another day on the extrapolation issue. i'm good with leaving them in.
i thought we were abandoning the mack fill till we added recursion to mack? |
|
Thanks. The NaN cells before the first positive sigma are already in 327ee23, and CI passed on it. The cutoff covers a second use of On the clrd columns, the counts look like this.
Snippetimport inspect
import textwrap
import warnings
import numpy as np
import chainladder as cl
from chainladder.utils import weighted_regression as wr
warnings.filterwarnings("ignore")
W = wr.WeightedRegression
head = W.loglinear_interpolation
src = textwrap.dedent(inspect.getsource(head))
def variant(old, new):
ns = {}
exec(src.replace(old, new), wr.__dict__, ns)
return ns["loglinear_interpolation"]
variants = {
"cutoff 3": head,
"cutoff 2": variant("< 3", "< 2"),
"no fallback": variant("if use_mack.any():", "if False:"),
"NaN below 3": variant("use_mack, mack_fill, fill_", "use_mack, 0, fill_"),
}
raw = {}
fill = W.sigma_fill
def spy(self, interpolation):
raw["sigma"] = np.array(self.sigma_, dtype=float)[:, 0, :, 0]
return fill(self, interpolation)
W.sigma_fill = spy
clrd = cl.load_sample("clrd")
for col in ["CumPaidLoss", "IncurLoss", "BulkLoss"]:
sig = {}
for name, fn in variants.items():
W.loglinear_interpolation = fn
sig[name] = cl.Development().fit(clrd[col]).sigma_.values[:, 0, 0, :]
npos = (np.nan_to_num(raw["sigma"]) > 0).sum(axis=1)
print(col, len(npos), "triangles")
print(" positive sigmas 0, 1, 2:", [int((npos == k).sum()) for k in (0, 1, 2)])
for name, s in sig.items():
gap = np.isnan(raw["sigma"]) & ~np.isnan(s)
print(f" {name}: fills {int(gap.sum())} sigmas in {int(gap.any(axis=1).sum())} triangles")So two choices.
I'd lean toward the first, since it keeps Mack out of this PR entirely. Which do you prefer? (This comment was written with Árbol, an AI assistant, under my direction.) |
Summary of Changes
WeightedRegression.loglinear_interpolationfills the sigmas thatDevelopmentcannot estimate by regressing log(sigma) on development period. It replaced a zero sigma with 1e-320 and fitted its log (-736.8) at full weight, so one zero-variation period could push the fills to 1.8e308 (details in #1609).This PR changes only that function:
mack_interpolationinstead. One point gives no slope, and a line through two points fits both exactly, so nothing is left to judge it by. With a cutoff of two, as in the issue, clrd IncurLoss still got a fill of 1,921,341 extrapolated from two sigmas, 0.0268 and 11.12.A regression test on clrd CumPaidLoss covers Brotherhood Mut Ins Co / prodliab (one positive sigma, where Mack has nothing to fill from and the old 1.8e308 fills become NaN), Great Amer Grp / medmal (two positive sigmas, where the fallback's 36-48 fill must equal the
sigma_interpolation="mack"value) and Buckeye Ins Grp / othliab (five positive sigmas and two zeros). For Buckeye the test refits the five positive sigmas withnp.polyfitand checks that the 12-24 and 108-120 fills lie on that line; the 12-24 fill is 7.1819 (it was 2.50e137).Related GitHub Issue(s)
Closes #1609
AI/LLM Usage
I used Árbol, an AI assistant running on Muse, under my direction to trace the cause, write the change and the test, run the comparisons and draft this description. I reviewed the change and the results.
Additional Context for Reviewers
Before and after on main ee2c24d, default settings:
total_mack_std_err_mack_std_err_sigma_interpolation="mack"results are byte-identical on every triangle checked.chainladder/developmentandchainladder/methodsgive 240 passed with this change and 238 passed on main (the difference is the two new test runs). The new test fails on main.Submitter's Checklist
[FIX],[FEAT],[DOCS],[TST],[CHORE], or[BRK]).Reviewer's Checklist