Repository navigation
[FEAT] Record the seed each random draw used as random_state_, and handle random_state as documented - #1605
[FEAT] Record the seed each random draw used as random_state_, and handle random_state as documented#1605scks1 wants to merge 9 commits into
random_state_, and handle random_state as documented#1605Conversation
…nted BootstrapODPSample and the Benktander family build their generators with sklearn.utils.check_random_state through a small helper, so None draws from the global np.random generator, an integer seeds a new RandomState and a RandomState instance is used as it is. The process-variance step offsets an integer seed by 1 without a truthiness test, so seed 0 now reproduces the ultimates as well as the resampled triangles. Integer seeds other than 0 give the same draws as before. Modules without a random.RandomState of their own, such as sparse, now draw through numpy, so apriori_sigma no longer raises AttributeError on a sparse triangle. Part of casact#1568. Developed with an AI assistant (Árbol, running Muse).
When random_state is None, BootstrapODPSample.fit and the Benktander family's fit draw one integer from the global np.random generator, use it for every draw and keep it as random_state_, so passing it back as random_state repeats the run. On a triangle with several columns and random_state=None, each column's child estimator draws its own integer, so the columns' residual draws stay independent as before, and np.random.seed before fit repeats the run. An integer or a RandomState instance passed in is kept as it is and shared by every column. The resampled Triangle carries random_state_ for the process-variance step, and Benktander.predict reuses it for the apriori draws. When apriori_sigma is 0, no draw is made and random_state_ is random_state. Closes casact#1568. Developed with an AI assistant (Árbol, running Muse).
Pyright Type CompletenessView the full Project (full
Other symbols referenced but not exported by
Symbols without documentation:
Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 14)
Patch symbol details
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1605 +/- ##
==========================================
+ Coverage 95.03% 95.06% +0.03%
==========================================
Files 100 100
Lines 6460 6510 +50
Branches 819 825 +6
==========================================
+ Hits 6139 6189 +50
Misses 203 203
Partials 118 118
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Developed with an AI assistant (Árbol, running Muse).
|
is the codecov miss legit? i can start the review if it's a false alarm (which does happen occasionally)? |
|
It's real, but it's only the GPU path. The three missed lines and the partial one are the cupy branch of I've pushed f6a4507, a test that covers that branch with a stand-in module, so it needs no cupy. In a local coverage run it covers the lines Codecov flagged, and the bootstrap, Benktander and Cape Cod tests pass. Codecov should clear once it reruns on the new head. (This comment was written with Árbol, an AI assistant, under my direction.) |
|
thanks for adding the tests! as I try to comprehend the overall approach of this PR, i'm wondering if we are only inches away from a dedicated |
|
Thanks! I don't think one exists yet. actsim's I'm happy to add one in this PR. Modeled on
If that shape works for you, I'll push it here. A better name is welcome. The three questions in the description are still open whenever you get to them. (This comment was written with Árbol, an AI assistant, under my direction.) |
|
can you clarify if those three questions are in regards to benktander or bootstrap? i think 2 and 3 are benktander and 1 is bootstrap but wanted to confirm |
|
Two of the three are about
(This comment was written with Árbol, an AI assistant, under my direction.) |
|
@scks1 sorry i forgot about this. can you address the conflict and we can close this out? |
Developed with an AI assistant (Árbol, running Muse).
|
No problem. I merged main into the branch and fixed the conflict in test_bootstrap.py. Both sides only added tests at the end of the file, so the merge keeps the tests from this PR and the new hat matrix fallback test from main. The PR's own changes are the same as before. The bootstrap and Benktander tests pass locally on the merged branch. The three questions in the description stay as the PR has them unless you want a change. I left the RandomStateMixin out of this push so the PR keeps its current scope, and I can open it as a follow-up if you want it. (This comment was written with Árbol, an AI assistant, under my direction.) |
to check my understanding, is this only an issue when random_state is None?
my vote is for redrawing when random_state=None. my rationale is this:
in the end state we should let the user choose. not sure if it's convenient to fit into this PR. we can keep the existing inconsistency and open up a new issue lastly, I'd prefer to get RandomStateMixin first. I get nervous whenever we have too many orphan private functions in utils. |
Add RandomStateMixin to utils/utility_functions.py, next to _record_seed and _check_random_state. Its _record_random_state sets random_state_ in fit, and its _get_random_state returns the generator for the recorded seed on the triangle's array module. BootstrapODPSample and Benktander use it in place of their own calls, and BornhuetterFerguson, CapeCod and ExpectedLoss get it through Benktander. Benktander passes draws=False when apriori_sigma is 0. The per-column seeds in BootstrapODPSample, the process-variance seed offset on the transformed Triangle and each class's random_state_ docstring stay as they were. Results do not change. Developed with an AI assistant (Árbol, running Muse).
With apriori_sigma above 0 and random_state=None, predict used to reuse the seed fit recorded as random_state_, so every predict repeated the fit's aprioris. predict now asks RandomStateMixin._get_random_state for a generator with redraw=True, which draws a new seed through _record_seed when random_state is None. An integer seed still gives the same aprioris in fit and every predict, and a RandomState instance keeps advancing. The random_state docstrings of Benktander, BornhuetterFerguson, CapeCod and ExpectedLoss describe the predict behavior. New tests cover predict for all four classes. fit and random_state_ do not change. Developed with an AI assistant (Árbol, running Muse).
|
Thanks. Yes, on 1 the mismatch only happens with An integer and a On 2, the second commit below makes Two commits on top of the merge are ready for the next push, with the mixin first as you asked.
The
Should I open the issue for letting users choose independent columns in (This comment was written with Árbol, an AI assistant, under my direction.) |
| return _axis_orders[_get_axis_number(axis)] | ||
|
|
||
|
|
||
| def _check_random_state(seed, xp=np): |
There was a problem hiding this comment.
should these three functions stay inside the mixin class?
| return _axis_orders[_get_axis_number(axis)] | ||
|
|
||
|
|
||
| def _check_random_state(seed, xp=np): |
There was a problem hiding this comment.
can you please add type hinting?
|
|
||
|
|
||
| def test_random_state_zero_reproduces_ultimates(raa): | ||
| """ |
There was a problem hiding this comment.
for this and the next two tests, do we have and/or need the converse test, i.e. different local/global seeds lead to different results?
| ``np.random.seed`` repeats the run, and that an integer seed is still shared | ||
| by every column. | ||
| """ | ||
| tri = cl.load_sample("raa") |
There was a problem hiding this comment.
if this test can work on sparse, please use the test fixture
| _process_variance_seed, | ||
| ) | ||
|
|
||
| random_state = _check_random_state(_process_variance_seed(self.random_state), xp) |
There was a problem hiding this comment.
is the process variance seed recorded (when self.random_state=None)? does it need to be?
| ) | ||
|
|
||
|
|
||
| def test_capecod_apriori_sigma_predict_redraws_without_a_seed( |
There was a problem hiding this comment.
there's a dedicated test_capecod.py
| raise ValueError("sample_weight is required.") | ||
|
|
||
| super().fit(X, y, sample_weight) | ||
| self._record_random_state(draws=self.apriori_sigma != 0) |
There was a problem hiding this comment.
the relationship between apriori_sigma, random_state and the redraw parameter seems needlessly complicated. do we really need that redraw parameter? is self.random_state is None not identical to redraw in all cases?
…hints _check_random_state, _process_variance_seed and _record_seed become static methods of RandomStateMixin, with type hints. The bootstrap process-variance step and its test call them through the class. Developed with an AI assistant (Árbol, running Muse).
A new test checks that different integer seeds, different global np.random seeds and RandomState instances built from different seeds give different resampled triangles and ultimates. The independent-columns test now takes the raa fixture, so it also runs on the sparse backend. Developed with an AI assistant (Árbol, running Muse).
Developed with an AI assistant (Árbol, running Muse).
|
Thanks for the review, Henry. I pushed a commit for each point and answer all seven comments below. Helpers in the mixin. Type hints. All five mixin methods now carry them. Converse tests. They were missing. Codeimport numpy as np
import chainladder as cl
raa = cl.load_sample("raa")
def total_ultimate(random_state):
sims = cl.BootstrapODPSample(n_sims=20, random_state=random_state).fit_transform(raa)
return round(float(np.nansum(cl.Chainladder().fit(sims).ultimate_.values)), 2)
print("seed 0", total_ultimate(0), "seed 1", total_ultimate(1))
np.random.seed(1)
a = total_ultimate(None)
np.random.seed(2)
print("global seed 1", a, "global seed 2", total_ultimate(None))
print("RandomState(42)", total_ultimate(np.random.RandomState(42)), "RandomState(43)", total_ultimate(np.random.RandomState(43)))Output Sparse fixture. Process-variance seed. Codeimport numpy as np
import chainladder as cl
from chainladder.utils.utility_functions import RandomStateMixin
raa = cl.load_sample("raa")
np.random.seed(2)
boot = cl.BootstrapODPSample(n_sims=50).fit(raa)
sims = boot.transform(raa)
print("random_state_", boot.random_state_, "triangle random_state", sims.random_state)
print("process-variance seed", RandomStateMixin._process_variance_seed(sims.random_state))
first = cl.Chainladder().fit(sims)
again = cl.BootstrapODPSample(n_sims=50, random_state=boot.random_state_).fit_transform(raa)
repeat = cl.Chainladder().fit(again)
print("refit repeats ultimate_", np.array_equal(first.ultimate_.values, repeat.ultimate_.values, equal_nan=True))
print("refit repeats process_variance_", np.array_equal(first.process_variance_.values, repeat.process_variance_.values, equal_nan=True))
inst = cl.BootstrapODPSample(n_sims=50, random_state=np.random.RandomState(9)).fit_transform(raa)
one = cl.Chainladder().fit(inst).ultimate_.values
two = cl.Chainladder().fit(inst).ultimate_.values
print("RandomState instance, two fits on the same sims match", np.array_equal(one, two, equal_nan=True))Output test_capecod.py. The CapeCod predict test now lives there. The redraw parameter. Codeimport numpy as np
import chainladder as cl
from chainladder.utils.utility_functions import RandomStateMixin as M
gen = cl.load_sample("genins")
weight = gen.latest_diagonal * 0 + 1
def refit_repeats():
model = cl.BornhuetterFerguson(apriori_sigma=0.5).fit(gen, sample_weight=weight)
seed = model.random_state_
repeat = cl.BornhuetterFerguson(apriori_sigma=0.5, random_state=seed).fit(gen, sample_weight=weight)
return np.array_equal(model.expectation_.values, repeat.expectation_.values)
print("with the redraw flag", refit_repeats())
def without_flag(self, xp=np, redraw=False):
if self.random_state is None:
return M._check_random_state(M._record_seed(None), xp)
return M._check_random_state(getattr(self, "random_state_", self.random_state), xp)
M._get_random_state = without_flag
print("with random_state is None only", refit_repeats())Output Codeimport numpy as np
import chainladder as cl
gen = cl.load_sample("genins")
weight = gen.latest_diagonal * 0 + 1
premium = gen.latest_diagonal * 0 + 8e6
forms = {"None": lambda: None, "int 7": lambda: 7, "RandomState(7)": lambda: np.random.RandomState(7)}
for estimator, sw in [(cl.Benktander, weight), (cl.BornhuetterFerguson, weight), (cl.ExpectedLoss, weight), (cl.CapeCod, premium)]:
for label, make in forms.items():
for sigma in (0.5, 0.0):
model = estimator(apriori_sigma=sigma, random_state=make()).fit(gen, sample_weight=sw)
fit = model.expectation_.set_backend("numpy").values
predicted = model.predict(gen, sample_weight=sw).expectation_.set_backend("numpy").values
print(f"{estimator.__name__:20s} {label:15s} {sigma} {np.array_equal(fit, predicted)}")Output The flag can still go if each caller passes its own seed. Codeimport numpy as np
import chainladder as cl
from chainladder.utils.utility_functions import RandomStateMixin as M
gen = cl.load_sample("genins")
weight = gen.latest_diagonal * 0 + 1
premium = gen.latest_diagonal * 0 + 8e6
current = cl.Benktander._get_benktander_aprioris
def seeded(self, X, sample_weight, redraw=False):
"""Draw the aprioris from a seed the caller picks, with no redraw branch."""
seed = M._record_seed(self.random_state) if redraw else self.random_state_
state = M._check_random_state(seed, X.get_array_module())
cov = self.apriori_sigma / self.apriori
sigma_log = np.sqrt(np.log1p(cov**2))
mu_log = np.log(self.apriori) - 0.5 * sigma_log**2
draws = state.lognormal(mu_log, sigma_log, X.shape[0])
apriori = sample_weight * draws.reshape(X.shape[0], -1)[..., None, None]
apriori.index = X.index.copy()
apriori.columns = sample_weight.columns
return apriori
def run(estimator, sw, random_state, method):
cl.Benktander._get_benktander_aprioris = method
np.random.seed(5)
model = estimator(apriori_sigma=0.5, random_state=random_state()).fit(gen, sample_weight=sw)
out = [model.expectation_.set_backend("numpy").values]
for _ in range(2):
out.append(model.predict(gen, sample_weight=sw).expectation_.set_backend("numpy").values)
cl.Benktander._get_benktander_aprioris = current
return out
forms = {"None": lambda: None, "int 7": lambda: 7, "RandomState(7)": lambda: np.random.RandomState(7)}
for estimator, sw in [(cl.Benktander, weight), (cl.BornhuetterFerguson, weight), (cl.ExpectedLoss, weight), (cl.CapeCod, premium)]:
for label, make in forms.items():
a, b = run(estimator, sw, make, current), run(estimator, sw, make, seeded)
print(f"{estimator.__name__:20s} {label:15s} {all(np.array_equal(x, y) for x, y in zip(a, b))}")Output The full suite passes with 1735 tests passed and 9 skipped. (This comment was written with Árbol, an AI assistant, under my direction.) |
henrydingliu
left a comment
There was a problem hiding this comment.
thanks for including the code and output along with the narrative. it's a lot easier to process!
fitmust draw from the seed it records inrandom_state_
this feels like a new scenario. when would we call fit twice? i'm looking through both existing tests and new tests from this PR and not finding any example. i'd venture that the overall design of these estimators is to fit once and maybe transform/predict multiple times. i believe this means that _record_random_state can key off of self.random_state completely?
| An integer seed is offset by 1 so the two sets of draws differ, including | ||
| for seed 0. A ``RandomState`` instance or ``None`` is passed through. | ||
| """ | ||
| if isinstance(seed, (int, np.integer)) and not isinstance(seed, bool): |
There was a problem hiding this comment.
seed is never a bool per type hinting?
|
Thanks, Henry. I worded that line badly. Nothing in the PR calls That case still needs Codeimport numpy as np
import chainladder as cl
from chainladder.utils.utility_functions import RandomStateMixin as M
raa = cl.load_sample("raa")
def repeats():
np.random.seed(3)
first = cl.BootstrapODPSample(n_sims=20).fit(raa)
again = cl.BootstrapODPSample(n_sims=20, random_state=first.random_state_).fit(raa)
a, b = first.resampled_triangles_.values, again.resampled_triangles_.values
return np.array_equal(a, b, equal_nan=True)
print("random_state_ repeats the run", repeats())
M._get_random_state = lambda self, xp=np, redraw=False: M._check_random_state(
self.random_state, xp
)
print("keyed off random_state alone", repeats())The redraw flag can go, though. In a local commit, Example from 6994622454import numpy as np
import chainladder as cl
from chainladder.utils.utility_functions import RandomStateMixin
raa = cl.load_sample("raa")
np.random.seed(2)
boot = cl.BootstrapODPSample(n_sims=50).fit(raa)
sims = boot.transform(raa)
print("random_state_", boot.random_state_, "triangle random_state", sims.random_state)
print("process-variance seed", RandomStateMixin._process_variance_seed(sims.random_state))
first = cl.Chainladder().fit(sims)
again = cl.BootstrapODPSample(n_sims=50, random_state=boot.random_state_).fit_transform(raa)
repeat = cl.Chainladder().fit(again)
print("refit repeats ultimate_", np.array_equal(first.ultimate_.values, repeat.ultimate_.values, equal_nan=True))
print("refit repeats process_variance_", np.array_equal(first.process_variance_.values, repeat.process_variance_.values, equal_nan=True))
inst = cl.BootstrapODPSample(n_sims=50, random_state=np.random.RandomState(9)).fit_transform(raa)
one = cl.Chainladder().fit(inst).ultimate_.values
two = cl.Chainladder().fit(inst).ultimate_.values
print("RandomState instance, two fits on the same sims match", np.array_equal(one, two, equal_nan=True))On the bool guard, you're right. The hints say Snippetimport numpy as np
from chainladder.utils.utility_functions import RandomStateMixin as M
seed = M._process_variance_seed(True)
print("process-variance seed for True:", seed)
print("first draw, main vs process variance:",
M._check_random_state(True, np).rand(), M._check_random_state(seed, np).rand())I can push it to this PR if that design looks right to you. (This comment was written with Árbol, an AI assistant, under my direction.) |
Summary of Changes
BootstrapODPSampleand the Benktander family now build their generators withsklearn.utils.check_random_state, sorandom_statebehaves as the docstrings say:Nonefollows the globalnp.randomstate, sonp.random.seedrepeats a run.0reproduces the ultimates as well as the triangles. The process-variance seed used to beNonefor 0.RandomStateinstance works. It used to raise aTypeError.With
random_state=None,fitdraws one integer, uses it for every draw and keeps it asrandom_state_, so passing it back repeats the run. On a triangle with several columns, each column draws its own integer, so the columns stay independent as on main, andnp.random.seedbeforefitrepeats the run. Integer seeds give the same results as before, with every column sharing the seed.This also fixes
BornhuetterFerguson(apriori_sigma=0.5)on a sparse triangle, which raisesAttributeError: 'function' object has no attribute 'RandomState'on main.Tests: six in
test_bootstrap.pyand two intest_benktander.py, on the numpy and sparse backends.Related GitHub Issue(s)
Closes #1568
AI/LLM Usage
This change was made with Árbol, an AI assistant running on Muse, working under my direction. It wrote the change and the tests, ran the checks and drafted this description. I reviewed the diff and the results before submitting.
Additional Context for Reviewers
Three questions:
.random_stateis nowrandom_state_, the seed the draws used, where it was the constructor value. Should it keep the constructor value?random_state=None,predictnow reuses the seedfitdrew instead of drawing a new one. Is that what you want?Nonegives independent columns. Should an integer seed give independent columns too? That would change seeded multi-column results.The cupy branch is untested (no GPU here). The bootstrap, methods, development and utilities tests pass with the uv.lock versions, and ruff is clean.
Submitter's Checklist
[FIX],[FEAT],[DOCS],[TST],[CHORE], or[BRK]).Reviewer's Checklist