Skip to content

[FEAT] Warn when a TweedieGLM or BarnettZehnwirth design matrix is rank deficient #1665

Description

@scks1

TweedieGLM and BarnettZehnwirth accept a design matrix with fewer independent columns than columns and fit it without a warning. The common way to get there is to add a calendar term to an origin and development design. In the frame these estimators build, valuation is a linear function of origin and development in every cell, so the three columns are linearly dependent. Kuang, Nielsen and Nielsen describe this identification problem for the extended chain-ladder model in "Identification of the age-period-cohort model and the extended chain-ladder model", Biometrika 95(4), 2008, pages 979 to 986, https://doi.org/10.1093/biomet/asn026.

On main at cc3335c, with alpha=0 now that #1663 passes it through:

import numpy as np
import chainladder as cl

gen = cl.load_sample("genins")
for f in ["C(development) + C(origin)",
          "C(development) + C(origin) + valuation",
          "development + origin + valuation"]:
    m = cl.TweedieGLM(design_matrix=f, alpha=0, max_iter=5000, tol=1e-10).fit(gen)
    X = np.asarray(m.model_.estimator_ml.named_steps["design_matrix"].transform(m.model_.df_))
    ult = cl.Chainladder().fit(m.transform(gen)).ultimate_.sum()
    print(f, X.shape[1], np.linalg.matrix_rank(X), round(ult, 2))

The loop prints these values.

design                                    columns  rank  total ultimate
C(development) + C(origin)                     19    19   53,038,954.62
C(development) + C(origin) + valuation         20    19   53,038,910.06
development + origin + valuation                4     3   55,442,708.27

Volume-weighted chain ladder gives 53,038,945.61. No warning appears in any of the three fits.

The forecasts hold steady because the same linear relation among origin, development and valuation also holds in the future cells. The companion paper by the same authors, "Forecasting with the age-period-cohort model and the extended chain-ladder model", Biometrika 95(4), 2008, pages 987 to 991, sets out when forecasts stay invariant. The coefficients carry no meaning on their own. In the linear design the fit returns 4.51786 for origin and -4.48402 for valuation, and the two nearly cancel.

BarnettZehnwirth shows the same thing on abc, and here one reported statistic changes.

formula                               columns  rank  mse_resid_  total ultimate
C(origin)+C(development)                   21    21    0.006842   15,447,587.43
C(origin)+C(development)+valuation         22    21    0.006997   15,447,587.43

The ultimate does not move, but mse_resid_ divides by 66 cells minus 22 coefficients where the design has only 21 independent columns, so mse_resid_ rises by a factor of 45/44 and std_residuals_ shrink by its square root.

I suggest a check at fit time in both estimators that compares np.linalg.matrix_rank of the design with its column count and raises a warning naming the formula when they differ. BarnettZehnwirth could also use the rank for the residual degrees of freedom. The change is small and self-contained, so it would suit a new contributor.

(This issue was written with Árbol, an AI assistant, under my direction.)

Activity

  1. henrydingliu commented on Nov 19, 2026

    @henrydingliu
    Member

    is this along the same lines as the warning added in commit 9fe6fec? i don't recall why i removed it.

  2. scks1 commented on Nov 19, 2026

    @scks1
    ContributorAuthor

    Yes, it aims at the same problem. #642 gives the reason the warning went away. @danielfong-act wrote that "any formula constructed by PTF_formula will trigger it." The 9fe6fec check counted formula terms without C(, and the trend segments that PTF_formula builds are all such terms.

    Counting terms misses in both directions. On abc at cc3335c, the categorical design with a calendar term slips past the count, and a full-rank PTF design trips it. A rank check gets both right.

    formula                                  linear terms  old warning  columns  rank
    C(origin) + development                             1        False       12    12
    C(origin) + C(development) + valuation              1        False       22    21
    development + origin + valuation                    3         True        4     3
    PTF alpha=[0], gamma=[0, 1], iota=[0, 1]            2         True        3     3
    
    Snippet for the table
    import numpy as np
    from patsy import ModelDesc
    import chainladder as cl
    
    
    def old_9fe6fec_count(formula):
        n = 0
        for term in ModelDesc.from_formula(formula).rhs_termlist[1:]:
            if "C(" not in term.factors[0].code:
                n += 1
        return n
    
    
    abc = cl.load_sample("abc")
    for kw in [dict(formula="C(origin) + development"),
               dict(formula="C(origin) + C(development) + valuation"),
               dict(formula="development + origin + valuation"),
               dict(alpha=[0], gamma=[0, 1], iota=[0, 1])]:
        bz = cl.BarnettZehnwirth(**kw).fit(abc)
        X = np.asarray(bz.model_.estimator_ml.named_steps["design_matrix"].transform(bz.model_.df_))
        n = old_9fe6fec_count(bz.formula)
        print(f"{kw}: {n} linear terms, old warning {n > 1}, columns {X.shape[1]}, rank {np.linalg.matrix_rank(X)}")

    old_9fe6fec_count copies the loop that 9fe6fec added to BarnettZehnwirth.fit.

    A warning that compares np.linalg.matrix_rank of the design with its column count stays silent on PTF formulas, so it would not run into the problem that removed the old one. I'll leave the change for a new contributor.

    (This comment was written with Árbol, an AI assistant, under my direction.)

  3. updated Priority Low, Scope Codebase and Effort Low on Nov 19, 2026
  4. added this to the v2.0.0 milestone on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Great First Contribution! 🌱Beginner friendly tickets with narrow scope and huge impact. Perfect to get your feet wet!

    Fields

    Priority

    Low

    Effort

    Low

    Scope

    Codebase

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions