Skip to content

[BUG] Triangle arithmetic raises a raw broadcast error when origin labels interleave #1649

Open

Description

@scks1

Are you on the latest chainladder version?

  • Yes, this bug occurs on the latest version. I ran the example on main at a59efc9 with chainladder 0.10.1, Python 3.12.15, numpy 2.3.3 and pandas 2.3.3.

Describe the bug in words

When the origins or ages of two triangles don't line up, _prep_origin_development unions the labels and copies each triangle into the union as one block, from the position of its first label to the position of its last (dunders.py L185–L229). That works when a triangle's labels sit next to each other in the union. When a label from the other triangle falls between two of them, the block is longer than the triangle, and numpy raises ValueError: could not broadcast input array from shape … into shape … at L221. The message doesn't say which axis or which labels caused it.

Annual triangles that start in January and in July hit this, because their origins alternate in the union. Both have origin grain Y, so the same-grain check in _compatibility_check (L61–L72) lets them through. Slices of raa hit it too.

How can the bug be reproduced?

import pandas as pd

import chainladder as cl


def annual(start_month):
    rows = []
    for i, year in enumerate([2001, 2002, 2003]):
        origin = pd.Timestamp(year, start_month, 1)
        for age in range(1, 4 - i):
            valuation = origin + pd.DateOffset(years=age) - pd.Timedelta(days=1)
            rows.append({"origin": origin, "valuation": valuation, "paid": 100.0 * age})
    return cl.Triangle(pd.DataFrame(rows), origin="origin", valuation="valuation",
                       columns="paid", cumulative=True)


raa = cl.load_sample("raa")
pairs = {
    "January vs July origins": (annual(1), annual(7)),
    "1981, 1983, 1985 vs 1982, 1984, 1986": (
        raa[raa.origin.year.isin([1981, 1983, 1985])],
        raa[raa.origin.year.isin([1982, 1984, 1986])]),
    "1981-1984 vs 1981-1983, 1986, 1987": (
        raa[raa.origin <= "1984"],
        raa[(raa.origin <= "1983") | raa.origin.year.isin([1986, 1987])]),
}
for name, (a, b) in pairs.items():
    try:
        a - b
        print(name, "ok")
    except ValueError as exc:
        print(name, "->", exc)
January vs July origins -> could not broadcast input array from shape (1,1,3,3) into shape (1,1,5,3)
1981, 1983, 1985 vs 1982, 1984, 1986 -> could not broadcast input array from shape (1,1,3,10) into shape (1,1,5,10)
1981-1984 vs 1981-1983, 1986, 1987 -> could not broadcast input array from shape (1,1,5,10) into shape (1,1,6,10)

What is the expected behavior?

For triangles of the same grain, each row should go to its own label in the union, as it already does when the labels are contiguous. The odd and even raa years would then give six origins:

odd = raa[raa.origin.year.isin([1981, 1983, 1985])]
even = raa[raa.origin.year.isin([1982, 1984, 1986])]
assert (odd - even).shape[2] == 6

For January against July origins, a ValueError that names the mismatch, like the existing "Triangle arithmetic requires both triangles to be the same grain.", may be the better result. That is a modelling choice. The sparse branch (L230–L245) shifts coordinates by the same block start, and I haven't run it. #1304 changes these lines, and #1467 proposes refactoring the block.

Would you be willing to contribute this ticket?

  • Yes, absolutely!
  • Yes, but I would like some help.
  • No.

(This issue was written with Árbol, an AI assistant, under my direction.)

Activity

  1. scks1 commented on Nov 17, 2026

    @scks1
    ContributorAuthor

    I ran the three pairs from the issue with the sparse backend on main a59efc9. Sparse does not raise. It returns a union triangle with wrong numbers and no warning.

    import pandas as pd
    
    import chainladder as cl
    
    
    def annual(start_month):
        rows = []
        for i, year in enumerate([2001, 2002, 2003]):
            origin = pd.Timestamp(year, start_month, 1)
            for age in range(1, 4 - i):
                valuation = origin + pd.DateOffset(years=age) - pd.Timedelta(days=1)
                rows.append({"origin": origin, "valuation": valuation, "paid": 100.0 * age})
        return cl.Triangle(pd.DataFrame(rows), origin="origin", valuation="valuation",
                           columns="paid", cumulative=True)
    
    
    raa = cl.load_sample("raa")
    pairs = {
        "January vs July origins": (annual(1), annual(7)),
        "1981, 1983, 1985 vs 1982, 1984, 1986": (
            raa[raa.origin.year.isin([1981, 1983, 1985])],
            raa[raa.origin.year.isin([1982, 1984, 1986])]),
        "1981-1984 vs 1981-1983, 1986, 1987": (
            raa[raa.origin <= "1984"],
            raa[(raa.origin <= "1983") | raa.origin.year.isin([1986, 1987])]),
    }
    for name, (a, b) in pairs.items():
        out = (a.set_backend("sparse") - b.set_backend("sparse")).set_backend("numpy")
        print(name)
        for origin, value in zip(out.odims, out.values[0, 0, :, 0]):
            print("  ", str(origin)[:10], value)

    Output at age 12:

    January vs July origins
       2001-01-01 100.0
       2001-07-01 nan
       2002-01-01 nan
       2002-07-01 -100.0
       2003-01-01 nan
       2003-07-01 nan
    1981, 1983, 1985 vs 1982, 1984, 1986
       1981-01-01 5012.0
       1982-01-01 3304.0
       1983-01-01 -4563.0
       1984-01-01 -1513.0
       1985-01-01 nan
       1986-01-01 nan
    1981-1984 vs 1981-1983, 1986, 1987
       1981-01-01 nan
       1982-01-01 nan
       1983-01-01 nan
       1984-01-01 4142.0
       1986-01-01 -557.0
       1987-01-01 nan
    

    The union path treats a missing side as zero, so matching each value to its own label would give 5012, -106, 3410, -5655, 1092 and -1513 for the odd and even years. Sparse gives 3304 for 1982, which is 3410 minus 106, and -4563 for 1983, which is 1092 minus 5655. The sparse branch adds one constant offset to each side's stored coordinates at L235 and L237. The left rows go to 1981 to 1983 and the right rows go to 1982 to 1984, whatever their labels say. Neither side puts anything in 1985 or 1986, so both come back nan. In the gap pair the right side starts at 1981, so 1984 reads 4142 again, the same mix of 1984 and 1986 that #1650 shows on equal-length axes. There 1981 to 1983 cancel to exact zeros, and those read back as nan. The January and July pair shows the same pattern.

    A fix for the dense block copy should cover this branch as well, and a test that runs each pair on both backends would catch both. #1650's three examples give the same numbers on sparse as on numpy.

    (This comment was written with Árbol, an AI assistant, under my direction.)

  2. henrydingliu commented on Nov 18, 2026

    @henrydingliu
    Member

    this is a little hard to follow. can you please reconstruct the example using small, hardcoded triangles?

  3. scks1 commented on Nov 18, 2026

    @scks1
    ContributorAuthor

    Thanks, here is a smaller version with two hardcoded triangles. Both have annual origins with two origins and two ages. One starts its years in January and the other in July.

    import pandas as pd
    
    import chainladder as cl
    
    
    def tri(origin, valuation, paid):
        df = pd.DataFrame({"origin": origin, "valuation": valuation, "paid": paid})
        return cl.Triangle(df, origin="origin", valuation="valuation",
                           columns="paid", cumulative=True)
    
    
    jan = tri(["2001-01-01", "2001-01-01", "2002-01-01"],
              ["2001-12-31", "2002-12-31", "2002-12-31"], [100.0, 150.0, 200.0])
    jul = tri(["2001-07-01", "2001-07-01", "2002-07-01"],
              ["2002-06-30", "2003-06-30", "2003-06-30"], [10.0, 15.0, 20.0])
    print(jan.origin_grain, jul.origin_grain)
    jan - jul

    On main at 0d35701 this prints

    Y Y
    Traceback (most recent call last):
      ...
      File "chainladder/core/dunders.py", line 221, in _prep_origin_development
        other_arr[:, :, rol:roh, rdl:rdh] = other.values
    ValueError: could not broadcast input array from shape (1,1,2,2) into shape (1,1,3,2)
    

    Both triangles have grain Y, so _compatibility_check lets them through. The union of origins then runs 2001-01, 2001-07, 2002-01, 2002-07, and L221 copies the July triangle into it as one block from its first origin to its last. That block covers three rows of the union, but the triangle has two.

    I would expect a ValueError that says the two triangles' origins don't line up, or a result that puts each row on its own origin in the union.

    (This comment was written with Árbol, an AI assistant, under my direction.)

  4. henrydingliu commented on Nov 19, 2026

    @henrydingliu
    Member

    jan.origin and jul.origin have different dtypes. that could be a condition we build into _compatibility_check?

  5. scks1 commented on Nov 19, 2026

    @scks1
    ContributorAuthor

    Yes, a dtype check would catch the January and July case. Their origins come out as period[Y-DEC] and period[Y-JUN].

    It wouldn't catch the odd and even raa years. Both halves are period[Y-DEC], and odd - even still raises the broadcast error at dunders.py L221. The union has five origins, and the copy writes each triangle into it as one block from its first origin to its last.

    period[Y-DEC] period[Y-JUN]
    period[Y-DEC] period[Y-DEC]
    could not broadcast input array from shape (1,1,3,10) into shape (1,1,5,10)
    
    Code
    import pandas as pd
    
    import chainladder as cl
    
    
    def tri(origin, valuation, paid):
        df = pd.DataFrame({"origin": origin, "valuation": valuation, "paid": paid})
        return cl.Triangle(df, origin="origin", valuation="valuation",
                           columns="paid", cumulative=True)
    
    
    jan = tri(["2001-01-01", "2001-01-01", "2002-01-01"],
              ["2001-12-31", "2002-12-31", "2002-12-31"], [100.0, 150.0, 200.0])
    jul = tri(["2001-07-01", "2001-07-01", "2002-07-01"],
              ["2002-06-30", "2003-06-30", "2003-06-30"], [10.0, 15.0, 20.0])
    raa = cl.load_sample("raa")
    odd = raa[raa.origin.year.isin([1981, 1983, 1985])]
    even = raa[raa.origin.year.isin([1982, 1984, 1986])]
    print(jan.origin.dtype, jul.origin.dtype)
    print(odd.origin.dtype, even.origin.dtype)
    try:
        odd - even
    except ValueError as exc:
        print(exc)

    So I'd take the dtype check as the first guard for mismatched fiscal years. Triangles on the same calendar would still need each row placed on its own label in the union. #1650 is the related case where only some labels differ and the arithmetic pairs them by position.

    (This comment was written with Árbol, an AI assistant, under my direction.)

  6. henrydingliu commented on Nov 19, 2026

    @henrydingliu
    Member

    the odd/even example looks like a slicing bug. this line should've stopped it.

  7. scks1 commented on Nov 19, 2026

    @scks1
    ContributorAuthor

    You're right, and the slice path explains how it gets past L59. _contig_slice turns any evenly spaced pick into a stepped slice, so origins [0, 2, 4] become slice(None, 5, 2). The boolean mask in the issue resolves to the same positions. L59 only catches uneven picks like [0, 1, 4].

    Arithmetic holds up without slicing. Odd and even raa years built straight from the CSV fill in every origin between their first and last year, and odd + even lines up on all ten origins with the totals intact.

    slice(None, np.int64(5), np.int64(2))
    (1, 1, 3, 10)
    (1, 1, 3, 10)
    Fancy indexing on origin/development is not supported.
    (1, 1, 10, 10) (1, 1, 9, 9) (1, 1, 10, 10)
    [1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990]
    True
    
    Code
    import numpy as np
    import pandas as pd
    
    import chainladder as cl
    from chainladder.core.slice import _LocBase
    
    raa = cl.load_sample("raa")
    print(_LocBase._contig_slice(np.array([0, 2, 4])))
    print(raa.iloc[:, :, [0, 2, 4], :].shape)
    print(raa[raa.origin.year.isin([1981, 1983, 1985])].shape)
    try:
        raa.iloc[:, :, [0, 1, 4], :]
    except ValueError as exc:
        print(exc)
    
    df = pd.read_csv(cl.__path__[0] + "/utils/data/raa.csv")
    
    
    def tri(years):
        part = df[df["origin"].isin(years)]
        return cl.Triangle(part, origin="origin", valuation="development",
                           columns="values", cumulative=True)
    
    
    odd = tri([1981, 1983, 1985])
    even = tri([1982, 1984, 1986])
    both = odd + even
    print(odd.shape, even.shape, both.shape)
    print(both.origin.year.tolist())
    print(np.nansum(both.values) == np.nansum(odd.values) + np.nansum(even.values))

    So slicing is the whole story here. Should a stepped origin or development slice raise the same error as an uneven pick? I can retitle #1649 to the slicing bug if that works for you.

    (This comment was written with Árbol, an AI assistant, under my direction.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions