Skip to content

[FIX] Reject ignore_index=True along origin and development in concat - #1534

Open
scks1 wants to merge 6 commits into
casact:mainfrom
scks1:fix-concat-ignore-index
Open

scks1 wants to merge 6 commits into
casact:mainfrom
scks1:fix-concat-ignore-index

Conversation

@scks1

@scks1 scks1 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary of Changes

With ignore_index=True, cl.concat renumbers the concatenation axis 0, …, n − 1. Along origin or development, that throws away the grain the labels carry. Origin labels became the integers 0, …, n − 1, and every origin printed as 1970 (#1489). Development labels became 0, 1, 2, … read as months, and latest_diagonal then raised ValueError: Slice returns empty Triangle. (#1490). As agreed in #1489, concat now raises a ValueError for ignore_index=True along origin or development. Index and columns accept it as before.

ValueError: ignore_index=True is not supported along the origin or development axis, because their labels carry the triangle's grain.

TriangleGroupBy aggregations stack one slice per group along the grouped axis, which can be origin or development (grain uses both). They now pass ignore_index=True only when grouping the index or columns, whose aggregated slices share placeholder labels that the helper replaces afterwards. Origin and development slices keep their own labels. grain gives the same results as before: it groups a valuation triangle, where each summed slice is labelled with its first origin or its own valuation date.

Calling groupby(..., axis=3) directly changes. On a development triangle, the aggregation gives every summed slice the same label, the triangle's valuation date, so on main the call relied on ignore_index=True and came back with development periods 0, …, n − 1: the same broken triangle as #1490, where latest_diagonal raises Slice returns empty Triangle. For raa's incremental triangle with development years summed in pairs, the call now raises from concat:

ValueError: Development periods repeat across the triangles: ['1990-12-31'].

If a group holds a single period, numpy raises a DTypePromotionError instead. On a valuation triangle, the same groupby now keeps each group's valuation date, where main also returned 0, …, n − 1. Nothing in the library or the tests groups along origin or development except grain.

The second commit takes up the point in #1489 that concat's errors are not helpful. Its other checks were bare asserts, so a failure was an AssertionError with no message. They now raise ValueErrors that say which axis differs or which labels repeat. Repeated labels are checked first, so the prism split from #1489 gets a pointer to +:

ValueError: Origin periods repeat across the triangles: ['2013-01-01']. To combine amounts for the same origin periods, add the triangles with + instead.

On main (with numpy 2.5.4), the same input raises numpy's ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions... from the development check, because the two pieces' development periods differ in length.

The third commit sorts imports in utility_functions.py, as the Ruff workflow's import-sorting step suggests. core/pandas.py and test_utilities.py are left as they are: #1304 edits the import block of core/pandas.py, and #1465 removes an import and several tests from test_utilities.py, so sorting either file here would conflict with those PRs.

The fourth and fifth commits follow review: the private _concat from the first commit is gone and groupby calls concat as described above, and concat's label helpers are now defined inside concat. The sixth removes the branches in concat that numbered origin and development labels 0, …, n − 1, which the new error makes unreachable.

Related GitHub Issue(s)

Fixes #1489
Fixes #1490

AI/LLM Usage

This change was written with Árbol, an AI assistant running on Muse, working under my direction. It wrote the code and tests, ran the checks listed below, and drafted this description. I reviewed the diff and the results before submitting.

Additional Context for Reviewers

Overlap with #1304. #1304 edits the same lines of concat (the odims/ddims checks and branches) and the same test, test_concat_ignore_index_axes, so whichever PR merges second will need a small rebase. I'm happy to do ours. This PR keeps odims/ddims, as the surrounding code on main does.

Exception type. concat now raises ValueError where it raised AssertionError (or numpy's shape error) before; every input that failed before still fails. test_concat_axis1_duplicate_columns is updated to match. If you would rather keep that out of this fix, I can move it to its own PR.

Tests (chainladder/utils/tests/test_utilities.py):

  • test_concat_ignore_index_axes keeps its index and columns cases. Its origin and development cases, which asserted that the call succeeded, are replaced by test_concat_ignore_index_origin_development_raises, parametrized over 2, 3, "origin", "development", -2 and -1.
  • New: test_concat_repeated_labels_raise (index, origin, development), test_concat_reports_repeated_origins_first, test_concat_misaligned_axes_raise.
  • The 24 new or changed test cases fail on main (e9cc6f7) and pass here.
  • test_grain.py passes unchanged. It covers the groupby path: if the aggregation kept passing ignore_index=True on every axis, as it does on main, 104 of its 112 cases would fail against the new concat.
  • I compared 23 groupby and grain results between main and this branch (values, the labels on all four axes, grain, is_val_tri, backend). 17 are identical; the 6 that differ are the direct development groupbys described above.
  • Full suite (Python 3.12.15, numpy 2.5.4, pandas 3.1.0): 1442 passed, 2 skipped, 8 failed. The same 8 fail on main in this environment: 2 test_heatmap_render cases need IPython, which isn't installed here, and 6 test_development.py cases that turn warnings into errors trip on numpy 2.5's DeprecationWarning about the generic timedelta unit, raised at core/base.py:706.
  • ruff check and ruff format --check are clean.

No docs page mentions ignore_index; the concat docstring now says which axes accept it and lists the errors.

Submitter's Checklist

  • I have reviewed and am adhering to the standards outlined in the project Governing Doc.
  • The PR subject title summarizes the changes, with one proper prefix ([FIX], [FEAT], [DOCS], [TST], [CHORE], or [BRK]).
  • I am a human (not a bot), and this PR form is written by a human.

Reviewer's Checklist

  • The implementation addresses the associated issue(s).
  • The implementation is appropriate, maintainable, and follows ARCHITECTURE.md.
  • PR subject title has the proper prefix and the subject is appropriate.
  • Relevant issue(s) are linked.
  • AI/LLM usage is disclosed and appropriate.
  • Documentation and tests are appropriate.
  • CI tests passed, or any failures are acceptable.
  • Leave a comment with the final recommendation (e.g. approve as is, request a secondary review, or flag an area for more review).

scks1 added 3 commits November 3, 2026 22:37
With ignore_index=True, concat renumbered origin or development labels
0, …, n - 1, which drops the grain the labels encode: origins became
nanosecond offsets from the Unix epoch (casact#1489), and development labels
were read as months (casact#1490). Following the discussion in casact#1489, concat
now raises a ValueError for ignore_index=True along either axis.

Groupby aggregations still stack one slice per group with
ignore_index=True along the grouped axis, which can be origin or
development, so they now call a private _concat that holds the existing
implementation.

Developed with an AI assistant (Árbol, running Muse).
concat checked its inputs with bare asserts, so a failed check surfaced
as an AssertionError with no message (casact#1489), and under python -O the
checks did not run at all. Each check now raises a ValueError that says
what went wrong:

- a label repeats along the concatenation axis. The message names the
  repeated labels, and for origin periods it points to adding the
  triangles with +, the route discussed in casact#1489.
- the index, origin periods or development periods differ along an
  axis other than the concatenation axis. Differing lengths get this
  message too, where numpy raised an "inhomogeneous shape" error before.

Repeated labels are reported first, so splitting a triangle at an origin
period and concatenating the pieces, the case in casact#1489, gets the hint
about + even when the pieces' development periods also differ. Every
input that failed a check before still fails, now with a ValueError.

Developed with an AI assistant (Árbol, running Muse).
Apply the import order that the Ruff workflow's "Suggest import
sorting for changed files" step proposes for utility_functions.py
(ruff 0.16.1 check --select I --fix). Only import lines move.
The step checks whole files, so this also sorts imports that were
already on main. core/pandas.py and test_utilities.py are left as
they are: casact#1304 edits the import block of core/pandas.py, and casact#1465
removes an import and several tests from test_utilities.py, so
sorting either file here would conflict with those PRs.

Developed with an AI assistant (Árbol, running Muse).
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 16.0% of exported symbols fully typed (256 / 1599)

Known Ambiguous Unknown Total
Project (head) 256 106 1237 1599

Other symbols referenced but not exported by chainladder: 24

Known Ambiguous Unknown Total
Other (head) 11 1 12 24

Symbols without documentation:

  • Functions without docstring: 340
  • Functions without default param: 0
  • Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 4)

Known Ambiguous Unknown Total
Patch 0 0 4 4
Patch symbol details
Symbol Status Change
chainladder.utils.tests.test_utilities.test_concat_ignore_index_origin_development_raises ❌ unknown new
chainladder.utils.tests.test_utilities.test_concat_misaligned_axes_raise ❌ unknown new
chainladder.utils.tests.test_utilities.test_concat_repeated_labels_raise ❌ unknown new
chainladder.utils.tests.test_utilities.test_concat_reports_repeated_origins_first ❌ unknown new

@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.40%. Comparing base (7157b3f) to head (8ecd1fd).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1534      +/-   ##
==========================================
+ Coverage   94.38%   94.40%   +0.01%     
==========================================
  Files          98       98              
  Lines        6216     6233      +17     
  Branches      799      806       +7     
==========================================
+ Hits         5867     5884      +17     
  Misses        225      225              
  Partials      124      124              
Flag Coverage Δ
unittests 94.40% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread chainladder/core/pandas.py Outdated
for i in self.groups.indices.values()
]
obj = concat(values, axis=self.axis, ignore_index=True)
obj = _concat(values, axis=self.axis, ignore_index=True)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm a little uncomfortable with making a private concat just for rejecting ignore_index=True. my 0-shot reasoning is this:

  • we are introducing this rejection because we don't want ill-formed triangle to exist
  • so assuming ill-formed triangles never existed, then grouping by axis=2 or axis=3 is always performed on a well-formed triangle, and would never result in an ill-formed triangle
  • this ignore_index=True should be necessary only when grouping axis=0 or axis=1
  • so would it work if we simply branched this concat call to use either ignore_index=True or ignore_index=False based on self.axis?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Done in 3775550: _concat is gone, and the groupby helper calls concat with ignore_index=self.axis in (0, 1). test_grain.py passes, and so does the rest of the suite, apart from the 8 environment failures main also has here (listed in the description).

One call doesn't fit the premise: groupby(..., axis=3) on a development triangle. The aggregation labels a summed development slice with the slice's valuation date (the "If axis is development, set the ddims to be the valuation date" step in core/pandas.py), and on a development triangle every slice has the same valuation date. On main, ignore_index=True then numbers the groups 0, …, n − 1, which gives the #1490 triangle: for raa's incremental triangle with development years summed in pairs, development is [0, 1, 2, 3, 4] and latest_diagonal raises Slice returns empty Triangle. With this change, concat raises instead:

ValueError: Development periods repeat across the triangles: ['1990-12-31'].

If a group holds a single development period, it keeps its integer age, and the call fails in numpy instead (DTypePromotionError). On a valuation triangle each group has its own valuation date, so the same groupby now keeps the dates (1982, 1984, …, 1990) and latest_diagonal works; on main it also came back labelled 0, …, 4. grain() is the only caller in the library that groups along origin or development, and its results are unchanged.

Neither error tells someone calling groupby what went wrong. Should the helper catch this case and raise a clearer error, or leave it to concat?

(The commit and this reply were made with Árbol, an AI assistant, under my direction.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueError: Development periods repeat across the triangles: ['1990-12-31'].

is it possible to raise this outside of groupby?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. concat raises it, so a direct call along the development axis hits it whenever two triangles share a development period:

raa = cl.load_sample("raa")
cl.concat([raa.iloc[..., :2], raa.iloc[..., 1:3]], axis=3)
# ValueError: Development periods repeat across the triangles: [24].

val = raa.dev_to_val()
cl.concat([val.iloc[..., :3], val.iloc[..., 2:5]], axis=3)
# ValueError: Development periods repeat across the triangles: ['1983-12-31'].

On main both calls fail with a bare AssertionError from the uniqueness assert in concat's development branch.

Within the library, groupby is the only caller that can now reach it with input main accepted, since it was the only one passing ignore_index=True. Every other concat call already used the default, so for those calls it fires only where main already failed an assert.

(This reply was made with Árbol, an AI assistant, under my direction.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these examples are concatting overlapping development periods. the call in the groupby helper supplies (theoretically) non-overlapping development slices.

i'm tryin to figure out whether the issue with the identical valuation date is an issue that other callers of concat could realistically run into, which would point to the fix belonging in concat and arguably in scope for this issue/PR. conversely, if this is due to a quirk in the groupby helper, we can close this out and do a follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's the groupby helper, so a follow-up sounds right. The repeated date comes from the aggregation that runs before concat: sum along development labels the summed column with the slice's valuation date, and on a development triangle the slices normally share the triangle's valuation date. For raa all ten columns report 1990-12-31. Ordinary non-overlapping slices keep their ages, or their dates on a valuation triangle, and concatenate fine on main and here.

A direct caller only meets it by doing the groupby by hand:

inc = cl.load_sample("raa").cum_to_incr()
halves = [inc.iloc[..., :5].sum("development"), inc.iloc[..., 5:].sum("development")]
cl.concat(halves, axis=3)
# main: AssertionError
# this PR: ValueError: Development periods repeat across the triangles: ['1990-12-31'].

On a valuation triangle the two sums are labelled 1985-12-31 and 1990-12-31, so that concat works, and with this PR the groupby keeps its dates.

After concatenating, the helper replaces the labels of index, column and origin groups, but it has no such step for development, so development groups reach concat with the aggregation's labels. A follow-up could label them in the helper before concatenating, for example with the group keys, as the origin branch does. That would also cover single-period groups, which keep their integer age and now fail in numpy (DTypePromotionError). grain() groups along development only after dev_to_val() and sets ddims itself afterwards, so it doesn't depend on those labels. Happy to open an issue for the follow-up.

(This reply was made with Árbol, an AI assistant, under my direction.)

Comment thread chainladder/utils/utility_functions.py Outdated
return [i.set_backend(backend) for i in objs]


def _labels_match(labels: list) -> bool:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if these aren't useful outside of concat, let's define these inside concat

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3e65a2f. They're only used by concat, so they're defined inside it now. _repeated became repeated_labels there, because check_unique_labels already has a local variable named repeated.

(The commit and this reply were made with Árbol, an AI assistant, under my direction.)

scks1 added 3 commits November 4, 2026 21:25
Drop the private _concat, as suggested in review. The groupby
aggregation helper now calls concat with ignore_index=True only when it
groups the index or columns: aggregating relabels those slices "(All)"
or 0, and the helper sets the real labels afterwards. Origin and
development slices are concatenated with their own labels.

grain() is unaffected. It groups a valuation triangle, where each summed
slice keeps its first origin or its own valuation date, so the labels
are distinct. Calling groupby(..., axis=3) directly on a development
triangle now raises, because summing a development slice labels it with
the triangle's valuation date and every group gets the same label; on
main that call returns development labels 0, ..., n - 1.

Developed with an AI assistant (Árbol, running Muse).
The three label helpers are used only by concat, so they move into it,
as suggested in review. _repeated becomes repeated_labels, because
check_unique_labels already uses repeated as a local name.

Developed with an AI assistant (Árbol, running Muse).
concat raises for ignore_index=True along origin or development before
it builds the new axis, so the branches that numbered origin and
development labels 0, ..., n - 1 can no longer run. Remove them.

Developed with an AI assistant (Árbol, running Muse).
@henrydingliu

Copy link
Copy Markdown
Member

I'm doing a pre-thanksgiving clean-up.

@scks1 apologies for forgetting about this PR. do you mind addressing the conflicts and we can close this out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants