Repository navigation
Conversation
With ignore_index=True, concat renumbered origin or development labels 0, …, n - 1, which drops the grain the labels encode: origins became nanosecond offsets from the Unix epoch (casact#1489), and development labels were read as months (casact#1490). Following the discussion in casact#1489, concat now raises a ValueError for ignore_index=True along either axis. Groupby aggregations still stack one slice per group with ignore_index=True along the grouped axis, which can be origin or development, so they now call a private _concat that holds the existing implementation. Developed with an AI assistant (Árbol, running Muse).
concat checked its inputs with bare asserts, so a failed check surfaced as an AssertionError with no message (casact#1489), and under python -O the checks did not run at all. Each check now raises a ValueError that says what went wrong: - a label repeats along the concatenation axis. The message names the repeated labels, and for origin periods it points to adding the triangles with +, the route discussed in casact#1489. - the index, origin periods or development periods differ along an axis other than the concatenation axis. Differing lengths get this message too, where numpy raised an "inhomogeneous shape" error before. Repeated labels are reported first, so splitting a triangle at an origin period and concatenating the pieces, the case in casact#1489, gets the hint about + even when the pieces' development periods also differ. Every input that failed a check before still fails, now with a ValueError. Developed with an AI assistant (Árbol, running Muse).
Apply the import order that the Ruff workflow's "Suggest import sorting for changed files" step proposes for utility_functions.py (ruff 0.16.1 check --select I --fix). Only import lines move. The step checks whole files, so this also sorts imports that were already on main. core/pandas.py and test_utilities.py are left as they are: casact#1304 edits the import block of core/pandas.py, and casact#1465 removes an import and several tests from test_utilities.py, so sorting either file here would conflict with those PRs. Developed with an AI assistant (Árbol, running Muse).
Pyright Type CompletenessView the full Project (full
Other symbols referenced but not exported by
Symbols without documentation:
Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 4)
Patch symbol details
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1534 +/- ##
==========================================
+ Coverage 94.38% 94.40% +0.01%
==========================================
Files 98 98
Lines 6216 6233 +17
Branches 799 806 +7
==========================================
+ Hits 5867 5884 +17
Misses 225 225
Partials 124 124
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| for i in self.groups.indices.values() | ||
| ] | ||
| obj = concat(values, axis=self.axis, ignore_index=True) | ||
| obj = _concat(values, axis=self.axis, ignore_index=True) |
There was a problem hiding this comment.
i'm a little uncomfortable with making a private concat just for rejecting ignore_index=True. my 0-shot reasoning is this:
- we are introducing this rejection because we don't want ill-formed triangle to exist
- so assuming ill-formed triangles never existed, then grouping by axis=2 or axis=3 is always performed on a well-formed triangle, and would never result in an ill-formed triangle
- this ignore_index=True should be necessary only when grouping axis=0 or axis=1
- so would it work if we simply branched this concat call to use either ignore_index=True or ignore_index=False based on
self.axis?
There was a problem hiding this comment.
Yes. Done in 3775550: _concat is gone, and the groupby helper calls concat with ignore_index=self.axis in (0, 1). test_grain.py passes, and so does the rest of the suite, apart from the 8 environment failures main also has here (listed in the description).
One call doesn't fit the premise: groupby(..., axis=3) on a development triangle. The aggregation labels a summed development slice with the slice's valuation date (the "If axis is development, set the ddims to be the valuation date" step in core/pandas.py), and on a development triangle every slice has the same valuation date. On main, ignore_index=True then numbers the groups 0, …, n − 1, which gives the #1490 triangle: for raa's incremental triangle with development years summed in pairs, development is [0, 1, 2, 3, 4] and latest_diagonal raises Slice returns empty Triangle. With this change, concat raises instead:
ValueError: Development periods repeat across the triangles: ['1990-12-31'].
If a group holds a single development period, it keeps its integer age, and the call fails in numpy instead (DTypePromotionError). On a valuation triangle each group has its own valuation date, so the same groupby now keeps the dates (1982, 1984, …, 1990) and latest_diagonal works; on main it also came back labelled 0, …, 4. grain() is the only caller in the library that groups along origin or development, and its results are unchanged.
Neither error tells someone calling groupby what went wrong. Should the helper catch this case and raise a clearer error, or leave it to concat?
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
There was a problem hiding this comment.
ValueError: Development periods repeat across the triangles: ['1990-12-31'].
is it possible to raise this outside of groupby?
There was a problem hiding this comment.
Yes. concat raises it, so a direct call along the development axis hits it whenever two triangles share a development period:
raa = cl.load_sample("raa")
cl.concat([raa.iloc[..., :2], raa.iloc[..., 1:3]], axis=3)
# ValueError: Development periods repeat across the triangles: [24].
val = raa.dev_to_val()
cl.concat([val.iloc[..., :3], val.iloc[..., 2:5]], axis=3)
# ValueError: Development periods repeat across the triangles: ['1983-12-31'].On main both calls fail with a bare AssertionError from the uniqueness assert in concat's development branch.
Within the library, groupby is the only caller that can now reach it with input main accepted, since it was the only one passing ignore_index=True. Every other concat call already used the default, so for those calls it fires only where main already failed an assert.
(This reply was made with Árbol, an AI assistant, under my direction.)
There was a problem hiding this comment.
these examples are concatting overlapping development periods. the call in the groupby helper supplies (theoretically) non-overlapping development slices.
i'm tryin to figure out whether the issue with the identical valuation date is an issue that other callers of concat could realistically run into, which would point to the fix belonging in concat and arguably in scope for this issue/PR. conversely, if this is due to a quirk in the groupby helper, we can close this out and do a follow-up.
There was a problem hiding this comment.
I think it's the groupby helper, so a follow-up sounds right. The repeated date comes from the aggregation that runs before concat: sum along development labels the summed column with the slice's valuation date, and on a development triangle the slices normally share the triangle's valuation date. For raa all ten columns report 1990-12-31. Ordinary non-overlapping slices keep their ages, or their dates on a valuation triangle, and concatenate fine on main and here.
A direct caller only meets it by doing the groupby by hand:
inc = cl.load_sample("raa").cum_to_incr()
halves = [inc.iloc[..., :5].sum("development"), inc.iloc[..., 5:].sum("development")]
cl.concat(halves, axis=3)
# main: AssertionError
# this PR: ValueError: Development periods repeat across the triangles: ['1990-12-31'].On a valuation triangle the two sums are labelled 1985-12-31 and 1990-12-31, so that concat works, and with this PR the groupby keeps its dates.
After concatenating, the helper replaces the labels of index, column and origin groups, but it has no such step for development, so development groups reach concat with the aggregation's labels. A follow-up could label them in the helper before concatenating, for example with the group keys, as the origin branch does. That would also cover single-period groups, which keep their integer age and now fail in numpy (DTypePromotionError). grain() groups along development only after dev_to_val() and sets ddims itself afterwards, so it doesn't depend on those labels. Happy to open an issue for the follow-up.
(This reply was made with Árbol, an AI assistant, under my direction.)
| return [i.set_backend(backend) for i in objs] | ||
|
|
||
|
|
||
| def _labels_match(labels: list) -> bool: |
There was a problem hiding this comment.
if these aren't useful outside of concat, let's define these inside concat
There was a problem hiding this comment.
Done in 3e65a2f. They're only used by concat, so they're defined inside it now. _repeated became repeated_labels there, because check_unique_labels already has a local variable named repeated.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
Drop the private _concat, as suggested in review. The groupby aggregation helper now calls concat with ignore_index=True only when it groups the index or columns: aggregating relabels those slices "(All)" or 0, and the helper sets the real labels afterwards. Origin and development slices are concatenated with their own labels. grain() is unaffected. It groups a valuation triangle, where each summed slice keeps its first origin or its own valuation date, so the labels are distinct. Calling groupby(..., axis=3) directly on a development triangle now raises, because summing a development slice labels it with the triangle's valuation date and every group gets the same label; on main that call returns development labels 0, ..., n - 1. Developed with an AI assistant (Árbol, running Muse).
The three label helpers are used only by concat, so they move into it, as suggested in review. _repeated becomes repeated_labels, because check_unique_labels already uses repeated as a local name. Developed with an AI assistant (Árbol, running Muse).
concat raises for ignore_index=True along origin or development before it builds the new axis, so the branches that numbered origin and development labels 0, ..., n - 1 can no longer run. Remove them. Developed with an AI assistant (Árbol, running Muse).
|
I'm doing a pre-thanksgiving clean-up. @scks1 apologies for forgetting about this PR. do you mind addressing the conflicts and we can close this out? |
Summary of Changes
With
ignore_index=True,cl.concatrenumbers the concatenation axis 0, …, n − 1. Along origin or development, that throws away the grain the labels carry. Origin labels became the integers 0, …, n − 1, and every origin printed as 1970 (#1489). Development labels became 0, 1, 2, … read as months, andlatest_diagonalthen raisedValueError: Slice returns empty Triangle.(#1490). As agreed in #1489,concatnow raises aValueErrorforignore_index=Truealong origin or development. Index and columns accept it as before.TriangleGroupByaggregations stack one slice per group along the grouped axis, which can be origin or development (grainuses both). They now passignore_index=Trueonly when grouping the index or columns, whose aggregated slices share placeholder labels that the helper replaces afterwards. Origin and development slices keep their own labels.graingives the same results as before: it groups a valuation triangle, where each summed slice is labelled with its first origin or its own valuation date.Calling
groupby(..., axis=3)directly changes. On a development triangle, the aggregation gives every summed slice the same label, the triangle's valuation date, so onmainthe call relied onignore_index=Trueand came back with development periods 0, …, n − 1: the same broken triangle as #1490, wherelatest_diagonalraisesSlice returns empty Triangle.For raa's incremental triangle with development years summed in pairs, the call now raises fromconcat:If a group holds a single period, numpy raises a
DTypePromotionErrorinstead. On a valuation triangle, the same groupby now keeps each group's valuation date, wheremainalso returned 0, …, n − 1. Nothing in the library or the tests groups along origin or development exceptgrain.The second commit takes up the point in #1489 that concat's errors are not helpful. Its other checks were bare
asserts, so a failure was anAssertionErrorwith no message. They now raiseValueErrors that say which axis differs or which labels repeat. Repeated labels are checked first, so the prism split from #1489 gets a pointer to+:On
main(with numpy 2.5.4), the same input raises numpy'sValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions...from the development check, because the two pieces' development periods differ in length.The third commit sorts imports in
utility_functions.py, as the Ruff workflow's import-sorting step suggests.core/pandas.pyandtest_utilities.pyare left as they are: #1304 edits the import block ofcore/pandas.py, and #1465 removes an import and several tests fromtest_utilities.py, so sorting either file here would conflict with those PRs.The fourth and fifth commits follow review: the private
_concatfrom the first commit is gone and groupby callsconcatas described above, and concat's label helpers are now defined insideconcat. The sixth removes the branches inconcatthat numbered origin and development labels 0, …, n − 1, which the new error makes unreachable.Related GitHub Issue(s)
Fixes #1489
Fixes #1490
AI/LLM Usage
This change was written with Árbol, an AI assistant running on Muse, working under my direction. It wrote the code and tests, ran the checks listed below, and drafted this description. I reviewed the diff and the results before submitting.
Additional Context for Reviewers
Overlap with #1304. #1304 edits the same lines of
concat(theodims/ddimschecks and branches) and the same test,test_concat_ignore_index_axes, so whichever PR merges second will need a small rebase. I'm happy to do ours. This PR keepsodims/ddims, as the surrounding code onmaindoes.Exception type.
concatnow raisesValueErrorwhere it raisedAssertionError(or numpy's shape error) before; every input that failed before still fails.test_concat_axis1_duplicate_columnsis updated to match. If you would rather keep that out of this fix, I can move it to its own PR.Tests (
chainladder/utils/tests/test_utilities.py):test_concat_ignore_index_axeskeeps its index and columns cases. Its origin and development cases, which asserted that the call succeeded, are replaced bytest_concat_ignore_index_origin_development_raises, parametrized over2,3,"origin","development",-2and-1.test_concat_repeated_labels_raise(index, origin, development),test_concat_reports_repeated_origins_first,test_concat_misaligned_axes_raise.main(e9cc6f7) and pass here.test_grain.pypasses unchanged. It covers the groupby path: if the aggregation kept passingignore_index=Trueon every axis, as it does onmain, 104 of its 112 cases would fail against the newconcat.grainresults betweenmainand this branch (values, the labels on all four axes, grain,is_val_tri, backend). 17 are identical; the 6 that differ are the direct development groupbys described above.mainin this environment: 2test_heatmap_rendercases need IPython, which isn't installed here, and 6test_development.pycases that turn warnings into errors trip on numpy 2.5'sDeprecationWarningabout the generic timedelta unit, raised atcore/base.py:706.ruff checkandruff format --checkare clean.No docs page mentions
ignore_index; theconcatdocstring now says which axes accept it and lists the errors.Submitter's Checklist
[FIX],[FEAT],[DOCS],[TST],[CHORE], or[BRK]).Reviewer's Checklist