Skip to content

[BUG] Filtering on valuation with a period-end date drops that period #1608

Open

Description

@scks1

Are you on the latest chainladder version?

  • Yes, this bug occurs on the latest version.

Describe the bug in words

Triangle.valuation stores each valuation at the last nanosecond of its period (1988-12-31 23:59:59.999999999 for annual raa), see base.py L743 and L748. pandas reads a date string, a bare year, a pd.Timestamp or a datetime bound as midnight. So a filter written with the period-end date drops that period:

  • raa[raa.valuation <= "1988-12-31"] and <= "1988" give the triangle as of 1987, not 1988.
  • raa.valuation == "1988-12-31" matches no cell, so raa[...] raises "Slice returns empty Triangle".
  • raa.valuation > "1988-12-31" keeps the 1988 cells.

The same happens at other grains: on the quarterly sample, valuation <= "1995-06-30" and <= "1995Q2" stop at 1995-03-31, and on monthly prism, <= "2017-08-31" stops at 2017-07-31.

The docs filter with the start of the next period (valuation < "2008" in the Friedland chapter 15 notebook) or compare against X.valuation_date, and both give the right answer. Nothing warns against the <= form, though, and it gives a different answer without an error. On raa it changes the chain ladder IBNR from 49,656.91 to 51,419.19.

From its diff, #1304 rewrites valuation but keeps the same period-end construction, so this would survive it. Any fix would touch the code #1304 changes.

How can the bug be reproduced?

import chainladder as cl

raa = cl.load_sample("raa")
a = raa[raa.valuation <= "1988-12-31"]
b = raa[raa.valuation < "1989"]
print(a.shape, a.valuation_date)
print(b.shape, b.valuation_date)
print((raa.valuation == "1988-12-31").sum())

On main ee2c24d (pandas 2.3.3):

(1, 1, 7, 7) 1987-12-31 23:59:59.999999999
(1, 1, 8, 8) 1988-12-31 23:59:59.999999999
0

What is the expected behavior?

assert raa[raa.valuation <= "1988-12-31"].valuation_date == raa[raa.valuation < "1989"].valuation_date
assert (raa.valuation == "1988-12-31").sum() == 8  # origins 1981 to 1988

Possible fixes, smallest first:

  1. A note in the triangle user guide and the valuation docstring saying to filter with < next period or against valuation_date.
  2. Store valuations at midnight on the period's last day, so a date-only bound compares as users expect. This changes every valuation_date timestamp, so it may fit better inside [FEAT]deprecate date dims #1304.
  3. A small helper such as tri.as_of("1988") that slices to the end of the named period.

Which would you prefer?

Would you be willing to contribute this ticket?

  • Yes, absolutely!
  • Yes, but I would like some help.
  • No.

(This issue was written with Árbol, an AI assistant, under my direction.)

Activity

  1. henrydingliu commented on Nov 14, 2026

    @henrydingliu
    Member

    did this exist on the prior version of L743? assuming it did, this is my take on the issue.

    the whole "microsecond before midnight" mental model for valuation date is actually fairly redundant after the triangle is constructed. when constructing a triangle, we want to make sure that losses reported and transactions recorded on 12/31 gets counted on 12/31. once we have a properly constructed triangle, do we ever need sub-second, or even sub-hour, level of precision?

    lastly, i don't think any of the 3 proposed next steps are mutually exclusive. docstring and the helper function can go in now. long term discusison can happen after #1304 lands

  2. scks1 commented on Nov 14, 2026

    @scks1
    ContributorAuthor

    Yes, it's old. valuation has built each period end as the start of the next period minus one time unit since at least v0.8.0 (- np.timedelta64(1, "ns") there, the same in v0.9.0, and at L698 in v0.10.1). Just before #1580 it sat at L702, and #1580 only added lines above it, which moved it to L743. The unit is now a nanosecond or a microsecond depending on the pandas version.

    Agreed on the plan. I'll open one PR with the note in the valuation docstring and the triangle user guide, plus an as_of() helper, and leave the stamping question until #1304 lands.

    One question before I write the helper. I'd make it keep every cell valued on or before the end of the named day or period: as_of("1988-12-31") and as_of("1988") both give raa as of 1988, and as_of("1988-06-30") gives 1987. Does that match what you have in mind?

    (This comment was written with Árbol, an AI assistant, under my direction.)

  3. henrydingliu commented on Nov 14, 2026

    @henrydingliu
    Member

    One question before I write the helper. I'd make it keep every cell valued on or before the end of the named day or period: as_of("1988-12-31") and as_of("1988") both give raa as of 1988, and as_of("1988-06-30") gives 1987. Does that match what you have in mind?

    aligned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions