Skip to content

[DISCUSSION] I/O backward compatibility #1456

Open

Description

@genedan

Triangle.__setstate__() is a method that will be added via #1303 to ensure the compatibility of pickles before and after the removal of kdims, vdims, odims, and ddims. This was done as a part of #601.

Once we are ready to drop the method, we can close this issue.

Edit: Renamed issue to be a general discussion on I/O backwards compatibility, and when/if to take up the task.

Activity

  1. henrydingliu commented on Oct 3, 2026

    @henrydingliu
    Member

    this is alarming. i will raise my concern on #601

  2. henrydingliu commented on Oct 3, 2026

    @henrydingliu
    Member

    also, do we need this for the various backend deprecations?

  3. genedan commented on Oct 3, 2026

    @genedan
    MemberAuthor

    Since a pickle preserves an object, it would preserve things like private attributes so yes, you might break pickling compatibility even if you keep the public API the same.

    Priyam's setstate approach is consistent with how Pandas deals with the problem. Future versions are able to read pickles from past versions within a supported window.

  4. genedan commented on Oct 3, 2026

    @genedan
    MemberAuthor

    Perhaps __setstate__() is here to stay, guaranteeing I/O compatibility for X minor versions? How about going back to the last major version?

    People always want their code to run faster, use less memory, etc. I anticipate internals changing indefinitely, something our successors will deal with as well.

  5. genedan commented on Oct 3, 2026

    @genedan
    MemberAuthor

    I asked Claude the following prompt:

    Can you see how far back Triangle I/O works in terms of version numbers, if being read by the latest release?

    The result was that it does not work reliably before 0.10.0 for pickles and 0.8.5 for JSON. That doesn't surprise me since we haven't been thinking about semver until recently. If you're curious, Pandas tests this in this file.

    Roping in @casact/chainladder-maintainers on how to think about this. How far back should we guarantee backwards-compatible I/O, and when should we actually start guaranteeing it? As far as I can tell, we don't support this yet, and Priyam's work was the first attempt at trying to do this.

    I think before we try any kinds of guarantees, we should make sure we have semi-stable formats for the various objects (pickle, JSON, dict, html, latex, etc., etc., etc.,). I don't think we have really tried this, we are just relying on John's prior work, which as far as I can tell, is still in alpha/beta phase.

    Do I think it's urgent? Not really, because I think actuaries still live in the land of copying and pasting things from the terminal/Jupyter into Excel, and we can tell tech-oriented ones to make sure the cl versions on both ends of the pipeline are the same. But I do think it's part of the work involved in eventually bringing the package from "interesting package bro" to "this is world class" level. We also have the option of leaving this as an open problem and proposing this to be one of the things that newcomers can take on.

    In order of priorities, I think supporting a wider variety of well-structured I/O formats (such as...) would be higher than backwards compatibility. This would make CL compatible with a greater number of technologies/work setups first, and then compatible with itself second.

  6. henrydingliu commented on Oct 4, 2026

    @henrydingliu
    Member

    The result was that it does not work reliably before 0.10.0 for pickles and 0.8.5 for JSON

    hmmm, i got some bug reports to file.

    I think supporting a wider variety of well-structured I/O formats (such as...)

    this is very doable. we just need a read_standard_frame() method that turns the output of the Triangle method below back into a Triangle

    def to_standard_frame(self):
        attrs = [_pattern, ...]
        return pd.concat([
            self.incr_to_cum().val_to_dev().to_frame(),
            pd.DataFrame(columns=[f"{x}={str(get_attr(self, x, None)}" for x in attrs]
        ])

    Do I think it's urgent? Not really

    i don't think the functionality itself is urgent. but i think we should get the prep work done sooner rather than later, i.e. taking inventory of all declared and ad-hoc attributes. let's discuss when you get to type-hinting on common.py.

  7. henrydingliu commented on Oct 4, 2026

    @henrydingliu
    Member

    a backward compat test for pickle is set up in #1465. i'm not committing to backward compat through 0.8.24 (first version on pandas 2). just wanted to set up a monitoring mechanism. we can set it to xfail at any time.

  8. priyam0k commented on Oct 4, 2026

    @priyam0k
    Contributor

    How far back should we guarantee backwards-compatible I/O, and when should we actually start guaranteeing it?

    Keeping __setstate__ active throughout 1.x makes sense so users upgrading from 0.10+ don't hit deserialization errors when loading legacy pickles with old kdims or vdims attributes. Guaranteeing compatibility all the way back to early 0.x builds creates high maintenance overhead, so aligning support with the current and previous minor or major window seems like a cleaner boundary.

    In order of priorities, I think supporting a wider variety of well-structured I/O formats (such as...) would be higher than backwards compatibility.

    Prioritizing standardized tabular serialization like Parquet, Arrow, or a standard frame export is definitely higher leverage than chasing pickle compatibility. Pickles are inherently tied to private class state, whereas schema-backed formats give stable interchange across versions and external tools. Open to other ideas on the support window.

  9. changed the title [-]Deprecate Triangle.__setstate__()[/-] [+][DISCUSSION] I/O backward compatibility[/+] on Oct 5, 2026
  10. genedan commented on Oct 5, 2026

    @genedan
    MemberAuthor

    Yeah agreed it's going to be too much work to get this feature to work on pre-1.0.0 releases. If we have a well-thought out Triangle.to_frame() method, we can piggy back off a lot of the more advanced formats that Pandas offers.

    It has been mentioned by Henry in the past that to_frame() is very hacky, and I would agree.

  11. henrydingliu commented on Oct 5, 2026

    @henrydingliu
    Member

    as promised, a demo to support the full range of Pandas-compatible formats is up. i only implemented read_html to illustrate the feasibility. but to implement everything else from this list should be mostly passthrough. take a look. if you are good with the idea, i can work on rest of the integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions