Engineering: Define portable, versioned persistence for scientific state

Current status (8 October 2026)

Merged: Decouple calculation presentation from input files #89 (frontend-independent requested/resolved presentation, formerly PR A) and Add multi-calculation portable RustiQ V1 containers #88 (portable multi-calculation .rustiq V1, formerly PR B). The Rust APIs RustiQData and RustiQBundle, portable snapshots, and typed AO ERI reuse are implemented; see docs/persistence-format-v1.md for the actual on-disk contract.
Merged frontend: Make Nickel authoritative for TOML runfiles #101 makes Nickel authoritative for TOML; Add native Nickel batch execution #103 is open and implements native Nickel batch execution and versioned batch JSON output.
In progress in PR Add portable artifact CLI workflow #108 (PR C): portable CLI --artifact/--reuse, artifact inspect/artifact verify, and direct reruns from .rustiq bundles, including multi-calculation matching against the already implemented V1 format. Remaining after PR C: converged HF/partial SCF/DIIS persistence (PR D).
Documentation: Document the project using the MdBook #106 owns the mdBook site and practical guides. Link to the existing authoritative V1 disk format instead of introducing new archive layouts or duplicating storage contracts in the book.

The detailed PR A/B specifications below are retained as design history, not open implementation tasks. Current implementations and tests take precedence over those historical proposals.

Current limitation

RustiQ can now reuse deterministic AO electron-repulsion integrals through its local directory-backed cache, and the project has a typed, storage-independent persistence layer. What is still missing is the portable user-facing artifact workflow.

The persistence work must serve two different needs without conflating them:

a machine-local cache, used as an implementation optimization and safe to delete at any time;
a portable .rustiq scientific artifact, deliberately kept, copied, archived, shared, or supplied to a later RustiQ calculation so expensive compatible work does not need to be repeated.

The immediate use case remains avoiding AO ERI recomputation. The broader goal is to let a future MP2 calculation reuse a compatible converged HF solution and its prerequisite artifacts rather than rerunning the whole scientific pipeline.

Refined persistence model

1. Cache and `.rustiq` are different products

                         RustiQ persistence

                ┌──────────────┴──────────────┐
                │                             │
          active local cache            portable .rustiq
          implementation policy         user-visible artifact
                │                             │
         transparent reuse             durable / movable
         disposable                    inspectable
         machine-local                 reusable as explicit input

Active local cache

The active cache is:

machine-local;
disposable;
optimized for hot-path access;
directory-backed;
selected by application-directory policy;
enabled by calculation configuration;
not part of a calculation's portable deliverables.

Its current AO ERI layout remains:

<cache-root>/
└── eri/
    └── <scientific-fingerprint>/
        ├── manifest.json
        └── arrays/
            └── integrals/
                └── ao-eri.npy

A cache miss, invalid entry, stale entry, unsupported representation, checksum failure, or truncated payload falls back to recomputation.

Portable `.rustiq`

A .rustiq is a portable RustiQ scientific artifact bundle.

It is not merely an exported cache entry and should not be described generically as a "checkpoint". A checkpoint/restart state is only one possible class of artifact that a .rustiq may contain.

A .rustiq is intended to preserve three distinct levels when available:

the exact original input sources as optional opaque provenance;
the normalized user request;
the resolved scientific calculation;

plus typed scientific artifacts, so RustiQ can:

inspect the original input provenance when it was captured;
inspect what the user requested semantically and what RustiQ actually resolved/executed;
determine which persisted artifacts are scientifically compatible with a new request;
reuse compatible deterministic artifacts;
later reuse converged HF state for post-HF calculations;
safely recompute only missing or incompatible stages;
exchange the same state across Linux, macOS, Windows, Rust, and Python.

For V1, a portable .rustiq is normatively a ZIP archive with ZIP64 support.

Multi-calculation portable container from V1

The Nickel frontend POC in #96 validated the frontend-neutral boundary as a non-empty collection of calculations:

ResolvedInput
└── calculations: 1..N

The portable V1 format should preserve that cardinality directly rather than freezing a single-calculation manifest and requiring an immediate V2 when native Nickel batches land.

A portable .rustiq therefore represents one or more calculation snapshots. A normal TOML run is simply the common case where the bundle contains exactly one calculation.

Conceptually:

portable .rustiq
├── format / producer
├── shared optional source provenance
└── calculations [1..N]
    ├── calculation entry
    │   ├── normalized request snapshot
    │   ├── resolved calculation snapshot
    │   └── typed scientific artifacts
    └── ...

This is a container-level multiplicity only:

do not introduce a scientific BatchConfig, SingleOrBatch, or batch execution type into rustiq-core;
each calculation retains its own normalized request, resolved scientific state, and artifact set;
compatibility and reuse remain per calculation and per artifact;
there is no aggregate batch scientific identity;
the root manifest must not pretend that several independent calculations share one scientific_identity;
shared input sources such as one .ncl file may be stored once as opaque bundle-level provenance and referenced by several calculation entries;
payload deduplication between calculations is optional future storage optimization and must not affect the semantic model;
the machine-local cache remains calculation/artifact-oriented and does not become a batch container.

The collection must be non-empty. An archive with zero calculation entries is invalid as a portable calculation bundle.

A multi-calculation bundle remains a snapshot, not an append-only history. If a new request resolves to calculations D and E while reusing artifacts from a source bundle containing A, B, and C, the newly written snapshot represents D and E (with any compatible reused artifacts). It does not automatically accumulate A, B, C, D, and E.

The stable per-calculation identifier and physical member layout were established in merged #88 and are specified in docs/persistence-format-v1.md. Such identifiers must support unambiguous manifest references without becoming an aggregate scientific identity or making user-visible labels/source order part of artifact compatibility.

Artifact bundle, not a mutable calculation database

The design intentionally borrows the useful idea behind EZFIO in Quantum Package: keeping enough structured scientific state to inspect, resume, and reuse a calculation. EZFIO exposes module-defined values through generated interfaces/providers and acts as a shared calculation-state store.

RustiQ deliberately stops short of making .rustiq an EZFIO-like mutable key/value database used directly by numerical algorithms.

The distinction is:

EZFIO-style calculation state	RustiQ `.rustiq`
mutable working calculation store	versioned scientific artifact snapshot
modules read/write named stored values	scientific code consumes typed Rust domain objects
storage can participate directly in runtime state flow	archive is read through typed/versioned artifact APIs
work/state can evolve in place	updates create a complete new snapshot and publish atomically
generic stored variables/arrays	explicit artifact representations and compatibility contracts
runtime/work state may live beside durable data	scratch/runtime state stays outside the portable bundle
persistence layer is central to the live calculation state	local cache/runtime state and portable artifact remain separate products

A .rustiq is therefore a versioned scientific artifact snapshot:

scientific code consumes typed Rust domain objects, not arbitrary paths/keys inside the archive;
algorithms do not open the archive and mutate fields during normal execution;
persisted state is exposed through typed/versioned artifact APIs such as RustiQData;
runtime scratch/work files remain outside the portable bundle;
machine-local cache state remains a separate disposable product;
a saved bundle may be completed with additional compatible artifacts, but completion creates a new complete snapshot and publishes it atomically.

In particular, avoid APIs resembling a generic mutable scientific database:

data.get("hf/max_iter")
data.set("scf/density")
data.set("mp2/work/block-17")

Prefer typed scientific boundaries:

request()
calculation()
get::<AoEriArtifact>()
get::<HfSolutionArtifact>()
get::<HfRestartArtifact>()

This distinction is architectural: .rustiq may support convenient read-only reuse, selective artifact reuse, and same-path restart/completion workflows at the CLI while remaining snapshot-oriented internally.

User-facing CLI workflow

The CLI contract should make the distinction explicit.

Produce a portable artifact

Proposed workflow:

rustiq run water.toml --artifact water.rustiq

The ordinary text/JSON calculation output remains presentation output. The .rustiq file is a separate durable scientific artifact.

Reuse a portable artifact in another calculation

rustiq run water-mp2.toml --reuse water-hf.rustiq

The TOML remains the requested calculation. The .rustiq is a source of already-computed scientific state.

A reuse source is read-only by default. rustiq run ... --reuse state.rustiq and rustiq artifact inspect state.rustiq must require only read access to the archive. The source may live on a read-only filesystem, removable media, shared storage, or otherwise be non-writable.

RustiQ should validate the archive, compare each useful persisted artifact against the effective current calculation, reuse compatible artifacts, and compute what is missing.

Conceptually:

water-mp2.toml              water-hf.rustiq
      │                           │
      │ requested calculation     │ reusable scientific state
      └────────────┬──────────────┘
                   ▼
             compatibility
                planning
                   │
        ┌──────────┼───────────┐
        │          │           │
      AO ERI     HF solution   MP2
       reuse        reuse      missing
        │          │           │
        └──────────┴─────┬─────┘
                         ▼
                        MP2

The portable artifact must therefore participate in the calculation dependency graph, not be bolted onto the end of the CLI as an export-only feature.

Produce a new artifact while reusing an old one

rustiq run water-mp2.toml \
    --reuse water-hf.rustiq \
    --artifact water-mp2.rustiq

The new artifact may contain compatible reused state together with newly computed artifacts. The source archive is not mutated as part of normal reuse.

Read-only, copy-on-write, and same-path modes

The path combination defines the mutation contract:

--reuse source.rustiq
    read-only reuse
    source is never modified

--reuse source.rustiq --artifact result.rustiq
    read-only source + new output snapshot
    source is never modified

--reuse state.rustiq --artifact state.rustiq
    explicit transactional replacement
    old snapshot is read-only during computation
    replacement happens only after a complete validated new snapshot exists

The first two modes must work even when the source archive itself is not writable. Same-path completion additionally requires that the destination path/parent can be replaced; inability to publish the replacement is an explicit error and must not damage the existing archive.

Complete/restart using the same `.rustiq` path

The CLI should also allow the explicit reuse input and artifact output to be the same path:

rustiq run water-hf.toml \
    --reuse water.rustiq \
    --artifact water.rustiq

This is the convenience workflow for progressively completing one portable scientific bundle.

Example progression:

water.rustiq
  │
  │ initial run
  ▼
AO ERI
  │
  │ later HF run reuses AO ERI
  ▼
AO ERI + HF partial/restart state
  │
  │ resumed/converged HF
  ▼
AO ERI + converged HF solution
  │
  │ later MP2 run reuses ERI + HF
  ▼
AO ERI + converged HF solution + MP2 artifacts

Once partial HF persistence exists, the same mechanism also supports an HF checkpoint/restart workflow:

rustiq run water-hf.toml \
    --reuse water.rustiq \
    --artifact water.rustiq

An interrupted or deliberately stopped SCF may leave a valid partial HF restart artifact in the newly published bundle; a later invocation can consume that state and continue according to the restart contract.

Same path does not mean in-place ZIP mutation. The implementation must:

open and validate the existing source bundle;
plan reuse per artifact;
compute missing/new state;
write a complete replacement bundle to a sibling temporary file;
preserve/re-emit reusable compatible artifacts through the typed persistence layer;
validate/finalize the new archive;
atomically replace the original path only after success.

If any step before publication fails, the original .rustiq remains unchanged and usable.

The output snapshots (request.json and calculation.json) describe the latest requested/resolved calculation represented by the completed bundle. This is not a historical event log.

Same-path completion is intended for a scientifically compatible continuation/extension of the bundle (for example HF -> MP2, or partial HF -> converged HF). Known artifacts that are incompatible with the new resolved calculation must not be silently retained as usable state. A future explicit migration/history mechanism is out of scope.

The explicit two-flag form remains intentional:

--reuse state.rustiq --artifact state.rustiq

It makes read/replace intent unambiguous. Merely naming an existing --artifact path must not implicitly turn creation into mutation.

Inspect an artifact

Add a dedicated artifact command, for example:

rustiq artifact inspect water.rustiq

Representative output should expose captured source provenance when present, plus the normalized request and resolved calculation, without requiring generic ZIP extraction:

RustiQ artifact: water.rustiq

Sources
  ✓ calculation.toml
  ✓ molecule.xyz

Requested
  Coordinate unit  Angstrom
  HF method        Auto
  Basis            cc-pVTZ
  Post-HF          MP2

Resolved
  Coordinate unit  Bohr
  HF method        RHF
  Molecule         H2O
  Basis            effective resolved cc-pVTZ

Artifacts
  ✓ AO ERI
  ✓ converged RHF solution
  ✓ MP2 result

Producer
  RustiQ 0.x

PR C also includes rustiq artifact verify PATH because full payload verification has a different cost profile from structural inspection. It eagerly loads supported numerical payloads (currently compact AO ERIs), checking integrity and representation decoding; unsupported representations fail explicitly. Verification does not execute a calculation or establish scientific validity.

Direct rerun from a portable bundle

PR #108 includes direct execution from a .rustiq bundle:

rustiq run water.rustiq

RustiQ reconstructs the archived resolved calculation collection, reuses compatible stored AO ERIs, and performs a fresh HF calculation (and MP2 when requested). This mode does not restore HF solutions, partial SCF state, or DIIS history; those remain PR D work. Multi-calculation bundles run as batches, and conflicting command-line options are rejected.

Selective artifact reuse

Reuse planning is per artifact and must allow a caller/policy to intentionally ignore an otherwise compatible artifact.

Conceptually, an artifact can be classified as:

reused
ignored by policy
incompatible
missing
invalid/corrupt
recomputed

"Ignored" has a narrow meaning:

do not load or consume that artifact for the current calculation;
continue planning with the next permitted source (for example local cache) or computation;
report that it was intentionally ignored rather than incompatible;
do not interpret ignore as deletion from the source bundle.

This supports cases such as forcing a fresh SCF while still reusing AO ERIs, bypassing a persisted HF solution for comparison/debugging, or later choosing which post-HF intermediates may participate in a run.

Artifact reuse selection and artifact retention/deletion are separate policies. A future explicit prune/drop operation may remove artifacts from a newly written bundle, but an ignore-for-reuse option must never silently delete them from the source.

The exact CLI syntax for include/exclude/ignore filters is intentionally deferred until the end of the persistence roadmap, after the basic portable-artifact workflow and HF persistence/restart are complete and stable. The persistence and compatibility planner must nevertheless avoid assuming that every compatible known artifact must always be consumed.

This means the implementation order is:

portable request/resolved models
        ↓
.rustiq container
        ↓
basic CLI reuse/output
        ↓
HF solution + restart state
        ↓
advanced artifact selection / ignore filters
        ↓
optional prune/drop policies

Advanced selection is therefore a follow-up ergonomics/policy feature, not part of the critical path for V1 reuse or restart.

Reuse precedence

When several sources can provide the same artifact, use this order:

explicit --reuse .rustiq
        ↓
enabled local cache
        ↓
computation

An explicitly supplied .rustiq is user intent and should therefore be considered before the local cache.

Safety rules differ slightly from cache behavior:

unreadable/corrupt/unsupported archive container or manifest -> explicit error;
malformed or corrupted required payload -> explicit error;
artifact intentionally ignored by reuse policy -> do not load it; report it as ignored and continue to the next source/computation;
valid archive with an artifact incompatible with the current request -> do not use that artifact; report the mismatch and continue with other compatible artifacts/recomputation;
missing artifact -> compute it;
compatible and permitted artifact -> reuse it.

Compatibility is per artifact, not one all-or-nothing archive digest. An HF archive must remain useful to a later MP2 request.

Portable source, request, and calculation layers

A digest is sufficient to validate an artifact, but it is not sufficient to explain where the calculation came from, what the user requested semantically, and what RustiQ actually resolved and executed.

These are three different concepts and must not be conflated:

original source(s)                optional opaque provenance
TOML + XYZ / Nickel / ...
        │
        ▼
normalized request                frontend-independent semantics
        │
        ▼
scientific resolution
        │
        ▼
resolved calculation              canonical scientific state

A portable .rustiq may therefore contain all three layers:

study.rustiq
├── manifest.json
├── sources/                     # shared optional provenance, never required scientifically
│   ├── study.ncl
│   └── molecule.xyz
└── calculations/
    ├── <calculation-id-a>/
    │   ├── request.json
    │   ├── calculation.json
    │   └── arrays/
    │       └── ...
    └── <calculation-id-b>/
        ├── request.json
        ├── calculation.json
        └── arrays/
            └── ...

The exact internal source member names are not a scientific contract. A future frontend may preserve a Nix file, several source files, or no source files at all.

Optional source provenance — exact original input

When the calculation came from textual/filesystem input, RustiQ should be able to preserve the original source bytes without making them authoritative scientific state.

Examples include:

original TOML text, including comments, whitespace, quoting style, field order, and omitted defaults;
original XYZ text, including its comment line and coordinate precision;
original Nickel source (or another future frontend source);
original source names/paths as provenance metadata where useful.

Source provenance is deliberately opaque to rustiq-core and to compatibility logic:

it is optional;
it is not required to inspect or scientifically reuse a .rustiq;
it does not participate in scientific identity or artifact compatibility;
it must not be parsed to reconstruct request.json or calculation.json;
a .rustiq produced directly through the Rust API is valid with no sources/ entries;
source paths are provenance strings only and must never be trusted as archive extraction paths.

Preserving a source is different from reproducing the original filesystem layout. Archive members must still obey the safe portable-path rules; any original filename/path retained for provenance must be metadata, not an extraction instruction.

`request.json` — normalized requested calculation

request.json records the effective request after frontend parsing/default application, but before scientific resolution/canonicalization.

It is the frontend-independent semantic representation of what was requested. Unlike the optional source layer, it deliberately removes syntax-only distinctions.

Examples of information that belongs here:

requested coordinate unit, e.g. angstrom;
requested HF method, including auto;
effective/defaulted HF options as requested before method resolution;
molecular charge and multiplicity;
requested post-HF method/settings;
a portable user-facing basis selector/label when one exists;
other frontend-independent options that affect what calculation was requested.

For example:

request.json
  coordinate_unit = angstrom
  hf.method       = auto
  basis           = sto-3g

The normalized request must not depend on:

comments or whitespace;
TOML/Nickel source ordering or syntax;
source spans;
quoting style;
original filesystem layout;
the distinction between a default omitted by the user and the same default written explicitly, once frontend defaults have been applied.

Therefore two syntactically different source files may legitimately produce the same request.json.

A requested basis name/label is provenance/presentation information, not sufficient scientific basis identity. Custom/local basis content is represented authoritatively by the resolved basis in calculation.json.

`calculation.json` — resolved scientific calculation

calculation.json records the canonical scientific state after resolution.

Examples:

coordinates canonicalized to Bohr;
HF Auto resolved to explicit RHF or UHF;
canonical molecule, charge, multiplicity, and electron state;
resolved basis contents and ordering, not only a basis name/path;
scientifically relevant effective HF settings;
requested/effective post-HF settings as required to interpret artifacts;
representation/algorithm versions affecting persisted results.

For example:

calculation.json
  coordinate_unit = bohr
  hf.method       = rhf
  basis           = <effective resolved AO data>

The resolved snapshot must not contain machine-local policy such as cache directories, terminal/reporting options, source spans, or non-scientific filesystem paths.

Presentation and reconstruction

The CLI/application layer may expose both exact provenance and canonical semantic renderings:

source provenance ─────────────► exact source display / export when available

normalized request ────────────► canonical TOML / future Nix / XYZ presentation

resolved calculation ─────────► resolved scientific presentation

This distinction preserves information without coupling persistence to one frontend.

For example:

source
  method = Auto
  units  = Angstrom
  comments / original formatting / path spelling preserved

request.json
  method = auto
  units  = angstrom
  defaults explicit semantically

calculation.json
  method = rhf
  units  = bohr

A canonical representation generated from request.json is guaranteed to be semantically equivalent, not byte-for-byte identical to the source. If the exact original source was captured, it can be displayed/exported separately.

Both request.json and calculation.json have explicit independent schema versions. Their wire schemas are independent of Rust struct layouts and the RustiQ package version.

The manifest should carry versioned, size- and SHA-256-validated references to the semantic snapshots and any captured source payloads. Scientific artifact compatibility is based on resolved scientific state, never on source bytes or source formatting.

Logical persistence format

The existing storage-independent V1 work remains valid.

Manifest

manifest.json is the bundle-level index. For portable V1 it contains a non-empty collection of calculation entries plus bundle metadata/shared provenance references.

Each calculation entry owns references to its versioned request snapshot, resolved calculation snapshot, and typed scientific artifacts. Scientific identities/compatibility metadata belong to the relevant calculation/artifact scope; there is no single root scientific identity for the whole bundle.

The common artifact envelope contains:

logical path;
byte size;
representation identifier;
SHA-256 digest;
representation-specific attributes.

Known representations decode attributes into strict typed metadata. Unknown representations retain their raw JSON metadata for forward-compatible inspection but cannot be used scientifically until supported.

Numerical arrays

Dense numerical arrays and packed tensors use NPY unless a future representation documents another encoding.

Current stable AO ERI path:

arrays/integrals/ao-eri.npy

Current representation:

rustiq-compact-eri-v1

NPY remains authoritative for dtype, endianness, shape, and storage order. Semantic indexing belongs to the representation contract, not the NPY header.

Scientific identity

Artifact identity is based on effective scientific inputs, never cache names, archive names, timestamps, compression, or Rust implementation details.

The existing scientific-identity-v1 AO ERI identity remains based on canonical effective geometry, resolved AO/basis data, ERI representation/computation version, Schwarz screening state, and other inputs that affect the numerical artifact.

The RustiQ package version is provenance, not automatic invalidation.

Integrity

Every persisted scientific payload has an independent SHA-256 digest.

ZIP CRCs may provide an additional transport/container check but are not the scientific integrity contract.

Portable container V1

A .rustiq V1 file MUST be a valid ZIP archive with ZIP64 support when required.

ZIP remains the V1 decision because the primary workload is seekable artifact reuse rather than pipe-oriented streaming:

direct lookup of named members through the central directory;
per-entry compression;
strong Windows/macOS/Linux tooling;
Python standard-library support through zipfile;
ZIP64 for large ERI/post-HF payloads.

Baseline policy:

large numerical arrays: Stored when fast access/low CPU is preferred;
metadata/small payloads: Deflated when beneficial;
compression method/level never participates in scientific identity.

The format is not NPZ and V1 does not add TAR, auto-detection, or a public generic archive abstraction.

The internal persistence machinery may dispatch between the existing folder storage and the ZIP implementation, but scientific/domain APIs must continue to expose typed artifacts rather than ZIP/filesystem concepts.

Persistence compatibility classes

Do not use one vague "checkpoint compatibility" promise.

Deterministic reusable artifacts

Examples:

overlap;
kinetic;
nuclear-attraction;
AO ERIs.

Compatibility is based on effective scientific identity plus representation/computation versions.

AO ERI reuse is implemented first.

Converged HF solution

Portable converged HF state should eventually include enough typed data to let a compatible post-HF request consume HF without running SCF again, including as appropriate:

resolved RHF/UHF method;
occupation counts;
MO coefficients;
orbital energies;
density;
SCF energy/diagnostics;
spin-resolved data;
prerequisite integral artifacts needed by the consumer.

This is the next important reuse target after AO ERIs because it enables the intended HF -> MP2 workflow.

Partial HF state and SCF restart

A portable .rustiq may contain a partial HF/SCF state when the calculation has not converged yet. This is distinct from a converged HF solution and should be useful for resuming interrupted or deliberately stopped calculations.

The restart contract must work with or without DIIS.

The restartable base state should contain the scientifically meaningful state required to continue the next SCF iteration, as applicable to RHF/UHF:

completed iteration number;
current density matrix/matrices;
current Fock matrix/matrices when required by the continuation contract;
current orbital coefficients and orbital energies when required;
current electronic/total energy and convergence diagnostics;
resolved RHF/UHF method, occupations, spin state, and compatibility metadata;
the integral/scientific identities on which the state depends;
convergence-controller state that affects the continuation.

When DIIS is disabled, this base state is sufficient for the defined non-DIIS restart path.

When DIIS is enabled, the restart representation may additionally persist the DIIS subspace/state needed to continue the same acceleration history, including the required Fock/error or residual vectors and any versioned DIIS-controller metadata.

A reader must not silently treat missing or incompatible DIIS state as an exact DIIS continuation. If the base HF state is otherwise scientifically compatible, RustiQ may explicitly downgrade to a documented restart/warm-start path that rebuilds DIIS history; that behavior must be reported to the user.

The persistence contract must therefore distinguish at least:

converged HF solution
partial HF restart state
partial HF restart state + DIIS continuation state

Restart compatibility is stricter than using a density/orbital state merely as an initial guess. Exact bit-for-bit trajectory reproduction is not promised unless a future representation explicitly defines and tests such a guarantee.

Post-HF artifacts

MP2 and later methods receive their own versioned artifact representations rather than serializing Rust implementation structures.

Reliability, security, and publication

Portable archives are untrusted input.

Readers must:

bound manifest/header/member sizes;
reject duplicate required logical paths;
reject absolute paths, parent traversal, backslashes/ambiguous platform paths, links, and special entries;
validate representation-specific metadata;
validate NPY dtype/shape/order/count;
reject object/pickle arrays;
verify payload size and SHA-256 before scientific use;
read known members directly rather than generically extracting the archive.

Portable archives must be published atomically:

existing/source .rustiq (optional)
      │ read/validate/reuse
      ▼
temporary replacement .rustiq
      │
      ├── write optional source provenance
      ├── write normalized request snapshot
      ├── write resolved calculation snapshot
      ├── copy/re-emit compatible reusable artifacts
      ├── write newly computed artifacts
      ├── compute/verify digests
      ├── write/finalize manifest
      ├── finalize ZIP/ZIP64
      ├── validate + flush/sync as appropriate
      ▼
atomic publish / atomic replacement

Creation of a new path may remain no-clobber by default. Explicit same-path --reuse PATH --artifact PATH is the narrowly defined case where atomic replacement is required. No code should rewrite ZIP members in place.

Where practical, irrelevant ZIP metadata such as timestamps and permissions should be normalized for reproducibility.

Implementation status and PR plan

The original four-PR plan is obsolete. PR A (#89) and PR B (#88) have now merged; the outstanding work is PR C (portable-artifact CLI) and PR D (HF solution/restart persistence). The detailed completed sections below remain as historical design records.

Completed

Centralize SCF integral production #81 — Centralize SCF integral production
- introduced ScfIntegrals / IntegralBuilder;
- moved RHF/UHF behind one integral-production boundary;
- established the insertion point for persistent integral reuse.
Add scientific persistence primitives #82 — Add scientific persistence primitives
- added manifest/schema primitives;
- added canonical AO ERI scientific identity;
- added NPY read/write and SHA-256 integrity;
- added Rust/Python NumPy interoperability.
Centralize application directories #86 — Centralize application directories
- moved machine-local directory policy into the CLI/application layer;
- established the correct place for cache path resolution.
Add directory-backed integral cache #84 — Add directory-backed integral cache
- added validated AO ERI reuse;
- added atomic publication and corruption fallback;
- added cache CLI management and human-readable aliases;
- kept the cache distinct from portable archives.
Add typed scientific persistence artifacts #87 — Add typed scientific persistence artifacts
- added RustiQData as the typed persistence facade;
- made manifest/storage/NPY orchestration internal;
- routed the directory-backed cache through the common persistence machinery;
- added internal folder storage plus a concrete dispatch point where ZIP/ZIP64 can be added without exposing a public generic backend;
- refined typed persistence errors and path validation.

Implementation sequence and merge discipline

The original dependency plan was (PR A and PR B are complete; continue with PR C and PR D in that order):

main
  │
  ├── PR A: frontend-independent resolved calculation presentation
  │      ↓ merge
  ├── PR B: portable .rustiq V1
  │      ↓ merge
  ├── PR C: CLI artifact workflow
  │      ↓ merge
  └── PR D: HF converged + partial/restart state
         ↓ merge
       main

Each PR targets main and must leave main in a complete, tested, useful state. Do not create a long-lived integration branch.

If work on a later PR must start before its prerequisite is merged, it may be temporarily stacked on the prerequisite branch. Once the prerequisite merges, rebase the dependent PR onto main.

Apply YAGNI throughout:

introduce an abstraction only when the current PR has a concrete consumer for it;
do not add placeholder variants, traits, backends, artifact kinds, commands, or configuration for hypothetical future work;
keep future possibilities in this issue/documentation rather than dormant production code;
extend the existing RustiQData and internal storage boundaries instead of creating parallel persistence paths;
do not refactor unrelated code unless it is required to establish the current PR's contract.

Before starting each PR, use the repository's current main and follow AGENTS.md. The requirements below define behavior; implementation details should reuse existing repository conventions where they already satisfy that behavior.

Completed PR A — decouple resolved calculation presentation from input formats (#89 merged)

Suggested branch:

refactor/decouple-calculation-presentation

Goal: establish two frontend-independent semantic views before persistence is introduced while keeping original input source text as an optional, separate provenance layer:

optional source provenance representing the exact input bytes when a frontend has them;
a normalized request view representing what the user asked for after frontend defaults are applied but before scientific resolution;
a resolved calculation view representing what RustiQ actually resolved and will execute.

The source layer preserves information for display/export/diagnostics, while request/resolved views remain the authoritative semantic inputs for persistence and compatibility.

Today the text-mode CLI prints:

ParsedRunFile::formatted_toml, produced by the TOML frontend with defaults inserted;
Geometry::to_string(), produced after loading the referenced XYZ file.

That means the presentation is coupled to the current TOML + XYZ frontend. This becomes a problem as soon as another input frontend is added, for example a Nix representation, and it also prevents a future .rustiq from reproducing the same effective calculation presentation without retaining TOML/XYZ source files.

The target flow is:

TOML + XYZ ─┐──────────────► optional exact source provenance
            │
future Nix ─┼──► normalized request ──► scientific resolution ──► resolved calculation
            │          │                                          │
other input ┘          ├──► canonical request presentation         ├──► canonical resolved presentation
                       │                                          │
                       └──► later request.json                     └──► later calculation.json

Both semantic views are format-independent. TOML, XYZ, Nix, or any future syntax are frontend adapters/renderers around them; they are not the scientific persistence contract.

The normalized request must preserve semantic choices that disappear during resolution, such as HF Auto and requested coordinate units such as Angstrom. The resolved view stores their scientific results, such as explicit RHF/UHF and canonical Bohr coordinates.

Canonical renderers produce a semantically equivalent representation from the normalized/resolved views. Separately, when original source bytes are available, the CLI may preserve/display them exactly. Comments, source ordering, omitted-vs-explicit defaults, source spans, and paths such as ../molecule.xyz remain frontend/source provenance and must not leak into the semantic request/resolved models.

Requirements:

identify the current frontend-independent configuration state after TOML parsing/default application and expose/shape it as the normalized request view without serializing TOML structs as a long-term domain contract;
preserve request semantics that disappear during scientific resolution, including:
- HfMethod::Auto;
- requested molecule coordinate units such as Angstrom;
- effective/defaulted HF/MP2 options;
- charge and multiplicity;
- portable basis label/selector provenance where available;
identify the existing resolved scientific state after CalculationBuilder::prepare() and reuse it as the resolved calculation view rather than introducing a duplicate scientific model;
ensure the resolved view exposes:
- atoms and coordinates in canonical Bohr;
- molecular charge and multiplicity;
- explicitly resolved RHF/UHF method;
- effective/resolved basis information;
- effective HF settings and requested/effective MP2 settings needed for presentation/persistence;
preserve exact TOML/XYZ source text as optional frontend provenance when available, without placing it in CalculationRequest or the resolved scientific view;
move canonical request/calculation presentation away from ParsedRunFile::formatted_toml;
provide canonical geometry presentation independent of the original XYZ source path/content;
allow text-mode rustiq run to show exact source provenance when available and semantic requested/resolved presentation from the corresponding views;
preserve source-aware diagnostics: original TOML spans/source text remain available for parse/configuration errors, but are not authoritative scientific state;
define CLI-side rendering adapters for the currently supported user-facing representations rather than putting TOML/XYZ serialization into rustiq-core;
keep cache paths, terminal/UI state, source paths, source spans, comments, formatting, and frontend syntax out of both semantic models;
document that future frontends such as Nix must adapt into the same normalized request/scientific resolution pipeline while optionally contributing opaque source provenance;
add tests showing:
- omitted defaults and explicitly written equivalent defaults normalize identically;
- HF Auto remains visible in the normalized request but resolves to explicit RHF/UHF;
- Angstrom input remains visible as requested units but resolved coordinates are Bohr;
- canonical rendering no longer requires the original geometry file after successful resolution;
- exact source display, when available, preserves comments/formatting/coordinate text without affecting semantic equality.

The purpose is not to make the core depend on TOML, XYZ, or Nix. The core exposes scientific meaning; the CLI/application layer decides how to render that meaning.

PR A implementation steps

trace the current RunFile -> rustiq_core::config -> CalculationBuilder/PreparedCalculation path and identify the smallest reusable normalized-request and resolved-calculation views;
add/adjust read-only accessors or dedicated semantic views only where existing APIs are insufficient;
make Auto versus resolved RHF/UHF and requested Angstrom versus resolved Bohr explicit in tests and presentation;
introduce CLI-side canonical renderers for normalized request/effective calculation configuration and geometry;
switch successful text-mode run presentation to those renderers;
keep parser/source structures solely for input parsing and source diagnostics;
add equivalence tests for omitted defaults versus explicitly written defaults;
add tests proving rendering does not require the original geometry file after successful preparation;
document the frontend boundary and the requirement that persistence consumes semantic request/resolved views, not frontend syntax.

PR A out of scope

writing or reading .rustiq;
defining ZIP/container behavior;
persisting calculation.json;
adding a Nix parser now;
reconstructing original source bytes when they were not captured;
treating comments, formatting, source paths, or source spans as scientific state;
HF result/restart persistence.

PR A merge gate

PR A is done when:

text-mode execution still shows an effective configuration with defaults and a geometry representation;
exact source provenance may still be displayed when captured, but canonical requested/resolved representations do not depend on it;
canonical representations are generated from semantic state rather than ParsedRunFile::formatted_toml or the original geometry source;
equivalent calculations with implicit versus explicit defaults normalize/render canonically equivalently;
source-aware TOML diagnostics still point to the original input;
the normalized request preserves semantic pre-resolution choices such as HF Auto and requested coordinate units;
the resolved scientific view contains explicit RHF/UHF and canonical Bohr coordinates;
both views are independent of TOML/XYZ and can later be serialized by .rustiq;
no persistence/container code is introduced.

Completed PR B — implement portable `.rustiq` V1 (#88 merged)

Suggested branch:

feature/add-rustiq-container

Goal: deliver a complete storage-level portable artifact capability in main: RustiQ can create and open a self-describing .rustiq through its Rust persistence API.

Historical planning note: #88 is merged, after #89 established the frontend-independent presentation boundary. No rebase or additional V1 schema redesign is pending under PR B.

This PR builds on PR A's optional source-provenance layer plus frontend-independent normalized-request and resolved-calculation views. It defines:

optional sources/ payloads as opaque provenance;
request.json as the versioned wire representation of the normalized request;
calculation.json as the versioned wire representation of the resolved scientific calculation.

Both are packaged with scientific artifacts in ZIP/ZIP64. Neither wire schema becomes the in-memory domain model, and the domain models must not depend on TOML, XYZ, Nix, or ZIP.

Requirements:

define and document independent versioned request.json and calculation.json schemas;
construct request.json from the normalized request view and calculation.json from resolved scientific inputs, never from source TOML/XYZ syntax;
include canonical molecule and resolved basis information;
include scientifically relevant method/settings needed to interpret persisted artifacts;
exclude cache/output/path/UI policy;
keep the snapshot schema independent from Rust struct serialization;
add ZIP/ZIP64 support;
add internal ZIP storage alongside the existing folder storage;
write/read manifest.json, optional captured sources/, request.json, calculation.json, and currently supported typed artifacts;
preserve lazy artifact loading where practical;
implement the defensive-reading rules above;
implement atomic archive publication;
normalize irrelevant ZIP metadata where practical;
add Rust write -> Rust read golden tests;
add Rust write -> Python read and Python fixture -> Rust read tests;
cover corruption, truncation, duplicate members, unsafe paths, unsupported versions, NPY portability, and ZIP64 scenarios;
keep ZIP types and physical-storage details out of the public scientific API.

After this PR, the Rust API must have a real use for every new persistence concept: a self-describing .rustiq can be written and read. The CLI does not need to expose it yet.

The first container implementation only needs to persist artifact types RustiQ actually supports at that point. It does not need speculative SCF restart or post-HF representations.

PR B implementation steps

consume the resolved scientific view established by PR A and inspect the existing RustiQData, manifest, artifact identity, NPY, and internal folder-storage boundaries from Add scientific persistence primitives #82/Add typed scientific persistence artifacts #87;
define normalized-request and resolved-calculation snapshot schemas plus golden JSON fixtures;
implement snapshot construction from the PR A semantic views;
implement internal ZIP/ZIP64 member access without exposing a generic storage trait;
route RustiQData portable open/write operations through that concrete ZIP implementation;
implement atomic archive publication and defensive member validation;
add Rust round-trip/corruption/security tests;
add Python/NumPy interoperability fixtures/tests;
update persistence-format documentation to describe the physical .rustiq V1 contract.

PR B out of scope

CLI flags/commands;
HF solution persistence;
partial SCF/restart state;
MP2 artifacts;
ZIP/TAR selection or archive auto-detection;
public generic storage/backend APIs;
cache import/export.

PR B merge gate

PR B is done when:

the Rust persistence API can atomically write a valid self-describing .rustiq containing a non-empty calculations[1..N] collection;
a one-calculation bundle round-trips through exactly the same collection model as a multi-calculation bundle;
every calculation entry has independent versioned request/resolved snapshots and typed artifact references;
optional source provenance can be shared at bundle scope without entering scientific identity or compatibility;
no aggregate batch scientific identity is introduced;
compatibility/integrity validation remains scoped to the relevant calculation/artifact;
an empty portable calculation collection is rejected;
the same API can reopen the bundle and lazily/readably recover supported artifacts for each calculation entry;
malformed/unsafe archives covered by the V1 threat model are rejected;
Rust/Python fixtures demonstrate documented JSON/ZIP/NPY interoperability for both single- and multi-calculation bundles;
existing folder-backed cache behavior and calculations remain unchanged;
no CLI behavior depends on unmerged future work.

Remaining PR C — integrate portable artifacts into the CLI

Suggested branch:

feature/add-artifact-cli

Goal: expose the already functional .rustiq persistence capability to users and make it a first-class reusable calculation artifact rather than an export-only file.

Initial CLI contract:

rustiq run calculation.toml --artifact result.rustiq
rustiq run study.ncl --artifact study.rustiq
rustiq run calculation.toml --reuse previous.rustiq
rustiq run calculation.toml --reuse previous.rustiq --artifact result.rustiq
rustiq run calculation.toml --reuse state.rustiq --artifact state.rustiq
rustiq artifact inspect result.rustiq
rustiq artifact verify result.rustiq
rustiq run result.rustiq

Requirements:

explicit --reuse PATH input;
explicit --artifact PATH output;
reuse precedence for each requested calculation/artifact: explicit archive -> enabled local cache -> computation;
open explicit reuse sources read-only and never require write access unless the same path is also explicitly selected as --artifact;
per-artifact compatibility/selection planning that can represent an intentionally ignored artifact independently from incompatibility, without exposing advanced user-facing include/exclude controls yet;
clear reporting of reused, ignored, incompatible, missing, and recomputed artifacts;
no silent use of invalid state;
artifact inspect shows both normalized requested settings and resolved scientific settings from the versioned snapshots, rather than reconstructing them from source syntax or ZIP extraction;
artifact verify eagerly loads and validates supported numerical payloads (currently compact AO ERIs), checks integrity and scientific representation, and fails explicitly for unsupported representations;
keep --format text|json independent from scientific artifact output;
source archive remains unchanged during normal reuse to a different output path;
allow --reuse PATH --artifact PATH when both paths identify the same bundle;
implement same-path completion as validated temporary rewrite + atomic replacement, never in-place ZIP mutation;
newly written/replaced archive may contain both reused and newly computed compatible artifacts;
write the latest non-empty resolved calculation collection to the output bundle, with independent request/resolved snapshots per calculation while retaining only scientifically compatible reusable known artifacts for the corresponding calculation entries;
never select calculations[0] implicitly: matching/reuse for multi-calculation bundles must be explicit and compatibility-driven per requested calculation;
allow rustiq run FILE.rustiq to rerun archived resolved calculation(s), reusing compatible AO ERIs from the bundle, and reject conflicting reuse/output/cache/basis-download options.

After this PR, the initial .rustiq workflow is user-accessible end to end. The PR should mostly orchestrate persistence APIs established by PR B rather than introduce another storage abstraction.

PR #108 also supports direct rustiq run FILE.rustiq. This reconstructs and reruns the archived resolved calculation(s), reusing compatible stored AO ERIs where available. It is a fresh HF/MP2 calculation; PR C does not persist or restore HF/SCF/DIIS state. Conflicting --reuse, --artifact, cache-directory, and basis-download options are rejected for this mode.

PR C implementation steps

add --reuse PATH and --artifact PATH to the existing run command without changing --format;
open and validate explicit reuse archives without write access before calculation orchestration;
implement per-artifact planning with the precedence explicit .rustiq -> enabled local cache -> computation, with a distinct ignored/policy-filtered state even if the first CLI version does not yet expose include/exclude flags;
feed compatible persisted artifacts through existing scientific boundaries rather than bypassing them;
atomically write the requested output archive from reused and newly computed compatible artifacts;
detect the explicit same-path --reuse/--artifact case and route it through a sibling temporary archive followed by atomic replacement after final validation;
ensure failure leaves the original same-path bundle byte-for-byte available/usable and never partially rewritten;
add the artifact inspect command using manifest/snapshot APIs, not generic ZIP extraction;
add artifact verify to eagerly validate supported numerical payloads, including checksum and representation decoding, with explicit errors for unsupported representations;
add text/JSON reporting for reuse decisions consistent with existing CLI output conventions;
add direct .rustiq rerun support using archived resolved snapshots and compatible stored AO ERIs, for both single- and multi-calculation bundles;
add text/JSON reporting for reuse decisions consistent with existing CLI output conventions;
add end-to-end CLI tests for reuse, incompatibility, corruption, cache fallback, computation fallback, new artifact creation, verification, single-/multi-calculation archive reruns, and successful/failed same-path completion.

PR C out of scope

HF/SCF state persistence;
DIIS persistence;
MP2 persistence;
local-cache/archive import/export;
user-facing artifact include/exclude/ignore CLI syntax;
advanced artifact-selection policies;
prune/delete semantics for artifacts inside a bundle;
new storage abstractions.

PR C merge gate

PR C is done when:

a normal TOML run can produce a portable .rustiq independently of presentation format;
another TOML run can explicitly reuse compatible artifacts from that archive even when the source archive is read-only;
reuse planning has a distinct representation for intentionally ignored artifacts and does not conflate ignore with incompatibility/deletion;
the same archive path can be used explicitly as both --reuse and --artifact, producing a validated atomic replacement rather than in-place mutation;
a failed same-path completion leaves the previous bundle intact;
explicit archive, local cache, and computation precedence is tested end to end;
invalid explicit archives fail explicitly while missing/incompatible individual artifacts safely fall through according to the contract;
rustiq artifact inspect reports the portable scientific contents without extracting arbitrary files;
rustiq artifact verify validates supported stored numerical payloads and clearly rejects unsupported representations;
rustiq run FILE.rustiq reruns archived calculations and reuses compatible AO ERIs without claiming HF/SCF/DIIS restart;
running without artifact options preserves existing behavior.

Remaining PR D — persist HF solutions and restartable partial SCF state

Suggested branch:

feature/persist-hf-state

Goal: extend the working portable-artifact pipeline with HF state in both useful forms:

a converged RHF/UHF solution that compatible post-HF methods such as MP2 can consume without rerunning SCF;
a partial restartable RHF/UHF state that can continue an interrupted/non-converged SCF, with or without DIIS.

Requirements:

define stable typed representations for converged RHF/UHF solutions;
define a stable typed representation for partial SCF restart state;
persist the minimum scientifically meaningful base state needed to continue SCF without DIIS;
when DIIS is active, persist the versioned DIIS continuation state required to preserve its usable history;
make DIIS state optional at the persistence-model level so restart is not coupled to DIIS being enabled;
distinguish converged, partial-restart, and partial-restart-with-DIIS state explicitly rather than inferring the mode from missing arrays;
version compatibility contracts independently of Rust structs;
validate dimensions, spin/method metadata, iteration/convergence metadata, identities, sizes, and digests;
let an MP2 request consume a compatible converged HF solution without rerunning SCF;
let an HF request resume a compatible partial state from the next defined SCF iteration;
support restart when DIIS was disabled;
support continuation of DIIS history when compatible DIIS state is present;
never silently claim exact DIIS continuation when DIIS state is absent or incompatible; an explicit fallback that rebuilds DIIS/warm-starts may be supported and must be reported;
recompute/restart safely when persisted HF state is missing or incompatible;
cover RHF and UHF;
verify numerical equivalence of fresh versus persisted converged HF -> MP2 flows;
test interrupted SCF -> persisted partial state -> resumed SCF for DIIS disabled and enabled;
expose all HF state through the same RustiQData facade rather than a CLI-specific path;
exercise the same-path completion workflow so an archive containing ERIs can be replaced by a new snapshot containing the same compatible ERIs plus partial/converged HF state;
exercise restart from a partial HF artifact in the same bundle path, followed by atomic replacement with the advanced/converged state.

After this PR, .rustiq supports both important HF workflows:

converged HF .rustiq ───────────────► MP2
                                         (no SCF rerun)

partial HF .rustiq ─────────────────► resumed SCF
partial HF + DIIS state .rustiq ────► resumed SCF + DIIS history

The guarantee is a scientifically defined restart/continuation contract. Bit-for-bit reproduction of the uninterrupted numerical trajectory is not required unless a representation explicitly promises it.

PR D implementation steps

Implement HF persistence incrementally inside this PR so each layer has its own tests before the next is connected:

D1 — converged HF solution

identify the minimum converged RHF/UHF state required by existing post-HF consumers;
define typed/versioned converged-HF representations and compatibility identity;
add RustiQData read/write support and round-trip validation;
connect compatible converged HF reuse to MP2;
prove that a reused compatible HF solution avoids rerunning SCF while preserving numerical results.

D2 — partial SCF restart without DIIS

identify the minimum state required by the existing RHF/UHF iteration loop to continue from the next defined iteration;
define a typed/versioned partial-HF representation distinct from converged HF;
persist iteration/convergence metadata and required matrices/vectors;
restore that state into the existing SCF machinery without a parallel solver path;
test interrupted -> persisted -> resumed calculations with DIIS disabled.

D3 — DIIS continuation state

identify the actual DIIS state used by the current implementation; persist only state needed for continuation;
add optional, explicitly versioned DIIS continuation data to partial HF persistence;
restore compatible DIIS history when present;
distinguish/report restart with preserved DIIS history from restart that rebuilds DIIS history;
test interrupted -> persisted -> resumed calculations with DIIS enabled, including missing/incompatible DIIS-state behavior.

D4 — end-to-end integration

exercise RHF and UHF for converged and partial paths;
exercise progressive same-bundle completion: ERI-only -> partial HF -> converged HF -> post-HF consumer;
verify that a partial HF checkpoint can be read from and replaced at the same .rustiq path without risking the prior valid snapshot on failure;
verify .rustiq inspection/reporting for HF artifact states;
verify corruption and compatibility fallbacks at each HF artifact level;
update persistence documentation and golden fixtures.

PR D out of scope

bit-for-bit reproduction of an uninterrupted SCF trajectory unless naturally guaranteed and explicitly documented;
broader weak-compatibility warm-start heuristics;
persisted MP2/post-HF intermediates/results;
new SCF algorithms introduced only to support persistence;
generic solver-state/checkpoint frameworks;
storage backends unrelated to the HF use case.

PR D merge gate

PR D is done when:

compatible converged RHF/UHF state can be persisted and reused by MP2 without rerunning SCF;
compatible partial RHF/UHF state can resume the existing SCF implementation with DIIS disabled;
compatible partial RHF/UHF + DIIS state can resume while preserving the defined DIIS continuation state;
absent/incompatible DIIS continuation data follows an explicit tested fallback/error contract and is never reported as preserved-DIIS continuation;
fresh and reuse/restart paths meet the documented numerical-equivalence expectations;
persistence-disabled calculations remain scientifically unchanged.

Follow-ups, not required for the first usable `.rustiq`

broader warm-start policies that intentionally accept weaker compatibility than the HF restart contract;
bit-for-bit/exact trajectory restart guarantees beyond the defined HF continuation contract;
persisted MP2/post-HF intermediates/results;
direct rustiq run file.rustiq when a concrete self-contained execution workflow is specified;
import/export between the local cache and portable archives if a concrete workflow justifies it;
alternative physical storage/container formats only if a real requirement appears;
after the core persistence + HF restart roadmap is complete: explicit artifact include/exclude/ignore filters for reuse planning;
after selection semantics are proven useful: explicit prune/drop commands or output-retention policies, separate from reuse-ignore semantics.

Validation and interoperability

Golden fixtures should cover, as relevant to implemented artifact types:

normalized request -> request.json -> normalized request semantic round trip;
resolved calculation -> calculation.json -> resolved calculation semantic round trip;
HF Auto request -> explicit RHF/UHF resolution preservation;
Angstrom request -> canonical Bohr resolution preservation;
Rust write -> Rust read;
Rust write -> Python read;
Python fixture -> Rust read;
matrix shape/order/value round trips;
compact ERI round trips;
little-/big-endian NPY portability;
RHF/UHF layouts;
checksum corruption rejection;
truncated member/archive rejection;
scientific-identity mismatch;
unknown representation/schema inspection versus scientific-use rejection;
ZIP64;
deterministic/reproducible ZIP metadata where implemented;
explicit archive reuse versus local-cache fallback;
HF -> MP2 reuse without rerunning compatible upstream work.

Python interoperability should continue to require only standard-library modules plus NumPy whenever possible.

Updated acceptance criteria

Non-goals for V1

a public generic storage/backend trait;
ZIP/TAR runtime selection;
archive auto-detection;
serializing Rust structs as the persistence contract;
requiring original TOML/XYZ/Nickel source bytes for scientific interpretation, compatibility, reuse, or restart;
reconstructing exact original source bytes or filesystem layout when source provenance was not captured;
turning .rustiq into an EZFIO-like generic mutable calculation database or runtime scratch store;
maintaining an append-only history/event log of every completion in V1;
introducing a scientific BatchConfig merely because the portable container can hold several calculations;
mutating ZIP members in place;
making the local cache itself portable;
requiring bit-for-bit trajectory reproduction for SCF restart.

Engineering: Define portable, versioned persistence for scientific state #79

Description

Current status (8 October 2026)

Current limitation

Refined persistence model

1. Cache and .rustiq are different products

Active local cache

Portable .rustiq

Multi-calculation portable container from V1

Artifact bundle, not a mutable calculation database

User-facing CLI workflow

Produce a portable artifact

Reuse a portable artifact in another calculation

Produce a new artifact while reusing an old one

Read-only, copy-on-write, and same-path modes

Complete/restart using the same .rustiq path

Inspect an artifact

Direct rerun from a portable bundle

Selective artifact reuse

Reuse precedence

Portable source, request, and calculation layers

Optional source provenance — exact original input

request.json — normalized requested calculation

calculation.json — resolved scientific calculation

Presentation and reconstruction

Logical persistence format

Manifest

Numerical arrays

Scientific identity

Integrity

Portable container V1

Persistence compatibility classes

Deterministic reusable artifacts

Converged HF solution

Partial HF state and SCF restart

Post-HF artifacts

Reliability, security, and publication

Implementation status and PR plan

Completed

Implementation sequence and merge discipline

Completed PR A — decouple resolved calculation presentation from input formats (#89 merged)

PR A implementation steps

PR A out of scope

PR A merge gate

Completed PR B — implement portable .rustiq V1 (#88 merged)

PR B implementation steps

PR B out of scope

PR B merge gate

Remaining PR C — integrate portable artifacts into the CLI

PR C implementation steps

PR C out of scope

PR C merge gate

Remaining PR D — persist HF solutions and restartable partial SCF state

PR D implementation steps

PR D out of scope

PR D merge gate

Follow-ups, not required for the first usable .rustiq

Validation and interoperability

Updated acceptance criteria

Non-goals for V1

Checklist

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. Cache and `.rustiq` are different products

Portable `.rustiq`

Complete/restart using the same `.rustiq` path

`request.json` — normalized requested calculation

`calculation.json` — resolved scientific calculation

Completed PR B — implement portable `.rustiq` V1 (#88 merged)

Follow-ups, not required for the first usable `.rustiq`