Selective/lazy from_res1d ingestion by quantity, for calibration loops

[jpalm3r]

Summary

from_res1d takes about 18-25 seconds on a moderately sized network file. The documented nodes=[...], reaches=[] filtering cuts that to around 5 seconds, but the remaining time is still mostly spent building topology rather than reading data.

In a calibration loop that rebuilds the network for every trial, this dominates the runtime — matching itself is already fast in comparison.

Request

Support selective or lazy ingestion by quantity, on top of the existing node/reach filtering, so calibration workflows can build only what a given trial needs.

Relevant entry points

• Network.from_res1d (src/modelskill/network.py:351-447)
• Network._load_res1d_network (network.py:448-487)

Found by a test user evaluating the network functionality ahead of a 1.4.0 release.

[jpalm3r]

I looked into this. Summary of what I found and a proposed approach.

How loading works today

from_res1d opens the file with no filters, then iterates every reach to build the full
topology. Per-location time series are read only for the selected nodes/reaches, via
node.to_dataframe(), which reads all quantities at that location.

Two separate costs

1. Topology — the loop over all reaches touches each reach's metadata and every
   gridpoint's chainage over the pythonnet/.NET boundary (376 gridpoints in the test file).
2. Data reads — the to_dataframe() calls, already gated by nodes/reaches.

On the small test file topology is ~40% of load time; on a large file it dominates, which
matches your report that the ~5s left after nodes=[...], reaches=[] is mostly topology.
A quantity filter reduces the data-read cost, not the topology cost. Worth flagging up
front so expectations are right.

mikeio1d already supports quantities=, but with a catch

Res1D(path, quantities=[...]) is fast, but it drops every location that lacks that
quantity — e.g. a WaterLevel-only node raises KeyError under a ["Discharge"] filter.
Passing the filter to the constructor would break the current guarantee that the full
topology is always built and that find()/recall() work on every node.

Per-location quantity reads do exist (node.WaterLevel.to_dataframe()), so we can filter
which quantities are read while still enumerating the full topology from an unfiltered file.

Proposed approach

Add a quantities parameter to from_res1d mirroring the nodes/reaches semantics
(None = all, str/list = subset, [] = none), applied at the read layer so the full
topology is preserved. A location that does not carry a requested quantity becomes
topology-only rather than an error.

Caveat and follow-up

This gives selective ingestion by quantity as requested, but does not touch the topology
building that dominates on large files. If calibration-loop speed is the real target, the
bigger win is likely reusing the topology across trials (build once, re-read only data per
trial) — worth a separate issue.

[crcoDHI]

Test user here: I profiled from_res1d on a large EPANET-backed .res file (7,955 nodes, 8,377 reaches) and found a couple of observations that may help refine the optimization work.

For this network, the dominant topology cost is not chainage handling. With nodes=[...] and reaches=[] (build time ~5–7 s), the main contributors are:

result_reach.py:_get_total_length: ~2.4 s across 8,377 calls.
Construction of 16,756 empty pandas DataFrames (two per reach for topology-only nodes): ~5 s.

Access to gridpoint and metadata information is only ~0.1 s in this case, although that may differ for river-network files.

The placeholder DataFrames also appear to be the root cause of the datetime-index issue I reported earlier. They propagate into _build_dataframe during concat and degrade the time index. Keeping None for topology-only nodes, or skipping empty DataFrames entirely, would both eliminate the bug and avoid ~16.7k DataFrame allocations per build.

I also confirmed the quantities= behavior on this file. Res1D(path, quantities=["Pressure"]) opens about 3.5× faster (3.2 s vs 11.1 s), but none of the 8,377 reaches are retained, so the topology is lost. That reinforces the need for the read-layer approach proposed above when working with mixed node/link quantity files.

Finally, topology reuse would be highly valuable for calibration workflows. In my case, the network topology remains fixed and only the result file changes between runs. An API along the lines of:
Network.from_res1d(new_res, topology=existing_network)
would fit that workflow well.

For reference, building the network from batched per-location reads (res.read(queries)) takes ~0.5 s for 50 sensors, with matching taking another ~0.9 s. That suggests a per-trial cost of roughly 1.5 s after file open, compared to 18–25 s today.

[jpalm3r]

Thanks @crcoDHI — that profile changed what this issue should fix.

Fixed on branch network-loading:

• fac935e7 — one shared empty DataFrame for topology-only nodes and gridpoints, instead of one
  allocation each. This is your 16,756 frames. _build_dataframe already skips empty frames before
  the concat, so the shared instance never reaches pandas — which also keeps the Nodes-filtered network cannot be matched — time coordinate degrades to object dtype #676 datetime fix
  intact.
• 12bb3f2c — read each selected node once. _init_node was reading a node once per reach
  endpoint, but _generate_graph keeps only the first copy, so the rest was discarded work. On our
  test file that is 236 reads for 119 nodes; on yours it would be 16,754 for 7,955.
• 22178858 — the quantities parameter, at the read layer as proposed. None = all, a str or
  list = that subset, [] = none. A location carrying none of the requested quantities becomes
  topology-only rather than raising, so it composes with nodes/reaches on mixed node/link files.
• 57e86117, d534e096 — the regression tests; 2c663898 — user guide.

On tests/testdata/network.res1d a filtered load goes from 176 ms to 132 ms under cProfile, with
378 empty-frame allocations down to 1 and 236 node reads down to 119. Our fixture is small and
carries exactly one quantity per location, so it verifies the quantity filter is correct but cannot
show a speedup or cover a location holding two quantities — a check against a larger file would be
worth having.

Three things from your comment are not in this branch, filed separately:

• Topology reuse, from_res1d(res, topology=existing_network) — Reuse network topology across from_res1d calls for calibration loops #682. This is the largest remaining
  win and it also sidesteps _get_total_length entirely, since the reach loop never runs.
• Batched reads via res.read(queries) — Batch per-location reads in from_res1d instead of one to_dataframe() per location #683.

[jpalm3r]

Addressed in #685: from_res1d now takes a quantities parameter (None = all, a string or list = a subset, [] = none), applied at the read layer so the full topology still gets built. A location that doesn't carry any requested quantity becomes topology-only instead of raising. I've added Closes #679 to #685's description so this closes automatically once it merges.