Template existing and planned (ECAA) generators by EllieKallmier · Pull Request #143 · Open-ISP/ISPyPSA

EllieKallmier · 2026-08-27T07:01:15Z

Adds generators_existing_planned to the new-format templater, and lays the
generator/storage split that storage_existing_planned will build on. A bit chunkier
than ideal prob but should be pretty similar to the new entrants so (hope) not too
much review work...

Everything comes from one IASR table —
existing_committed_anticipated_additional_generator_summary — which already lists one
row per real generating unit (DUID-level), generators and storage together. So the module
splits that table in two, then merges per-unit properties onto the generator side from six
further IASR tables. storage_existing_planned is left at just the split and column rename
here; property merges for that one coming up.

src/ispypsa/templater/
├── existing_planned.py      → new module: gen/storage split, PHES routing, property merges
├── mappings.py              → _GENERATORS_EXISTING_PLANNED_PROPERTY_MAP; key_col rename
├── helpers.py               → shared helpers consolidated here (see Refactors)
├── create_template.py       → wires the new table into the new-format branch
└── geography.py, transmission.py, new_entrants.py
                             → helpers moved out to helpers.py
src/ispypsa/iasr_table_caching/local_cache.py
                             → registers the seven newly-consumed IASR tables
tests/test_workbook_table_cache/7.5/
                             → frozen cache rebuilt with those tables

Things worth knowing before reading the diff

PHES routing can't rely on Technology Type alone. Wivenhoe and Shoalhaven are
labelled plain "Hydro" in the summary, but have real entries in the PHES properties table
(and the numbers cross-check — Wivenhoe's 3000 MWh / 285 MW ≈ the properties table's 10h).
Routing by technology string alone would send them to generators and silently never read
their PHES properties. So the split is a technology match union name-presence in the PHES
properties table -> noting it could just be one or the other

_validate_phes_routing then fails loud if any PHES station is missing from the summary
altogether — the only way to catch a future Wivenhoe-shaped surprise, since routing's own
presence check can't distinguish "absent" from "missed". Lower Tumut is the one
documented tolerated exception: it's Tumut 3's reversible pump-turbine subset (units 1-3),
see #131 comment thread.

Two things to note:

minimum_load for coal now comes from Typical Lowest Band. See #142
KiataWF1 vs KIATAWF1 is handled as a named-constant correction, not by fuzzy
matching — fuzz.ratio scores it 50, well below any threshold that would be safe to apply
across 642 IDs. It's the only ID mismatch in the whole set. More detail in comments.

Known gap - not fixed here 25 REZ-located rows use sub-zone ids (Q8a/Q8b/
Q8c) that renewable_energy_zones doesn't carry — only the parent Q8. Same class as
the existing #133 non-REZ placeholder gap, different cause, so it follows the same
precedent: named as an exception in the CLI-level geo_id coverage test rather than solved
here. Fix to come shortly - also flagging a future need to reassess when we move to v7.8
workbook which introduces a (differently-purposed/different coverage and impact) split up of
N9 -> N9a / N9b.

Refactors

Mostly consolidation, so both new-format modules draw on one set of helpers instead of
reaching sideways into each other:

Moved into helpers.py:
- _set_geo_id (was duplicated in both modules),
- the geo_id/granularity trio from geography.py,
- _assert_table_valid and the property-map machinery from new_entrants.py, and
- _build_geo_region_lookup from transmission.py
_map_geo_id_to_granularity now handles all three granularities and leaves REZ geo_ids
untouched in every case, so that rule lives in one place rather than being reimplemented
per module.
New entrant's property maps rename technology_col → key_col, so one merge helper
serves both modules. The maps gain a numeric flag (default true) controlling whether a
merged value is coerced with pd.to_numeric(errors="raise") — an input-side typo guard;
only commissioning_date opts out.
- There's a note-to-self in there that I'm happy to take opinions on re: float vs int
  dtype enforcement (whether to implement here or let validator apply) - my thought is
  towards implementing here to keep validator's scope tight but it's a small fix thing
  so just left simple as-is for the moment.
commissioning_date gets an explicit reformat: the parser emits ISO strings, the schema
declares %d/%m/%Y.

…move some functions to helpers, pull in required tables

…tor shared helpers Functions (and mappings) that were shared/very similar across new_entrants templater module are pulled out and gently refactored to become generic helpers.

codecov · 2026-09-03T07:56:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/ispypsa/iasr_table_caching/local_cache.py	`74.28% <ø> (ø)`
src/ispypsa/templater/create_template.py	`93.25% <100.00%> (+0.23%)`	⬆️
src/ispypsa/templater/existing_planned.py	`100.00% <100.00%> (ø)`
src/ispypsa/templater/geography.py	`93.93% <ø> (-0.80%)`	⬇️
src/ispypsa/templater/helpers.py	`100.00% <100.00%> (ø)`
src/ispypsa/templater/mappings.py	`100.00% <100.00%> (ø)`
src/ispypsa/templater/new_entrants.py	`100.00% <100.00%> (ø)`
src/ispypsa/templater/transmission.py	`98.80% <100.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nick-gorman

@EllieKallmier, generally I think this looks great and was pretty easy to follow. I just found a few pretty small things, I'll leave it up to you what you reckon is worth actioning.

nick-gorman · 2026-09-09T04:29:47Z

+
+# Case mismatch between maximum_capacity's IASR ID and the summary's: the 'safe'
+# fuzzy-matching threshold (90) would miss (fuzz.ratio("KiataWF1", "KIATAWF1") == 50).
+_MAXIMUM_CAPACITY_ID_TYPO_FIX = dict(


I think this constant would read better with a generic name like IASR_TYPO_FIXES, because then the fixes themselves are defined on a table by table basis. Does that make sense?

Yep I agree! I'll leave this for now as I have an update coming up that this name change will fit very nicely into :)

nick-gorman · 2026-09-09T04:37:33Z

+    I/O Example (subset of columns; regional_granularity="sub_regions"):
+        existing_committed_anticipated_additional_generator_summary:
+            IASR ID / DLT names  Power Station  Technology Type      REZ ID  Sub-region
+            BW01                 Bayswater      Steam Sub Critical   NA      CNSW
+
+        maximum_capacity_existing_committed_anticipated_additional_generators:
+            IASR ID  Installed capacity (MW)  Commissioning date
+            BW01     660.0                    NaN
+
+        returns:
+            name  power_station  geo_id  capacity  commissioning_date
+            BW01  Bayswater      CNSW    660.0     NaN


Maybe slightly reformatted and expanded I/O example like below?

Suggested change

I/O Example (subset of columns; regional_granularity="sub_regions"):

existing_committed_anticipated_additional_generator_summary:

IASR ID / DLT names Power Station Technology Type REZ ID Sub-region

BW01 Bayswater Steam Sub Critical NA CNSW

maximum_capacity_existing_committed_anticipated_additional_generators:

IASR ID Installed capacity (MW) Commissioning date

BW01 660.0 NaN

returns:

name power_station geo_id capacity commissioning_date

BW01 Bayswater CNSW 660.0 NaN

I/O Example (subset of columns):

iasr_tables:

existing_committed_anticipated_additional_generator_summary:

IASR ID / DLT names Power Station Technology Type REZ ID Sub-region

BW01 Bayswater Steam Sub Critical NA CNSW

maximum_capacity_existing_committed_anticipated_additional_generators:

IASR ID Installed capacity (MW) Commissioning date

BW01 660.0 NaN

... plus the other property tables (see _GENERATORS_EXISTING_PLANNED_PROPERTY_MAP)

regional_granularity: "nem_regions"

sub_regional_geography:

geo_id geo_type region_id

CNSW subregion NSW

returns:

name power_station geo_id capacity commissioning_date

BW01 Bayswater NSW 660.0 NaN # CNSW -> NSW via sub_regional_geography

nick-gorman · 2026-09-09T05:36:24Z

+    Only generators whose ``technology`` appears in the table's own 'Technology Type'
+    column are fuzzy-matched and
+
+    I/O Example (property_spec = _COAL_MINIMUM_LOAD_PROPERTY; value_col abbreviated,


I think I have a preference for expanding the I/O examples. So in this case showing property_spec and maybe nesting coal_minimum_stable_level under an iasr_tables: headning. Up to you if you think its worth it.

nick-gorman · 2026-09-10T06:02:41Z

+    """
+    resolved = _fuzzy_match_names(
+        names,
+        table_keys,


Maybe this is over kill. But the fuzzy matching would be slightly hardended by filtering out known storage unit names from table_keys before passing to _fuzzy_match_names.

nick-gorman · 2026-09-10T06:12:56Z

+            ValueError: PHES properties station(s) not found in the summary:
+                        ['Some New Station']
+    """
+    if summary.empty:


Not a big one, but I reckon, given that this is the templater we could assume summary always contains some data. I think if you did change this is would just impact a couple of tests.

nick-gorman · 2026-09-10T06:13:07Z

+            BW01     Bayswater             660.0     NaN                 10.05
+            HUNTER1  Hunter Power Station  375.0     2025-08-01          10.93
+    """
+    if generators.empty:


I might leave this one for the moment - but definitely open to remove in future - just for a little (low cost?) hedge. Because this is catching only when the 'generators' part of the summary table is empty rather than the whole summary table it's just a tiny bit less of a given(?). Though I totally accept it's defensive and we're not at the stage of pre-filtering inputs or anything so if you do feel like the cons outweigh the (admittedly minute) pros I will happily delete hahaha 😆

nick-gorman · 2026-09-10T06:13:17Z

+            ANGAS1  Reciprocating engine  3.0
+            Q1G1    Large scale Solar PV  NaN   # neither coal nor gas
+    """
+    if generators.empty:


nick-gorman · 2026-09-10T06:20:25Z

+    "closure_year": dict(
+        table="expected_closure_years",
+        key_col="IASR ID",
+        value_col="Expected Closure Year (Calendar year)",


Could replace numeric with a declared dtype so that the closure can be output as a int rather than a float. Maybe a nitpick, hmmm.

Suggested change

value_col="Expected Closure Year (Calendar year)",

value_col="Expected Closure Year (Calendar year)",

dtype="int"

I have also been wondering about this - at the moment I'm not 100% convinced that the benefits of doing something like this (non-float looking year values is kinda the main one) outweighs the added complexity due to NaN presence. I'm also thinking about the round-trip when tables get saved to and read from csv (with potentially still empty columns) and needing to keep re-setting dtypes at those boundaries, + validation probably will (/could) have some type-setting role I imagine.

There's also a kind of question about where this mapping 'faces'/what it's kind of telling you about, which at the moment has been pretty much all source-related. A new field that's telling you what will happen to the output is a slightly different direction - but that's a very very small difference (and could be argued that it's still telling about the source too) imo.

Anyway I might open an issue just to keep this comment + thoughts somewhere and am totally open to implementing this if we decide it's worth it!

nick-gorman · 2026-09-10T06:23:35Z

+       geo_id (REZ ID with Sub-region fallback — see helpers._set_geo_id) and relabels
+       it to ``regional_granularity`` (REZ-located rows stay untouched at every
+       granularity, matching new_entrants.py's convention — see
+       _relabel_geo_id_to_granularity).


Not sure _relabel_geo_id_to_granularity is used anywhere.

nick-gorman · 2026-09-10T06:23:51Z

+    _get_property_value_map,
+    _group_properties_by_source,
+    _is_storage_row,
+    _is_subregion_geo_id,


_is_subregion_geo_id unused?

Addresses some comments from PR review: others have been addressed in a new issue (#145) or in-file comments/notes to be revisited in upcoming PRs.

EllieKallmier added 3 commits August 25, 2026 18:03

add initial existing/planned templating skeleton: split gen/storage, …

7a7bc6c

…move some functions to helpers, pull in required tables

add generator property merges to existing/planned templater and refac…

52a8d02

…tor shared helpers Functions (and mappings) that were shared/very similar across new_entrants templater module are pulled out and gently refactored to become generic helpers.

add comments for context on fix items to come

1febb23

EllieKallmier added type: feature New feature or request module: templater Covers contents of `templater` module labels Aug 27, 2026

EllieKallmier requested a review from nick-gorman August 27, 2026 07:01

EllieKallmier mentioned this pull request Sep 3, 2026

Fix new entrant renaming #144

Merged

add generators_existing_planned output table to expected list

48ca299

nick-gorman approved these changes Sep 10, 2026

View reviewed changes

EllieKallmier mentioned this pull request Sep 14, 2026

Templater property maps: where and how to enforce/set/change dtypes? #145

Open

expand I/O examples, pass known IDs to exclude from fuzzy mapping

7aab360

Addresses some comments from PR review: others have been addressed in a new issue (#145) or in-file comments/notes to be revisited in upcoming PRs.

EllieKallmier merged commit 5b351e4 into main Sep 14, 2026
15 checks passed

EllieKallmier deleted the template-existing-planned-skeleton branch September 14, 2026 05:37

EllieKallmier mentioned this pull request Sep 30, 2026

Template existing planned storage #149

Open

	value_col="Expected Closure Year (Calendar year)",
	value_col="Expected Closure Year (Calendar year)",
	dtype="int"

Conversation

EllieKallmier commented Aug 27, 2026

Things worth knowing before reading the diff

Refactors

Uh oh!

codecov Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nick-gorman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nick-gorman Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nick-gorman Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Sep 3, 2026 •

edited

Loading

nick-gorman Sep 10, 2026 •

edited

Loading

nick-gorman Sep 10, 2026 •

edited

Loading