DigitalPebble · jnioche · Sep 11, 2026 · Sep 4, 2026 · Sep 10, 2026 · Sep 10, 2026
diff --git a/docs/src/modules.md b/docs/src/modules.md
@@ -342,6 +342,19 @@ in countries with sub-national data (currently the US and India), the carbon int
 taken from the Ember value for the state hosting the data centre; otherwise the
 country-level Ember value is used.
 
+The figures are keyed by region and year, like [PWUE](#pwue): a line item gets the figure of
+the year it was incurred in, read from its usage date. When Ember has not published that year
+yet (the figures of a year come out in the course of the following one, some series lag
+further), the latest year published for the region is used; rows without a usable date get the
+latest year as well, and usage before 2022, the first year the file carries, gets the 2022
+figure. Ember revises published figures and its most recent year can include estimates, so
+refreshing the file can move past years too.
+
+The figures follow
+[Ember's methodology](https://files.ember-energy.org/public-downloads/ember_electricity_data_methodology.pdf):
+full lifecycle emissions, including upstream methane, supply chain and manufacturing, with all
+gases converted to CO2 equivalent over 100 years.
+
 The data is loaded from `ember/ember_co2_intensity.csv`, which is generated from
 [`cloud_regions.json`](#cloud-region-metadata) — see the scripts under
 [`scripts/`](https://github.com/DigitalPebble/spruce/tree/main/scripts) and the dedicated
@@ -351,7 +364,7 @@ refresh it.
 | | |
 |---|---|
 | **Class** | `com.digitalpebble.spruce.modules.ember.AverageCarbonIntensity` |
-| **Reads** | `region` |
+| **Reads** | `region`, the usage date of the line item |
 | **Writes** | `carbon_intensity` |
 
 ## Stage 4 — Impacts

diff --git a/scripts/README.md b/scripts/README.md
@@ -21,7 +21,7 @@ scripts/fix_cloud_regions.sh src/main/resources/cloud_regions.json
 scripts/fetch_ember_co2_intensity.sh src/main/resources/cloud_regions.json
 ```
 
-End result: `src/main/resources/ember/ember_co2_intensity.csv`, columns `provider,region,gCO2_per_kWh`.
+End result: `src/main/resources/ember/ember_co2_intensity.csv`, columns `provider,region,year,gCO2_per_kWh`, one row per region and year from 2022 on.
 
 ## Scripts
 
@@ -71,7 +71,7 @@ Usage: `./fix_cloud_regions.sh [cloud_regions.json]` (no default — pass the pa
 
 ### `fetch_ember_co2_intensity.sh`
 
-Downloads three Ember CSVs and emits one CSV row per keyed cloud region:
+Downloads three Ember CSVs and emits one CSV row per keyed cloud region and year:
 
 - `yearly_full_release_long_format.csv` — per-country power-sector intensity.
 - `us_yearly_full_release_long_format.csv` — per-US-state intensity.
@@ -80,27 +80,30 @@ Downloads three Ember CSVs and emits one CSV row per keyed cloud region:
 Filtering and reduction (all datasets):
 
 - `Unit == "gCO2/kWh"`.
-- Keep only the row with the highest `Year` per ISO3 code / state code.
+- Keep every year from `FROM_YEAR` (2022) on, per ISO3 code / state code.
 - Country rows are further restricted to countries that appear in
   `cloud_regions.json` (one alias: Ember's "United States of America" ↔
   cloud_regions' "United States").
 
 Joining to cloud regions:
 
-- For every keyed region under `aws`/`gcp`/`azure.cloud_regions`, emit
-  `(provider, region_code, gCO2_per_kWh)`.
+- For every keyed region under `aws`/`gcp`/`azure.cloud_regions`, emit one
+  `(provider, region_code, year, gCO2_per_kWh)` row per year.
 - For regions whose country has a sub-national source (US, India),
   reverse-geocode the region's `latitude`/`longitude` via OpenStreetMap
   Nominatim (`zoom=5`, read `address["ISO3166-2-lvl4"]`) to get an ISO
   3166-2 subdivision code (e.g. `US-VA`, `IN-MH`) and use the
   state-level Ember value.
-- If the subdivision can't be resolved (e.g. the coordinates point at DC,
-  which isn't a state) or has no Ember entry, fall back to the
-  country-level value.
-- Nominatim results are cached in `geocode_cache.tsv` (`lat`, `lon`,
-  ISO 3166-2 code). Repeat runs are free; the 1 req/sec rate limit only
-  matters on the first run or when new regions in supported countries
-  appear. Delete the cache to force a refresh.
+- If a region in one of those countries has no coordinates, or Nominatim
+  places its coordinates in no subdivision, or Ember has no figure for that
+  subdivision, the script stops without writing the csv and lists the
+  regions with the reason: each of them would otherwise get the national
+  figure.
+- Resolved subdivisions are cached in `scripts/.geocode_cache` (`lat`,
+  `lon`, ISO 3166-2 code); failed requests are not, so they are retried on
+  the next run. Repeat runs are free; the 1 req/sec rate limit only matters
+  on the first run or when new regions in supported countries appear.
+  Delete the cache to force a refresh.
 - `_unresolved` entries are skipped (no region code to emit).
 
 Sub-national configuration lives in two arrays at the top of the script:
@@ -121,7 +124,7 @@ Usage: `./fetch_ember_co2_intensity.sh [cloud_regions.json] [output.csv]`.
 Environment variables:
 
 - `EMBER_GEOCACHE` — override the geocode cache path (default
-  `./geocode_cache.tsv`).
+  `scripts/.geocode_cache`).
 
 ## Cloud region water data
 

diff --git a/scripts/fetch_ember_co2_intensity.sh b/scripts/fetch_ember_co2_intensity.sh
@@ -8,13 +8,13 @@
 #   - Unit == "gCO2/kWh"
 # Country CSV:
 #   - ISO 3 code non-empty, Area in cloud_regions.json country set.
-#   - One value per ISO3: the row with the highest Year.
+#   - One value per ISO3 and year, from FROM_YEAR on.
 # Sub-national CSVs (US, India):
 #   - State code non-empty.
-#   - One value per state code: the row with the highest Year.
+#   - One value per state code and year, from FROM_YEAR on.
 # Join:
-#   - For every keyed region under aws/gcp/azure.cloud_regions, emit
-#     (provider, region_code, gCO2_per_kWh).
+#   - For every keyed region under aws/gcp/azure.cloud_regions, emit one
+#     (provider, region_code, year, gCO2_per_kWh) row per year.
 #   - For regions in a country with a sub-national source, reverse-geocode
 #     the region's lat/lon via OpenStreetMap Nominatim to get the ISO 3166-2
 #     subdivision code and use the sub-national value. If the subdivision
@@ -25,7 +25,7 @@
 #   - _unresolved entries are skipped (no region code).
 #   - Regions with no country/subdivision match are skipped.
 #
-# Output columns: provider,region,gCO2_per_kWh
+# Output columns: provider,region,year,gCO2_per_kWh
 #
 # Usage:  ./fetch_ember_co2_intensity.sh [cloud_regions.json] [output.csv]
 #
@@ -39,6 +39,9 @@ PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
 URL="https://files.ember-energy.org/public-downloads/yearly_full_release_long_format.csv"
 CLOUD_REGIONS="${1:-cloud_regions.json}"
 OUTPUT="${2:-$PROJECT_ROOT/src/main/resources/ember/ember_co2_intensity.csv}"
+# First year of figures to ship. Usage before it gets the figures of this year (see
+# AverageCarbonIntensity); the PUE / WUE file starts the same year.
+FROM_YEAR=2022
 GEO_CACHE="${EMBER_GEOCACHE:-$SCRIPT_DIR/.geocode_cache}"
 NOMINATIM_UA="ember-cloud-region-script/1.0"
 
@@ -76,7 +79,9 @@ tmp_countries="$(mktemp)"
 tmp_lookup="$(mktemp)"
 tmp_sub_lookup="$(mktemp)"
 tmp_sub_csv="$(mktemp)"
-trap 'rm -f "$tmp_csv" "$tmp_countries" "$tmp_lookup" "$tmp_sub_lookup" "$tmp_sub_csv"' EXIT
+tmp_alias="$(mktemp)"
+tmp_out="$(mktemp)"
+trap 'rm -f "$tmp_csv" "$tmp_countries" "$tmp_lookup" "$tmp_sub_lookup" "$tmp_sub_csv" "$tmp_alias" "$tmp_out"' EXIT
 
 echo "Downloading $URL..." >&2
 curl --fail -sSL "$URL" -o "$tmp_csv"
@@ -93,9 +98,9 @@ jq -r '
     | .[]
 ' "$CLOUD_REGIONS" > "$tmp_countries"
 
-# Build country -> latest gCO2/kWh lookup (TSV: country<TAB>value).
-# Canonicalise the one known name mismatch so the join works.
-awk -F',' -v countries_file="$tmp_countries" '
+# Build country -> gCO2/kWh by year lookup (TSV: country<TAB>year<TAB>value),
+# from FROM_YEAR on. Canonicalise the one known name mismatch so the join works.
+awk -F',' -v countries_file="$tmp_countries" -v from="$FROM_YEAR" '
     BEGIN {
         while ((getline line < countries_file) > 0) {
             if (line != "") ok[line] = 1
@@ -104,45 +109,29 @@ awk -F',' -v countries_file="$tmp_countries" '
         if (ok["United States"]) ok["United States of America"] = 1
     }
     NR == 1 { next }
-    $15 == "gCO2/kWh" && $2 != "" && ($1 in ok) {
-        iso = $2
-        year = $3 + 0
-        if (year > best_year[iso]) {
-            best_year[iso] = year
-            best_country[iso] = $1
-            best_value[iso] = $16
-        }
-    }
-    END {
-        for (iso in best_value) {
-            name = best_country[iso]
-            if (name == "United States of America") name = "United States"
-            print name "\t" best_value[iso]
-        }
+    $15 == "gCO2/kWh" && $2 != "" && ($1 in ok) && $3 + 0 >= from {
+        name = $1
+        if (name == "United States of America") name = "United States"
+        print name "\t" $3 "\t" $16
     }
 ' "$tmp_csv" > "$tmp_lookup"
 
-# Build combined sub-national lookup (TSV: ISO_3166-2_code<TAB>value) by
-# pulling each configured source and prefixing bare state codes.
-# Ember sub-national CSVs share a schema: col 4=State code, 6=Year, 10=Unit,
-# 11=Value.
+# Build combined sub-national lookup (TSV: ISO_3166-2_code<TAB>year<TAB>value)
+# by pulling each configured source and prefixing bare state codes.
+# Ember sub-national CSVs share a schema: State code, Year, Unit and Value are
+# the 4th, 6th, 10th and 11th of 13 columns. A state name with a comma
+# ("Washington, D.C.") is quoted and shifts the columns after it, so the fields
+# are counted from the end of the line: the last three columns are Value,
+# YoY absolute change and YoY % change.
 : > "$tmp_sub_lookup"
 for entry in "${SUBNATIONAL[@]}"; do
     IFS='|' read -r sn_country sn_url sn_prefix <<< "$entry"
     echo "Downloading $sn_url..." >&2
     curl --fail -sSL "$sn_url" -o "$tmp_sub_csv"
-    awk -F',' -v prefix="$sn_prefix" '
+    awk -F',' -v prefix="$sn_prefix" -v from="$FROM_YEAR" '
         NR == 1 { next }
-        $10 == "gCO2/kWh" && $4 != "" {
-            code = $4
-            year = $6 + 0
-            if (year > best_year[code]) {
-                best_year[code] = year
-                best_value[code] = $11
-            }
-        }
-        END {
-            for (code in best_value) print prefix code "\t" best_value[code]
+        $(NF-3) == "gCO2/kWh" && $(NF-9) != "" && $(NF-7) + 0 >= from {
+            print prefix $(NF-9) "\t" $(NF-7) "\t" $(NF-2)
         }
     ' "$tmp_sub_csv" >> "$tmp_sub_lookup"
 done
@@ -151,11 +140,10 @@ done
 # even when the two systems use different codes for the same region.
 for alias in "${SUBNATIONAL_ALIASES[@]}"; do
     IFS='|' read -r nom_code ember_code <<< "$alias"
-    val=$(awk -F'\t' -v k="$ember_code" '$1==k {print $2; exit}' "$tmp_sub_lookup")
-    if [[ -n "$val" ]]; then
-        printf '%s\t%s\n' "$nom_code" "$val" >> "$tmp_sub_lookup"
-    fi
+    awk -F'\t' -v k="$ember_code" -v n="$nom_code" -v OFS='\t' '$1 == k {print n, $2, $3}' \
+        "$tmp_sub_lookup" >> "$tmp_alias"
 done
+cat "$tmp_alias" >> "$tmp_sub_lookup"
 
 # Check whether a country has a sub-national source configured.
 has_subnational() {
@@ -168,67 +156,85 @@ has_subnational() {
 }
 
 # Reverse-geocode (lat, lon) -> ISO 3166-2 subdivision code via Nominatim,
-# with caching. Returns empty if no subdivision can be resolved.
+# with caching. Prints the code, or nothing when Nominatim places the point in
+# no subdivision; returns 1 when the request fails. Only resolved codes are
+# cached, so a failed request is retried on the next run.
 touch "$GEO_CACHE"
 geo_to_subdivision() {
     local lat="$1" lon="$2"
     local hit
     hit=$(awk -F'\t' -v lat="$lat" -v lon="$lon" '
-        $1 == lat && $2 == lon { print $3; found=1; exit }
+        $1 == lat && $2 == lon && $3 != "" { print $3; found=1; exit }
         END { if (!found) exit 1 }
     ' "$GEO_CACHE") && { printf '%s' "$hit"; return; }
 
     sleep 1  # respect Nominatim 1 req/sec policy
     local resp code
     resp=$(curl --fail -sSL -A "$NOMINATIM_UA" \
         "https://nominatim.openstreetmap.org/reverse?format=jsonv2&zoom=5&lat=${lat}&lon=${lon}" \
-        2>/dev/null || echo '{}')
+        2>/dev/null) || return 1
     code=$(jq -r '.address["ISO3166-2-lvl4"] // ""' <<< "$resp")
     # Validate shape: "XX-..." where XX is a 2-letter country code.
     if [[ ! "$code" =~ ^[A-Z]{2}-[A-Z0-9]+$ ]]; then code=""; fi
-    printf '%s\t%s\t%s\n' "$lat" "$lon" "$code" >> "$GEO_CACHE"
+    if [[ -n "$code" ]]; then
+        printf '%s\t%s\t%s\n' "$lat" "$lon" "$code" >> "$GEO_CACHE"
+    fi
     printf '%s' "$code"
 }
 
-lookup_value() {
-    awk -F'\t' -v key="$1" -v file="$2" '
-        BEGIN {
-            while ((getline line < file) > 0) {
-                split(line, a, "\t")
-                if (a[1] == key) { print a[2]; exit }
-            }
-        }
-    '
+# Print "year<TAB>value" lines for a key, in year order.
+lookup_years() {
+    awk -F'\t' -v key="$1" '$1 == key { print $2 "\t" $3 }' "$2" | sort -n
 }
 
 mkdir -p "$(dirname "$OUTPUT")"
 
+# Regions of a country with state figures that would get the national figure,
+# each with the reason. The csv is written only when there are none.
+problems=()
+
 # Emit a row per keyed cloud region.
 {
     echo "# https://ember-energy.org/creative-commons/"
     echo "# Creative Commons Attribution Licence (CC-BY-4.0)"
-    echo "#provider,region,gCO2_per_kWh"
+    echo "#provider,region,year,gCO2_per_kWh"
     while IFS=$'\t' read -r provider region country lat lon; do
-        value=""
-        if [[ -n "$lat" && -n "$lon" ]] && has_subnational "$country"; then
-            code=$(geo_to_subdivision "$lat" "$lon")
-            if [[ -n "$code" ]]; then
-                value=$(lookup_value "$code" "$tmp_sub_lookup")
+        years=""
+        if has_subnational "$country"; then
+            if [[ -z "$lat" || -z "$lon" ]]; then
+                problems+=("$provider $region: no coordinates in $CLOUD_REGIONS")
+            elif ! code=$(geo_to_subdivision "$lat" "$lon"); then
+                problems+=("$provider $region: Nominatim request failed for $lat,$lon")
+            elif [[ -z "$code" ]]; then
+                problems+=("$provider $region: no subdivision at $lat,$lon")
+            else
+                years=$(lookup_years "$code" "$tmp_sub_lookup")
+                if [[ -z "$years" ]]; then
+                    problems+=("$provider $region: no Ember figure for $code")
+                fi
             fi
         fi
-        if [[ -z "$value" ]]; then
-            value=$(lookup_value "$country" "$tmp_lookup")
+        if [[ -z "$years" ]]; then
+            years=$(lookup_years "$country" "$tmp_lookup")
         fi
-        if [[ -n "$value" ]]; then
-            echo "$provider,$region,$value"
+        if [[ -n "$years" ]]; then
+            awk -F'\t' -v p="$provider" -v r="$region" '{ print p "," r "," $1 "," $2 }' <<< "$years"
         fi
     done < <(jq -r '
         ["aws","gcp","azure"][] as $p
         | .[$p].cloud_regions // {}
         | to_entries[]
         | [$p, .key, .value.country, .value.latitude, .value.longitude] | @tsv
     ' "$CLOUD_REGIONS")
-} > "$OUTPUT"
+} > "$tmp_out"
+
+if (( ${#problems[@]} )); then
+    echo "error: these regions would get the national figure instead of their state's:" >&2
+    printf '  %s\n' "${problems[@]}" >&2
+    echo "$OUTPUT left unchanged" >&2
+    exit 1
+fi
+mv "$tmp_out" "$OUTPUT"
 
 rows=$(($(wc -l < "$OUTPUT") - 1))
 echo "Wrote $OUTPUT ($rows rows)" >&2
diff --git a/scripts/fix_cloud_regions.sh b/scripts/fix_cloud_regions.sh
@@ -9,6 +9,8 @@
 #     East US 2 at Richmond; East US is actually Ashburn/Sterling).
 #   - Fixes AWS us-east-1 coordinates (upstream points to Washington DC;
 #     the region is actually in Ashburn/Loudoun County, VA).
+#   - Fixes Azure US Gov Virginia coordinates (upstream points to Washington
+#     DC; the region is in Virginia).
 #   - Fixes AWS cn-northwest-1 (upstream labels it "Ningxiang" in Hunan;
 #     the region is in Ningxia, near Yinchuan).
 #   - Fixes AWS ca-central-1 coordinates (upstream points ~50 km west of
@@ -99,6 +101,12 @@ jq '
     | .azure.cloud_regions.eastus.longitude  = "-77.487442"
     | .azure.cloud_regions.eastus.metro_area = "Ashburn"
     | .azure.cloud_regions.eastus.name       = "East US (Ashburn)"
+
+    # US Gov Virginia is in Virginia, not Washington DC. Microsoft names only the
+    # state, so the coordinates are those of Richmond: any point in the state
+    # selects the state figure.
+    | .azure.cloud_regions.usgovvirginia.latitude  = "37.540700"
+    | .azure.cloud_regions.usgovvirginia.longitude = "-77.433654"
 ' "$FILE" > "$tmp"
 
 mv "$tmp" "$FILE"

diff --git a/src/main/java/com/digitalpebble/spruce/RowColumn.java b/src/main/java/com/digitalpebble/spruce/RowColumn.java
@@ -99,7 +99,7 @@ public Integer getYear(Row r) {
             return null;
         }
         if (value instanceof java.sql.Timestamp timestamp) {
-            return timestamp.toLocalDateTime().getYear();
+            return timestamp.toInstant().atZone(ZoneOffset.UTC).getYear();
         }
         if (value instanceof java.time.Instant instant) {
             return instant.atZone(ZoneOffset.UTC).getYear();