Conversation
|
csv file too wide, draft for now |
|
Reworked the monthly file, see the last commit. Ready for a look. |
is there a difference in the methodology used by Ember for monthly vs yearly? |
|
Should |
|
The carbon intensity is systematically lower than using yearly averages. We need to make sure that the datasets in Ember use the same methodology. I would expect that over a year the figures would be roughly the same. |
Agreed, this needs checking. Ember's methodology documents differences in sources and coverage: Sweden uses Eurostat yearly and mostly ENTSO-E monthly, with gross vs net generation for the EU data. I havent pinned down how much of the gap this explains. I'll compare the monthly values weighted by electricity generation against the annual figure for the same country and year.
Yes, PWUE already uses it in this PR. That's where I extracted the lookup from, so both modules use the same date precedence. The other places need full timestamps or intervals. |
|
Pushed the comparison, weighted by generation over 2022-2025: yearly / monthly falls within 0.9-1.1 in 63 of 64 US state-years and 99 of 131 country-years, the country gaps go both ways. The script now keeps the months only for areas passing every complete year since 2022, the rest stay on the yearly figure; the ones left out are listed at the top of the CSV. Ireland just misses the cutoff at 0.898, 15% would keep it. Does 10% sound reasonable? |
|
From https://files.ember-energy.org/public-downloads/ember_electricity_data_methodology.pdf Estimating latest yearly data |
jnioche
left a comment
jnioche
left a comment
There was a problem hiding this comment.
While we are here: update the documentation by linking to https://files.ember-energy.org/public-downloads/ember_electricity_data_methodology.pdf
and explain that the Ember figures
include full lifecycle
emissions including upstream methane, supply chain and manufacturing emissions, and include all gases, converted into CO2 equivalent over a 100-year timescale.
|
Still seeing a 20% drop for some regions (us-east-1 / eu-west-2) over a 12 months period. |
|
feel free to provide changes on the branch |
Regenerated with scripts/fetch_ember_co2_intensity.sh from the current Ember release: 78 of the 157 regions move, most by a few gCO2/kWh. The larger shift is us-east-1, from 384.4 to 327.17, which now matches the Virginia figure used for the Azure region in the same state.
|
@jnioche some investigation I would like to share: Ember has yearly figures back to 2000, so the middle step is trivial: all year from 2022 in the csv and keep the year of the usage (like PWUE) - Also tried scaling the monthly profile to annual figure: looks ok on past years, but monthly estimates would need validating. Something for a PR later? On the 20%: partly of it is the refresh rather than the monthly figures - us-east-1 on main was mapped to the US national figure instead of Virginia's (eastus in the same state already had the right one). The rest is the monthly series itself, which goes away with the yearly-per-year step anyway. |
thanks @dpol1 let's do that for now
Can look at monthly again later in a separate PR
this formulation is not quite clear for me (maybe because it is AI generated?): refresh? yearly-per-year step? Anyway, the main point here is that |
Hand-typed, but drafted with an AI, hence the jargon ;-) Plain version: "yearly per year" = one figure per region and year, each line uses the one of its billing year. I will replace the commits on this branch with that and keep the monthly ones on a side branch.
The csv in main was generated when cloud_regions.json still placed us-east-1 in Washington DC. Ember has no figure for DC, so the script used the US average, which is what it does when a state has no figure. The coordinates were corrected later, the csv was not regenerated. usgovvirginia still has the DC coordinates, I will fix that too. To make this visible: the csv gets a column with the Ember area each region uses (us-east-1 → US-VA), and a test checks that every US and Indian region uses a state figure. OK with the extra column? |
or change |
|
Yep, fair point - let me think about it |
The csv carries every year from 2022 on, one row per region and year, and AverageCarbonIntensity picks the figure of the year the usage was billed in, as PWUE does: the latest earlier year when Ember has not published that one, the latest year when the row has no date. The usage date lookup moves out of PWUE into UsageDate, and timestamps are read in UTC instead of the JVM zone.
usgovvirginia gets coordinates in Virginia; cloud_regions.json placed it in Washington DC. The script reads Ember's state files by counting fields from the end of the line, so "Washington, D.C." no longer shifts the columns and drops the state. When a region in a country with state figures would get the national figure, the script lists the reasons and leaves the csv unchanged. Failed Nominatim requests are not cached and cached empty codes are ignored.
dpol1
changed the title
Done: the script now stops, lists the US and Indian regions that would get the national figure with the reason, and leaves the csv untouched. One correction on DC: Ember does have a figure for it, the script split "Washington, D.C." on the comma and lost the row. Fixed, usgovvirginia too. Branch replaced with approach we discussed. The monthly ones sit on a side branch for a later PR. |
|
Thanks @dpol1 |
2 participants
One figure per region and year from 2022 on; each line uses the figure of the year it was billed in, as PWUE does: the latest earlier year when Ember has not published that one, the latest year when the row has no date.
The first commit regenerates the csv from the current Ember files. The third places usgovvirginia in Virginia, keeps Ember's "Washington, D.C." rows, which the parser split on the comma and dropped, and stops the script, csv untouched, when a region in the US or India would get the national figure.
The monthly figures stay on a branch for a later PR.