Skip to content

Commit 4568302

Browse files
CameronBrooks11CameronBrooks11
committed
feat: publish the article analysis pipeline
The compute behind the articles on flowmatrixai.com: fetch of every public source by recorded URL and SHA-256, the storm multiplier (EIA-861), the EAGLE-I outage aggregates, the penetration gap (ACS x EAGLE-I), the Google Trends seven-day window, and the ten article exhibits, with the computed CSVs committed as fixtures and a test suite that reproduces the published figures and regenerates every exhibit byte-identically. Code is MIT; the committed outputs are CC BY 4.0 because the EAGLE-I inputs are. AGENTS.md states the rules for a public repository.
0 parents  commit 4568302

60 files changed

Lines changed: 46231 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.editorconfig‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
root = true
2+
3+
[*]
4+
charset = utf-8
5+
end_of_line = lf
6+
insert_final_newline = true
7+
trim_trailing_whitespace = true
8+
indent_style = space
9+
indent_size = 2
10+
11+
[*.py]
12+
indent_size = 4
13+
14+
[*.md]
15+
trim_trailing_whitespace = false

‎.github/CODEOWNERS‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# The maintainer of record. A GitHub handle is the one personal identifier this public repo carries.
2+
* @CameronBrooks11

‎.github/pull_request_template.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
## Summary
2+
3+
<!-- What changed and why. This repository is public: the PR body is public too. -->
4+
5+
## Changes
6+
7+
-
8+
9+
## Published figures
10+
11+
<!-- If a committed CSV, SVG or a value in tests/reference.py changed: which figure,
12+
from what to what, why, and which article needs the matching edit. Otherwise "none". -->
13+
14+
## Testing
15+
16+
<!-- How was this verified? `just check && just test` output, or the CI run. -->
17+
18+
## Checklist
19+
20+
- [ ] CI (`ok`) is green
21+
- [ ] Nothing here names a private repository, internal tooling, a client or a person (see AGENTS.md)
22+
- [ ] Any new raw source is registered in `fetch.py` with URL, SHA-256 and licence
23+
- [ ] README / METHODS updated if the method or a source changed

‎.github/workflows/ci.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
check:
13+
name: check
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
- uses: astral-sh/setup-uv@v6
18+
with:
19+
enable-cache: true
20+
- uses: extractions/setup-just@v3
21+
- run: just setup
22+
- run: just check
23+
24+
test:
25+
name: test
26+
runs-on: ubuntu-latest
27+
steps:
28+
- uses: actions/checkout@v4
29+
- uses: astral-sh/setup-uv@v6
30+
with:
31+
enable-cache: true
32+
- uses: extractions/setup-just@v3
33+
- run: just setup
34+
# Small sources are fetched so the reproduction tests run in CI. The EAGLE-I
35+
# archive (11.6 GB) is not; its tests read the committed aggregates and skip
36+
# the full-table check.
37+
- run: just fetch-sources eia861 acs
38+
- run: just test
39+
40+
ok:
41+
name: ok
42+
needs: [check, test]
43+
if: ${{ always() }}
44+
runs-on: ubuntu-latest
45+
steps:
46+
- run: |
47+
results="${{ join(needs.*.result, ',') }}"
48+
case "$results" in *failure*|*cancelled*) exit 1 ;; esac
49+
echo ok

‎.github/workflows/secret-scan.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: secret-scan
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
schedule:
8+
- cron: "0 6 * * 1" # weekly full-history sweep, Mondays 06:00 UTC
9+
workflow_dispatch:
10+
11+
permissions:
12+
contents: read
13+
14+
jobs:
15+
gitleaks:
16+
uses: FlowMatrix-AI/.github/.github/workflows/gitleaks.yml@fdd0291829cdff1867bc59dc9afc6524642dabbc # main

‎.gitignore‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Environment: no env file is needed here, but never let one in
2+
.env
3+
.env.*
4+
!.env.example
5+
6+
# Python
7+
__pycache__/
8+
.venv/
9+
venv/
10+
*.egg-info/
11+
dist/
12+
build/
13+
.pytest_cache/
14+
.ruff_cache/
15+
16+
# OS / editor
17+
.DS_Store
18+
*.log
19+
.idea/
20+
.vscode/
21+
22+
# Fetched raw sources are restored by `just fetch-sources` (URL + SHA-256
23+
# recorded in src/article_data/fetch.py); never committed.
24+
outputs/raw/
25+
26+
# Large regenerated outputs (the small aggregates beside them are committed fixtures)
27+
outputs/computed/eaglei-county-month.csv
28+
outputs/computed/eaglei-2025-county-week.csv

‎AGENTS.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# AGENTS.md
2+
3+
**THIS REPOSITORY IS PUBLIC.** Every file, commit message, branch name, issue,
4+
pull request and review comment in it is visible to anyone on the internet, and
5+
the methods page on flowmatrixai.com links here. Read this section before doing
6+
anything else.
7+
8+
## Hard rules for a public repo
9+
10+
The test for anything you are about to add: **if it would not be fine on the
11+
front page of the site, it does not belong here.** Concretely:
12+
13+
1. **Never name or link a private repository.** Not in code, comments,
14+
docstrings, commit messages, issues or PR bodies. "Moved from the internal
15+
research repo" is fine; the repo's name and URL are not. If a private repo
16+
is the origin of a figure, describe the public source the figure was
17+
computed from instead.
18+
2. **Never reference internal tooling, portals, infrastructure or operations.**
19+
No internal hostnames, dashboards, deploy targets, CRM or ticketing systems,
20+
session or workflow identifiers, or filesystem paths from a particular
21+
machine. The only hosts that may appear are the public data sources and
22+
flowmatrixai.com.
23+
3. **Never name a client, a prospect, or an individual.** Utilities, agencies
24+
and counties appear here as rows in public datasets and may be named as
25+
such. A commercial relationship with anyone is never disclosed. No personal
26+
names paired with a title or employer, no email addresses, no phone numbers,
27+
no personal profile URLs. The one exception is the maintainer's own GitHub
28+
handle in `CODEOWNERS`.
29+
4. **Never include commercial terms, pricing, revenue, pipeline, funnel figures
30+
or internal strategy.** This repo explains how published numbers were
31+
computed, nothing about how the business runs.
32+
5. **Every dataset comes from a public source with its URL recorded.** A raw
33+
input is added by registering it in `src/article_data/fetch.py` with its
34+
URL, SHA-256 and licence; a pinned input under `outputs/inputs/` records
35+
where and when it was pulled. Nothing arrives by hand-copy from a private
36+
file. If a source is not public, the analysis that needs it does not go
37+
here.
38+
6. **No credentials, and no pointers to where a credential is stored.** The
39+
pipeline needs no API key by design (ACS is read from the keyless summary
40+
files); keep it that way rather than adding a keyed endpoint.
41+
7. **Commit messages, branch names and issues are public too.** Write them as
42+
if they were part of the README. No internal ticket references, no client
43+
or project codenames, no "per the call with X".
44+
45+
When in doubt, leave it out and ask. A leak here is permanent: history is
46+
public and a force-push does not remove what has already been cloned.
47+
48+
## Project
49+
50+
The data and code behind the articles on flowmatrixai.com. Every published
51+
figure is produced from a public source that is fetched by recorded URL,
52+
verified against a recorded SHA-256, and reduced by code in this repository.
53+
The computed outputs and the article exhibits are committed so a reader can
54+
check a number without running anything.
55+
56+
## Commands
57+
58+
```bash
59+
just setup # uv sync --frozen
60+
just check # ruff format --check, ruff check, pyright (what CI runs)
61+
just test # pytest; reproduction tests skip until sources are fetched
62+
just fetch-sources eia861 acs # verified downloads of the small sources into outputs/raw/
63+
just pipeline --stream # every analysis; streams the 11.6 GB EAGLE-I archive year by year
64+
just exhibits # redraw the article SVGs from the committed CSVs
65+
```
66+
67+
## Layout
68+
69+
```
70+
pyproject.toml, uv.lock project + locked environment (Python 3.12, uv)
71+
src/article_data/
72+
fetch.py source registry (URL, SHA-256, licence) and downloader
73+
storm_multiplier.py EIA-861 SAIDI with / without major event days
74+
eaglei.py EAGLE-I county-month and 2025 county-week aggregates
75+
penetration_gap.py ACS qualifying homes x EAGLE-I exposure
76+
trends.py Google Trends decay after the 2024 hurricanes
77+
exhibits/ the article SVGs; figstyle.py holds brand tokens and determinism
78+
cli.py the `article-data` command
79+
tests/ pytest suite; reference.py holds the published figures
80+
outputs/
81+
METHODS.md per-analysis source, licence, method, known limits
82+
raw/ fetched sources (gitignored, restored by `article-data fetch`)
83+
inputs/ pinned inputs that cannot be re-fetched byte-for-byte; brand tokens; fonts
84+
computed/ outputs; small ones committed as fixtures
85+
exhibits/ the article SVGs, committed and regenerated by the tests
86+
```
87+
88+
## Rules specific to the pipeline
89+
90+
- **Do not commit anything under `outputs/raw/`.** Raw sources are fetched by
91+
script and verified by checksum; committed outputs are small aggregates only.
92+
- **Do not hand-edit a committed CSV under `outputs/computed/`.** Regenerate
93+
it. If a number changes, change `tests/reference.py` and the article in the
94+
same PR, with the reason in the PR body.
95+
- **Do not hand-edit an SVG under `outputs/exhibits/`.** The tests regenerate
96+
every exhibit and assert it is byte-identical to the committed file. Change
97+
the drawing code, run `just exhibits`, commit the result.
98+
- **Do not change `HASH_SALT` in `exhibits/figstyle.py`.** It seeds
99+
matplotlib's element ids; changing it rewrites every committed SVG for no
100+
visible gain.
101+
- **Do not re-pull the Google Trends series casually.** The pinned series under
102+
`outputs/inputs/` is the published one; a re-pull is a re-baseline of the
103+
shipped numbers, done deliberately with the tests and the article updated
104+
together.
105+
- **Do not switch the ACS step to the Census API.** The API redirects to an
106+
HTML page with HTTP 200 when the key is missing, so a naive fetch looks like
107+
it worked. The pipeline reads the keyless table-based summary files for that
108+
reason, and it keeps this repo free of any credential.
109+
- **A checksum mismatch is an error, not a warning.** If an upstream file
110+
changes, record the new SHA-256 in the same PR that re-verifies the figures
111+
against it; never loosen the check.
112+
- **Licences travel with the data.** EAGLE-I is CC BY 4.0, which is why the
113+
committed outputs carry attribution and are released under `LICENSE-DATA`.
114+
A new source's licence is recorded in `fetch.py` and, if it constrains
115+
redistribution, in `outputs/METHODS.md` and the README.
116+
117+
## Conventions
118+
119+
- Conventional Commits (`type(scope): description`), one logical change per
120+
commit, no AI attribution trailers.
121+
- `just check && just test` before every commit; CI runs the same gates plus
122+
a secret scan, and `ok` must be green to merge.
123+
- Squash merge; delete the head branch on merge.

‎LICENSE‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
MIT License
2+
3+
Copyright (c) 2026 FlowMatrix AI
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.
22+
23+
---
24+
25+
This licence covers the code in this repository: src/, tests/, pyproject.toml,
26+
uv.lock, the justfile and the CI configuration. The committed data outputs
27+
(outputs/computed/, outputs/exhibits/, outputs/METHODS.md) are released under
28+
CC BY 4.0; see LICENSE-DATA. The Inter fonts under outputs/inputs/fonts/ are
29+
under the SIL Open Font License; see the OFL.txt beside them.

0 commit comments

Comments
 (0)