Skip to content

About

Python data fetchers for KeynesWatch macroeconomic dashboards, collecting U.S. and China economic, fiscal, monetary, and labor-market data from public sources into MySQL.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

Keynes Watch — Data Pipeline

Keynes Watch collects U.S. and Chinese macroeconomic data into MySQL tables and local CSV files. This is the standalone data pipeline that powered keyneswatch.com, released under the MIT License. It contains no Flask application, charts, or website deployment configuration; the gallery below shows what the site built from these tables.

项目安排: keyneswatch.com 网站已下线;本仓库保留为可独立运行的数据采集项目。 需要最新数据的使用者,请配置自己的数据库、API 密钥并运行或定时执行采集程序。 网站各页面的截图见下文画廊。

The hosted website has been taken offline. This repository lets you run the pipeline yourself; it does not provide a hosted data feed or promise continued updates to upstream integrations. Updates run only when you invoke the fetchers or configure your own scheduler. Historical coverage and availability depend on each source.

The project was created by Chenning Xu. The September 2026 synchronization is described in the comparison and migration notes.

What the data looked like on keyneswatch.com

The website rendered these tables as interactive Plotly pages behind a Flask app. The screenshots below come from a local run of that site in September 2026 against a database filled by this pipeline. Each caption names the table(s) the chart reads, so you can find the matching fetcher below.

Keynes Watch home page

United States

Monthly Treasury Statement
Calendar year-to-date deficit, one line per year. Table mts.

Monthly Treasury Statement, deficit YTD by year
Weighted average maturity
Face-value-weighted remaining maturity of marketable Treasury debt, computed by this pipeline from security-level MSPD records. Table treasury_average_maturity.

Treasury weighted average maturity since 2001
Kalecki profits equation
Corporate profits decomposed quarterly from NIPA into investment, government saving, foreign saving, dividends and personal saving. Table kalecki_equation.

Kalecki profits equation, quarterly
Sectoral financial balances
Private, government and foreign net lending as a share of GDP; the three sum to zero by identity. Table kalecki_equation.

Three-sector financial balances
Overnight rates: EFFR and SOFR
The SOFR percentile distribution and the effective funds rate inside the FOMC target range. Table onrates.

EFFR and SOFR percentiles within the target range
Reserve balances scaled by GDP
Reserve balances at the Fed divided by nominal GDP. Tables fred_wresbal and fred_gdp.

Reserve balances relative to nominal GDP
Treasury General Account
Daily closing balance, including the 2023 debt-limit drawdown and rebuild. Table tga_balance.

Daily TGA balance
Withheld taxes
Daily withheld income and payroll taxes, cumulative year-to-date and aligned by calendar position. Table withheld_tax.

Withheld taxes cumulative YTD by year
Treasury debt held by the public, share of GDP
Total public debt held by the public, from the MSPD, divided by nominal GDP. Tables total_debt_outstanding and fred_gdp.

Treasury debt held by the public as a share of GDP
Yield curve on two dates
Nominal constant-maturity curves overlaid for shape comparison. Table yield_curve.

Nominal yield curve on two dates
Statutory debt limit
Gaps in the line mark suspension periods. Table debt_limit.

Statutory debt limit with suspension gaps
Average interest rate on Treasury debt
The rate the Treasury pays on outstanding marketable securities. Table treasuries_average_yields.

Average interest rate on marketable Treasury debt

China

Interest rate corridor
SLF 7-day ceiling, IOER floor, 7-day reverse repo rate and the FDR007 fixing. Tables pboc_repo_fixing, pboc_omo and pboc_policy_rates.

China interest rate corridor
PBOC balance sheet
Asset side: FX reserves, claims on government and claims on depository institutions. Table pboc_balance_sheet.

PBOC balance sheet, asset side
Kalecki profits equation, China
The same identity rebuilt annually from the NBS flow-of-funds accounts. Table cn_flow_of_funds.

Kalecki profits equation for China
Sectoral financial balances, China
Private, government and foreign net lending as a share of GDP. Table cn_flow_of_funds.

Three-sector financial balances for China
Money supply
M1 and M2 growth with the M1 minus M2 gap shaded. Table pboc_money_supply.

M1 and M2 growth with the gap shaded
Total social financing by component
Monthly flow, stacked by instrument. Table pboc_social_financing.

Total social financing monthly flow by component
70-city house prices
New-home index year on year, averaged by city tier. Table nbs_house_price.

70-city house price index by tier
Land transfer revenue
Cumulative year-to-date proceeds from land-use-right sales, by year. Table mof_land_revenue.

Land transfer revenue cumulative YTD by year
Loan Prime Rate
1-year and 5-year-plus LPR. Table pboc_lpr.

Loan Prime Rate, 1-year and 5-year-plus
Required reserve ratio
Large and small-to-medium institutions, with the spread shaded. Table pboc_rrr.

Required reserve ratios for large and small institutions

Data sources

Source Main tables / files
fred Claims, payrolls, unemployment, CPI, GDP, reserve balances, Fedwire monthly statistics
bea BEA NIPA data for the Kalecki equation and three-sector balances
fiscal TGA balance, debt limit, Treasury outstanding, average maturity, average yields, Monthly Treasury Statement, withheld tax
nyfed New York Fed repo / reverse-repo operations and overnight rates
treasury Nominal and real Treasury yield curves
indeed Indeed Hiring Lab wage and job-posting CSV snapshots
pboc LPR, money supply, social financing, credit, reserve ratios, SHIBOR, policy rates, balance sheet, OMO, FR/FDR repo fixing rates
nbs Real estate climate, 70-city house prices, macro real estate indicators, annual flow-of-funds accounts, monthly retail sales
mof Land transfer revenue from monthly fiscal reports

The fetchers use official APIs and releases, AKShare adapters, and Indeed's published datasets. Some policy-rate history is maintained as seed values in the code. See the source modules for endpoints, units, and transformation rules.

Most sources create their own tables in an existing MySQL database. The indeed source writes files under fetch_data/github/ and does not need MySQL or an API key. Downloaded data is excluded from git; the small public examples under tests/fixtures/ are regression inputs, not a distributable dataset.

Setup

Use Python 3.11 (the tested version) and MySQL. From the repository root:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cp .env.example .env

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
Copy-Item .env.example .env

Create your database using a MySQL administrator account, for example:

CREATE DATABASE keyneswatch CHARACTER SET utf8mb4;

Configure your own database user with permission to read/write data and create or alter tables and indexes in that database. Populate .env with its connection details. Fetchers do not create the database or database user.

Variable Required for Description
DB_HOST Database-backed sources MySQL host
DB_PORT Optional MySQL port; defaults to 3306
DB_USER Database-backed sources MySQL user
DB_PASSWORD Database-backed sources MySQL password
DB_NAME Database-backed sources Existing database name
FRED_API_KEY fred Your FRED API key
BEA_API_KEY bea Your BEA API key
CN_PROXY Optional HTTP/SOCKS proxy URL for supported China fetchers

.env is ignored by git. No production credentials, certificates, database exports, or private server settings are needed.

Usage

List source groups and target tables without fetching data or connecting to MySQL:

python -m fetch_data.run --list

Run all sources, one source, or a matching table group:

python -m fetch_data.run
python -m fetch_data.run --source fred
python -m fetch_data.run --source fred --series claims
python -m fetch_data.run --source pboc --series repo_fixing

The first run can take considerably longer than subsequent updates. Run only the sources you need. A failed series causes the runner to return a nonzero exit code; other series still run. Re-running accepts revisions or resumes incremental downloads according to each source's strategy.

The optional Bash helper creates a virtual environment if needed and installs requirements when they change. It resolves paths from the checkout, so it can be run from any working directory:

bash scripts/update_all.sh
FETCH_SOURCES="fred bea fiscal" bash scripts/update_all.sh

It supports PROJECT_DIR, VENV_DIR, LOG_FILE, and FETCH_SOURCES overrides. Its console output is also saved to update.log by default. On Windows, use the Python commands above, or run the helper through Git Bash.

For unattended updates, configure your own cron job or Windows Task Scheduler task. For example, a daily cron entry (in the machine's local timezone):

0 9 * * * /bin/bash /path/to/Keynes-Watch/scripts/update_all.sh

Each source run records series outcomes, durations, and errors in logs/fetch_status.json. A filtered run preserves the previous results for untouched series; check each series' finished timestamp when assessing freshness. This file is local runtime state and is ignored by git. Schedule runs sequentially within a checkout; status-file locking is process-local.

Updating an existing database

Ordinary runs now re-read revised FRED observations, repair MTS fiscal-year mapping and Treasury maturity summaries, refill lagging NY Fed operation types, and correct PBOC zero-allotment notices to a NULL rate with zero volume. See migration notes for targeted commands.

NBS house prices use the newer city-price API with response-city validation; retail sales and flow-of-funds also use the newer data-release API. Official press releases provide fallbacks where implemented. NBS throttling can stop API requests for the remainder of a run; retry later instead of increasing parallelism. The national real-estate macro fetcher still uses the legacy easyquery endpoint with a press-release fallback.

Optional NBS full-history refreshes, in Bash:

NBS_HOUSE_FULL=1 python -m fetch_data.run --source nbs --series house_price
NBS_RETAIL_FULL=1 python -m fetch_data.run --source nbs --series retail_sales

In PowerShell, set $env:NBS_HOUSE_FULL = '1' (or NBS_RETAIL_FULL), run the corresponding Python command, then remove the temporary setting with Remove-Item Env:NBS_HOUSE_FULL (or Env:NBS_RETAIL_FULL).

Tests

python -m pip install -r requirements-test.txt
python -m pytest -q
python -m compileall -q fetch_data

Tests use curated public fixtures and isolated database/network substitutes; they do not load .env or update external data. CI runs these checks and CLI discovery without API keys or a live database. It does not deploy the website or schedule data collection. Tests verify parsing and update behavior, not the continued availability of every external endpoint.

License and repository scope

MIT applies to this project's code. Upstream data remains subject to its providers' terms. Keep local secrets, downloaded datasets, logs, certificates, and database dumps out of commits. The website code and its deployment history are maintained separately from this data-only repository.

About

Python data fetchers for KeynesWatch macroeconomic dashboards, collecting U.S. and China economic, fiscal, monetary, and labor-market data from public sources into MySQL.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages