Riskline — Telling real mortgage defaults apart from pandemic payment pauses

Riskline looks at real U.S. home-loan records and answers a deceptively simple question: which loans actually went bad? Getting that right turns out to be the whole game — because during COVID, the usual way of counting "bad loans" is badly wrong. Once the number is fixed, Riskline uses it to make lending decisions: who to approve, and what to charge.

---

The finding, in one paragraph

A "bad loan" (a default) normally means the borrower fell far enough behind that the lender expects a loss. The common shortcut is to call any loan that falls 90 days behind a default. By that shortcut, 4.6% of 2019 home loans look bad. But most of those borrowers did nothing wrong: during the pandemic, the government let millions of homeowners pause their payments (a program called forbearance), and almost all of them started paying again and fully caught up. Once you separate those pauses from real defaults, the true rate is 0.29% — about 16 times lower. A lender using the shortcut would treat 2019 borrowers as far riskier than they really were, and overcharge them.

The usual shortcut	After the fix
4.6% of 2019 loans look "bad"	0.29% actually went bad

What's really going on

Think of a loan's "days late" counter. Normally, if it climbs to 90+ days, something has gone wrong. But under the pandemic relief program, a homeowner could pause payments with no penalty — and while paused, that same counter climbs to 90 and 120 days anyway, even though nothing bad happened. When the pause ended, most people resumed and caught up with no loss.

So a model that naively counts "90 days late = bad" learns that 2019 was almost a crisis year. It wasn't. Riskline fixes this by reading the specific fields in the data that mark a pandemic pause, and separating a loan that truly defaulted from a loan that was paused and recovered. That corrected number is the foundation everything else is built on — the risk model, the pricing, and the fairness check.

What the data covers

Riskline uses real loan records from Freddie Mac (a government-sponsored mortgage company that publishes anonymized loan data). It looks at five sample years chosen to cover very different conditions:

• 2006 and 2007 — the run-up to the housing crash (lots of real defaults)
• 2013 — a calm, healthy period
• 2018 and 2019 — the pandemic window

That's about 250,000 loans and 6 million monthly payment records. Each loan is also matched to the economic conditions when it was made — unemployment, house prices, and mortgage rates.

What we found

The pandemic gap is real and measurable. The two ways of counting agree in every period except 2018–2019. That gap is exactly the paused-and-recovered loans — proof it's a pandemic effect, not real risk.

When loans go bad tells you why. Loans from the 2007 crisis went bad fast — a sign of weak lending. The 2019 loans only "went bad" late (12–24 months in, which lands in 2020–21) — the tell-tale fingerprint of pandemic pauses, not bad loans. The 2007 crisis year was about ten times worse than the calm 2013 year.

One month stands out across twenty years. The share of loans slipping from on-time to 30 days late sits near 0.6% for two decades — then spikes above 3% in April 2020 before falling back. That single month is the biggest jump in the whole record.

Ranking risk and pricing risk are two different skills. An early version of the model correctly ranked which loans were riskier, yet still predicted the level of risk about 55× too high. Getting the order right isn't the same as getting the number right — so we check both, separately.

A housing downturn hurts through severity, not frequency. If we simulate a recession, the number of defaults barely moves — but the loss on each one jumps (because homes sell for less). That's the correct way a housing shock does its damage.

The lending decision is fair. Grouped by neighborhood, the approve/decline decision passes the standard U.S. fair-lending test (the "four-fifths rule") with room to spare. The small remaining gap tracks a genuinely higher default rate in those areas, not bias in the model.

The full six-page Power BI report is at powerbi/riskline_portfolio.pbix (PDF: powerbi/exports/riskline_portfolio.pdf); the other pages cover risk by borrower group and by state.

Does the model work?

We trained the model on old years (2006, 2007, 2013) and tested it on years it had never seen (2018, 2019) — the honest way to find out if it works on the future.

In plain terms: Riskline's model sorts safe loans from risky ones better than the traditional banking scorecard, and — just as important — its predicted risk levels match what actually happened. It's both well-ordered and accurate.

Technical scorecard (for the curious)

Tested out-of-time on 2018–2019:

Measure (plain meaning)	Old approach (WOE scorecard)	Riskline (LightGBM, calibrated)
AUC — separates risky from safe (0.5 = coin flip, 1.0 = perfect)	0.721	0.745
KS — same idea, different scale	0.346	0.383
Calibration error — do predicted %s match reality (lower is better)	0.004	0.005 (must be ≤ 0.02)
Expected loss at 85% approval		20.4 bps (0.20%)
Approval rate within a 0.30% loss budget		95%

Loss when a loan does go bad averages 52% of the balance, computed from 7,580 real crisis-era losses. Break-even interest rates run from 7.5% (safest borrowers) to 8.8% (riskiest). Charts below come from the Python pipeline.

How it's built

Real data flows through a standard pipeline and ends in a dashboard and a set of lending decisions:

Freddie Mac loan data  +  economic data (unemployment, house prices, rates)
        |
        v  Spark  —  clean and organize
   Bronze  raw records, with automatic quality checks
        v
   Silver  the corrected "real default vs pandemic pause" label
        v
   Gold    the analysis tables (trends, risk by group, by state, payoffs)
        |            |
        |            v  BigQuery + dbt  (organized SQL, 24 automated checks)
        |            v  Power BI  (six-page report)
        v
   Model   traditional scorecard vs. a modern, calibrated model, with reason codes
        v
   Decide  who to approve, what to charge, recession stress test, fairness check

Layer	Tool	What it does
Processing	PySpark, Delta Lake	Builds the loan table and the corrected labels
Warehouse	Google BigQuery, dbt	Organized SQL from raw to report-ready, with tests
Dashboard	Power BI	Six-page report a non-technical reader can browse
Modeling	scikit-learn, LightGBM	Old-school scorecard vs. a modern, calibrated model
Explainability	SHAP	Plain reasons for each approve/decline (required by law)
Tracking	MLflow	Records every model run
CI	GitHub Actions	Runs the tests automatically on every change

The whole thing can be built and tested with no real data and no accounts, using a built-in generator that produces look-alike files with known patterns planted in them. There are 27 Python tests and 24 database tests.

Try it (offline, no cloud needed)

python3.10 -m venv .venv && source .venv/bin/activate
export JAVA_HOME=/opt/homebrew/opt/openjdk@17   # Spark needs Java
pip install -e ".[dev]"
make test        # 27 tests
make smoke       # end-to-end check through Spark
make silver      # build the loan table + corrected labels
make gold        # the analysis tables
make model       # old scorecard vs. modern model
make decision    # approvals, pricing, stress test
make mlflow-ui   # browse model runs

The cloud path is make load-bigquery then make dbt-build (see dbt/riskline_dbt/README.md).

Getting the real data

This repo ships the code, not the data or any secrets (both are excluded by .gitignore; Freddie Mac's license forbids redistributing the data).

• Offline: the built-in generator makes look-alike files — run the make steps above with no data and no accounts.
• Real data: register at Freddie Mac's data portal, download the sample files into data/raw/sample_<year>/, add free FRED and U.S. Census API keys to .env (see .env.example), then make macro.
• Cloud: with your own free BigQuery sandbox, run make load-bigquery and make dbt-build.

More detail

docs/CASE-STUDY.md	Short write-up that leads with the finding
docs/BUILD-ANALYSIS.md	Full build account and phase log
docs/data_dictionary.md	Data layout, pandemic-relief fields, quality checks
docs/decisions/	Why Power BI, why BigQuery
PRODUCT-SPEC.md, BUILD-GUIDE.md	Requirements and build plan

A note on scope

The numbers above come from the five real Freddie Mac sample years. The code scales to the full dataset; the numbers would firm up at full scale. The warehouse is BigQuery (the plan originally assumed Snowflake — see ADR-002), and the Spark pipeline runs locally, not on Databricks.