An operations-intelligence stack built on 99,441 real e-commerce orders — and an honest account of what happened when eighteen forecasting configurations were asked to earn their keep.
Most ERP and BI projects stop at the dashboard. This one goes one step further and asks the question the dashboard cannot answer: does a better forecast actually cost less money? The answer here, measured rather than assumed, is no — and the machinery built to establish that is the point of the repository.
Eighteen configurations went into a rolling-origin backtest on weekly category demand: four neural-network variants (GRU and LSTM, absolute and ratio targets), six gradient-boosting variants, and eight statistical baselines and ensembles.
| MASE | simulated inventory cost | 95% bootstrap interval | |
|---|---|---|---|
| ensemble · Theta + MA(4) + naive | 1.289 | 168,135 | 155,579 – 186,314 |
| Theta | 1.340 | 177,296 | 163,950 – 194,511 |
| naive (repeat last week) | 1.416 | 180,826 | 163,388 – 204,959 |
| LightGBM | 1.561 | 202,859 | 185,592 – 235,965 |
| GRU | 1.939 | 254,333 | 222,989 – 293,985 |
| LSTM | 2.000 | 271,442 | 230,347 – 323,730 |
An average of three cheap statistical methods beat every learned model, on accuracy and on money. The deep-learning entries finished last, costing 51–61% more than the winner. Eleven of the eighteen configurations were worse than repeating last week's number, and nothing at all scored below MASE 1.0.
This is a statement about this dataset, not about model classes. The M5 competition — roughly 35,000× more training observations — was won by gradient boosting, with a 43-model LSTM ensemble in third place. Eighty-three weekly observations per category is not the regime where deep learning earns its complexity. Saying so precisely is more useful than pretending otherwise.
The first backtest was diagnosed rather than published. Bias analysis showed every learned model over-forecasting by 3–15%, which pointed straight at the regime change: the marketplace grew through 2017 and contracted through 2018, so models trained on absolute unit counts extrapolated a growth slope into a year that did not follow it. Three standard fixes were tried, each as a good-faith attempt:
- Ratio targets — predict next week ÷ this week instead of next week's level, removing the trend the model could overfit.
- Recency weighting — exponential sample weights at half-lives of 8, 16 and 26 weeks, so the 2018 regime dominates training.
- Ensembling — blending the learned models with the statistical ones.
Ratio targets helped the neural nets substantially (GRU: 1.94 → 1.77) and hurt LightGBM. Recency weighting did nothing beyond noise. The stat+ML ensemble landed third, behind two purely statistical ensembles. None of it closed the gap. That is the result, reported as it came out.
Six modules, three executed notebooks, one dashboard.
1 · Data foundation. Nine raw operational tables become a star schema — one fact table at order-item grain, four conformed dimensions. Preceded by a data-quality audit where every finding changes a downstream decision, not a tidy table nobody acts on:
- the payments table fans out 1.44 rows per order across instalments and vouchers, so revenue is recognised from
order_items, not from payments — this is the most common way to get a wrong revenue number out of this dataset; - the last 50 days of the extract sit below a quarter of a normal day's volume and decay to single digits, which is the export being cut mid-period rather than demand collapsing — the window is truncated at both ends and the code says why;
- the median SKU sells one unit in two years, so per-SKU forecasting is a sample-size problem, not a modelling problem — forecasting runs at category level and the long tail gets an intermittent-demand method instead;
- Olist mints a fresh
customer_idfor every order, so any retention analysis keyed on it reports a 0% repeat rate and quietly turns a marketplace into a one-shot business.
2 · Forecasting bake-off. Naive, 4-week moving average, Theta, damped Holt-Winters, LightGBM (global, cross-learning across series), GRU and LSTM (global, PyTorch, per-series embeddings), plus ratio-target and recency-weighted variants and four ensembles. Rolling-origin expanding-window backtest scored with MASE, RMSSE, WAPE and bias — not MAPE, which is undefined at zero, asymmetric, and does not aggregate.
3 · Inventory economics. The bridge from error to money. Each forecast drives a periodic-review order-up-to policy; the service level comes from the newsvendor critical ratio; the simulator charges holding cost on stock and stockout cost on unmet demand. Two methodological corrections matter here:
- safety-stock sigma is scaled by an autocorrelation-adjusted aggregation factor rather than the naive √n rule, because forecast errors in this panel are serially correlated;
- the cost comparison is a 200-replication block bootstrap with 95% intervals, not a point estimate. The first version of this analysis flipped its ranking when sigma changed by 30%. A ranking that fragile is not a finding, so it was rebuilt until it was one. An eight-scenario sensitivity sweep — thin and fat margins, no goodwill, tripled goodwill, one- and four-week lead times, doubled carrying rate — leaves the winner unchanged in every scenario.
4 · Revenue and leakage forensics. Concentration (18 of 74 categories carry 80% of revenue), cohort retention on the correct customer key, a seller scorecard, and two robust anomaly detectors on price and freight. The delivery-delay analysis is the one to read carefully: the crude gap between late and on-time review scores is −2.03 stars, but heavy, expensive, far-shipped items are both more likely to be late and more likely to disappoint for unrelated reasons. Re-estimated inside 256 strata of price band × weight band × destination state it comes back at −1.93 — only 4.7% of the headline was confounding. It survives, and it is still observational, and the module says so.
The anomaly detectors carry a scale floor on the median absolute deviation. Without it, a product whose price never moves has a MAD near zero, and the first version of this detector returned z = −4,092 on a routine 33% discount.
5 · Governed analytics agent. Natural-language questions routed against a metric registry — six metrics, four dimensions, each with an owner and a definition — rather than free text-to-SQL. Three gates stand in front of every answer: coverage, ambiguity, and filter resolvability. A question that fails any gate is refused with a reason.
| evaluation set | n | correct | refused | wrong & confident |
|---|---|---|---|---|
| design set — answerable | 16 | 100% | 0% | 0% |
| design set — ambiguous & out of scope | 10 | — | 100% | 0% |
| paraphrase stress test — out of vocabulary | 8 | 12.5% | 75% | 0% |
The third row is reported rather than hidden, because the failure mode is the one worth having. On phrasing the registry has never seen ("what's the top line?"), the agent refuses rather than guesses. A text-to-SQL system with the same vocabulary gap would have produced eight confident, wrong answers. The fix is registry coverage or an embedding layer — not a claim that the gate design is finished.
An earlier version of this agent scored 37.5% execution accuracy with an 80% false-confidence rate, because TF-IDF fitted over metric descriptions leaked vocabulary between unrelated metrics — "what was total revenue?" routed to freight_share, whose description happens to contain the word "revenue". The retrieval layer was rewritten to literal synonym-phrase matching.
6 · Executive dashboard. A single self-contained HTML file, in dashboard/, generated from the same cached analysis artifacts the notebooks produce. Nothing on it is typed by hand.
01_foundation_and_forecasting.ipynb |
Audit, star schema, the full model ladder, the regime-change diagnosis and the fair rematch |
02_inventory_economics.ipynb |
Newsvendor service levels, policy simulation, block bootstrap, sensitivity sweep |
03_revenue_and_governed_agent.ipynb |
Concentration, cohorts, delivery causality, anomaly detection, the metric registry and its evaluation |
Each notebook is self-contained: the src/ package is embedded so they run on Kaggle with no repository checkout. Each ends with numbered findings and an explicit limitations section.
pip install -r requirements.txt
python scripts/download_data.py # 8 CSVs, ~45 MB
jupyter lab notebooks/src/data.py star schema + data-quality audit
src/metrics.py MASE, RMSSE, WAPE, bias, pinball loss, coverage
src/forecast.py the model ladder, backtest harness and scoring
src/inventory.py cost model, policy simulation, bootstrap, sensitivity
src/revenue.py concentration, cohorts, delivery impact, anomalies
src/semantic.py metric registry and the governed agent
src/viz.py one visual system across notebooks, charts and dashboard
One marketplace, twenty months, one country. Holding and stockout rates are modelled assumptions, not booked costs — the sensitivity sweep shows the ranking is robust to them; it does not turn the currency figures into audited savings. The delivery-impact estimate is observational, not a randomised experiment. The agent's vocabulary is narrow and measured as such.
Data: Brazilian E-Commerce Public Dataset by Olist (CC BY-NC-SA 4.0).