Does forecasting CPU spikes help cloud provisioning, and how would you know?
Agulló et al. (2025, Future Generation Computer Systems 170, 107833) propose changing what a forecaster is trained to predict so that it sees CPU spikes coming, and report that this doubles spike-prediction scores. SpikeCheck checks how those scores are computed, rebuilds one part of the experiment on the public Alibaba 2018 cluster trace, and tests what makes the method work. That part is Strategy 1 with the TempInc model. The paper's Google data, other five models and Strategy 2 weren't run (scope and why).
Read the project story: the problems, decisions and fixes behind SpikeCheck, in SOART format.
- Check the paper's metrics. Its over-provisioning metric divides by about 1e-10, and its spike precision rewards forecasts that lag (review, ADR-001).
- Rebuild the data: stream the 29 GB Alibaba container trace into 5-minute steps and keep 5,946 long-running containers (ADR-005).
- Train the authors' model on different targets. The model, inputs and data stay the same; only what it's asked to predict changes (ADR-004).
- Score fairly. Allocations take effect after a 5, 15 or 30-minute delay (ADR-002), and forecasters are compared at equal over-provisioning, with three seeds each (ADR-003).
- Split the spikes by type: seasonal, follow-on, isolated, or unknown when there isn't a day of history (ADR-009).
- Test whether the hold needs to be learned, against a rule fixed before a single test run (ADR-010).
| Finding | Evidence |
|---|---|
| The paper's headline reproduces on its own evaluator | Spike F1 0.41 → 0.68 for the TempInc model (paper: 0.27 → 0.64) |
| Its over-provisioning numbers are an artefact | Every forecaster scores 10¹⁵ to 10¹⁷, even repeating the last value |
| With no actuation delay, a plain classifier read at an upper quantile is as good or better | Same recall as the proposed method with 14 to 29% less over-provisioning, across three seeds |
| With a 15-minute delay, the proposed method wins, because its target looks ahead | Spike coverage 0.57 against 0.43 at about the same over-provisioning |
| Overall, looking ahead is what helps and holding the allocation after a spike hurts | A target that only looks ahead beats the paper's at every budget, in all three seeds: +8 points of coverage and 18% less throttling at slack 0.6 |
| The look-ahead has to be at least as long as the delay | A 30-minute look-ahead covers 0.74 of spikes at a 30-minute delay against 0.59 for the 25-minute one (slack 0.8, all three seeds), with no cost at shorter delays |
| One overall score hides where each method gains | At a 15-minute delay and slack 0.8, the look-ahead beats the paper's target on seasonal (+0.030) and isolated (+0.045) spikes and trails it on follow-on spikes (−0.084). The follow-on result depends on the 6-step definition |
| Computing the hold instead of learning it passes three of four pre-set checks, so the complete hypothesis isn't supported | Computed minus learned at slack 0.8: overall +0.011 and +0.008 and throttled CPU −5.4% and −5.7% at 5 and 15 minutes. Follow-on −0.004 passes at 5 minutes, and −0.008 fails the 1-point margin at 15. No hold still has the best overall coverage |
The details and every caveat are in the replication write-up.
The replication exposed limitations in the paper's evaluation. In the computed-hold test, calculating the known part of the target and forecasting only the future gave higher overall coverage and less throttled CPU than learning the whole target, at a small follow-on cost that I can't rule out exceeding my margin at 15 minutes. I haven't established that follow-on protection depends on actuation delay, or that using a hold is better than having no hold.
The full conclusion covers what was reproduced, where the evidence stops and what can't be claimed.
| Part | State |
|---|---|
| Review of the paper | Done |
| First replication (paper reproduced, quantile readout) | Done |
| Horizon vs hold, three seeds | Done |
| 30-minute look-ahead, three seeds | Done |
| Rebuild: every result regenerated from the committed scripts | Done |
| Spike types: definitions fixed before any scores, three seeds, six label settings (ADR-009) | Done |
| Computed hold: rule fixed first, test run once (ADR-010) | Done |
| Trace-driven autoscaler simulation | Next |
| Strategy 2 (altered loss), calibration, time and cluster holdouts | Backlog |
Six diagrams in docs/architecture.md: the data pipeline, how a forecast becomes a score, the paper's metrics against the corrected ones, which time steps each target covers, the experiment grid, and the TempInc model.
flowchart LR
T[("Alibaba 2018 trace<br/>29 GB")] --> S["stream + prep<br/>5,946 traces"]
S --> M["authors' TempInc model<br/>one per target and seed"]
M --> E["corrected metrics<br/>+ authors' evaluator"]
E --> R["results/"]
| ADR | Decision |
|---|---|
| 001 | Score with corrected metrics, and keep the authors' evaluator alongside |
| 002 | Score forecasts under actuation delay |
| 003 | Compare at equal over-provisioning, with seed-paired differences |
| 004 | Change one thing at a time: only the target varies |
| 005 | Rebuild the preprocessing, keeping the authors' choices where known |
| 006 | Keep the data and the authors' code out of the repo; pin their commit |
| 007 | Date commits by when each file's content actually existed |
| 008 | Publish only results the committed code can regenerate |
| 009 | Define mutually exclusive spike types |
| 010 | Compute the hold instead of learning it, tested against a rule fixed first |
scripts/ pipeline: stream_alibaba, prep, train, baselines, metrics, evaluation, spike types, computed hold, runners
results/ replication/, horizon/, horizon30/, spike_types/, computed_hold/: CSV, JSON, plots
docs/ review, replication write-up, architecture diagrams, project story, ADRs
data/ downloaded and generated data, models (not committed)
upstream/ the authors' code at a pinned commit (not committed)
LOG.md one line per session
NEXT.md the next single task
BACKLOG.md parked ideas
LICENSE MIT, for the code
LICENSE-docs CC BY 4.0, for the docs and results
Python 3.12, about 1 GB of free disk, and a few hours.
scripts/setup.sh # authors' code at commit 98f0d51, plus packages
scripts/get_data.sh # stream and prepare the trace (about 2.5 h at 4 MB/s)
scripts/run_all.sh # first replication, about 1.5 hThe horizon experiments use scripts/run_seeds.sh and scripts/overnight.sh; see the replication write-up. Training a model takes about 30 minutes and up to 2.5 GB of memory on an 8 GB laptop.
- Agulló, F., Gutierrez-Torre, A., Torres, J., Berral, J. Ll. (2025). Enhancing the output of time series forecasting algorithms for cloud resource provisioning. Future Generation Computer Systems 170, 107833. https://doi.org/10.1016/j.future.2025.107833
- Authors' code: https://github.com/FerranAgulloLopez/ResourceForecasting (Apache-2.0)
- Alibaba cluster trace 2018: https://github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018
- The code is under the MIT License (LICENSE).
- The docs and results are under CC BY 4.0 (LICENSE-docs). Reuse them as long as you credit the source.
- Two things aren't covered.
results/replication/chosen_containers.csvcomes from the Alibaba 2018 trace, which Alibaba publishes without a licence, so it stays under Alibaba's terms. The authors' code, whichscripts/setup.shdownloads, stays under its own Apache-2.0 licence.