Why Synthetic Data Matters: Stress-Testing Forecast Models Beyond Historical Limits

The Comfort of Backtests — and the Risk They Hide

It’s easy to feel confident when your model’s backtest looks good. The fit is clean, error metrics are low, and forecasts match up nicely against held-out data. In a world addicted to dashboards and KPIs, this looks like success.

But for anyone who’s been on the wrong side of a forecast, especially in financial markets, we know what comes next: the world changes — and your model doesn’t.

Most forecasting systems today are trained and validated on historical data. That’s not a flaw; it’s just the standard approach. But history, as we know, is only a partial teacher. It reflects what happened, not what could have.

If we rely exclusively on it, we risk building fragile systems that only look good in hindsight.

Real Data Is Biased Toward “Normal”

One of the hardest truths in forecasting is that the most important events are often the least represented in your dataset.

Tail risks, structural breaks, liquidity shocks, energy crises — they might happen once in a decade, or once in a generation. But when they do, they dominate outcomes.

And yet, your model likely hasn’t seen them. Not enough times to generalise. Not in enough forms to build meaningful intuition.

This is especially problematic for time-series models, which often assume some form of continuity: that yesterday’s pattern says something about tomorrow. That’s fine—until the pattern breaks.

At that moment, accuracy becomes less important than adaptability. And most models are never tested for that.

Synthetic Data Is a Way to Ask Better Questions

Synthetic data isn’t a magic fix. But it is a powerful lens.

It gives us a way to create scenarios that haven’t happened yet, or haven’t happened often enough to train on. We can use it to build data that’s deliberately difficult: with fat tails, regime shifts, clustered volatility, and correlated shocks — all the stuff that tends to break models.

Using GAN-based methods like those proposed by Wiese et al. (2020), synthetic data enables you to simulate and train for the unthinkable. But we’re not limited to just price dynamics.

At Ahead, our models offer the flexibility to generate rich, domain-relevant scenarios by conditioning diffusion models across a wide set of variables — from macroeconomic indicators to market microstructure — and across broad cross-sections of the market. That means we can explore the behaviours of multiple asset classes, stress entire portfolios, and test allocation strategies under conditions that never occurred… but easily could.

In other words, we’re not just generating noise. We’re engineering risk. And that’s a very different proposition.

We often think of synthetic data as something to plug into a pipeline. But really, it’s a way to interrogate that pipeline. To see how your forecasting system responds not just to more data — but to data with sharper edges.

You Can’t See Fragility Until You Create Friction

One of the most telling experiments we’ve run involved stress-testing forecasting models on synthetic financial data that mimicked crisis regimes.

These were models that had been validated — cleanly — on M4-style datasets. But when we ran them against synthetic sequences that included sudden spikes in volatility, jumps in return distribution, and autocorrelated shocks, their performance dropped significantly. Calibration broke. Confidence intervals widened. And in some cases, point forecasts completely missed the directional move.

Now, this isn’t about blaming the model. It’s about realising that standard evaluation methods don’t tell you the full story. You need new stress environments to surface new vulnerabilities.

The KDE plot below shows how forecast errors diverge between real and synthetic conditions. The right tail — where the synthetic data lives — is precisely where the model becomes least reliable.

Resilience Needs Rehearsal

No one builds an aircraft without wind tunnel testing. No one deploys a power grid without blackout simulation. In every high-stakes system, we assume things will go wrong — and we simulate failure before it happens.

Forecasting deserves the same treatment.

The world we’re forecasting into isn’t a continuation of the past. It’s a moving target. Monetary policy regimes change. Energy markets restructure. Climate events disrupt supply chains. Digital assets shift correlations in unexpected ways.

You can’t wait for those events to appear in your data. By then, it’s too late.

Synthetic data lets you rehearse volatility, not just react to it. It doesn’t predict the next crisis — but it makes sure your model isn’t seeing one for the first time when it happens.

A Shift in Mindset: From Prediction to Preparation

If there’s one thing synthetic data teaches you, it’s that good performance on a stable dataset doesn’t mean your model is robust. It might just mean your test was too easy.

What we need is a shift in how we evaluate forecasting systems — not just in terms of metrics like MAE or MAPE, but in terms of questions like:

• What happens when the regime shifts?

• Where does your model’s uncertainty grow fastest?

• How often is your confidence misplaced?

  
  

These are the questions synthetic data helps answer.

And increasingly, they’re the questions investors, clients, and boards want answers to.

Closing Thought: Risk Lives in the Edges

At its core, synthetic data is about imagination — but grounded in statistical structure.

It lets you ask: What would my model do if the world changed in this specific way?

And then: Am I okay with that answer?

If you’re building models that influence investment, allocation, or policy decisions — it’s not enough to know what works most of the time.

You need to know what breaks it, and how.

That’s what synthetic data is for.

References

• Ovadia, Y. et al. (2019). Can You Trust Your Model’s Uncertainty? NeurIPS. arXiv

• Wiese, J. et al. (2020). Generating Realistic Financial Time Series with GANs. arXiv