Backtesting & Validation
How Backtesting Works, And What It Can't Tell You
Backtesting replays a strategy against historical data. What the process does, why results look better than they were, and how to read a report honestly.
3 min read
All topics
Hot Topic
Backtesting & Validation
Testing a strategy against history: how engines work, why results flatter, and what validation can and cannot establish.
Definition
Trending- What is backtesting & validation?
- Backtesting is the process of running a trading strategy against historical market data to measure how it would have performed, which establishes whether a rule set had an edge in the period tested but not whether that edge persists.
2 pieces published on this topic · Updated
Understanding backtesting & validation
A backtest answers one narrow question well: given this exact rule set, applied to this exact historical period, with these assumptions about costs and fills, what would have happened? Every word in that sentence is carrying real weight, and most disappointing live results trace back to one of them having been quietly ignored.
The value is real but negative in character. A backtest is far better at eliminating strategies than at endorsing them. A rule set that fails over ten years of data has been usefully ruled out. A rule set that succeeds has passed one test, on data that already exists, in a period that will not repeat. This is a much weaker statement than it feels like.
The failure modes are well documented and mostly structural rather than mathematical: lookahead bias, where a rule reads information it could not have had; survivorship bias, where delisted instruments are missing from the data; and overfitting, where a strategy has been tuned until it describes the noise in one particular history. None of these show up as an error. They show up as good results.
Common questions
- How much history should a backtest cover?
- Enough to include conditions unlike the present, at minimum one significant drawdown and one period of low volatility. A strategy tested only across a rising market has been tested against one regime, however many years that regime lasted.
- What does a good backtest result actually prove?
- That a rule set had an edge in the period tested, under the stated assumptions. It does not establish that the edge exists outside that period, and the more parameters were tuned to produce the result, the weaker even the original claim becomes.
- Why do live results usually fall short of backtested ones?
- Most often costs and fills. Backtests commonly assume execution at a price the strategy could not have traded at, and model transaction costs as a flat percentage when several Indian charges are charged differently on a buy than on a sell. Both errors flatter frequent trading specifically.
Guides
Blog posts covering this topic.
Technical articles
Engineering deep-dives covering this topic.
Related topics
Put this into practice
Describe a strategy in plain language, backtest it against historical data, and paper trade it before any capital is committed.