How to Read a Backtest Honestly: The Checklist

A four-item checklist for evaluating any backtest result: tails, regime concentration, sample size, and costs. Run it against every result, your own most of all.

A backtest ends with a set of numbers: a mean return, a hit rate, a distribution of outcomes. The temptation, especially when the result supports a thesis you already wanted to be true, is to anchor on the mean and treat it as the answer. That is how a 4 percent mean cumulative abnormal return over 60 days becomes "the strategy returns 4 percent every two months," a sentence with no information in it.

This post is the antidote: a fixed, four-item checklist you can run against any result, whether it came out of your own notebook, a paper, or a vendor's pitch deck. One scoping note first. The checklist assumes the study was built on clean inputs. If the universe silently excluded delisted names or the signal peeked at data it could not have had, no amount of careful reading fixes it; those failures are covered separately in survivorship bias, delisting bias, and look-ahead bias. What follows is the next stage: the numbers were computed correctly, and now you have to decide what they mean.

Item 1: Tails

A positive mean can hide fatal tails. The mean tells you where the center of the distribution is; it says nothing about what the worst outcomes cost, and the worst outcomes are what remove you from the game.

So never read the mean alone. Read five numbers together:

• Mean: the center of the distribution.
• Median: whether the center is being propped up by outliers in one direction.
• Hit rate: the fraction of events that went your way, which is the shape of your day-to-day experience.
• Dispersion: the standard deviation of outcomes, which is how variable trade-to-trade life will be.
• t-statistic: roughly how many standard errors the mean sits from zero, which is how likely the effect is real rather than luck.

Two concrete results make the point. A 6 percent mean with a 35 percent hit rate and a t-statistic of 1.4 is almost certainly a few extreme outliers wearing a strategy costume; stress the sample slightly and it dissolves. A 0.8 percent mean with a 70 percent hit rate and a t-statistic of 4 is in many ways more interesting: the magnitude is smaller, but the consistency is real. A result worth trading typically has all five numbers in coherent agreement: mean meaningfully different from zero, median the same sign as the mean, hit rate above 50 percent, and a t-statistic of at least 2.5 or 3 on a sample large enough to mean something. A suspicious result has some of these features and not the others.

Item 2: Regime concentration

Split every result by period and look at where the profit actually lives. An edge that shows up across years and across market conditions is a property of the event you are studying. An edge whose entire profit sits inside one hot stretch, one panic, or one bubble year is a property of that regime, and regimes end without filing notice.

This is not a hypothetical failure mode; it is arguably the most common one. A strategy averaged over five years can owe its whole mean to six months of them. The mechanical check is simple: group events by year, or by the risk regime they occurred in, and recompute the mean per group. If the sign flips or the effect vanishes outside one group, you have not found an edge. You have found a memory of a particular market.

Item 3: Sample size

Eight events is a story. Three hundred events is a distribution.

Small samples fail you twice. Statistically, the standard error of a mean shrinks with the square root of the sample size, so a mean computed over a handful of events is compatible with almost any true effect, including zero and including the opposite sign. Psychologically, small samples are where narrative sneaks in: with eight events you remember each one, you can explain each one, and the explanations feel like understanding. They are not evidence.

There is no magic threshold, and the honest posture is a sliding scale. Tens of events support a hypothesis worth pursuing. Hundreds support a measured effect with a believable confidence interval. When an event class is genuinely rare, the discipline is to say so and size positions accordingly, not to dress eight anecdotes in a decimal point.

Item 4: Costs and borrow

Every number you report should be net of what it costs to be in the trade. For long strategies that means commissions, bid-ask spreads, and slippage, the gap between the price you modeled and the price you actually got. For short strategies it also means the borrow: the daily fee you pay to hold someone else's shares, which for the distressed names event studies love can run from negligible to triple-digit annualized rates.

The procedure is straightforward: for each event, look up the borrow rate over the holding window, multiply by the day count, subtract from the position return. The subtle trap is missing data. Names with no recorded borrow rate are often missing precisely because they were hard to borrow and expensive, so dropping them silently flatters the result. The conservative fix is to assume the universe median rate plus a premium for the missing names, and keep them in the sample. A gross-of-cost result is not a smaller version of the truth; it can be the opposite sign of the truth. Plenty of published event effects are real, statistically robust, and worth exactly nothing after the borrow.

The whole checklist in one place

• Tails. Read the mean, median, hit rate, dispersion, and t-statistic together. Coherent agreement or it does not count.
• Regimes. Split by period. Profit concentrated in one regime is a memory, not an edge.
• Sample size. Eight events is a story; three hundred is a distribution. Scale your confidence to the count.
• Costs. Report net of commissions, slippage, and borrow. Handle missing cost data conservatively.

A result that survives all four items is evidence. A result you have not attacked is a hope with a decimal point. The order matters less than the habit: the person your backtest is most likely to fool is you, because you are the one who wants it to work. Reading a result honestly is the second half of a skill whose first half is knowing what counts as a real edge in the first place: a mechanism tells you why an effect should exist, and this checklist tells you whether it actually does.

Practice it against real results

This checklist is taught, and then repeatedly enforced, in Systematic Trading with Market Data, the interactive course from the quant behind Alphanume Research and The Quant Galore. The lesson this post condenses, Reading a Result Honestly, sits in the research-methods module, and every study in the later modules is graded against it by name. When a result in the course looks weak on one of the four axes, the lesson says so, in those words.

The lessons run real Python against real market data in the browser, with no videos and no setup. The first module is free with no account needed; start with the first lesson or browse the full syllabus, and see pricing for access to the rest.