Forty green trades in a row can still be luck, and thirty red ones can still hide a real edge. The only way to tell the difference is to stop staring at the equity curve and ask the boring statistical question: could these results have come from a system whose true expectancy is zero?
The null hypothesis: your edge is zero
Every honest evaluation starts by assuming the thing you want to disprove. Here the null hypothesis is that your true average R-multiple — profit or loss measured in units of initial risk — is exactly zero. In other words, you have no edge, and the profit you see is variance dressed up as skill.
Your job is not to prove you have an edge. It is to gather enough evidence that the “no edge” story becomes implausible. That framing matters because it forces you to account for the trades that went against you and the ones you got lucky on. A run of wins does not reject the null on its own; a distribution of results that sits convincingly above zero does.
Work in R, not dollars. If you risked the same fraction on every trade, a +2R and a -1R are directly comparable regardless of account size or instrument. That normalization is what makes the test valid across a mixed book of futures and FX. If you have never standardized your trades this way, start with the r-multiple primer.
The t-statistic on average R
The test is a one-sample t-test. You need three numbers from your closed trades:
- n — the number of trades
- mean R — the average R-multiple (this is your sample expectancy)
- s — the standard deviation of your R-multiples
The t-statistic is:
t = mean_R / (s / sqrt(n))The denominator, s / sqrt(n), is the standard error of your mean — how much your average R would wobble if you re-ran the same system on a fresh sample of trades. A large t means your mean is many standard errors above zero, which is hard to explain by chance.
Notice what drives t up: a bigger edge, a tighter spread of outcomes, and more trades. Two traders with the same average R are not equally convincing — the one with the calmer, more repeatable distribution has the stronger statistical case.
p-values and what significance really means
Convert t into a p-value: the probability of seeing results at least this good if the null were true and your edge really were zero. Since you only care about beating zero, use a one-sided test.
A common threshold is p < 0.05, roughly a t of 1.65 or higher for a decent sample. That does not mean “95% chance the edge is real.” It means: if you had no edge, you’d see numbers this favorable less than 5% of the time. That is evidence, not proof, and it says nothing about how big the edge is.
| t-statistic | one-sided p-value | reading |
|---|---|---|
| ~1.3 | ~0.10 | weak, keep sampling |
| ~1.65 | ~0.05 | conventional significance |
| ~2.33 | ~0.01 | strong |
| ~3.1 | ~0.001 | very strong |
Report the actual number alongside your mean R rather than a green/red verdict. A p of 0.04 and a p of 0.0004 are different worlds, and collapsing them into “significant” throws away information.
Sample size and the power to detect an edge
Power is the flip side of significance: the chance your test catches a real edge that genuinely exists. Small samples have low power, so a thin, wide-spread book can hide a true edge and return a non-significant result. Absence of significance is not evidence of no edge — it is often just too few trades.
As a rough feel: a small but real edge with typical trade-outcome variance needs a few hundred trades before the test reliably fires. A large, clean edge can show up in far fewer. Rather than chase a magic count, watch the trend — the standard error shrinks with the square root of n, so each doubling of your trade count only tightens the estimate by about 40%. Pair the t-test with an expectancy calculation so you are tracking effect size and significance together, not one in isolation.
Multiple-testing: why backtests overfit
Here is where most edges evaporate. If you test twenty parameter sets, one will clear p < 0.05 by pure chance even when none has an edge — that is what a 5% false-positive rate means. Backtest optimizers do exactly this at scale, quietly running thousands of variants and handing you the luckiest curve.
Defenses:
- Split your data. Fit on one period, confirm on untouched out-of-sample data.
- Adjust the threshold. With k tests, a crude Bonferroni correction divides your p-cutoff by k. Twenty variants means demanding p < 0.0025, not 0.05.
- Count every variant you tried, including the ones you discarded. The ones you don’t report are exactly the ones that inflate false positives.
A spreadsheet makes this easy to fudge because nothing forces you to log the failed experiments. If you have been optimizing in Excel, the honesty gap is worth reading about in the Shibiki vs. spreadsheet comparison.
From significant to tradable: the gap
Statistical significance is necessary, not sufficient. A significant edge can still be untradable once reality intrudes:
- Costs and slippage shave every R, and a thin edge can cross into negative after fills.
- Regime change means the sample that proved your edge may not describe the market you trade next month. Significance is backward-looking.
- A wide confidence interval on your win rate — the range your true rate plausibly sits in — can span from marginal to great even when the point estimate looks fine.
This is why Shibiki reports live edge health per strategy with a Wilson confidence interval rather than a single win-rate number. As trades accumulate, the interval tightens, and you watch an edge earn its confidence instead of assuming it from a lucky streak. Significance is the entry ticket. Durability, tested continuously against fresh trades, is what you actually risk money on. To go deeper on the metric your t-test is really measuring, read the trading expectancy explainer.
Related: Trading expectancy · Expectancy calculator · Shibiki vs. spreadsheet