A demonstration you can run yourself
Below is a search over a market that contains no signal at all — a random walk with zero drift, built in your browser. Each strategy is a different way of trading that noise. Watch the best one get better.
Every curve you just watched was generated from the same worthless data. There was no edge to find, because the price series was drawn from a coin flip. And yet a few hundred attempts produced a strategy with a Sharpe ratio most people would call publishable, and one or two would call a career.
This is not a trick of the simulation. It is arithmetic. Run 190 configurations over three years of noise and the median outcome — not the lucky tail, the median — is a best-of-grid Sharpe near 1.14. Half of all such searches do better than that on data with nothing in it.
The number that misleads you is not the Sharpe ratio. It is the number of things you tried before you kept one, and that number never appears in the backtest.
The deflated Sharpe ratio asks a single question: given how hard you looked, how good would the best result look if nothing worked? Then it asks whether yours beat that, and by enough to matter. Put your own backtest in.
Count honestly: every parameter combination, every universe, every time you looked at the result and went back to change something. Nobody ever tried it once.
A Gaussian has kurtosis 3. Negative skew and fat tails mean the Sharpe ratio is measured less precisely than it looks, so the same number proves less.
One assumption you cannot see: the Sharpe ratios across your attempts are taken to have spread 0.5 annualised, the value used in the source paper’s own example. A grid whose configurations differ more than that raises the noise threshold and makes the verdict harsher than what you see here.
The defaults are the worked example from the paper that introduced the statistic: a 2.5 Sharpe over five years, found after 100 attempts, on returns with the negative skew and fat tails a real strategy has. It does not clear the bar. The paper's own conclusion is that this discovery would have had to come after no more than 46 independent attempts to count as evidence — and the calculator puts the breakeven at 47. Drag the attempts slider to 250 and a career-making number falls to 0.81.
Prediction ledgers fail differently. If you forecast 40 tickers every morning, a year of work looks like 10,000 calls. It isn't. When the market rallies, most of your "up" calls are right at the same time, for the same reason. Those 40 rows are one observation wearing forty hats.
How much the calls you make on one morning stand or fall together. For equity direction it is rarely below 0.2, because every name shares the market factor.
There is a third trap this page does not simulate, and it is the most common of all: a hit rate above 50% in a market that rose 58% of the time. A forecaster who says "up" every single day scores 58% while carrying no information whatsoever. The fix is the Pesaran–Timmermann test, which compares your hit rate against the one your own directional bias earns for free. The command-line tool runs it; a spreadsheet won't.
This page is the argument. luckfilter.py is the instrument — one file,
no dependencies, no install. It runs the demonstration above on real data and prints a
verdict card.
python3 luckfilter.py --demo # mine noise, watch it die
python3 luckfilter.py returns.csv --trials 200 # one series, declare the search
python3 luckfilter.py grid.csv # many columns, many candidates
python3 luckfilter.py calls.csv --directional # a ledger of up/down calls
What it runs, and where each test comes from:
| Test | Answers | Source |
|---|---|---|
| PSR | Is the Sharpe above zero, allowing for skew and fat tails? | Bailey & López de Prado 2012 |
| DSR | Is it above what the best of N worthless attempts would score? | Bailey & López de Prado 2014 |
| MinTRL | How long a record would settle it? | Bailey & López de Prado 2012 |
| PBO | Does picking the in-sample winner generalise, or is it a coin flip? | Bailey, Borwein, López de Prado & Zhu 2016 |
| Reality Check | Could a search this wide produce a winner this good by chance? | White 2000 |
| SPA | Same question, robust to padding the search with junk. | Hansen 2005 |
| Pesaran–Timmermann | Is the hit rate better than your own bias earns for free? | Pesaran & Timmermann 1992 |
| Henriksson–Merton | The same, exactly, when the sample is small. | Henriksson & Merton 1981 |
| Design effect | How many independent observations do you really have? | Kish 1965; Donner 1986 |
| Murphy decomposition | Do your stated confidences mean anything? | Murphy 1973 |
A warning that applies to the whole enterprise. Passing these checks does not mean a strategy works. It means one specific way of being wrong has been ruled out. The deflated Sharpe in particular is only as honest as the trial count you feed it, and that number lives in your memory, not in your data.