Methods
How a grade is made
Every grade on this site comes from the open-source Embargo engine (version 0.1.0). The site only shows what the engine returns.
1. Count the tries
If you test 200 versions of an idea, the best one will look good even when none of them has an edge. The engine asks how many versions you tried and raises the bar to match: it compares your result with the best result that many no-skill strategies would typically reach. That comparison is the luck-corrected confidence (the deflated Sharpe ratio of Bailey and López de Prado).
2. Seal the last part
The final 30% of the data is set aside and checked on its own. In a Quick Check it is split after the fact, so it is a weaker test than a holdout that was sealed before any tuning.
3. Race random strategies
The engine reshuffles your own returns in blocks to build thousands of strategies with the same ups and downs but no skill, and counts how often they do as well as yours: random strategies produce a result this good X times in 100. With several columns it also checks whether the best version in development stays the best on fresh data (probability of backtest overfitting).
4. Look for leaks
Results that are too smooth, too often positive or implausibly strong trip a leakage check and get an F, because they usually mean the backtest saw the future.
5. Grade
| Grade | Meaning |
|---|---|
| A | Looks like skill |
| B | Promising |
| C | Could be either, leaning good |
| D | Could be either |
| E | Probably luck |
| F | Too good: check for leakage |
A grade is a method check on the file you uploaded. It does not show that any returns were earned and says nothing about the future. Quick Check reports are assurance level L0: the tries are self-declared.
How well the engine does its job
From the Embargo engine's published validation report. Each row is a large simulation where the right answer is known in advance.
It rarely calls luck skill
1,000 sets of no-skill strategies for each row, three years of daily data. How often the engine still said skill_supported (grade A or B):
| Tries declared | Wrongly judged skilled |
|---|---|
| 1 | 4.6% |
| 10 | 0.0% |
| 50 | 0.0% |
| 200 | 0.1% |
At one try the rate sits just under the 5% that a 95% confidence bar allows; counting the tries is what pushes it towards zero.
Strategies with no edge that were tuned 20 times in the engine's Lab: 0.0% judged skilled.
It catches leaks
Strategies built with a known look-ahead leak, and how often the engine's leakage checks flagged them:
| Planted leak | Caught |
|---|---|
| The outcome copied into a feature | 100% |
| Backfilled data | 99.6% |
| Centred moving average (looks ahead) | 98.4% |
| Trading on the same bar as the signal | 100% |
It finds real skill when there is enough data
Strategies with a real edge (true Sharpe ratio 1.5): how often the engine judged them skilled.
| Years of daily data | Judged skilled |
|---|---|
| 1 | 47% |
| 3 | 82% |
| 5 | 95% |
With one year of data, a real edge is missed about half the time: a C or D is often a call for more data, not a verdict on the idea.
Speed: a full report on 5,000 days × 200 strategies takes about 3.6 seconds.
The same checks on this site's 365 daily puzzles
Each daily puzzle hides one planted edge (true Sharpe 1.2) among the three luckiest of 10, 100 or 1,000 zero-skill monkeys. How the engine graded them:
| Monkeys | Luckiest looked A or B | Luckiest graded E honestly | Planted edge graded A or B |
|---|---|---|---|
| 10 | 103 of 365 | 259 of 365 | 287 of 365 |
| 100 | 340 of 365 | 286 of 365 | 282 of 365 |
| 1,000 | 355 of 365 | 309 of 365 | 286 of 365 |
"Looked" means graded on the development part alone with one try declared. The planted edge uses the first seed whose draw shows its true strength on both halves, so the lesson is not lost to an unlucky draw; the rule is in scripts/gen_data.py.
Reproduce everything
pip install vendor/embargo-0.1.0-py3-none-any.whl python scripts/gen_data.py # rebuilds public/data/ byte for byte