Abstract
A researcher who backtests many trading strategies and keeps the best one faces two questions: how much to believe the winner's backtest, and whether searching more candidates leads to a better strategy. We show that, when strategies are sampled at random from a fixed pool, the second question has an exact answer: the expected out-of-sample payoff at any search width is a known weighted average of the leaderboard, the out-of-sample payoff of each in-sample rank. We measure that leaderboard for 652 mechanical trading signals on U.S. mega-cap stocks (2010-2025) and on industry and size/value portfolios (1970-2025), trading one day after the signal and paying realistic spreads. Three results stand out. The backtest always overstates, and the overstatement grows with search width. Net of costs, searching more candidates picks a less costly strategy, mainly slow momentum, so the chosen strategy improves relative to a narrow search on the diversified panels. But at no width does any rule, including shrinkage and combination, find a strategy with a reliably positive return after costs.