Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games
Abstract
Engine self-play allows chess openings to be compared under controlled conditions, without the selection effects of human game databases. When openings differ only slightly, however, rankings built from a few hundred games can be unstable. In Study 1, strength-limited Stockfish 15.1 played 17,400 games: ten forced White first moves and ten forced Black replies to 1.e4, each at three strength settings. The leading opening changed between settings, twice by a single game. There was no statistically reliable evidence, though, that differences among openings depended on strength (White p = .311, Black p = .843). The design had limited sensitivity to isolated strength-dependent effects: a 12-point effect was detected by the omnibus test in only about 39% of simulated studies. A Connect Four positive control detected an observed 22.5-point center-column advantage, demonstrating sensitivity to large opening differences. With 290 games per opening, the observed leader was the true best of ten openings in only about 44% of simulated studies assuming a between-opening standard deviation of three percentage points. Follow-up studies with complete game records showed that the settings were ordered in direct matches but did not form a consistent Elo scale, that win and draw rates differed across hardware and implementation configurations, and that an apparent 1.d3 weakness was not confirmed in new games. With Stockfish 19 and a fixed node budget, survey-selected replies showed no statistically reliable advantage over free engine choice. Opening leaderboards from a few hundred engine games can overstate certainty.
Related
- Unidentified prediction-risk rankings under spatial sampling with fixed inclusion probabilities
- Halve-Then-Greedy: Adaptive Sampling for Posterior Success with Near-Optimal Instance Dependence
- Edge-Defect Spectral Methods for Higher-Order Rankings of Spanning Tree Counts
- Conditional independence and graphical models for rankings
- Simultaneous Inference for Centre-Outward Rankings of Random Objects
BibTeX
@misc{flick2026opening,
title = {{Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games}},
author = {Conrad Flick and Jaysen Jensen and Jingyu Huang and Mario Camara Paslar},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.37988},
eprint = {2610.37988},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.37988},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games
AU - Conrad Flick
AU - Jaysen Jensen
AU - Jingyu Huang
AU - Mario Camara Paslar
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.37988
UR - https://arxiv.org/abs/2610.37988
AB - Engine self-play allows chess openings to be compared under controlled conditions, without the selection effects of human game databases. When openings differ only slightly, however, rankings built from a few hundred games can be unstable. In Study 1, strength-limited Stockfish 15.1 played 17,400 games: ten forced White first moves and ten forced Black replies to 1.e4, each at three strength settings. The leading opening changed between settings, twice by a single game. There was no statistically reliable evidence, though, that differences among openings depended on strength (White p = .311, Black p = .843). The design had limited sensitivity to isolated strength-dependent effects: a 12-point effect was detected by the omnibus test in only about 39% of simulated studies. A Connect Four positive control detected an observed 22.5-point center-column advantage, demonstrating sensitivity to large opening differences. With 290 games per opening, the observed leader was the true best of ten openings in only about 44% of simulated studies assuming a between-opening standard deviation of three percentage points. Follow-up studies with complete game records showed that the settings were ordered in direct matches but did not form a consistent Elo scale, that win and draw rates differed across hardware and implementation configurations, and that an apparent 1.d3 weakness was not confirmed in new games. With Stockfish 19 and a fixed node budget, survey-selected replies showed no statistically reliable advantage over free engine choice. Opening leaderboards from a few hundred engine games can overstate certainty.
ER -