Benchmarks in Leipzig
Abstract
Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the three-day workshop Benchmarks in Leipzig with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100~questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs and their predecessors, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive. In September 2026, we added a fourth stage in which the next generation of models attempted all 100 questions once more, after which only 1 question remains unsolved.
Related
- How Reusable Are Benchmarks with Richer Feedback?
- When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks
- Least-false Cox coefficients under affine follow-up contamination: exact continuous- and grouped-time benchmarks
- Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation
- Finite-Sample Calibration and Power Benchmarks in Two-Sample Random-Integration Tests
BibTeX
@misc{balakin2026benchmarks,
title = {{Benchmarks in Leipzig}},
author = {Andrei Balakin and Miklós Bóna and Marie-Charlotte Brandenburg and Clara Briand and Veronica Calvo Cortes and Shelby Cox and Jesus A. De Loera and Danai Deligeorgaki and Hannah Friedman and Tim Gehrunger and Chiara Giardino and Stephen Griffeth and Baran Hashemi and Elena Hoster and Alexander Ivanov and Nupur Jain and Aryaman Jal and Leonie Kayser and Joris Koefler and Kevin Kühn and Mario Kummer and Felix Lotter and René Marczinzik and Victor S. Miller and Alejandro Morales and Greta Panova and Gianni Petrella and Nathan Pflueger and Lakshmi Ramesh and Nikolas Rieke and Carlos Rodriguez and Andrea Rosana and Flavio Salizzoni and Otto T. P. Schmidt and Sven Ulf Schmitz and Lina Maria Simbaqueba Marin and Luca Sodomaco and Christian Stump and Bernd Sturmfels and Alexander Taveira Blomenhofer and Simon Telen and Philipp Tuchel and Emil Verkama and Carl Felix Waller and Julian Weigert and Annette Werner and Nathan Williams and Claudius Zibrowius},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.05818},
eprint = {2606.05818},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.05818},
}
RIS
TY - UNPB
TI - Benchmarks in Leipzig
AU - Andrei Balakin
AU - Miklós Bóna
AU - Marie-Charlotte Brandenburg
AU - Clara Briand
AU - Veronica Calvo Cortes
AU - Shelby Cox
AU - Jesus A. De Loera
AU - Danai Deligeorgaki
AU - Hannah Friedman
AU - Tim Gehrunger
AU - Chiara Giardino
AU - Stephen Griffeth
AU - Baran Hashemi
AU - Elena Hoster
AU - Alexander Ivanov
AU - Nupur Jain
AU - Aryaman Jal
AU - Leonie Kayser
AU - Joris Koefler
AU - Kevin Kühn
AU - Mario Kummer
AU - Felix Lotter
AU - René Marczinzik
AU - Victor S. Miller
AU - Alejandro Morales
AU - Greta Panova
AU - Gianni Petrella
AU - Nathan Pflueger
AU - Lakshmi Ramesh
AU - Nikolas Rieke
AU - Carlos Rodriguez
AU - Andrea Rosana
AU - Flavio Salizzoni
AU - Otto T. P. Schmidt
AU - Sven Ulf Schmitz
AU - Lina Maria Simbaqueba Marin
AU - Luca Sodomaco
AU - Christian Stump
AU - Bernd Sturmfels
AU - Alexander Taveira Blomenhofer
AU - Simon Telen
AU - Philipp Tuchel
AU - Emil Verkama
AU - Carl Felix Waller
AU - Julian Weigert
AU - Annette Werner
AU - Nathan Williams
AU - Claudius Zibrowius
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2606.05818
UR - https://arxiv.org/abs/2606.05818
AB - Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the three-day workshop Benchmarks in Leipzig with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100~questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs and their predecessors, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive. In September 2026, we added a fourth stage in which the next generation of models attempted all 100 questions once more, after which only 1 question remains unsolved.
ER -