How to grade the accuracy of the BLAS
Abstract
Motivated by accelerating machine learning (ML), many computer vendors and chip manufacturers are building accelerators for matrix multiplication, which save time and energy by operating in the lower precisions needed for ML. This has in turn motivated many efforts to use these accelerators to provide faster matrix multiplication implementations with the higher precision required by many other linear algebra applications. Motivated by the large design space of algorithms for approximating higher precision, with significant performance/accuracy tradeoffs, we provide a benchmark to "grade the accuracy" of a matrix multiplication implementation (or the BLAS more generally), ranging from an "A" for attaining the classic floating point error bound, to a "C" for attaining a weaker but still useful bound, that is satisfied by Strassen-like algorithms. We also propose "ungameable" tests that vendors or users can run to verify their promised accuracy, and describe how these different grades impact the accuracy of applications like LU, QR, and Cholesky decomposition. Our test code is publicly available at github.com/Reference-LAPACK/grading-the-BLAS for developers and users, and we also plan to publicly release all our test results.
Related
- Domain-Specific Mathematics Performance of Grade 8 Students: Evidence from a K to 12 Achievement Test
- A Math Circle in Third Grade
- Auslander-type grade conditions and subcategories related to syzygy modules
- Sharing a unit, measuring a length: a didactic analysis of a teaching progression for fractions from Grade 1 to Grade 8
- Two teams that cancel: a didactic analysis of a teaching progression for signed numbers from Grade 5 to Grade 7
BibTeX
@misc{demmel2026grade,
title = {{How to grade the accuracy of the BLAS}},
author = {James Demmel and Greg Henry and Igor Kozachenko and Julien Langou and Xiaoye Sherry Li and Jason Riedy and Jackson Vanover},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12307},
eprint = {2609.12307},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12307},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How to grade the accuracy of the BLAS
AU - James Demmel
AU - Greg Henry
AU - Igor Kozachenko
AU - Julien Langou
AU - Xiaoye Sherry Li
AU - Jason Riedy
AU - Jackson Vanover
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.12307
UR - https://arxiv.org/abs/2609.12307
AB - Motivated by accelerating machine learning (ML), many computer vendors and chip manufacturers are building accelerators for matrix multiplication, which save time and energy by operating in the lower precisions needed for ML. This has in turn motivated many efforts to use these accelerators to provide faster matrix multiplication implementations with the higher precision required by many other linear algebra applications. Motivated by the large design space of algorithms for approximating higher precision, with significant performance/accuracy tradeoffs, we provide a benchmark to "grade the accuracy" of a matrix multiplication implementation (or the BLAS more generally), ranging from an "A" for attaining the classic floating point error bound, to a "C" for attaining a weaker but still useful bound, that is satisfied by Strassen-like algorithms. We also propose "ungameable" tests that vendors or users can run to verify their promised accuracy, and describe how these different grades impact the accuracy of applications like LU, QR, and Cholesky decomposition. Our test code is publicly available at github.com/Reference-LAPACK/grading-the-BLAS for developers and users, and we also plan to publicly release all our test results.
ER -