Ozaki Scheme II Is Fast on CPUs Too: Multiple-Precision Matrix Multiplication on Intel AMX-INT8 and Arm SVE2-i8mm
Abstract
We implement Ozaki Scheme II (residue number system + Chinese remainder theorem), which reduces multiple-precision dense matrix multiplication to a sequence of low-precision, high-throughput integer or floating-point GEMMs on CPUs. Two backends are built on top of a shared CRT reconstruction stage: (a) exact INT8 x INT8 -> INT32 tile products on Intel AMX, and (b) binary64 DGEMM, the CPU construction of the original Ozaki Scheme II paper. On a two-socket Xeon Gold 6526Y (Emerald Rapids, 32 cores), we evaluate significand precisions of 53-2048 bits and matrix dimensions N = 256-8192. The results are always within 1 ulp of a high-precision MPFR reference (essentially correctly rounded), while running up to 167x faster than a naive MPFR matrix product, up to 588x faster than BNCmatmul's Strassen multiplication, and 9-78x faster than Ozaki Scheme I (FP64 slicing + OpenBLAS DGEMM). The break-even point between the two backends is approximately N = 2048: below it, the binary64 backend wins thanks to its smaller number of moduli; above it, the AMX-INT8 backend wins as the GEMMs dominate. We further port the implementation to AArch64 (NVIDIA GB10: Cortex-X925 x 10 + Cortex-A725 x 10). Since this machine lacks SME/SME2, the INT8 kernel uses the SMMLA matrix-product instruction of the SVE2 i8mm extension. We obtain an exact INT8 GEMM sustaining 6.5 TOPS and, still within 1 ulp across all conditions, speedups of 14-89x over BNCmatmul's Ozaki Scheme I (OpenBLAS-linked routine) and 6-19x over a fairness-adjusted OzI-best variant. The paper also includes a tutorial introduction to Ozaki Scheme II (Section "Introduction to Ozaki Scheme II") and a quantitative explanation of why this seemingly GPU-specific technique is fast on CPUs as well.
Related
- Accelerating Multiple-Precision LU Decomposition with Ozaki Scheme II
- Adaptive accuracy-driven splitting in Ozaki matrix multiplication and application to block LU factorization
- Emulating DGEMM and ZGEMM with INT8 Matrix Engines on Intel Arc GPUs: Ozaki Scheme II on the Xe2 Architecture
- Fast and Stable Arbitrary Precision Approximate Matrix Multiplication
- Approximate Matrix Multiplication through Joint Leverage Score Sampling
BibTeX
@misc{kouya2026ozaki,
title = {{Ozaki Scheme II Is Fast on CPUs Too: Multiple-Precision Matrix Multiplication on Intel AMX-INT8 and Arm SVE2-i8mm}},
author = {Tomonori Kouya},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27831},
eprint = {2609.27831},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27831},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Ozaki Scheme II Is Fast on CPUs Too: Multiple-Precision Matrix Multiplication on Intel AMX-INT8 and Arm SVE2-i8mm
AU - Tomonori Kouya
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.27831
UR - https://arxiv.org/abs/2609.27831
AB - We implement Ozaki Scheme II (residue number system + Chinese remainder theorem), which reduces multiple-precision dense matrix multiplication to a sequence of low-precision, high-throughput integer or floating-point GEMMs on CPUs. Two backends are built on top of a shared CRT reconstruction stage: (a) exact INT8 x INT8 -> INT32 tile products on Intel AMX, and (b) binary64 DGEMM, the CPU construction of the original Ozaki Scheme II paper. On a two-socket Xeon Gold 6526Y (Emerald Rapids, 32 cores), we evaluate significand precisions of 53-2048 bits and matrix dimensions N = 256-8192. The results are always within 1 ulp of a high-precision MPFR reference (essentially correctly rounded), while running up to 167x faster than a naive MPFR matrix product, up to 588x faster than BNCmatmul's Strassen multiplication, and 9-78x faster than Ozaki Scheme I (FP64 slicing + OpenBLAS DGEMM). The break-even point between the two backends is approximately N = 2048: below it, the binary64 backend wins thanks to its smaller number of moduli; above it, the AMX-INT8 backend wins as the GEMMs dominate. We further port the implementation to AArch64 (NVIDIA GB10: Cortex-X925 x 10 + Cortex-A725 x 10). Since this machine lacks SME/SME2, the INT8 kernel uses the SMMLA matrix-product instruction of the SVE2 i8mm extension. We obtain an exact INT8 GEMM sustaining 6.5 TOPS and, still within 1 ulp across all conditions, speedups of 14-89x over BNCmatmul's Ozaki Scheme I (OpenBLAS-linked routine) and 6-19x over a fairness-adjusted OzI-best variant. The paper also includes a tutorial introduction to Ozaki Scheme II (Section "Introduction to Ozaki Scheme II") and a quantitative explanation of why this seemingly GPU-specific technique is fast on CPUs as well.
ER -