Right Precision, Right Place: Runtime Adaptations for Mixed-Precision Cholesky Factorization
Abstract
Modern GPU hardware accelerators and tensor cores offer substantial performance advantages through low-precision arithmetic, but exploiting these capabilities for scientific computing requires careful monitoring of numerical accuracy. We present an adaptive mixed-precision Cholesky factorization that dynamically selects the arithmetic precision of individual matrix tiles according to their evolving numerical properties during the Cholesky factorization. Unlike approaches that determine precision statically from the initial matrix generation or rely on iterative refinement to fully recover digits of accuracy lost during a low-precision factorization stage, our method uses a robust error-controlled numerical analysis to make informed, fine-grained precision decisions throughout the course of the computation. This approach prevents oversolving by enabling the factorization to exploit low-precision tensor-core operations where they are numerically safe, while retaining higher precision for computation that is more sensitive to rounding errors.
The proposed method is integrated into the PaRSEC dynamic runtime system, enabling precision selection, task scheduling, online data representation / conversion, and data movement to be coordinated across heterogeneous CPU-GPU architectures. By reducing both the memory footprint and the volume of data transferred across the systems' memory hierarchy, the approach addresses not only arithmetic throughput but also the data movement bottleneck that increasingly limits the scalability of scientific applications on heterogeneous systems. We demonstrate the effectiveness of the approach on representative climate applications matrix kernels, achieving up to 70% performance improvement on NVIDIA GH200 GPUs against the static precision selection without compromising application-level accuracy. On multiple GPUs, our approach reaches 3.2× the FP64 baseline throughput and exceeds 1 Pflop/s. Our methodology highlights the importance of a co-design across the scientific application, the dynamic runtime systems, and the hardware architectures, for the development of error-aware matrix algorithms.
Related
- Dense Matrices Are Alike; Sparse Matrices Are Sparse in Their Own Way: A Structure-Adaptive Tile Cholesky Factorization
- Reduced and Mixed-Precision Block Householder QR Factorization
- Singular Cholesky Fibers over Finite Fields
- Hidden Convexity in Multi-Retailer Cold-Chain Allocation: A Cholesky-Spectral Second-Order Cone Programming Approach
- CAST: Canonical Approximate Schur Tree for Approximate Cholesky on Graphs
BibTeX
@misc{cao2026right,
title = {{Right Precision, Right Place: Runtime Adaptations for Mixed-Precision Cholesky Factorization}},
author = {Qinglei Cao and Theo Mary and Rabab Alomairy and Hatem Ltaief and David E Keyes},
howpublished = {HAL},
year = {2026},
month = oct,
url = {https://hal.science/hal-05793453v1},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Right Precision, Right Place: Runtime Adaptations for Mixed-Precision Cholesky Factorization
AU - Qinglei Cao
AU - Theo Mary
AU - Rabab Alomairy
AU - Hatem Ltaief
AU - David E Keyes
PY - 2026
DA - 2026/10/12
PB - HAL
UR - https://hal.science/hal-05793453v1
AB - Modern GPU hardware accelerators and tensor cores offer substantial performance advantages through low-precision arithmetic, but exploiting these capabilities for scientific computing requires careful monitoring of numerical accuracy. We present an adaptive mixed-precision Cholesky factorization that dynamically selects the arithmetic precision of individual matrix tiles according to their evolving numerical properties during the Cholesky factorization. Unlike approaches that determine precision statically from the initial matrix generation or rely on iterative refinement to fully recover digits of accuracy lost during a low-precision factorization stage, our method uses a robust error-controlled numerical analysis to make informed, fine-grained precision decisions throughout the course of the computation. This approach prevents oversolving by enabling the factorization to exploit low-precision tensor-core operations where they are numerically safe, while retaining higher precision for computation that is more sensitive to rounding errors. The proposed method is integrated into the PaRSEC dynamic runtime system, enabling precision selection, task scheduling, online data representation / conversion, and data movement to be coordinated across heterogeneous CPU-GPU architectures. By reducing both the memory footprint and the volume of data transferred across the systems' memory hierarchy, the approach addresses not only arithmetic throughput but also the data movement bottleneck that increasingly limits the scalability of scientific applications on heterogeneous systems. We demonstrate the effectiveness of the approach on representative climate applications matrix kernels, achieving up to 70% performance improvement on NVIDIA GH200 GPUs against the static precision selection without compromising application-level accuracy. On multiple GPUs, our approach reaches 3.2× the FP64 baseline throughput and exceeds 1 Pflop/s. Our methodology highlights the importance of a co-design across the scientific application, the dynamic runtime systems, and the hardware architectures, for the development of error-aware matrix algorithms.
ER -