Two bits about lossy compression: On the limits of compression in cosmology
Abstract
Astronomy is in an era of enormous sky surveys and of the large simulation suites needed to interpret them, both costly to store and slow to share. Simulation outputs are stored as 32-bit floats, yet numerical noise and astrophysical uncertainties make better than percent-level pixel accuracy unnecessary. Because cosmological fields are statistically homogeneous with nearly Gaussian mode amplitudes, classic rate-distortion results apply directly, and non-Gaussian structure permits further compression. We use the scientific compression package SZ3, and a neural compressor that fixes the quantization and learns the probability of each quantization bin with an autoregressive transformer. SZ3 beats the Gaussian approach only where the field is smooth on the pixel scale or its values pile up at a single point, while the neural approach matches or beats SZ3 on every field we consider and is 9 to 27\% below the Gaussian-optimal coder at fixed distortion, with the largest margin where the field is most non-Gaussian. Because the quantizer, not the network, sets the error, a poorly trained model can waste bits but never cost accuracy. A model trained only on weak lensing convergence maps transfers to N-body density fields without retraining, suggesting it has learned generic properties of cosmological structure rather than features of one dataset. Eulerian grids need only 1-4 bits per pixel (bpp) at percent-level accuracy, and particle displacements and velocities 5-6 at the precisions they require. The approach carries over to observational data: on Rubin Observatory Data Preview 1 coadds, with the quantization step set to a quarter of the background noise, the fine-tuned transformer needs 4.0 bpp, 27\% fewer than the Gaussian coder.
Related
- Options for Compression of radio interferometry data: lossy compression of visibilities and lossless compression of uv-visibility grids for the MHONGOOSE survey
- What we have not learned about the dark energy equation of state
- Cosmology with weak lensing surveys
- On entropic cosmology, late-time acceleration and the quantum bounce
- Theoretical consistency and observational constraints on entropic cosmology
BibTeX
@misc{tohfa2026bits,
title = {{Two bits about lossy compression: On the limits of compression in cosmology}},
author = {Hurum Maksora Tohfa and Matthew McQuinn},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.40239},
eprint = {2609.40239},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.40239},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Two bits about lossy compression: On the limits of compression in cosmology
AU - Hurum Maksora Tohfa
AU - Matthew McQuinn
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.40239
UR - https://arxiv.org/abs/2609.40239
AB - Astronomy is in an era of enormous sky surveys and of the large simulation suites needed to interpret them, both costly to store and slow to share. Simulation outputs are stored as 32-bit floats, yet numerical noise and astrophysical uncertainties make better than percent-level pixel accuracy unnecessary. Because cosmological fields are statistically homogeneous with nearly Gaussian mode amplitudes, classic rate-distortion results apply directly, and non-Gaussian structure permits further compression. We use the scientific compression package SZ3, and a neural compressor that fixes the quantization and learns the probability of each quantization bin with an autoregressive transformer. SZ3 beats the Gaussian approach only where the field is smooth on the pixel scale or its values pile up at a single point, while the neural approach matches or beats SZ3 on every field we consider and is 9 to 27\% below the Gaussian-optimal coder at fixed distortion, with the largest margin where the field is most non-Gaussian. Because the quantizer, not the network, sets the error, a poorly trained model can waste bits but never cost accuracy. A model trained only on weak lensing convergence maps transfers to N-body density fields without retraining, suggesting it has learned generic properties of cosmological structure rather than features of one dataset. Eulerian grids need only 1-4 bits per pixel (bpp) at percent-level accuracy, and particle displacements and velocities 5-6 at the precisions they require. The approach carries over to observational data: on Rubin Observatory Data Preview 1 coadds, with the quantization step set to a quarter of the background noise, the fine-tuned transformer needs 4.0 bpp, 27\% fewer than the Gaussian coder.
ER -