Optimizing the extraction of information from redshift probability distribution functions
Abstract
Photometric redshifts are essential for large-scale structure analyses, yet extracting optimal point estimates and reliability measures from the probability distribution functions (PDZs) delivered by modern photo-z pipelines remains challenging. We introduce turboPDZ, a machine-learning framework using PDZ features and associated observables to optimize redshift point estimates and derive a data-driven reliability score. We apply it to the three independent HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) across Wide and Deep/UltraDeep layers. Each PDZ is compressed via PCA and combined with summary descriptors. A multilayer perceptron, optimized with Optuna under a composite objective (σNMAD and η0.15), predicts zml. A second network, trained in log-space with MAE and calibrated post hoc, yields an empirical error scale σml, from which the reliability score rml is derived via percentile ranking. Relative to catalog point estimates, zml improves both metrics in all six pipeline-layer combinations. Absolute median bias is smaller in four configurations and tied in two. The reliability score achieves the lowest retained-fraction AUC in 10 of 12 native pipeline-layer/metric comparisons. For the Mizuki template-fitting pipeline, catalog reliability indicators have AUC values up to about 13 times larger than those of rml. Feature importance shows that point estimation is dominated by PCA and location statistics, whereas reliability estimation gives greatest weight to peak, PCA, and shape information. The framework is survey-agnostic, but each PDZ dataset requires its own fitted preprocessing, PCA basis, and trained models. The code will be released as a reusable Python library at \href{https://github.com/valerio-marra/turboPDZ}{github.com/valerio-marra/turboPDZ}.
Related
- νGundam: A unified framework for point-redshift and probability-density-function-aware spatial correlation functions
- On the Probability Distribution and Null-hypothesis Testing of Cross Correlation for Light Curves in Active Galactic Nuclei
- On the Probability Distribution and Null-hypothesis Testing of Cross-correlation for Light Curves in Active Galactic Nuclei
- Data Representation Matters: Optimizing Machine Learning for X-ray Source Classification by Leveraging Spatio-Spectral Information
- Baryonic Imprints on DM Halos: characterizing the full concentration-mass probability distribution with CAMELS
BibTeX
@misc{duarte2026optimizing,
title = {{Optimizing the extraction of information from redshift probability distribution functions}},
author = {Rodrigo Duarte and Valerio Marra},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.26822},
eprint = {2607.26822},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.26822},
}
RIS
TY - UNPB
TI - Optimizing the extraction of information from redshift probability distribution functions
AU - Rodrigo Duarte
AU - Valerio Marra
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2607.26822
UR - https://arxiv.org/abs/2607.26822
AB - Photometric redshifts are essential for large-scale structure analyses, yet extracting optimal point estimates and reliability measures from the probability distribution functions (PDZs) delivered by modern photo-z pipelines remains challenging. We introduce turboPDZ, a machine-learning framework using PDZ features and associated observables to optimize redshift point estimates and derive a data-driven reliability score. We apply it to the three independent HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) across Wide and Deep/UltraDeep layers. Each PDZ is compressed via PCA and combined with summary descriptors. A multilayer perceptron, optimized with Optuna under a composite objective (σ_NMAD and η_0.15), predicts z_ml. A second network, trained in log-space with MAE and calibrated post hoc, yields an empirical error scale σ_ml, from which the reliability score r_ml is derived via percentile ranking. Relative to catalog point estimates, z_ml improves both metrics in all six pipeline-layer combinations. Absolute median bias is smaller in four configurations and tied in two. The reliability score achieves the lowest retained-fraction AUC in 10 of 12 native pipeline-layer/metric comparisons. For the Mizuki template-fitting pipeline, catalog reliability indicators have AUC values up to about 13 times larger than those of r_ml. Feature importance shows that point estimation is dominated by PCA and location statistics, whereas reliability estimation gives greatest weight to peak, PCA, and shape information. The framework is survey-agnostic, but each PDZ dataset requires its own fitted preprocessing, PCA basis, and trained models. The code will be released as a reusable Python library at \href{https://github.com/valerio-marra/turboPDZ}{github.com/valerio-marra/turboPDZ}.
ER -