PreprintCC BY 4.0HAL
Activation-Matched Gibbs Initialization for Neural Networks
Abstract
Initialization plays an important role in neural network training. A classical way to address this task relies on Gaussian weights and preactivations, motivated in part by their analytical tractability and their emergence in large-width approximations. We propose an activation-induced initialization based on a statistical-mechanical viewpoint: for activations that arise as the gradient of a Legendre potential, the latter defines a Gibbs family of preactivation distributions. This perspective allows the preactivation distribution, rather than only its scale, to be adapted to the activation geometry.
Related
- Optimal Initialization Scale for Neural Networks with Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective
- Critical initialization destabilizes higher input derivatives in wide scalar-input networks
- Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
- Correlated initialization of deep residual networks
- ENFR: Evidence Networks with Flow-Matched References for Amortized Simulation-Based Model Inference
BibTeX
@misc{pintosuarez2026activationmatched,
title = {{Activation-Matched Gibbs Initialization for Neural Networks}},
author = {Diego Pinto-Suárez and Jordan Patracone and Marc Sebban},
howpublished = {HAL},
year = {2026},
month = oct,
url = {https://hal.science/hal-05789261v2},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Activation-Matched Gibbs Initialization for Neural Networks
AU - Diego Pinto-Suárez
AU - Jordan Patracone
AU - Marc Sebban
PY - 2026
DA - 2026/10/19
PB - HAL
UR - https://hal.science/hal-05789261v2
AB - Initialization plays an important role in neural network training. A classical way to address this task relies on Gaussian weights and preactivations, motivated in part by their analytical tractability and their emergence in large-width approximations. We propose an activation-induced initialization based on a statistical-mechanical viewpoint: for activations that arise as the gradient of a Legendre potential, the latter defines a Gibbs family of preactivation distributions. This perspective allows the preactivation distribution, rather than only its scale, to be adapted to the activation geometry.
ER -