A Composite Activation Function for Learning Stable Binary Representations
Abstract
Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.
Related
- Into the danger zone: stable extrapolation in high-dimensional function and operator learning
- Protected Logical Qudits in Kitaev Quantum Double Models via Stable Representations
- Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection
- On the log-concavity of the composite Bessel function $x^αJ_{ν}\left( βx^γ\right) $
- An analogue of Γ-function for irreducible cuspidal representations of the group GL(2) over a two-dimensional local field
BibTeX
@misc{park2026composite,
title = {{A Composite Activation Function for Learning Stable Binary Representations}},
author = {Seokhun Park and Choeun Kim and Kwanho Lee and Sehyun Park and Insung Kong and Yongdai Kim},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.11558},
eprint = {2605.11558},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.11558},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Composite Activation Function for Learning Stable Binary Representations
AU - Seokhun Park
AU - Choeun Kim
AU - Kwanho Lee
AU - Sehyun Park
AU - Insung Kong
AU - Yongdai Kim
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2605.11558
UR - https://arxiv.org/abs/2605.11558
AB - Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.
ER -