Phase Transitions in Learning Driven by Extremal Data
Abstract
We study an extremal regression problem, an example of multiple instance learning (MIL), where the predictors are given as multisets, called bags. The bag elements (called instances) possess hidden labels which are pooled to produce an observed bag-level label. Max pooling is common in MIL applications, but gives rise to vanishing gradients. To handle this, modern MIL pipelines approximate the max with a softmax, treating the sharpness as a learnable parameter, denoted λ. When the max pooling model is the true data generating process, learning will be successful when λ is increased during gradient descent, thereby driving the dynamics into regions of parameter space with vanishing gradients. To understand the conditions under which gradient descent breaks down, we study a minimal model and derive a low dimensional projection of the gradient flow in the joint limit of large bags n and high dimension d with α=ln n/d fixed. The flow has two distinct phase transitions: an intrinsic freezing transition related to the random energy model of disordered systems, and a retrieval transition occurring in modern Hopfield networks. Numerical simulation confirms both the phase diagram and the convergence of the finite-size training gradient to its predicted asymptotic form.
Related
- Successive Phase Transitions from a Composite Fermion Liquid to a Fractional Quantum Hall State at ν=3/2 Driven by In-Plane Magnetic Field
- Field-driven quantum phase transitions in a spin-1/2 Heisenberg antiferromagnet on an extended Lieb lattice
- Occupancy Driven Phase Transitions in Gapped Systems
- Effects of Nonmagnetic Impurities on Nonequilibrium Phase Transitions in Microwave-Driven Superconductors
- Universal splitting of nonequilibrium phase transitions in driven Potts heat engines
BibTeX
@misc{levien2026phase,
title = {{Phase Transitions in Learning Driven by Extremal Data}},
author = {Ethan Levien and Trevor GrandPre},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.29531},
eprint = {2610.29531},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.29531},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Phase Transitions in Learning Driven by Extremal Data
AU - Ethan Levien
AU - Trevor GrandPre
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.29531
UR - https://arxiv.org/abs/2610.29531
AB - We study an extremal regression problem, an example of multiple instance learning (MIL), where the predictors are given as multisets, called bags. The bag elements (called instances) possess hidden labels which are pooled to produce an observed bag-level label. Max pooling is common in MIL applications, but gives rise to vanishing gradients. To handle this, modern MIL pipelines approximate the max with a softmax, treating the sharpness as a learnable parameter, denoted λ. When the max pooling model is the true data generating process, learning will be successful when λ is increased during gradient descent, thereby driving the dynamics into regions of parameter space with vanishing gradients. To understand the conditions under which gradient descent breaks down, we study a minimal model and derive a low dimensional projection of the gradient flow in the joint limit of large bags n and high dimension d with α=ln n/d fixed. The flow has two distinct phase transitions: an intrinsic freezing transition related to the random energy model of disordered systems, and a retrieval transition occurring in modern Hopfield networks. Numerical simulation confirms both the phase diagram and the convergence of the finite-size training gradient to its predicted asymptotic form.
ER -