Optimization with Region-Reduced ReLU Neural Networks
Abstract
Optimization of mathematical models involving integer decisions and neural networks with ReLU activation (ReLU ANNs) is a challenging task. Nevertheless, such models are an enabling technology in many application domains. A prominent example is superstructure optimization in chemical engineering, where ReLU ANNs are frequently employed as surrogate models for complex nonlinear processes. We survey recent developments in this area. We argue that in addition to network size and training options of the ANNs, the ReLU activation geometry and the number of linear regions on the domain of interest have a strong impact on computational optimization performance. While standard model compression approaches such as structured pruning reduce network size, they do not explicitly address geometric considerations. Therefore, we propose a novel \emph{region-reduced} model compression approach that combines the stabilization of unstable neurons and the merging of redundant neurons to reduce the number of linear regions while maintaining predictive accuracy through error compensation. We evaluate our method against standard compression approaches on multiple optimization use cases. First, the two-dimensional peaks function for which we can visualize the activation geometry. Second, on optimization over individual surrogate ReLU ANNs for three chemical processes, and third, on a hybrid superstructure optimization problem that involves the three ReLU ANNs, additional process submodels, and binary variables. The results for the superstructure problem demonstrate the large potential of region reduction with a decrease of 40\% to 50\% in computational time, yielding solutions closer than 1\% to the reference at negligible effort of obtaining the compressed model.
Related
- Distributional Regression for Arbitrary Response Variables Using ReLU Neural Networks
- Optimal approximation of smooth functions by ReLU neural networks
- Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks
- Minimax rates for learning spectral Barron functions by deep ReLU neural networks
- Schauder and Riesz Bases for Higher-Dimensional Lebesgue Spaces Inspired by ReLU Neural Networks
BibTeX
@misc{plate2026optimization,
title = {{Optimization with Region-Reduced ReLU Neural Networks}},
author = {Christoph Plate and Caroline Ganzer and Mirko Hahn and Alexander Klimek and Heyuan Liu and Sebastian Sager and Kai Sundmacher and Hanna Wilhelm},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31380},
eprint = {2609.31380},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31380},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimization with Region-Reduced ReLU Neural Networks
AU - Christoph Plate
AU - Caroline Ganzer
AU - Mirko Hahn
AU - Alexander Klimek
AU - Heyuan Liu
AU - Sebastian Sager
AU - Kai Sundmacher
AU - Hanna Wilhelm
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.31380
UR - https://arxiv.org/abs/2609.31380
AB - Optimization of mathematical models involving integer decisions and neural networks with ReLU activation (ReLU ANNs) is a challenging task. Nevertheless, such models are an enabling technology in many application domains. A prominent example is superstructure optimization in chemical engineering, where ReLU ANNs are frequently employed as surrogate models for complex nonlinear processes. We survey recent developments in this area. We argue that in addition to network size and training options of the ANNs, the ReLU activation geometry and the number of linear regions on the domain of interest have a strong impact on computational optimization performance. While standard model compression approaches such as structured pruning reduce network size, they do not explicitly address geometric considerations. Therefore, we propose a novel \emph{region-reduced} model compression approach that combines the stabilization of unstable neurons and the merging of redundant neurons to reduce the number of linear regions while maintaining predictive accuracy through error compensation. We evaluate our method against standard compression approaches on multiple optimization use cases. First, the two-dimensional peaks function for which we can visualize the activation geometry. Second, on optimization over individual surrogate ReLU ANNs for three chemical processes, and third, on a hybrid superstructure optimization problem that involves the three ReLU ANNs, additional process submodels, and binary variables. The results for the superstructure problem demonstrate the large potential of region reduction with a decrease of 40\% to 50\% in computational time, yielding solutions closer than 1\% to the reference at negligible effort of obtaining the compressed model.
ER -