Reinforcement-Learning-Enabled Supervisory Control of a Full-Scale Anaerobic Digestion Facility Using a Calibrated Digital Twin.
- 1School of Civil and Environmental Engineering, Georgia Institute of Technology, Atlanta, Georgia30318, United States.
- 2Jimmy and Rosalynn Carter School of Public Policy, Georgia Institute of Technology, Atlanta, Georgia30332, United States.
Abstract
Anaerobic digestion (AD) is a cornerstone of energy recovery and decarbonization at wastewater resource recovery facilities (WRRFs), yet full-scale digesters are often operated conservatively due to delayed process feedback, nonlinear stability constraints, and limited observability. This study presents a novel reinforcement learning (RL)-enabled supervisory control framework for full-scale AD that integrates a calibrated mechanistic digital twin model (DTM) with offline policy learning and counterfactual uncertainty evaluation. Specifically, the DTM was calibrated to reproduce control-relevant digester behavior. Using 2.7 years of simulated operation, a batch-constrained Q (BCQ)-reinforcement learning policy was trained to maximize biogas production while penalizing instability associated with pH excursions and elevated volatile fatty acids (VFAs)/alkalinity (ALK) ratios. Counterfactual evaluation against historical, greedy, and heuristic controls used 1 year unseen DTM simulation and 1 year of full-scale operational data. The RL policy achieved comparable mean daily biogas production (+0.35%) in simulated testing. Under the 1 year operational data, it achieved plant-equivalent mean production of 9802 m3/day, corresponding to a modest potential cumulative improvement (+2.7%, 93 405 m3/year) subject to substantial uncertainty P(Δ > 0) = 0.524. These results indicate that offline-trained RL, combined with a mechanistic digital twin and counterfactual evaluation, provides a feasible and interpretable framework for developing and assessing supervisory AD control without live-plant experimentation while showing that the resulting benefit must be interpreted under explicit model-plant uncertainty.
Related
- Assessment of fate of contaminants of emerging concern in forward osmosis-reverse osmosis-anaerobic digestion hybrid: A literature-based approach
- Durable superhydrophobic interfaces sustain long-term ammonia recovery from real anaerobic digestion effluent
- Integrated Intensification and Nutrient Recovery Strategies in Two-Stage Anaerobic Co-Digestion of Sewage Sludge and the Organic Fraction of Municipal Solid Waste: State of the Art, Engineering Challenges of Scale-Up, and Perspectives
- Simulation-Based Supervisory Monitoring of Photovoltaic Mismatch in a Hybrid Greenhouse Microgrid
- DeepWaveRL: Self-Supervised Full Waveform Inversion via Reinforcement Learning
BibTeX
@article{yunus2026reinforcementlearningenabled,
title = {{Reinforcement-Learning-Enabled Supervisory Control of a Full-Scale Anaerobic Digestion Facility Using a Calibrated Digital Twin.}},
author = {Ahmed I Yunus and Srinivas Jalla and Joe F Bozeman and Yongsheng Chen},
journal = {Environmental science \& technology},
year = {2026},
month = oct,
doi = {10.1021/acs.est.6c05822},
url = {https://europepmc.org/article/MED/42917018},
note = {CC BY},
}
RIS
TY - JOUR
TI - Reinforcement-Learning-Enabled Supervisory Control of a Full-Scale Anaerobic Digestion Facility Using a Calibrated Digital Twin.
AU - Ahmed I Yunus
AU - Srinivas Jalla
AU - Joe F Bozeman
AU - Yongsheng Chen
PY - 2026
DA - 2026/10/01
JO - Environmental science & technology
DO - 10.1021/acs.est.6c05822
UR - https://europepmc.org/article/MED/42917018
AB - Anaerobic digestion (AD) is a cornerstone of energy recovery and decarbonization at wastewater resource recovery facilities (WRRFs), yet full-scale digesters are often operated conservatively due to delayed process feedback, nonlinear stability constraints, and limited observability. This study presents a novel reinforcement learning (RL)-enabled supervisory control framework for full-scale AD that integrates a calibrated mechanistic digital twin model (DTM) with offline policy learning and counterfactual uncertainty evaluation. Specifically, the DTM was calibrated to reproduce control-relevant digester behavior. Using 2.7 years of simulated operation, a batch-constrained Q (BCQ)-reinforcement learning policy was trained to maximize biogas production while penalizing instability associated with pH excursions and elevated volatile fatty acids (VFAs)/alkalinity (ALK) ratios. Counterfactual evaluation against historical, greedy, and heuristic controls used 1 year unseen DTM simulation and 1 year of full-scale operational data. The RL policy achieved comparable mean daily biogas production (+0.35%) in simulated testing. Under the 1 year operational data, it achieved plant-equivalent mean production of 9802 m3/day, corresponding to a modest potential cumulative improvement (+2.7%, 93 405 m3/year) subject to substantial uncertainty P(Δ > 0) = 0.524. These results indicate that offline-trained RL, combined with a mechanistic digital twin and counterfactual evaluation, provides a feasible and interpretable framework for developing and assessing supervisory AD control without live-plant experimentation while showing that the resulting benefit must be interpreted under explicit model-plant uncertainty.
SN - 0013-936X
SN - 1520-5851
ER -