LongHaul-Bench: A Reproducible Benchmark for Long-Horizon Reliability of Self-Improving LLM Agents Under Industrial Edge Constraints
Abstract
Self-improving agents built on large language models (LLMs) are increasingly proposed for industrial fault diagnosis, but it is not known whether the experience such an agent accumulates over months of operation makes it more reliable or slowly corrupts it, least of all under edge-deployment constraints. We present LongHaul-Bench, a seed-reproducible synthetic industrial environment and protocol that follows an agent's knowledge state over about 1,000 sequential episodes with a quantized 3B-parameter model and a hard experience-memory budget. Across five independently generated worlds, experiential learning lifts exact accuracy over a frozen control by 3.8 points (complete rank separation; Holm-corrected p = .048). Plausibly corrupted feedback induces dose-dependent knowledge rot (-6.0 points at 40% corruption; -2.3, not significant, at 20%), whereas blatantly invalid feedback is inert. We explain these effects with a recall-precision decomposition computed on episode-matched pairs: memory adds about 9 points when a recall surfaces the correct precedent and removes about 20 points when it misleads, so memory helps only above a break-even recall precision of roughly 0.7. Plausible corruption leaves the cost of a misleading recall unchanged and instead lowers recall precision (0.81, 0.72, 0.65 at 0/20/40%); the precision drop alone accounts for the full loss at 20% and three-quarters of it at 40%. LLM-free manual-consistency gates recover the damage by restoring precision to 0.78, yet suppress two-thirds of memory's benefit when manuals are incomplete, and memory's value is an order of magnitude larger for a weaker 3B model family (+28 vs. +2.3 points). Generator, harness, run logs, and analysis scripts are released at https://github.com/vahit19/longhaul-bench.
Related
- Accuracy, Reliability, and Efficiency of a MACE Machine-Learning Interatomic Potential for Aspirin: A Reproducible Benchmark for Molecular Simulation
- A Reproducible Benchmark for Leakage-Resistant Machine Learning on Multi-Site Brain Connectivity Graphs
- MultiCaRe-Thorax: A Curated Benchmark Dataset and Reproducible Labeling Framework for Multi-Label Thoracic Disease Classification in Chest X-Ray Imaging
- TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text
- Improving low-cost NO₂ sensor reliability under real-world air quality conditions by calibrating distributions instead of data points
BibTeX
@misc{feryad2026longhaulbench,
title = {{LongHaul-Bench: A Reproducible Benchmark for Long-Horizon Reliability of Self-Improving LLM Agents Under Industrial Edge Constraints}},
author = {Vahit Feryad},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23624799},
url = {https://zenodo.org/records/23624799},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - LongHaul-Bench: A Reproducible Benchmark for Long-Horizon Reliability of Self-Improving LLM Agents Under Industrial Edge Constraints
AU - Vahit Feryad
PY - 2026
DA - 2026/10/24
PB - Zenodo
DO - 10.5281/zenodo.23624799
UR - https://zenodo.org/records/23624799
AB - Self-improving agents built on large language models (LLMs) are increasingly proposed for industrial fault diagnosis, but it is not known whether the experience such an agent accumulates over months of operation makes it more reliable or slowly corrupts it, least of all under edge-deployment constraints. We present LongHaul-Bench, a seed-reproducible synthetic industrial environment and protocol that follows an agent's knowledge state over about 1,000 sequential episodes with a quantized 3B-parameter model and a hard experience-memory budget. Across five independently generated worlds, experiential learning lifts exact accuracy over a frozen control by 3.8 points (complete rank separation; Holm-corrected p = .048). Plausibly corrupted feedback induces dose-dependent knowledge rot (-6.0 points at 40% corruption; -2.3, not significant, at 20%), whereas blatantly invalid feedback is inert. We explain these effects with a recall-precision decomposition computed on episode-matched pairs: memory adds about 9 points when a recall surfaces the correct precedent and removes about 20 points when it misleads, so memory helps only above a break-even recall precision of roughly 0.7. Plausible corruption leaves the cost of a misleading recall unchanged and instead lowers recall precision (0.81, 0.72, 0.65 at 0/20/40%); the precision drop alone accounts for the full loss at 20% and three-quarters of it at 40%. LLM-free manual-consistency gates recover the damage by restoring precision to 0.78, yet suppress two-thirds of memory's benefit when manuals are incomplete, and memory's value is an order of magnitude larger for a weaker 3B model family (+28 vs. +2.3 points). Generator, harness, run logs, and analysis scripts are released at https://github.com/vahit19/longhaul-bench.
ER -