LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning
Abstract
Vision-based reinforcement learning holds strong potential for decision-making in controlled environment agriculture (CEA). However, its development is hindered by the scarcity of labelled crop images. To address this gap, LettuceVisSim, a lettuce growth simulator that generates labelled time series of crop images, was developed and validated. The simulator contains a process-based model (PBM) for shoot dry weight dynamics, a canopy layout algorithm for deriving canopy layout representations from shoot dry weight, and a Unity rendering engine for image generation. Five findings support the simulator. First, the PBM reproduced shoot dry weight under dynamic plant-density management with R2=0.84. Second, a piecewise cubic regression mapped shoot dry weight to potential projected area with R2=0.94. Third, the canopy layout representation was validated using 12 experimental datasets each having different dynamic environmental and spacing conditions. It reproduced the ground coverage ratio dynamics observed in measured images, achieving R2=0.84 when driven by measured shoot dry weight and R2=0.40 (0.76 excluding one outlier) when driven by PBM-simulated values. Fourth, the Unity rendering engine converted canopy layout representations into RGB and segmentation images at less than 10~ms. Fifth, a demonstration showed that a lighting-control policy can be learned and applied by observing only crop images that were generated with LettuceVisSim, providing a proof of concept of vision-based reinforcement learning in CEA using LettuceVisSim.
Related
- Active learning strategy for excursion-set confidence regions of functional simulator outputs
- Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
- Active learning strategies for the estimation of a feasible set defined from a vector output black-box simulator
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
BibTeX
@misc{zhu2026lettucevissim,
title = {{LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning}},
author = {Ziye Zhu and Bert van 't Ooster and Congcong Sun and Eldert van Henten and Sjoerd Boersma},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12505},
eprint = {2609.12505},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12505},
}
RIS
TY - UNPB
TI - LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning
AU - Ziye Zhu
AU - Bert van 't Ooster
AU - Congcong Sun
AU - Eldert van Henten
AU - Sjoerd Boersma
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.12505
UR - https://arxiv.org/abs/2609.12505
AB - Vision-based reinforcement learning holds strong potential for decision-making in controlled environment agriculture (CEA). However, its development is hindered by the scarcity of labelled crop images. To address this gap, LettuceVisSim, a lettuce growth simulator that generates labelled time series of crop images, was developed and validated. The simulator contains a process-based model (PBM) for shoot dry weight dynamics, a canopy layout algorithm for deriving canopy layout representations from shoot dry weight, and a Unity rendering engine for image generation. Five findings support the simulator. First, the PBM reproduced shoot dry weight under dynamic plant-density management with R^2=0.84. Second, a piecewise cubic regression mapped shoot dry weight to potential projected area with R^2=0.94. Third, the canopy layout representation was validated using 12 experimental datasets each having different dynamic environmental and spacing conditions. It reproduced the ground coverage ratio dynamics observed in measured images, achieving R^2=0.84 when driven by measured shoot dry weight and R^2=0.40 (0.76 excluding one outlier) when driven by PBM-simulated values. Fourth, the Unity rendering engine converted canopy layout representations into RGB and segmentation images at less than 10~ms. Fifth, a demonstration showed that a lighting-control policy can be learned and applied by observing only crop images that were generated with LettuceVisSim, providing a proof of concept of vision-based reinforcement learning in CEA using LettuceVisSim.
ER -