Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat
Abstract
The wheat canopy genome harbors abundant yet untapped genetic variation that could be harnessed to enhance yield potential. The green area index (GAI) is a structural metric that reflects the photosynthetically active canopy surface and is closely linked to final grain yield. Current image-based GAI retrieval methods often suffer from signal saturation and coarse structural depiction, constraining downstream genetic analyses. To address this limitation, we constructed a comprehensive image dataset spanning eight field experiments across China and France, encompassing approximately 600 genotypes under six distinct management regimes. Leveraging this diverse data, we developed a multimodal deep-learning framework augmented by simulated-to-realistic (sim2real) synthetic data transfer. This framework fuses nadir and oblique RGB images with accumulated thermal time to produce high-precision, time-series GAI estimates. Validated on independent testing datasets from both China and France, the multimodal approach demonstrated robust performance with an accuracy of R 2 = 0.88 and an RMSE of 0.49 m 2 m -2 , representing an improvement of about 22% over the traditional gap fraction method. In three site-year field experiments involving 565 genotypes, the GAI dynamics derived from the multimodal approach showed higher broad-sense heritability (0.20-0.48) than those from the gap fraction approach (0.02-0.13) and stronger genotypic correlations with yield (0.19-0.40 versus 0.09-0.31). Furthermore, genetic analysis confirmed the biological fidelity of the estimated traits, identifying loci that co-localize with known architectural regulators such as Rht-D1 , TaTB1-4D , and TaBGC1-4D . Consistently, the multimodal-derived phenotypes were specifically enriched in cell-wall remodeling and hormonal signaling pathways (e.g., brassinosteroid) that directly regulate canopy expansion. Overall, the proposed method offers a powerful tool for unlocking genetic gain in canopy architecture and accelerating canopy-targeted wheat improvement.
Related
- GAMPoint: Geometry-augmented vision transformer with a multimodal-ready interface for 3D point-cloud segmentation
- Generative-augmented multimodal neuroaffective decoding for product evaluation: toward neurofeedback-driven human-in-the-loop co-design
- Cophylogeny simulators are not interchangeable: similarities, differences and structural biases in synthetic host-symbiont
- High-Resolution and Multimodal Structural Phenotyping of Intermediate and Advanced Non-Neovascular Age-Related Macular Degeneration
- iNaturalist records reveal shifts in phenology of caterpillar (Lepidoptera) activity in response to warmer spring temperatures
BibTeX
@article{song2026syntheticaugmented,
title = {{Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat}},
author = {Zhenming Song and Chen Zhu and Rui Yu and Pingtao Jiang and Yangmingrui Gao and Benoit de Solan and Jianhui Wu and Dejun Han and Yanfeng Ding and Frédéric Baret and Ruixi Han and Shouyang Liu},
journal = {Plant Phenomics},
year = {2026},
month = oct,
doi = {10.1016/j.plaphe.2026.100192},
url = {https://hal.inrae.fr/hal-05781675v1},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat
AU - Zhenming Song
AU - Chen Zhu
AU - Rui Yu
AU - Pingtao Jiang
AU - Yangmingrui Gao
AU - Benoit de Solan
AU - Jianhui Wu
AU - Dejun Han
AU - Yanfeng Ding
AU - Frédéric Baret
AU - Ruixi Han
AU - Shouyang Liu
PY - 2026
DA - 2026/10/06
JO - Plant Phenomics
DO - 10.1016/j.plaphe.2026.100192
UR - https://hal.inrae.fr/hal-05781675v1
AB - The wheat canopy genome harbors abundant yet untapped genetic variation that could be harnessed to enhance yield potential. The green area index (GAI) is a structural metric that reflects the photosynthetically active canopy surface and is closely linked to final grain yield. Current image-based GAI retrieval methods often suffer from signal saturation and coarse structural depiction, constraining downstream genetic analyses. To address this limitation, we constructed a comprehensive image dataset spanning eight field experiments across China and France, encompassing approximately 600 genotypes under six distinct management regimes. Leveraging this diverse data, we developed a multimodal deep-learning framework augmented by simulated-to-realistic (sim2real) synthetic data transfer. This framework fuses nadir and oblique RGB images with accumulated thermal time to produce high-precision, time-series GAI estimates. Validated on independent testing datasets from both China and France, the multimodal approach demonstrated robust performance with an accuracy of R 2 = 0.88 and an RMSE of 0.49 m 2 m -2 , representing an improvement of about 22% over the traditional gap fraction method. In three site-year field experiments involving 565 genotypes, the GAI dynamics derived from the multimodal approach showed higher broad-sense heritability (0.20-0.48) than those from the gap fraction approach (0.02-0.13) and stronger genotypic correlations with yield (0.19-0.40 versus 0.09-0.31). Furthermore, genetic analysis confirmed the biological fidelity of the estimated traits, identifying loci that co-localize with known architectural regulators such as Rht-D1 , TaTB1-4D , and TaBGC1-4D . Consistently, the multimodal-derived phenotypes were specifically enriched in cell-wall remodeling and hormonal signaling pathways (e.g., brassinosteroid) that directly regulate canopy expansion. Overall, the proposed method offers a powerful tool for unlocking genetic gain in canopy architecture and accelerating canopy-targeted wheat improvement.
SN - 2643-6515
ER -