pxGPT: a schema-constrained multimodal large language model framework for plant phenotyping
- 1Biotechnology Center in Southern Taiwan, Academia Sinica , Tainan 711010,
- 2Agricultural Biotechnology Research Center, Academia Sinica , Taipei 115201,
- 3Biotechnology Center, National Chung Hsing University , Taichung 402202,
Abstract
Summary: Deep learning can scale image-based plant phenotyping but typically targets predefined traits, whereas manual phenotypic description is more flexible but difficult to scale. We present Phenotype eXplorer Generative Plant Traits (pxGPT), a Python framework that converts multi-view plant images into structured phenotype records. The workflow combines free-text description, dataset-specific schema generation with human review, and schema-constrained information extraction. We evaluated the extraction stage on three Brassica datasets using 78 973 annotations from 10 annotators who had no access to pxGPT predictions. For 38–45 qualitative traits per dataset, pxGPT-human agreement ranged from 0.572 to 0.712, compared with 0.620 to 0.739 agreement among human annotators. All 32 736 required trait fields were present; 93 values were empty, and no qualitative descriptor violations were recorded. For selected morphological structures, pooled false-positive rates were 0.5%–8.0%, while sensitivity varied across structure types in one mature-plant dataset (37.5%–100%). The curated schemas were applied unchanged with two commercial models and one locally hosted open-weight model, with differences in mean accuracy of 0.03 or less. In the evaluated Brassica datasets, these results support schema-guided extraction of visible plant traits without separate model training for each trait. Human-guided schema development offers a path to extend pxGPT to other crops and imaging conditions, with accuracy requiring validation in each new setting.
Availability and implementation: The software is available at https://github.com/YaoChengLab/pxgpt and an archival version of the source code is available on Zenodo (https://doi.org/10.5281/zenodo.22931745). Curated schemas used in this study, annotation guides, phenotype records, and human annotations are available at https://doi.org/10.5281/zenodo.22907554.
Related
- Multimodal cellular context enables molecular phenotyping from single-molecule trajectories
- Multimodal phenotyping reveals plant-food-related dietary information in serum Raman spectra of community-dwelling Japanese adults
- Predicted Effector Gene Aggregation, Standards and Unified Schema (PEGASUS): A Community Framework for Effector Gene Reporting
- High-Resolution and Multimodal Structural Phenotyping of Intermediate and Advanced Non-Neovascular Age-Related Macular Degeneration
- AltraFlowSOM: A Semi-Supervised Framework for Imaging Mass Cytometry Phenotyping
BibTeX
@article{zheng2026pxgpt,
title = {{pxGPT: a schema-constrained multimodal large language model framework for plant phenotyping}},
author = {Po-Xing Zheng and Jheng-Yan Ou and Yu-Lin Wu and Ruei-Shiuan Wang and Ting-Li Wu and Te-Chang Hsu and Yao-Cheng Lin},
journal = {Bioinformatics},
year = {2026},
month = oct,
doi = {10.1093/bioinformatics/btag781},
url = {https://academic.oup.com/bioinformatics/article/doi/10.1093/bioinformatics/btag781/8956098},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - pxGPT: a schema-constrained multimodal large language model framework for plant phenotyping
AU - Po-Xing Zheng
AU - Jheng-Yan Ou
AU - Yu-Lin Wu
AU - Ruei-Shiuan Wang
AU - Ting-Li Wu
AU - Te-Chang Hsu
AU - Yao-Cheng Lin
PY - 2026
DA - 2026/10/20
JO - Bioinformatics
DO - 10.1093/bioinformatics/btag781
UR - https://academic.oup.com/bioinformatics/article/doi/10.1093/bioinformatics/btag781/8956098
AB - Summary: Deep learning can scale image-based plant phenotyping but typically targets predefined traits, whereas manual phenotypic description is more flexible but difficult to scale. We present Phenotype eXplorer Generative Plant Traits (pxGPT), a Python framework that converts multi-view plant images into structured phenotype records. The workflow combines free-text description, dataset-specific schema generation with human review, and schema-constrained information extraction. We evaluated the extraction stage on three Brassica datasets using 78 973 annotations from 10 annotators who had no access to pxGPT predictions. For 38–45 qualitative traits per dataset, pxGPT-human agreement ranged from 0.572 to 0.712, compared with 0.620 to 0.739 agreement among human annotators. All 32 736 required trait fields were present; 93 values were empty, and no qualitative descriptor violations were recorded. For selected morphological structures, pooled false-positive rates were 0.5%–8.0%, while sensitivity varied across structure types in one mature-plant dataset (37.5%–100%). The curated schemas were applied unchanged with two commercial models and one locally hosted open-weight model, with differences in mean accuracy of 0.03 or less. In the evaluated Brassica datasets, these results support schema-guided extraction of visible plant traits without separate model training for each trait. Human-guided schema development offers a path to extend pxGPT to other crops and imaging conditions, with accuracy requiring validation in each new setting. Availability and implementation: The software is available at https://github.com/YaoChengLab/pxgpt and an archival version of the source code is available on Zenodo (https://doi.org/10.5281/zenodo.22931745). Curated schemas used in this study, annotation guides, phenotype records, and human annotations are available at https://doi.org/10.5281/zenodo.22907554.
SN - 1367-4803
SN - 1367-4811
ER -