Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes
- 1Stanford University
Abstract
BackgroundImage-based dietary assessment is valuable for precision diabetes care. Multimodal large language models (mLLMs) may substantially facilitate dietary self-monitoring. However, comprehensive evaluation of their real-world accuracy remains limited.
MethodsWe developed SeeFood, an open-source benchmark, and evaluated eight commercial and open-source mLLMs on 80 real-world meal photographs from the SNAPMe dataset for nutrient and energy estimation (calories, carbohydrates, protein, fat, fiber), food-item identification, and day-level diet quality (Healthy Eating Index, HEI-2020). Three finalists were evaluated on the complete dataset (1680 photographs, 275 participant-days), and the best model was analyzed by meal type, portion size, sizing-marker use, ingredient visibility and meal complexity.
ResultsGPT-4o mini achieved the lowest errors, with a significant advantage for carbohydrates; the three finalists performed similarly on other outcomes and on HEI-2020. Mean absolute percentage errors (MAPE) remained high (52%-90%), and the direction of bias was model-specific. The carbohydrate mean absolute error was 13.3 g (95% CI 12.4-14.3), and only 57% of meals were within {+/-}10 g, a clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were in the range reported for human estimation on other images ([~]55% MAPE), whereas drink-only images had the highest relative errors (312% MAPE) because of dissolved sugars. Food items were reliably identified when fully visible (Dice = 0.74), and HEI-2020 estimates showed moderate agreement with reference values (RMSE = 12.5 points; r = 0.61). Larger portions were increasingly underestimated, and sizing references did not improve accuracy. Newer or larger models and chain-of-thought prompting did not consistently improve performance.
ConclusionsCurrent mLLMs can extract meaningful dietary information from real-world meal photographs, but substantial errors in nutrient quantification and portion-size estimation limit their use for precision diabetes management. Integrating visual assessment with additional contextual information may enable more reliable, lower-burden dietary monitoring.
Related
- Clinical performance and practical limitations of large language models in respiratory medicine
- Expert-Guided Visual Correction for Characterizing Diagnostic Performance and Error Patterns of Multimodal Large Language Models Using Periodontal In-Service Examination Images
- Diagnostic and treatment recommendation performance of multimodal large language models in failed or painful total hip arthroplasty: an expert-rated benchmark study.
- Nutritional Accuracy and Clinical Applicability of Large Language Model-Generated Therapeutic Diet Plans for Phenylketonuria: A Comparative Simulation Study
- Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study
BibTeX
@misc{gravellier2026imagebased,
title = {{Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes}},
author = {L. Gravellier and C. Zhang and Y. Wu and V. Parameswaran and A. Scheideman and M. Shao and E. Zhang and D. Perelman and C. D. Gardner and D. C. Klonoff and H. Raeder and M. P. Snyder and T. Wang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.21.26365949},
url = {https://www.medrxiv.org/content/10.64898/2026.10.21.26365949v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes
AU - L. Gravellier
AU - C. Zhang
AU - Y. Wu
AU - V. Parameswaran
AU - A. Scheideman
AU - M. Shao
AU - E. Zhang
AU - D. Perelman
AU - C. D. Gardner
AU - D. C. Klonoff
AU - H. Raeder
AU - M. P. Snyder
AU - T. Wang
PY - 2026
DA - 2026/10/21
PB - medRxiv
DO - 10.64898/2026.10.21.26365949
UR - https://www.medrxiv.org/content/10.64898/2026.10.21.26365949v1
AB - BackgroundImage-based dietary assessment is valuable for precision diabetes care. Multimodal large language models (mLLMs) may substantially facilitate dietary self-monitoring. However, comprehensive evaluation of their real-world accuracy remains limited. MethodsWe developed SeeFood, an open-source benchmark, and evaluated eight commercial and open-source mLLMs on 80 real-world meal photographs from the SNAPMe dataset for nutrient and energy estimation (calories, carbohydrates, protein, fat, fiber), food-item identification, and day-level diet quality (Healthy Eating Index, HEI-2020). Three finalists were evaluated on the complete dataset (1680 photographs, 275 participant-days), and the best model was analyzed by meal type, portion size, sizing-marker use, ingredient visibility and meal complexity. ResultsGPT-4o mini achieved the lowest errors, with a significant advantage for carbohydrates; the three finalists performed similarly on other outcomes and on HEI-2020. Mean absolute percentage errors (MAPE) remained high (52%-90%), and the direction of bias was model-specific. The carbohydrate mean absolute error was 13.3 g (95% CI 12.4-14.3), and only 57% of meals were within {+/-}10 g, a clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were in the range reported for human estimation on other images ([~]55% MAPE), whereas drink-only images had the highest relative errors (312% MAPE) because of dissolved sugars. Food items were reliably identified when fully visible (Dice = 0.74), and HEI-2020 estimates showed moderate agreement with reference values (RMSE = 12.5 points; r = 0.61). Larger portions were increasingly underestimated, and sizing references did not improve accuracy. Newer or larger models and chain-of-thought prompting did not consistently improve performance. ConclusionsCurrent mLLMs can extract meaningful dietary information from real-world meal photographs, but substantial errors in nutrient quantification and portion-size estimation limit their use for precision diabetes management. Integrating visual assessment with additional contextual information may enable more reliable, lower-burden dietary monitoring.
ER -