Effects of Dish Name, Visible Amount, Persona, and Explicit Instruction on Energy Estimates From Food Photographs by 26 Vision-Language Models: Comparative Evaluation Study
- 1Research Institute of Info-Communication Medicine
Abstract
BackgroundVision-language models (VLMs) are being considered for estimating energy intake from meal photographs. Prompt descriptions such as a Japanese-dietitian persona shift the estimates, suggesting their use for calibration; what the estimates rest on was unknown.
ObjectiveWe examined whether prompt descriptions can serve as calibration, at which training stage responses to a nationality persona and to an amount instruction arise, and whether estimates follow the visible amount or one associated with the identified dish.
MethodsEleven commercial and 15 open-weight models were evaluated on NutriImage (887 Japanese single-dish photographs) and SNAPMe (1,134 US meal photographs) under eight prompt conditions (no description, five personas, and smaller- or larger-portion instructions). With one model, images were cropped or replaced by the dish name. Before-and after-meal photographs of 804 meals (ACETADA) were estimated. Pre-trained and post-trained weights of Gemma 3 and InternVL3.5 8B were compared by likelihood scoring. Prompts were compared with numerical calibration.
ResultsWhere the dish was named as in the full image, a 25% crop changed neither the median absolute percentage error (MedAPE; 33.8% vs 34.4%) nor the persona shift (-20 vs -20 kcal); dish names alone gave similar accuracy (MedAPE 31.7%); nearly empty trays still received 39%-75% of the before-meal estimate, often described as already eaten. Between models, the persona shift was related to baseline bias (r = -0.77) and less to instruction following (r = -0.36); the instruction shift showed the reverse (r = -0.14 and -0.95). In Gemma 3, the Japan-US contrast was 2-4 times larger before post-training, and the instruction response 2-4 times larger after it (InternVL3.5 8B: 3 and 1.4 times). Across images, the Japan-US contrast was uniform on US meals and, for Japanese dishes in the commercial models, Gemma 3, and Qwen3-VL 32B, larger where overestimated. In these models, the persona lowered MedAPE for Japanese dishes as much as, or slightly more than, subtracting the mean bias, and no more than dividing by a ratio; on US meals it did not; in open-weight families that responded little, only numerical calibration lowered error. The smaller-portion instruction raised MedAPE in models with little bias (eg, 30.0% to 50.2%).
ConclusionsIn the model examined most closely, the estimates behaved as if anchored on an amount associated with the identified dish and only partly corrected by the visible amount. Prompt descriptions moved them by amounts depending on model and cuisine, not on the correction needed; a persona is therefore not proposed as a general means of calibration. In a facility, the findings suggest showing the identified dish for correction, calibrating by a ratio measured on reference meals with the deployed model, keeping amount statements out of the prompt, and not deriving intake from before- after-meal differences.
Related
- Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
- Divergent Cardiovascular Disease Burden in Hungary and Iraq: A Comparative Analysis of Global Burden of Disease Estimates
- Evaluation of accelerated whole-body diffusion weighted imaging with deep learning reconstruction in patients with metastatic prostate cancer: assessment of image quality and ADC estimates.
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
BibTeX
@misc{nakagawa2026effects,
title = {{Effects of Dish Name, Visible Amount, Persona, and Explicit Instruction on Energy Estimates From Food Photographs by 26 Vision-Language Models: Comparative Evaluation Study}},
author = {S. Nakagawa and A. Yamamoto},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.09.26365134},
url = {https://www.medrxiv.org/content/10.64898/2026.10.09.26365134v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Effects of Dish Name, Visible Amount, Persona, and Explicit Instruction on Energy Estimates From Food Photographs by 26 Vision-Language Models: Comparative Evaluation Study
AU - S. Nakagawa
AU - A. Yamamoto
PY - 2026
DA - 2026/10/09
PB - medRxiv
DO - 10.64898/2026.10.09.26365134
UR - https://www.medrxiv.org/content/10.64898/2026.10.09.26365134v1
AB - BackgroundVision-language models (VLMs) are being considered for estimating energy intake from meal photographs. Prompt descriptions such as a Japanese-dietitian persona shift the estimates, suggesting their use for calibration; what the estimates rest on was unknown. ObjectiveWe examined whether prompt descriptions can serve as calibration, at which training stage responses to a nationality persona and to an amount instruction arise, and whether estimates follow the visible amount or one associated with the identified dish. MethodsEleven commercial and 15 open-weight models were evaluated on NutriImage (887 Japanese single-dish photographs) and SNAPMe (1,134 US meal photographs) under eight prompt conditions (no description, five personas, and smaller- or larger-portion instructions). With one model, images were cropped or replaced by the dish name. Before-and after-meal photographs of 804 meals (ACETADA) were estimated. Pre-trained and post-trained weights of Gemma 3 and InternVL3.5 8B were compared by likelihood scoring. Prompts were compared with numerical calibration. ResultsWhere the dish was named as in the full image, a 25% crop changed neither the median absolute percentage error (MedAPE; 33.8% vs 34.4%) nor the persona shift (-20 vs -20 kcal); dish names alone gave similar accuracy (MedAPE 31.7%); nearly empty trays still received 39%-75% of the before-meal estimate, often described as already eaten. Between models, the persona shift was related to baseline bias (r = -0.77) and less to instruction following (r = -0.36); the instruction shift showed the reverse (r = -0.14 and -0.95). In Gemma 3, the Japan-US contrast was 2-4 times larger before post-training, and the instruction response 2-4 times larger after it (InternVL3.5 8B: 3 and 1.4 times). Across images, the Japan-US contrast was uniform on US meals and, for Japanese dishes in the commercial models, Gemma 3, and Qwen3-VL 32B, larger where overestimated. In these models, the persona lowered MedAPE for Japanese dishes as much as, or slightly more than, subtracting the mean bias, and no more than dividing by a ratio; on US meals it did not; in open-weight families that responded little, only numerical calibration lowered error. The smaller-portion instruction raised MedAPE in models with little bias (eg, 30.0% to 50.2%). ConclusionsIn the model examined most closely, the estimates behaved as if anchored on an amount associated with the identified dish and only partly corrected by the visible amount. Prompt descriptions moved them by amounts depending on model and cuisine, not on the correction needed; a persona is therefore not proposed as a general means of calibration. In a facility, the findings suggest showing the identified dish for correction, calibrating by a ratio measured on reference meals with the deployed model, keeping amount statements out of the prompt, and not deriving intake from before- after-meal differences.
ER -