Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
- 1Research Institute of Info-Communication Medicine
Abstract
BackgroundEstimating nutrient intake from meal photographs with vision-language models (VLMs) is now practical, but patient photographs often cannot leave the institution. We developed PLANS (Personal-device Local Assessment of Nutrients System), in which a small open VLM runs entirely on a personal device.
MethodsOpen VLMs were compared on two image sets with reference values (NutriImage, 887 Japanese meals; SNAPMe, 1,134 US meals), and the selected model (Qwen3-VL-4B, 4-bit) was run on an iPad Air without network access. Energy (primary outcome), protein, fat, carbohydrate and salt were evaluated by median absolute percentage error (MedAPE) and Spearmans {rho}. The effect of supplying the food weight, including errors in it, was examined.
ResultsEnergy estimates on the iPad were identical to those of a desktop-computer run for 84-86% of images (other values, 75-87%). MedAPE on the iPad was 37.0% and 36.5% for energy (26-28% of images within {+/-}20%), 43-65% for the macronutrients and 68-101% for salt ({rho} 0.51-0.85). On the desktop computer, supplying the true weight reduced MedAPE of the 4B model on NutriImage from 37.0% to 32.5% for energy and from 100.0% to 66.8% for salt; energy MedAPE stayed below the base value for declared weights of 0.5-1.25 times the true weight or rounded to 50 g, but rose markedly when the container weight was included.
ConclusionsAn offline VLM on a tablet reproduced server-side energy accuracy and is a candidate tool for dietary counselling where photographs cannot leave the institution; given the wide limits of agreement, its usefulness in practice remains to be examined.
Related
- Development and quality evaluation of lyophilized plasma external quality assessment samples for coagulation testing in resource-limited settings
- Development and Population-Based Prognostic Utility Assessment of an ICD-10-CM-Based Prediction Model to Identify Left Ventricular Systolic Dysfunction: Discrimination Performance and Population-Based Evaluation in 83,357 Patients with Acute Coronary Syndrome
- RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment
- Development and psychometric evaluation of a pre-sarcopenia risk assessment scale for sedentary young adults with type 2 diabetes mellitus
- Effects of Dish Name, Visible Amount, Persona, and Explicit Instruction on Energy Estimates From Food Photographs by 26 Vision-Language Models: Comparative Evaluation Study
BibTeX
@misc{nakagawa2026development,
title = {{Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study}},
author = {S. Nakagawa and A. Yamamoto},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.11.26365273},
url = {https://www.medrxiv.org/content/10.64898/2026.10.11.26365273v1},
note = {CC BY-ND},
}
RIS
TY - UNPB
TI - Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
AU - S. Nakagawa
AU - A. Yamamoto
PY - 2026
DA - 2026/10/11
PB - medRxiv
DO - 10.64898/2026.10.11.26365273
UR - https://www.medrxiv.org/content/10.64898/2026.10.11.26365273v1
AB - BackgroundEstimating nutrient intake from meal photographs with vision-language models (VLMs) is now practical, but patient photographs often cannot leave the institution. We developed PLANS (Personal-device Local Assessment of Nutrients System), in which a small open VLM runs entirely on a personal device. MethodsOpen VLMs were compared on two image sets with reference values (NutriImage, 887 Japanese meals; SNAPMe, 1,134 US meals), and the selected model (Qwen3-VL-4B, 4-bit) was run on an iPad Air without network access. Energy (primary outcome), protein, fat, carbohydrate and salt were evaluated by median absolute percentage error (MedAPE) and Spearmans {rho}. The effect of supplying the food weight, including errors in it, was examined. ResultsEnergy estimates on the iPad were identical to those of a desktop-computer run for 84-86% of images (other values, 75-87%). MedAPE on the iPad was 37.0% and 36.5% for energy (26-28% of images within {+/-}20%), 43-65% for the macronutrients and 68-101% for salt ({rho} 0.51-0.85). On the desktop computer, supplying the true weight reduced MedAPE of the 4B model on NutriImage from 37.0% to 32.5% for energy and from 100.0% to 66.8% for salt; energy MedAPE stayed below the base value for declared weights of 0.5-1.25 times the true weight or rounded to 50 g, but rose markedly when the container weight was included. ConclusionsAn offline VLM on a tablet reproduced server-side energy accuracy and is a candidate tool for dietary counselling where photographs cannot leave the institution; given the wide limits of agreement, its usefulness in practice remains to be examined.
ER -