Benchmarking Open-Source Vision–Language Models for Food Image Recognition and Portion Size Estimation
4 affiliations
- 1Dipartimento di Neuroscienze, Sezione di Biofisica, Università Cattolica del Sacro Cuore, 00168 Rome, Italy
- 2UOC Fisica per le Scienze della Vita, Fondazione Policlinico Universitario Agostino Gemelli, IRCCS (Istituto di Ricovero e Cura a Carattere Scientifico), 00168 Rome, Italy
- 3Gruppo Fastal Blu Sistemi, Via Nomentana 263, 00161 Rome, Italy
- 4Digital Innovation Hub Roma, Chirale S.r.l., Via Ignazio Persico 32-46, 00154 Rome, Italy
Abstract
Background/Objectives: Accurate recognition of food items and estimation of portion sizes from images is a key challenge in advancing dietary assessment and nutritional monitoring. In this study, we benchmark twenty-six configurations (seventeen model releases) of state-of-the-art open-source Vision–Language Models (VLMs), varying in size (≤15 GB) and number of parameters (<25B), on real-world food photographs collected in a university canteen, doing so to evaluate their potential for supporting automated nutrition logging and health interventions. Methods: We construct a curated benchmark dataset with human-annotated labels, including both food item names and scale-measured portion sizes, and assess model predictions based on recognition performance and mean absolute error in the portion estimate. The semantic matcher used to score food recognition is validated against a panel of human judges, and all results are reported with bootstrap confidence intervals and stochastic replicates. Results: Our experiments show that the VLM qwen2.5vl:7b achieves the highest overall performance in food identification (F1=0.68), outperforming other open-source models in terms of precision and F1 score. Under the human-validated semantic matcher, it also outperforms the more recent successors of the same model families, which nevertheless improve portion estimation and, under a threshold-free LLM judge less sensitive to verbose descriptions, match or exceed it in recognition. For comparison, we evaluate five proprietary models (four OpenAI generations and one Anthropic model) as a benchmark and find that they tend to outperform open-source VLMs in both food recognition (best F1=0.77, GPT-4o) and portion size estimation error; notably, however, the most recent proprietary generations do not improve on their predecessors for this task under our evaluation protocol. Conclusions: These findings highlight the potential and current limitations of general-purpose VLMs in fine-grained food understanding tasks, with implications for the design of scalable, vision-based dietary assessment systems in digital health.
Related
- Benchmarking Open-Source Vision-Language Models for Brain Metastasis Assessment on Single-Slice Contrast-Enhanced MRI
- Development and validation of a food photography atlas for portion size estimation of traditional Croatian dishes in adults.
- Unifying Vision and Language: Benchmarking End to End Transformer Model Against the ClipCap Framework
- Benchmarking modular skills and fixed orchestration across six large language models in a meta-analysis task
- BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
BibTeX
@article{marchetti2026benchmarking,
title = {{Benchmarking Open-Source Vision–Language Models for Food Image Recognition and Portion Size Estimation}},
author = {Tommaso Marchetti and Stefano Capezzone and Rosita Esposito and Marco De Spirito and Giuseppe Maulucci},
journal = {Nutrients},
year = {2026},
month = oct,
doi = {10.3390/nu18213582},
url = {https://www.mdpi.com/2072-6643/18/21/3582},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Benchmarking Open-Source Vision–Language Models for Food Image Recognition and Portion Size Estimation
AU - Tommaso Marchetti
AU - Stefano Capezzone
AU - Rosita Esposito
AU - Marco De Spirito
AU - Giuseppe Maulucci
PY - 2026
DA - 2026/10/29
JO - Nutrients
DO - 10.3390/nu18213582
UR - https://www.mdpi.com/2072-6643/18/21/3582
AB - Background/Objectives: Accurate recognition of food items and estimation of portion sizes from images is a key challenge in advancing dietary assessment and nutritional monitoring. In this study, we benchmark twenty-six configurations (seventeen model releases) of state-of-the-art open-source Vision–Language Models (VLMs), varying in size (≤15 GB) and number of parameters (<25B), on real-world food photographs collected in a university canteen, doing so to evaluate their potential for supporting automated nutrition logging and health interventions. Methods: We construct a curated benchmark dataset with human-annotated labels, including both food item names and scale-measured portion sizes, and assess model predictions based on recognition performance and mean absolute error in the portion estimate. The semantic matcher used to score food recognition is validated against a panel of human judges, and all results are reported with bootstrap confidence intervals and stochastic replicates. Results: Our experiments show that the VLM qwen2.5vl:7b achieves the highest overall performance in food identification (F1=0.68), outperforming other open-source models in terms of precision and F1 score. Under the human-validated semantic matcher, it also outperforms the more recent successors of the same model families, which nevertheless improve portion estimation and, under a threshold-free LLM judge less sensitive to verbose descriptions, match or exceed it in recognition. For comparison, we evaluate five proprietary models (four OpenAI generations and one Anthropic model) as a benchmark and find that they tend to outperform open-source VLMs in both food recognition (best F1=0.77, GPT-4o) and portion size estimation error; notably, however, the most recent proprietary generations do not improve on their predecessors for this task under our evaluation protocol. Conclusions: These findings highlight the potential and current limitations of general-purpose VLMs in fine-grained food understanding tasks, with implications for the design of scalable, vision-based dietary assessment systems in digital health.
SN - 2072-6643
ER -