IoT-enabled multimodal large language model-based framework for compositional food recognition, nutrient profiling, NOVA classification, and cancer risk mitigation
- 1Department of Science and Environmental Studies, The Education University of Hong Kong, Hong Kong, Hong Kong SAR, China
- 2Department of Health and Physical Education, The Education University of Hong Kong, Hong Kong, Hong Kong SAR, China
- 3Department of Psychology, The Education University of Hong Kong, Hong Kong, Hong Kong SAR, China
Abstract
BackgroundAutomated dietary assessment increasingly struggles to evaluate industrial food processing levels, revealing a systemic gap between the visual presentation of meals and their underlying ultra-processed food composition. Although existing dietary monitoring tools can effectively track caloric and macronutrient intake, they lack the qualitative processing assessments required to identify hidden additives and map dietary risk triage quadrants.MethodsThis study introduces a modular, IoT-enabled multimodal dietary decision-support framework that decouples visual food recognition, standardized database retrieval, and qualitative language reasoning for automated NOVA processing classification and 2D decision matrix mapping. Operating on isolated single-dish meal instances, the system employs a four-phase cascaded pipeline that integrates vision-language models, database retrieval, and large language models (LLMs) with strict JSON schema enforcement. The platform was evaluated using the Nutrition-NOVA validation benchmark, a harmonized multimodal dataset constructed through dense vector-space semantic linkage between quantitative profiles from the Center for Food Safety database and qualitative NOVA tiers from Open Food Facts. This benchmark addresses a 75.0% metadata sparsity deficit identified in legacy repositories.ResultsIn experimental evaluations on the Nutrition-NOVA benchmark, the baseline Qwen-VL-Max architecture achieved an adjacent accuracy of 77.7%. Furthermore, enforcing schema modularity enabled complete plug-and-play backend replaceability between commercial cloud application programming interfaces (APIs) and private, on-premises open-weight models without requiring architectural retraining.DiscussionThe proposed framework establishes an efficient, safety-first screening tool for digital healthcare decision support. By isolating qualitative LLM reasoning from visual perceptual noise and database metadata sparsity, the system successfully bridges the gap between intuitive meal perception and point-of-care dietary risk triage, offering a non-invasive and scalable solution for automated dietary auditing and longitudinal population health monitoring.
Related
- An uncertainty-aware multimodal deep learning framework for Alzheimer’s disease classification using MRI and electronic health records
- HalluciGuard-eye: knowledge graph-enhanced retrieval-augmented generation for hallucination detection and mitigation in ophthalmic multimodal large language models
- Preoperative recognition of appendiceal diverticulitis: a case report with multimodal correlation
- The Impact of Bias Mitigation on Fairness and Accuracy in Automated Skin Lesion Classification
- Artificial Intelligence-Driven Nanobody Engineering for Pathogen Recognition: From Functional Epitope Prioritization to Multimodal Foundation Models
BibTeX
@article{he2026iotenabled,
title = {{IoT-enabled multimodal large language model-based framework for compositional food recognition, nutrient profiling, NOVA classification, and cancer risk mitigation}},
author = {Zebang He and Steve W. Y. Mung and Peggy Pui Lai Or and Chetwyn C. H. Chan},
journal = {Frontiers in Nutrition},
year = {2026},
month = oct,
doi = {10.3389/fnut.2026.1970182},
url = {https://doi.org/10.3389/fnut.2026.1970182},
note = {CC BY},
}
RIS
TY - JOUR
TI - IoT-enabled multimodal large language model-based framework for compositional food recognition, nutrient profiling, NOVA classification, and cancer risk mitigation
AU - Zebang He
AU - Steve W. Y. Mung
AU - Peggy Pui Lai Or
AU - Chetwyn C. H. Chan
PY - 2026
DA - 2026/10/26
JO - Frontiers in Nutrition
DO - 10.3389/fnut.2026.1970182
UR - https://doi.org/10.3389/fnut.2026.1970182
AB - BackgroundAutomated dietary assessment increasingly struggles to evaluate industrial food processing levels, revealing a systemic gap between the visual presentation of meals and their underlying ultra-processed food composition. Although existing dietary monitoring tools can effectively track caloric and macronutrient intake, they lack the qualitative processing assessments required to identify hidden additives and map dietary risk triage quadrants.MethodsThis study introduces a modular, IoT-enabled multimodal dietary decision-support framework that decouples visual food recognition, standardized database retrieval, and qualitative language reasoning for automated NOVA processing classification and 2D decision matrix mapping. Operating on isolated single-dish meal instances, the system employs a four-phase cascaded pipeline that integrates vision-language models, database retrieval, and large language models (LLMs) with strict JSON schema enforcement. The platform was evaluated using the Nutrition-NOVA validation benchmark, a harmonized multimodal dataset constructed through dense vector-space semantic linkage between quantitative profiles from the Center for Food Safety database and qualitative NOVA tiers from Open Food Facts. This benchmark addresses a 75.0% metadata sparsity deficit identified in legacy repositories.ResultsIn experimental evaluations on the Nutrition-NOVA benchmark, the baseline Qwen-VL-Max architecture achieved an adjacent accuracy of 77.7%. Furthermore, enforcing schema modularity enabled complete plug-and-play backend replaceability between commercial cloud application programming interfaces (APIs) and private, on-premises open-weight models without requiring architectural retraining.DiscussionThe proposed framework establishes an efficient, safety-first screening tool for digital healthcare decision support. By isolating qualitative LLM reasoning from visual perceptual noise and database metadata sparsity, the system successfully bridges the gap between intuitive meal perception and point-of-care dietary risk triage, offering a non-invasive and scalable solution for automated dietary auditing and longitudinal population health monitoring.
SN - 2296-861X
ER -