Internal Consistency Does Not Validate an LLM-Assisted Nutrition Pipeline: A Reference-Standard-Free Forensic Reliability Audit of 13,073 Free-Text Meal Records
- 1Neptura, Libano 731040, Tolima, Colombia
- 2Facultad de Ciencias Humanas, Corporación Universitaria Minuto de Dios—UNIMINUTO, Bogotá D.C. 111021, Colombia
- 3Facultad de Ingeniería, Universidad Distrital Francisco José de Caldas, Bogotá D.C. 111711, Colombia
Abstract
Background/Objectives: Large language model (LLM) pipelines increasingly convert free-text meal descriptions into nutrient values, yet reference-standard validation is rarely feasible in production. We asked whether the controls such pipelines conventionally pass suffice and built a reference-standard-free battery any pipeline can run on its output. Methods: We audited 13,073 meals logged by 513 adults through a WhatsApp conversational agent in rural Colombia (9 February 2026 to 6 August 2026). The pipeline is a two-stage cascade of commercial LLMs from different providers: a vision model renders photographs as text, while a text model maps these to nutrients. Six pre-specified tests were applied, including test–retest ICC(1,1) over recurring strings, stratified by modality because 48.3% of records carried a photograph. Results: The pipeline passed every conventional control (Atwater-reconstructed energy r = 0.9951; median relative error 0.0000) but failed reproducibility. Across 191 recurring strings (944 meals), energy ICC point estimates were 0.7061 (95% CI 0.5419–0.8217) text only and 0.5258 (0.3778–0.6513) with photographs; none reached 0.75. Modality is a design control, not an established effect, and the coefficient depends on which inputs recur, so it is not intrinsic. Under the strictest identity available—byte-identical text, no photograph—ICC was 0.6944. Intermediate-text inspection exposed a failure invisible at the output: 1.44% of photograph-accompanied records carried a vision-stage artefact string versus 0.01% of text-only records (ICC 0.0911). Conclusions: Internal consistency does not validate an LLM-assisted nutrition pipeline. This design measures instability under recurring inputs without isolating its cause. Prospective test–retest reproducibility is a cheap, necessary acceptance criterion, and in a cascade, the intermediate representation must also be inspected.
Related
- Structural validity and internal consistency of the Client Satisfaction Questionnaire-8 among parents of children with ADHD: a cross-sectional study
- Psychometric validation of the persian short form of the physical literacy in children questionnaire in Iranian children aged 4-12 years: factorial validity, internal consistency, and gender-based measurement invariance.
- Intra- and Inter-Operator Reliability of Internal Jugular Vein Ultrasound Image Acquisition by Ultrasound-Naive Nurses.
- A 3D-printed phantom to validate subject orientation in 3D imaging and recordings
- Explainable machine learning and multi-layer transcriptomic analysis identify and validate prognostic biomarkers for breast cancer survival.
BibTeX
@article{torres2026internal,
title = {{Internal Consistency Does Not Validate an LLM-Assisted Nutrition Pipeline: A Reference-Standard-Free Forensic Reliability Audit of 13,073 Free-Text Meal Records}},
author = {Jhony Fernando López Torres and Juan Gabriel Castañeda and Edwin Rivas},
journal = {Nutrients},
year = {2026},
month = sep,
doi = {10.3390/nu18193233},
url = {https://europepmc.org/article/PMC/PMC13667244},
note = {CC BY},
}
RIS
TY - JOUR
TI - Internal Consistency Does Not Validate an LLM-Assisted Nutrition Pipeline: A Reference-Standard-Free Forensic Reliability Audit of 13,073 Free-Text Meal Records
AU - Jhony Fernando López Torres
AU - Juan Gabriel Castañeda
AU - Edwin Rivas
PY - 2026
DA - 2026/09/30
JO - Nutrients
DO - 10.3390/nu18193233
UR - https://europepmc.org/article/PMC/PMC13667244
AB - Background/Objectives: Large language model (LLM) pipelines increasingly convert free-text meal descriptions into nutrient values, yet reference-standard validation is rarely feasible in production. We asked whether the controls such pipelines conventionally pass suffice and built a reference-standard-free battery any pipeline can run on its output. Methods: We audited 13,073 meals logged by 513 adults through a WhatsApp conversational agent in rural Colombia (9 February 2026 to 6 August 2026). The pipeline is a two-stage cascade of commercial LLMs from different providers: a vision model renders photographs as text, while a text model maps these to nutrients. Six pre-specified tests were applied, including test–retest ICC(1,1) over recurring strings, stratified by modality because 48.3% of records carried a photograph. Results: The pipeline passed every conventional control (Atwater-reconstructed energy r = 0.9951; median relative error 0.0000) but failed reproducibility. Across 191 recurring strings (944 meals), energy ICC point estimates were 0.7061 (95% CI 0.5419–0.8217) text only and 0.5258 (0.3778–0.6513) with photographs; none reached 0.75. Modality is a design control, not an established effect, and the coefficient depends on which inputs recur, so it is not intrinsic. Under the strictest identity available—byte-identical text, no photograph—ICC was 0.6944. Intermediate-text inspection exposed a failure invisible at the output: 1.44% of photograph-accompanied records carried a vision-stage artefact string versus 0.01% of text-only records (ICC 0.0911). Conclusions: Internal consistency does not validate an LLM-assisted nutrition pipeline. This design measures instability under recurring inputs without isolating its cause. Prospective test–retest reproducibility is a cheap, necessary acceptance criterion, and in a cascade, the intermediate representation must also be inspected.
SN - 2072-6643
ER -