A Shadow-Mode Benchmark of Frontier Vision-Language and Decision Models for Parenteral Vial Inspection
Abstract
A reproducible public-data benchmark (not a GMP line qualification) asking, with honest metrics, whether frontier AI can be trusted in the decision step of pharmaceutical visual inspection. Built on the public ResiDual dataset (Rosati, Marchesini, Lumetti, et al., "Identifying Impurities in Liquids of Pharmaceutical Vials," ICPR 2024, DOI 10.1007/978-3-031-78447-7_18; 730 spin-stop sequences / 13,880 frames / 5 classes). Images are NOT redistributed — only code, predictions, metrics, and audit trails; obtain ResiDual from UNIMORE AImageLab and cite the authors.
Study 1 (pre-registered): a trained ResNet-18 baseline vs a zero-shot panel (GPT-4o, GPT-5.1, Gemini-2.5-Pro, Qwen2.5-VL-72B, Claude-Sonnet-5.5, Grok-4.7) and two Jev 1.14 arms (image + text-judge), scored sequence-level with recall / specificity / AUROC + silent-miss + ECE calibration + per-impurity recall. Finding: VLMs rank dirty above clean well (AUROC 0.81–0.99) but cannot be trusted to decide zero-shot — they either reject most clean vials or silently miss defects; no model is calibrated (ECE 0.13–0.41); a small trained CNN solves the clean benchmark.
Study 2: an under-trained (imperfect) CNN baseline on degraded images (recall 1.00, specificity 0.656, 11 false-rejects, 0 misses), then a decision-model shadow layer (Jev 1.14, pre-registered; plus exploratory OpenAI GPT-6 Luna Decisions API and GPT-6.1 chat) tested for whether it can recover false-rejects without clearing real defects. Cross-vendor finding: no decision-model, in either mode, is a safe recovery layer — every layer that recovers false-rejects also clears real defects; the only "safe" ones recover almost nothing. This holds across two independent vendors (TypeSafe Jev and OpenAI Luna Decisions).
Scope / honesty: public-data benchmark on (degraded) research images, NOT a validated GMP line; Jev 1.14 image = experimental preview snapshot; the GPT-6.1/Luna arms were not in the frozen pre-registration (labeled exploratory); zero-shot contamination not excludable (the trained CNN is the controlled anchor); "PoD" is a class-level proxy. Pre-registered on OSF (DOI 10.17605/OSF.IO/UH9ZA; protocol frozen before analysis, git commit fe4dba9). Independent research; no employer; no monetization.
Related
- Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models
- Training-Label Efficiency in Vision-Based Inspection of LNG Tank Cables: A Duplicate-Aware and Calibration-Controlled Benchmark of Supervised and Normal-Only Anomaly Detection
- A Macro-to-Micro Framework for Bridge-Deck Pavement-Distress Inspection Using LiDAR-Based Screening and YOLOv8s Image Detection: A Case Study of the Jingzhou Yangtze River Highway Bridge and Public Benchmark Data
- Scoring the NIH Stroke Scale from bedside exam transcripts: a synthetic benchmark of a decision model and a large language model
- An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph
BibTeX
@misc{pandey2026shadowmode,
title = {{A Shadow-Mode Benchmark of Frontier Vision-Language and Decision Models for Parenteral Vial Inspection}},
author = {Gourav Pandey},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23462165},
url = {https://zenodo.org/records/23462165},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Shadow-Mode Benchmark of Frontier Vision-Language and Decision Models for Parenteral Vial Inspection
AU - Gourav Pandey
PY - 2026
DA - 2026/10/19
PB - Zenodo
DO - 10.5281/zenodo.23462165
UR - https://zenodo.org/records/23462165
AB - A reproducible public-data benchmark (not a GMP line qualification) asking, with honest metrics, whether frontier AI can be trusted in the decision step of pharmaceutical visual inspection. Built on the public ResiDual dataset (Rosati, Marchesini, Lumetti, et al., "Identifying Impurities in Liquids of Pharmaceutical Vials," ICPR 2024, DOI 10.1007/978-3-031-78447-7_18; 730 spin-stop sequences / 13,880 frames / 5 classes). Images are NOT redistributed — only code, predictions, metrics, and audit trails; obtain ResiDual from UNIMORE AImageLab and cite the authors. Study 1 (pre-registered): a trained ResNet-18 baseline vs a zero-shot panel (GPT-4o, GPT-5.1, Gemini-2.5-Pro, Qwen2.5-VL-72B, Claude-Sonnet-5.5, Grok-4.7) and two Jev 1.14 arms (image + text-judge), scored sequence-level with recall / specificity / AUROC + silent-miss + ECE calibration + per-impurity recall. Finding: VLMs rank dirty above clean well (AUROC 0.81–0.99) but cannot be trusted to decide zero-shot — they either reject most clean vials or silently miss defects; no model is calibrated (ECE 0.13–0.41); a small trained CNN solves the clean benchmark. Study 2: an under-trained (imperfect) CNN baseline on degraded images (recall 1.00, specificity 0.656, 11 false-rejects, 0 misses), then a decision-model shadow layer (Jev 1.14, pre-registered; plus exploratory OpenAI GPT-6 Luna Decisions API and GPT-6.1 chat) tested for whether it can recover false-rejects without clearing real defects. Cross-vendor finding: no decision-model, in either mode, is a safe recovery layer — every layer that recovers false-rejects also clears real defects; the only "safe" ones recover almost nothing. This holds across two independent vendors (TypeSafe Jev and OpenAI Luna Decisions). Scope / honesty: public-data benchmark on (degraded) research images, NOT a validated GMP line; Jev 1.14 image = experimental preview snapshot; the GPT-6.1/Luna arms were not in the frozen pre-registration (labeled exploratory); zero-shot contamination not excludable (the trained CNN is the controlled anchor); "PoD" is a class-level proxy. Pre-registered on OSF (DOI 10.17605/OSF.IO/UH9ZA; protocol frozen before analysis, git commit fe4dba9). Independent research; no employer; no monetization.
ER -