Vision-Language Agents for Active Perception in Optics Laboratories
Abstract
Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.
Related
- Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures
- Partial vision leads to an unexpected emergent collective behavior in active aligning particles
- Region-Aware Degradation-Conditioned Vision-Language Learning for Blind Image-Quality Assessment of Power Field-Operation Imagery
- Toward a social psychology of AI: language-model agents reproduce human-like minimal-group bias
- Quantum Entangled Multimodal Fusion Networks (QEMFN): Resource-Aware Hybrid Vision-Language Fusion via Trainable Entanglement
BibTeX
@misc{lopez2026visionlanguage,
title = {{Vision-Language Agents for Active Perception in Optics Laboratories}},
author = {Ryan Lopez and Sachin Vaidya and Seou Choi and Serena Landers and Marin Soljačić},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32918},
eprint = {2609.32918},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32918},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Vision-Language Agents for Active Perception in Optics Laboratories
AU - Ryan Lopez
AU - Sachin Vaidya
AU - Seou Choi
AU - Serena Landers
AU - Marin Soljačić
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32918
UR - https://arxiv.org/abs/2609.32918
AB - Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.
ER -