Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition

Abstract

This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis

Hypothesis-Driven Autonomous Materials Synthesis with Multimodal LLM Agents
Izumi Takahara, Kazunori Nishio, Akira Aiba +4
arXiv16 Sep 2026PreprintCC BY 4.0Full textCondensed matterarXiv
Mined from Scientific Literature: Process Schemas for Atomic Layer Deposition and Etching in Materials Science
Sameer Sadruddin, Eleni Poupaki, Alex Watkins +5
arXiv10 Sep 2026PreprintCC BY 4.0Full textCondensed matterarXiv
Scaling LLM Agents for Materials Design through Hierarchical Collective Reasoning
Jaehwan Choi, Yousung Jung
arXiv15 Sep 2026PreprintarXiv licenceFull textCondensed matterarXiv
OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
Humza Nusrat, Kundan Thind
arXiv27 Oct 2026PreprintCC BY 4.0Full textBiological & medical physicsarXiv
Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments
Jiaqi Cai, Siyu Cheng, William S. Huxter +10
arXiv30 Oct 2026PreprintarXiv licenceFull textCondensed matterarXiv

Cite

BibTeX RIS

BibTeX

@misc{yanguasgil2026evaluating,
  title = {{Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition}},
  author = {Angel Yanguas-Gil},
  howpublished = {arXiv},
  year = {2026},
  month = aug,
  doi = {10.48550/arxiv.2608.29309},
  eprint = {2608.29309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.29309},
}

RIS

TY  - UNPB
TI  - Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition
AU  - Angel Yanguas-Gil
PY  - 2026
DA  - 2026/08/29
PB  - arXiv
DO  - 10.48550/arxiv.2608.29309
UR  - https://arxiv.org/abs/2608.29309
AB  - This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis
ER  -

Abstract

Related

Cite