OptiMatLLM: A Benchmark Suite and a Lightweight Domain-Specific LLM for Optical Materials Research
4 affiliations
- 1Key Laboratory of Spectral Imaging Technology
- 2Xi’an Institute of Optics and Precision Mechanics of the Chinese Academy of Sciences
- 3School of Software
- 4Nanjing University of Information Science & Technology
Abstract
The design of optical materials relies on highly specialized knowledge, which creates a major bottleneck for data-driven scientific discovery in the optical material domain. Large language models open new possibilities for scientific research, but general-purpose models struggle with the domainspecific physical logic and terminology. More importantly, no standardized evaluation benchmark exists to systematically assess how well LLMs perform on optical materials tasks. To fill this gap, we first construct two evaluation benchmarks, Optic_Choice and Optic_QA, designed specifically for the optical materials domain. We build a high-quality instruction-tuning dataset of 23,225 question-answer pairs through an automated multi-stage pipeline from a curated corpus of papers and textbook. We then develop OptiMatLLM, a 7-billion-parameter LLM fine-tuned LLM on this corpus. Evaluation results show that OptiMatLLM achieves top performance on our optical benchmarks, reaching 0.79 accuracy on Optic_Choice, while also performing competitively on broader scientific tests such as CMHChem, ChemBench, and MaScQA. The model retains strong general capabilities as well, scoring 0.91 on the GSM8K mathematical reasoning benchmark. We further provide a systematic comparison of multiple leading models on these benchmarks, establishing a reference for future work. All benchmarks, dataset, and model are publicly released to support further research on LLMs for optical and optoelectronic materials design.
Related
- Rescuable Stagnation: A Diagnostic for Benchmark Suitability in LLM-Guided Materials Optimization
- An Audited Open Record of Three Deep-Blue TADF Host Materials with Benchmark Tasks for Machine Learning
- MatUQ: a benchmark for uncertainty-aware out-of-distribution materials property prediction with graph neural networks
- Microplastic Release from Drinking Water Contact Materials: Material-Specific Responses to Chlorination and Implications for Consumer Exposure
- Workflow design as a research variable in computational materials discovery
BibTeX
@misc{wang2026optimatllm,
title = {{OptiMatLLM: A Benchmark Suite and a Lightweight Domain-Specific LLM for Optical Materials Research}},
author = {Xinyi Wang and Youle Wang and Lei Zhang},
howpublished = {ChemRxiv},
year = {2026},
month = sep,
doi = {10.26434/chemrxiv.15004991/v2},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15004991/v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - OptiMatLLM: A Benchmark Suite and a Lightweight Domain-Specific LLM for Optical Materials Research
AU - Xinyi Wang
AU - Youle Wang
AU - Lei Zhang
PY - 2026
DA - 2026/09/04
PB - ChemRxiv
DO - 10.26434/chemrxiv.15004991/v2
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15004991/v2
AB - The design of optical materials relies on highly specialized knowledge, which creates a major bottleneck for data-driven scientific discovery in the optical material domain. Large language models open new possibilities for scientific research, but general-purpose models struggle with the domainspecific physical logic and terminology. More importantly, no standardized evaluation benchmark exists to systematically assess how well LLMs perform on optical materials tasks. To fill this gap, we first construct two evaluation benchmarks, Optic_Choice and Optic_QA, designed specifically for the optical materials domain. We build a high-quality instruction-tuning dataset of 23,225 question-answer pairs through an automated multi-stage pipeline from a curated corpus of papers and textbook. We then develop OptiMatLLM, a 7-billion-parameter LLM fine-tuned LLM on this corpus. Evaluation results show that OptiMatLLM achieves top performance on our optical benchmarks, reaching 0.79 accuracy on Optic_Choice, while also performing competitively on broader scientific tests such as CMHChem, ChemBench, and MaScQA. The model retains strong general capabilities as well, scoring 0.91 on the GSM8K mathematical reasoning benchmark. We further provide a systematic comparison of multiple leading models on these benchmarks, establishing a reference for future work. All benchmarks, dataset, and model are publicly released to support further research on LLMs for optical and optoelectronic materials design.
ER -