Benchmarking thermoelectric property extraction from scientific text
Abstract
Databases of thermoelectric properties are built largely by extracting values from the literature, and the choice of extraction method determines how many records are recovered and how many are wrong. We compare five methods on the same paragraph-level task: a corpus-informed regular-expression baseline, the published ChemDataExtractor thermoelectric parsers, a MaterialsBERT-based named-entity model with rule-based linking, a single-prompt large language model (LLM) and an LLM agent. The methods were developed on a 100-paragraph development sample (the rules of the regular-expression baseline earlier, on the whole corpus), frozen, and run on a 200-paragraph test sample drawn by stratified sampling from 311 papers (mostly arXiv preprints), with a paper-level split between the two samples. A record counts as correct only if property, value, temperature and material all match the gold standard. On the 160 in-scope test paragraphs (241 core records), the single-prompt LLM reached F1 = 0.79 (95 % CI 0.71–0.87) and the agent 0.73 (0.64–0.83), against 0.22 for the regular-expression baseline and 0.06 for the ChemDataExtractor and NER pipelines. Both LLM methods recover property, value and temperature with F1 above 0.94; most of their remaining losses arise from the material criterion; in most of these cases one material name contains the other. We found no evidence of an accuracy improvement from the tested agent configuration, which differs from the single prompt in several components at once. We release the protocol, code, gold standard and all method outputs in a public repository.
Related
- Two-Dimensional C60 Fullerene Networks (Graphullerene): Structure–Property Relations, Quantitative Benchmarking and Future Directions
- Automated Extraction and FAIRification of Chemical Structures from Scientific Publications
- A computed thermoelectric feature database for 50,992 GNoME materials
- Analytical and experimental study of a thermoelectric generator system for low grade chimney waste heat recovery
- Materials design strategies for green thermoelectric generators enabling sustainable and scalable energy harvesting
BibTeX
@misc{uyanık2026benchmarking,
title = {{Benchmarking thermoelectric property extraction from scientific text}},
author = {Berna Uyanık},
howpublished = {ChemRxiv},
year = {2026},
month = oct,
doi = {10.26434/chemrxiv.15010913/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010913/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Benchmarking thermoelectric property extraction from scientific text
AU - Berna Uyanık
PY - 2026
DA - 2026/10/22
PB - ChemRxiv
DO - 10.26434/chemrxiv.15010913/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010913/v1
AB - Databases of thermoelectric properties are built largely by extracting values from the literature, and the choice of extraction method determines how many records are recovered and how many are wrong. We compare five methods on the same paragraph-level task: a corpus-informed regular-expression baseline, the published ChemDataExtractor thermoelectric parsers, a MaterialsBERT-based named-entity model with rule-based linking, a single-prompt large language model (LLM) and an LLM agent. The methods were developed on a 100-paragraph development sample (the rules of the regular-expression baseline earlier, on the whole corpus), frozen, and run on a 200-paragraph test sample drawn by stratified sampling from 311 papers (mostly arXiv preprints), with a paper-level split between the two samples. A record counts as correct only if property, value, temperature and material all match the gold standard. On the 160 in-scope test paragraphs (241 core records), the single-prompt LLM reached F1 = 0.79 (95 % CI 0.71–0.87) and the agent 0.73 (0.64–0.83), against 0.22 for the regular-expression baseline and 0.06 for the ChemDataExtractor and NER pipelines. Both LLM methods recover property, value and temperature with F1 above 0.94; most of their remaining losses arise from the material criterion; in most of these cases one material name contains the other. We found no evidence of an accuracy improvement from the tested agent configuration, which differs from the single prompt in several components at once. We release the protocol, code, gold standard and all method outputs in a public repository.
ER -