Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
Abstract
The emergence of foundation models for molecular property prediction requires a high degree of AI data readiness, including reliable metadata annotation. However, both public repositories and industrial screening databases suffer from missing, inconsistent, or conflated assay annotations. In this work, we quantify the extent of missing annotations in PubChem for the BioAssay Ontology (BAO) assay format and physical detection method fields and investigate whether open-source and proprietary large language models (LLMs) can reliably predict and audit metadata annotations directly from the assay text. In our assessment, we found that the annotation coverage across PubChem's ∼2 million bioassays is critically sparse, 36\% lacking an assay format, 89\% a BioAssay type, and >99.9\% any BAO-mapped assay format or detection technology term. This motivates the need for automated test-metadata curation. Using evaluation sets derived from PubChem and ChEMBL, we assess the agreement of seven open-source and proprietary LLMs with existing silver labels. Recall is at least 0.96 for biochemical and cell-based assay formats, with a similar pattern for detection technology, although disagreements increase on under-represented classes. Manual inspection shows that many of these disagreements trace back to inconsistencies between silver sources rather than to LLM error. Moreover, in a qualitative study with a senior industrial curator, LLM-generated evidence prompted the expert to revise some of their own labels, showing LLMs can flag potentially mislabeled assays. Across the study, performance differences between proprietary and open-source models were small. Together, these results suggest LLMs can support the large-scale annotation and auditing of assay metadata, though per-class reliability estimates and targeted human review remain necessary before such labels enter downstream ML pipelines.
Related
- Language models annotate mass cytometry clusters within the range where human experts disagree
- Development of a peptide-based bioassay for zearalenone detection via phage display and in silico analysis
- In Situ Reproductive Bioassay with Caged Gammarus fossarum (Crustacea): Part 2—Evaluating the Relevance of Using a Molt Cycle Temperature‐Dependent Model as a Reference to Assess Toxicity in Freshwater Monitoring
- Toxicity of sediment-bound pollutants in the Seine estuary, France, using a Eurytemora affinis larval bioassay
- Can agents reliably design therapeutic antibodies?
BibTeX
@misc{weesep2026llms,
title = {{Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?}},
author = {Laura van Weesep and Riccardo Tedoldi and Jens Sjölund and Hossein Azizpour and Susanne Winiwarter and Ola Engkvist and Jon Paul Janet and Samuel Genheden and Juan Viguera Diez},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01616},
eprint = {2610.01616},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01616},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
AU - Laura van Weesep
AU - Riccardo Tedoldi
AU - Jens Sjölund
AU - Hossein Azizpour
AU - Susanne Winiwarter
AU - Ola Engkvist
AU - Jon Paul Janet
AU - Samuel Genheden
AU - Juan Viguera Diez
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01616
UR - https://arxiv.org/abs/2610.01616
AB - The emergence of foundation models for molecular property prediction requires a high degree of AI data readiness, including reliable metadata annotation. However, both public repositories and industrial screening databases suffer from missing, inconsistent, or conflated assay annotations. In this work, we quantify the extent of missing annotations in PubChem for the BioAssay Ontology (BAO) assay format and physical detection method fields and investigate whether open-source and proprietary large language models (LLMs) can reliably predict and audit metadata annotations directly from the assay text. In our assessment, we found that the annotation coverage across PubChem's ∼2 million bioassays is critically sparse, 36\% lacking an assay format, 89\% a BioAssay type, and >99.9\% any BAO-mapped assay format or detection technology term. This motivates the need for automated test-metadata curation. Using evaluation sets derived from PubChem and ChEMBL, we assess the agreement of seven open-source and proprietary LLMs with existing silver labels. Recall is at least 0.96 for biochemical and cell-based assay formats, with a similar pattern for detection technology, although disagreements increase on under-represented classes. Manual inspection shows that many of these disagreements trace back to inconsistencies between silver sources rather than to LLM error. Moreover, in a qualitative study with a senior industrial curator, LLM-generated evidence prompted the expert to revise some of their own labels, showing LLMs can flag potentially mislabeled assays. Across the study, performance differences between proprietary and open-source models were small. Together, these results suggest LLMs can support the large-scale annotation and auditing of assay metadata, though per-class reliability estimates and targeted human review remain necessary before such labels enter downstream ML pipelines.
ER -