Beyond Search Scores: Retrieval Quality, Computational Cost, and LLM Judgment Sensitivity in a Multilingual Research Archive
- 1State University of Trade and Economics, Kyiv, Ukraine
Abstract
Research archives mix languages, file formats and retrieval cues, making the choice between lexical, dense and hybrid search an empirical question.
We study these trade-offs in archive-index, a local system combining SQLite BM25, Ukrainian/Russian lemma expansion and multimodal embeddings.
The constructed UA-PAR collection contains 7,070 public-source files, 200 model-written queries and 5,572 pooled query–document pairs assessed by language models.
On its 170 answerable queries, mean nDCG@10 under reviewed Gemini labels is 0.674 for multilingual-E5, 0.633 for EmbeddingGemma 2, 0.629 for adaptive fusion, 0.482 for lemmatized BM25 and 0.450 for plain BM25.
Adaptive fusion exceeds BM25 after Holm correction but has no demonstrated advantage over dense EG2 or reciprocal-rank fusion; evidence for the lemma gain depends on the relevance judge.
Restricting EG2 to the same text candidates as E5 changes its score by less than 0.002.
In a separate 50-record extension with initial Gemini labels, EG2 and adaptive fusion obtain 0.759 and 0.756, versus 0.721 for E5; the cohort contains 49 distinct wordings.
Further experiments distinguish ordinary from condensed GGUF evaluation, optimized vector selection from the actual archive-ranking path, and source-dataset image/speech matching from model-judged text retrieval.
Costs come from one laptop in an ordinary desktop session; the private-archive study reports operational aggregates.
The original answerable targets are all notes, and no human relevance audit is included.
The results characterize implemented retrieval trade-offs under these conditions rather than establish a generally superior fusion rule.
Related
- Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
- AFP assistant: a retrieval-augmented generation and large language model-powered multilingual polio chatbot for low-resource language communities
- Regime-aware retrieval of lower-tropospheric stability and a sensitivity-weighted diagnostic of cross-site transferability over heterogeneous land surfaces
- Computational aberration-retrieval with entangled photons
- Computational identification of candidate biomarkers and compounds associated with skeletal muscle insulin sensitivity improvement mediated by physical activity: an integrative multi-omics and computational chemistry study.
BibTeX
@misc{pylypenko2026beyond,
title = {{Beyond Search Scores: Retrieval Quality, Computational Cost, and LLM Judgment Sensitivity in a Multilingual Research Archive}},
author = {Oleksandr Pylypenko},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23602155},
url = {https://zenodo.org/records/23602155},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Beyond Search Scores: Retrieval Quality, Computational Cost, and LLM Judgment Sensitivity in a Multilingual Research Archive
AU - Oleksandr Pylypenko
PY - 2026
DA - 2026/10/23
PB - Zenodo
DO - 10.5281/zenodo.23602155
UR - https://zenodo.org/records/23602155
AB - Research archives mix languages, file formats and retrieval cues, making the choice between lexical, dense and hybrid search an empirical question. We study these trade-offs in archive-index, a local system combining SQLite BM25, Ukrainian/Russian lemma expansion and multimodal embeddings. The constructed UA-PAR collection contains 7,070 public-source files, 200 model-written queries and 5,572 pooled query–document pairs assessed by language models. On its 170 answerable queries, mean nDCG@10 under reviewed Gemini labels is 0.674 for multilingual-E5, 0.633 for EmbeddingGemma 2, 0.629 for adaptive fusion, 0.482 for lemmatized BM25 and 0.450 for plain BM25. Adaptive fusion exceeds BM25 after Holm correction but has no demonstrated advantage over dense EG2 or reciprocal-rank fusion; evidence for the lemma gain depends on the relevance judge. Restricting EG2 to the same text candidates as E5 changes its score by less than 0.002. In a separate 50-record extension with initial Gemini labels, EG2 and adaptive fusion obtain 0.759 and 0.756, versus 0.721 for E5; the cohort contains 49 distinct wordings. Further experiments distinguish ordinary from condensed GGUF evaluation, optimized vector selection from the actual archive-ranking path, and source-dataset image/speech matching from model-judged text retrieval. Costs come from one laptop in an ordinary desktop session; the private-archive study reports operational aggregates. The original answerable targets are all notes, and no human relevance audit is included. The results characterize implemented retrieval trade-offs under these conditions rather than establish a generally superior fusion rule.
ER -