Test-Time Augmentation for LLMs: Where to Inject Diversity at Inference Time
Abstract
Aggregating several LLM responses to the same question can improve accuracy, but every method that does so must decide where the responses should differ. Established test-time scaling methods such as self-consistency vary only the reasoning path, resampling from a fixed input. The input modification offers several further options: the wording of the question can be rephrased, the spelling can be perturbed, and the accompanying images can be transformed. We study Test-Time Augmentation (TTA), which aggregates predictions across transformed versions of the input, and use it to ask where diversity is best injected. Comparing these options across six benchmarks at a matched number of inference calls, we find that where the diversity comes from matters more than how much of it there is. Meaning-preserving paraphrases of the input yield the largest accuracy gains (+1.80 pp over a single call) and reach peak accuracy with the fewest samples. Yet, they are not the most diverse: character-level noise makes the responses disagree far more often (37.8% versus 27.4%) while gaining less accuracy, and perturbing two modalities at once falls below the single-call baseline. Not all diversity is equally useful: perturbations that keep the meaning intact produce disagreement that aggregation can use, while those that corrupt it produce disagreement that is wrong. Overall, our results suggest that for mid-tier LLMs such as Claude Haiku, varying how a question is phrased is a better use of an inference call than varying the reasoning path. Our TTA implementation is available at github.com/aws-samples/sample-genai-reflection-for-bedrock.
Related
- Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation
- [DeepGOF] Where Does a Logistic Risk Model Fail? An Audited Neural Goodness-of-Fit Test for Model Development and External Validation
- Where Do Two Populations of Persistence Diagrams Differ? Calibrated Local Inference at a Fixed Budget
- What Does a Benford Test Actually Test? Marginal Conformity, Sampling Structure, and Forensic Inference
- Spectra: Exact Component Transport for Test-Time Prior Adaptation in Simulation-Based Inference
BibTeX
@misc{kozodoi2026testtime,
title = {{Test-Time Augmentation for LLMs: Where to Inject Diversity at Inference Time}},
author = {Nikita Kozodoi and Zainab Afolabi and Jack Butler},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.09351},
eprint = {2608.09351},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.09351},
}
RIS
TY - UNPB
TI - Test-Time Augmentation for LLMs: Where to Inject Diversity at Inference Time
AU - Nikita Kozodoi
AU - Zainab Afolabi
AU - Jack Butler
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2608.09351
UR - https://arxiv.org/abs/2608.09351
AB - Aggregating several LLM responses to the same question can improve accuracy, but every method that does so must decide where the responses should differ. Established test-time scaling methods such as self-consistency vary only the reasoning path, resampling from a fixed input. The input modification offers several further options: the wording of the question can be rephrased, the spelling can be perturbed, and the accompanying images can be transformed. We study Test-Time Augmentation (TTA), which aggregates predictions across transformed versions of the input, and use it to ask where diversity is best injected. Comparing these options across six benchmarks at a matched number of inference calls, we find that where the diversity comes from matters more than how much of it there is. Meaning-preserving paraphrases of the input yield the largest accuracy gains (+1.80 pp over a single call) and reach peak accuracy with the fewest samples. Yet, they are not the most diverse: character-level noise makes the responses disagree far more often (37.8% versus 27.4%) while gaining less accuracy, and perturbing two modalities at once falls below the single-call baseline. Not all diversity is equally useful: perturbations that keep the meaning intact produce disagreement that aggregation can use, while those that corrupt it produce disagreement that is wrong. Overall, our results suggest that for mid-tier LLMs such as Claude Haiku, varying how a question is phrased is a better use of an inference call than varying the reasoning path. Our TTA implementation is available at github.com/aws-samples/sample-genai-reflection-for-bedrock.
ER -