Unifying Conformal Language Tasks with In-Context Ensembles
Abstract
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
Related
- On the direct SLE-martingale approach to the integrability of Conformal Loop Ensembles
- Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
- When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning
- Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift
- Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models
BibTeX
@misc{huang2026unifying,
title = {{Unifying Conformal Language Tasks with In-Context Ensembles}},
author = {Xiao Shi Huang and Chen-Yuan Lin and Bruce Kuwahara and Kin Kwan Leung and Jesse C. Cresswell},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.03005},
eprint = {2609.03005},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.03005},
}
RIS
TY - UNPB
TI - Unifying Conformal Language Tasks with In-Context Ensembles
AU - Xiao Shi Huang
AU - Chen-Yuan Lin
AU - Bruce Kuwahara
AU - Kin Kwan Leung
AU - Jesse C. Cresswell
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2609.03005
UR - https://arxiv.org/abs/2609.03005
AB - Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
ER -