Robust Active Learning for Few-Shot Example Selection in Text-to-SQL
Abstract
Domain-specific text-to-SQL systems ground a large language model by retrieving annotated few-shot examples, and each example needs expert-written SQL. We treat the choice of which queries to annotate as constrained experimental design on the low-dimensional manifold of query embeddings, with query-dependent annotation noise, a partition matroid constraint that spreads selections across semantic domains, and an unknown covariance structure. We propose a stratified greedy algorithm that maximizes a heteroscedastic information-gain objective. We prove that the objective is monotone and submodular under query-dependent noise, so stratified greedy selection carries a 1/2-approximation guarantee under the partition constraint. Under kernel misspecification the guarantee degrades by an additive spectral term; we compute it on both experimental pools and find it too large for the bound to be quantitatively informative. To connect the design objective to the downstream task, we give a retrieval model that bounds few-shot accuracy from below by per-domain fill distance, demonstration noise, and domain coverage, and we calibrate its locality assumption on both pools. On an enterprise supply-chain corpus and on the BIRD benchmark, the selected banks improve cross-domain retrieval and end-to-end LLM SQL over random and distance-based selection at the same annotation budget. Stratified controls and pre-specified tests show that the gain comes from the partition constraint: uniform sampling within each stratum matches the full method in the oracle-label evaluations, farthest-point selection within strata adds a little at small budgets, and the noise weighting has no measurable effect. The practical advice is to annotate one example per domain per batch from the first batch on.
Related
- Learning from pandemics: Feature selection and clustering for robust digital disease nowcasting
- Covariate Selection for Doubly Robust Double/debiased Machine Learning Estimators for Causal Inference
- Towards optimal selection schedules for online active learning
- Numerical Realization of an Entropy-Based Selection Principle in the Example of Kelvin-Helmholtz Instability
- Robust Model Order Selection via Dithered Differential Step-Down Thresholding
BibTeX
@misc{pourhabib2026robust,
title = {{Robust Active Learning for Few-Shot Example Selection in Text-to-SQL}},
author = {Arash Pourhabib},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.10125},
eprint = {2606.10125},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.10125},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Robust Active Learning for Few-Shot Example Selection in Text-to-SQL
AU - Arash Pourhabib
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2606.10125
UR - https://arxiv.org/abs/2606.10125
AB - Domain-specific text-to-SQL systems ground a large language model by retrieving annotated few-shot examples, and each example needs expert-written SQL. We treat the choice of which queries to annotate as constrained experimental design on the low-dimensional manifold of query embeddings, with query-dependent annotation noise, a partition matroid constraint that spreads selections across semantic domains, and an unknown covariance structure. We propose a stratified greedy algorithm that maximizes a heteroscedastic information-gain objective. We prove that the objective is monotone and submodular under query-dependent noise, so stratified greedy selection carries a 1/2-approximation guarantee under the partition constraint. Under kernel misspecification the guarantee degrades by an additive spectral term; we compute it on both experimental pools and find it too large for the bound to be quantitatively informative. To connect the design objective to the downstream task, we give a retrieval model that bounds few-shot accuracy from below by per-domain fill distance, demonstration noise, and domain coverage, and we calibrate its locality assumption on both pools. On an enterprise supply-chain corpus and on the BIRD benchmark, the selected banks improve cross-domain retrieval and end-to-end LLM SQL over random and distance-based selection at the same annotation budget. Stratified controls and pre-specified tests show that the gain comes from the partition constraint: uniform sampling within each stratum matches the full method in the oracle-label evaluations, farthest-point selection within strata adds a little at small budgets, and the noise weighting has no measurable effect. The practical advice is to annotate one example per domain per batch from the first batch on.
ER -