Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models

Abstract

Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or rationales, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.

Learning Choice Model Trees for Feature-Based Multi-Product Pricing: Exact Optimization and Field Evidence
Jiajie Zhang, Yanqiu Ruan, Xiao Jin +1
arXiv15 Sep 2026PreprintarXiv licenceFull textOptimisation & control StatisticsarXiv
FEMA-Long: Modeling unstructured covariances for discovery of time-dependent effects in large-scale longitudinal datasets
Pravesh Parekh, Nadine Parker, Diliana Pecheva +23
PLoS Genetics11 Jun 2026Peer-reviewedCC BYFull text5 citedStatisticsOpenAlex
The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics
Zong Shang, Tomoya Wakayama, Guillaume Lecué +1
arXiv30 Jul 2026PreprintarXiv licenceFull textStatisticsarXiv
Statistical Feature Augmentation for Anomaly Detection in Dynamic Graphs
Philipp Schlinge, Jean-Luc Schnipper, Martin Atzmueller
arXiv2 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Interpretable Intelligent Fault Diagnosis of Active Magnetic Bearing–Rotor Systems Based on Multidimensional Feature Fusions with Robustness Assessment
Ruikun Li, Yuntian Ta, Qitao Yin +2
Mathematics25 Oct 2026Peer-reviewedCC BY 4.0General mathematicsCrossref

Cite

BibTeX RIS

BibTeX

@misc{arnaudo2026interpretable,
  title = {{Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models}},
  author = {Baptiste Arnaudo and Amine Bennouna and Sébastien Martin},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.38082},
  eprint = {2610.38082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38082},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models
AU  - Baptiste Arnaudo
AU  - Amine Bennouna
AU  - Sébastien Martin
PY  - 2026
DA  - 2026/10/27
PB  - arXiv
DO  - 10.48550/arxiv.2610.38082
UR  - https://arxiv.org/abs/2610.38082
AB  - Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or rationales, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.
ER  -

Abstract

Related

Cite