A Distributional Framework for Behavioral Characterization and Risk Management of Large Language Models
Abstract
Large language models are stochastic by nature. Repeated executions under the same experimental conditions may produce different responses. We propose a framework that can be applied across the model lifecycle, from task design to residual behavior and continuous monitoring. The method consists of building an empirical response distribution estimated through Monte Carlo simulation. Semantic interventions are introduced to study how these distributions change. We illustrate the framework with two different tasks, borrower classification and music album recommendation, and quantify the resulting distributional changes. The framework is linked to model risk management and governance by measuring robustness and control effectiveness, and characterizing residual behavior. Since it relies only on observed responses, it can also be applied to black-box models accessed through commercial APIs.
Related
- When are dividend barriers optimal? A distributional characterization in renewal risk models
- Local Whittle estimation for distributional time series: A regularization framework in Bayes Hilbert space
- Quantifying Behavioral Tails in Black-Box Language Models
- Large-Scale Comparison of Metaheuristic Algorithms via Behavioral Verification of Large Language Model-Generated Codes
- Right, Wrong, or Unfinished: A Competing Risks Framework for Language-Model Reasoning under Token Budgets
BibTeX
@misc{garcialorite2026distributional,
title = {{A Distributional Framework for Behavioral Characterization and Risk Management of Large Language Models}},
author = {David García-Lorite and Raul Merino},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.12718},
eprint = {2610.12718},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.12718},
}
RIS
TY - UNPB
TI - A Distributional Framework for Behavioral Characterization and Risk Management of Large Language Models
AU - David García-Lorite
AU - Raul Merino
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.12718
UR - https://arxiv.org/abs/2610.12718
AB - Large language models are stochastic by nature. Repeated executions under the same experimental conditions may produce different responses. We propose a framework that can be applied across the model lifecycle, from task design to residual behavior and continuous monitoring. The method consists of building an empirical response distribution estimated through Monte Carlo simulation. Semantic interventions are introduced to study how these distributions change. We illustrate the framework with two different tasks, borrower classification and music album recommendation, and quantify the resulting distributional changes. The framework is linked to model risk management and governance by measuring robustness and control effectiveness, and characterizing residual behavior. Since it relies only on observed responses, it can also be applied to black-box models accessed through commercial APIs.
ER -