Black-box model classification under the discriminative factorization
Abstract
Access to modern generative systems is often restricted to querying an API (the ``black-box'' setting) and many properties of the system are unknown to the user at inference time. Recent work has shown that low-dimensional representations of models based on the relationship of their embedded responses to a set of queries are useful for inferring model-level properties. However, the quality of these representations is highly sensitive to the query set. We introduce the \emph{discriminative factorization} to distinguish between high- and low-quality query sets in the context of black-box model-level classification. Under this framework, the probability of chance-level classification is bounded by a term that decays exponentially in the query budget. On three auditing tasks, estimated factorization parameters predict the empirical performance decay rate. We conclude by showing that query sets selected using the estimated factorization reproduce the empirical ordering of oracle query sets.
Related
- Training More Robust Classification Model via Discriminative Loss and Gaussian Noise Injection
- A classification of modular functors via factorization homology
- Geodesics in Black-Hole Spacetimes: A Qualitative Classification Using Vieta's Formulae
- Sharp Probabilistic Normwise Backward Error Bounds for LU Factorization
- A Categorical Realization of the (2-)Category of Monoids via Sch{ü}tzenberger Categories and Strict Factorization Systems
BibTeX
@misc{helm2026blackbox,
title = {{Black-box model classification under the discriminative factorization}},
author = {Hayden Helm and Merrick Ohata and Carey Priebe},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.07878},
eprint = {2605.07878},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07878},
}
RIS
TY - UNPB
TI - Black-box model classification under the discriminative factorization
AU - Hayden Helm
AU - Merrick Ohata
AU - Carey Priebe
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2605.07878
UR - https://arxiv.org/abs/2605.07878
AB - Access to modern generative systems is often restricted to querying an API (the ``black-box'' setting) and many properties of the system are unknown to the user at inference time. Recent work has shown that low-dimensional representations of models based on the relationship of their embedded responses to a set of queries are useful for inferring model-level properties. However, the quality of these representations is highly sensitive to the query set. We introduce the \emph{discriminative factorization} to distinguish between high- and low-quality query sets in the context of black-box model-level classification. Under this framework, the probability of chance-level classification is bounded by a term that decays exponentially in the query budget. On three auditing tasks, estimated factorization parameters predict the empirical performance decay rate. We conclude by showing that query sets selected using the estimated factorization reproduce the empirical ordering of oracle query sets.
ER -