Data Attribution via Sketched Metadifferentiation
Abstract
Data attribution seeks to quantify how individual training examples shape a model's predictions and underpins problems including data valuation, machine unlearning, and model interpretability. Despite having a long line of work, computationally scalable methods often struggle to predict the effect of removing training data in neural networks due to their non-convex nature. To overcome this challenge, metagradient-based methods such as MAGIC (Ilyas and Engstrom, 2025) differentiate each prediction through the entire training run and compute its exact influence with respect to the training data, but require a separate run for every prediction. To reduce this cost, we cast budgeted attribution as estimating a large influence matrix from a small number of measurements. We show that the measurements most appropriate for recovering this matrix differ from those best suited for attribution itself. We then present two algorithms, MAGE and SPELL, suited for reconstruction and attribution respectively, that run on existing metagradient machinery at no extra cost. Empirical studies demonstrate strong performance over existing baselines across training scales and measurement budgets.
Related
- An Equivalence result for sketched Anderson Acceleration and sketched GMRES
- Real-Arithmetic Sketched Eigensolvers for Symmetric Matrices
- BLOKS: Scaling Kernel Solvers with Sketched Block Preconditioning
- Sketched Calibration for Conformal Prediction under Covariate Shift
- Randomized Symplectic Model Reduction: One-Pass Basis Construction and Sketched Hamiltonian Assembly
BibTeX
@misc{chen2026data,
title = {{Data Attribution via Sketched Metadifferentiation}},
author = {Yuxi Chen and Hamza Golubovic and Han Tong and Arian Maleki and Andrew Ilyas},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.15044},
eprint = {2609.15044},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.15044},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Data Attribution via Sketched Metadifferentiation
AU - Yuxi Chen
AU - Hamza Golubovic
AU - Han Tong
AU - Arian Maleki
AU - Andrew Ilyas
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.15044
UR - https://arxiv.org/abs/2609.15044
AB - Data attribution seeks to quantify how individual training examples shape a model's predictions and underpins problems including data valuation, machine unlearning, and model interpretability. Despite having a long line of work, computationally scalable methods often struggle to predict the effect of removing training data in neural networks due to their non-convex nature. To overcome this challenge, metagradient-based methods such as MAGIC (Ilyas and Engstrom, 2025) differentiate each prediction through the entire training run and compute its exact influence with respect to the training data, but require a separate run for every prediction. To reduce this cost, we cast budgeted attribution as estimating a large influence matrix from a small number of measurements. We show that the measurements most appropriate for recovering this matrix differ from those best suited for attribution itself. We then present two algorithms, MAGE and SPELL, suited for reconstruction and attribution respectively, that run on existing metagradient machinery at no extra cost. Empirical studies demonstrate strong performance over existing baselines across training scales and measurement budgets.
ER -