Why Forget-Only Unlearning Needs Memorization
Abstract
Machine unlearning asks for a deletion algorithm whose output is close to retraining from scratch without the selected forget examples. In this work, we study forget-only unlearning, where the deletion algorithm receives only the trained model and the examples to forget, with no retained data or extra training information. We ask whether forget-only unlearning is always possible. We first show that this depends on the learning method: different datasets can produce the same trained model but require very different outputs after the same examples are removed. Using this observation, we derive lower bounds on how accurately unlearning can match retraining and instantiate them for several standard learning algorithms. We then ask what must be true when forget-only unlearning succeeds. To this end, we derive lower bounds on what an algorithm must memorize about the training data to handle arbitrary deletion requests. For simple threshold learners, the required information can be as large as the entire dataset, even though ordinary training keeps only one boundary point. Overall, our results show that information discarded during ordinary learning may be needed later for deletion, so models designed for forget-only unlearning may need to retain more information than standard training does.
Related
- Posterior Frequencies and the Cost of Not Fitting: A Correction to a Memorization Lower Bound
- Valid Early Detection of Memorization in Diffusion Models
- Memorization Capacity of Invariant Convolutional Neural Nets
- Why Constrained Updates Still Forget: A Functional-Path Analysis of Continual SFT
- Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
BibTeX
@misc{radic2026forgetonly,
title = {{Why Forget-Only Unlearning Needs Memorization}},
author = {Luka Radić and Vikrant Singhal and Amartya Sanyal},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10519},
eprint = {2610.10519},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10519},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Why Forget-Only Unlearning Needs Memorization
AU - Luka Radić
AU - Vikrant Singhal
AU - Amartya Sanyal
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10519
UR - https://arxiv.org/abs/2610.10519
AB - Machine unlearning asks for a deletion algorithm whose output is close to retraining from scratch without the selected forget examples. In this work, we study forget-only unlearning, where the deletion algorithm receives only the trained model and the examples to forget, with no retained data or extra training information. We ask whether forget-only unlearning is always possible. We first show that this depends on the learning method: different datasets can produce the same trained model but require very different outputs after the same examples are removed. Using this observation, we derive lower bounds on how accurately unlearning can match retraining and instantiate them for several standard learning algorithms. We then ask what must be true when forget-only unlearning succeeds. To this end, we derive lower bounds on what an algorithm must memorize about the training data to handle arbitrary deletion requests. For simple threshold learners, the required information can be as large as the entire dataset, even though ordinary training keeps only one boundary point. Overall, our results show that information discarded during ordinary learning may be needed later for deletion, so models designed for forget-only unlearning may need to retain more information than standard training does.
ER -