Valence-Aware Gradient Memory and Fractional Kernel Comparisons for LoRA-Based Language Model Unlearning
- 1Bauman Moscow State Technical University, 105005 Moscow, Russia
Abstract
Selective machine unlearning aims to suppress targeted information while preserving retained and general knowledge. We investigate valence-aware gradient memory within low-rank adaptation (LoRA), separating retention-supporting and forgetting-related directions. Earlier experiments used Gamma-scaled exponential moving averages rather than genuine fractional dynamics; we preserve that distinction and introduce a normalized finite-history Grünwald–Letnikov (GL) fractional-integral accumulator with algebraically decaying coefficients. The study progresses from synthetic transformer optimization and pretrained text classification to causal language model experiments, including a new controlled evaluation on TOFU Forget10 using Qwen2.5-0.5B-Instruct. A SciQ- and WikiText-regularized teacher reduces reference-answer NLL on original and paraphrased TOFU questions while exhibiting a small World Facts penalty in a separate audit. From this fixed teacher, four unlearning methods were compared for 120 updates over three random seeds. Dual GL yields the largest paraphrased forget-set NLL increase (2.169±0.331), exceeding dual EMA (1.964±0.250), but incurs greater retain-set (1.043±0.207 versus 0.914±0.147) and World Facts (1.333±0.645 versus 1.153±0.567) degradation. Exact paired sign-flip tests do not establish statistical significance (p=0.25 for GL versus EMA). The stronger suppression achieved with fractional memory is accompanied by greater disruption of retained and general factual knowledge, suggesting a shift in model behavior rather than more selective forgetting. Whether the targeted information becomes genuinely inaccessible rather than merely less likely to be reproduced remains unresolved.
Related
- Failure of the Pólya-Szegő inequality for the fractional gradient
- Hölder gradient estimates for fractional p-caloric functions
- Large deviations, gradient flows and fluctuating hydrodynamics for fractional diffusion
- Kernel representations of fractional gradient bilinear forms. Part I. Constant coefficients
- Kernel representations of fractional gradient bilinear forms. Part II. Variable coefficients
BibTeX
@article{borodulin2026valenceaware,
title = {{Valence-Aware Gradient Memory and Fractional Kernel Comparisons for LoRA-Based Language Model Unlearning}},
author = {Aleksei Borodulin and Andrei Gantimurov and Vladimir Nelyub and Vadim Tynchenko and Ivan Malashin},
journal = {Fractal and Fractional},
year = {2026},
month = oct,
doi = {10.3390/fractalfract10110746},
url = {https://www.mdpi.com/2504-3110/10/11/746},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Valence-Aware Gradient Memory and Fractional Kernel Comparisons for LoRA-Based Language Model Unlearning
AU - Aleksei Borodulin
AU - Andrei Gantimurov
AU - Vladimir Nelyub
AU - Vadim Tynchenko
AU - Ivan Malashin
PY - 2026
DA - 2026/10/26
JO - Fractal and Fractional
DO - 10.3390/fractalfract10110746
UR - https://www.mdpi.com/2504-3110/10/11/746
AB - Selective machine unlearning aims to suppress targeted information while preserving retained and general knowledge. We investigate valence-aware gradient memory within low-rank adaptation (LoRA), separating retention-supporting and forgetting-related directions. Earlier experiments used Gamma-scaled exponential moving averages rather than genuine fractional dynamics; we preserve that distinction and introduce a normalized finite-history Grünwald–Letnikov (GL) fractional-integral accumulator with algebraically decaying coefficients. The study progresses from synthetic transformer optimization and pretrained text classification to causal language model experiments, including a new controlled evaluation on TOFU Forget10 using Qwen2.5-0.5B-Instruct. A SciQ- and WikiText-regularized teacher reduces reference-answer NLL on original and paraphrased TOFU questions while exhibiting a small World Facts penalty in a separate audit. From this fixed teacher, four unlearning methods were compared for 120 updates over three random seeds. Dual GL yields the largest paraphrased forget-set NLL increase (2.169±0.331), exceeding dual EMA (1.964±0.250), but incurs greater retain-set (1.043±0.207 versus 0.914±0.147) and World Facts (1.333±0.645 versus 1.153±0.567) degradation. Exact paired sign-flip tests do not establish statistical significance (p=0.25 for GL versus EMA). The stronger suppression achieved with fractional memory is accompanied by greater disruption of retained and general factual knowledge, suggesting a shift in model behavior rather than more selective forgetting. Whether the targeted information becomes genuinely inaccessible rather than merely less likely to be reproduced remains unresolved.
SN - 2504-3110
ER -