Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
Abstract
Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Related
- Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning
- Operational Indirect Rate-Distortion Functions for Classification Tasks
- Cooperative Multi-Task Semantic Communication for Joint Classification and Regression Tasks
- Mapping Artificial Intelligence for Personalised Learning based on Learning Style: Scopus-Based Bibliometric Analysis (2006-2026)
- Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
BibTeX
@misc{liang2026learning,
title = {{Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks}},
author = {Haodong Liang and Yanhao Jin and Krishnakumar Balasubramanian and Lifeng Lai},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02437},
eprint = {2610.02437},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02437},
}
RIS
TY - UNPB
TI - Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
AU - Haodong Liang
AU - Yanhao Jin
AU - Krishnakumar Balasubramanian
AU - Lifeng Lai
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.02437
UR - https://arxiv.org/abs/2610.02437
AB - Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
ER -