FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training
Abstract
Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches 0.6476 ± 0.0139 over eight seeds (median 0.6481; paired 95% interval [+0.079,+0.122]) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.
Related
- Training-Free Multi-Agent Cooperative Embodied Search from Uncertain Vision-Language Priors
- Projected Primal–Dual Gradient Preference Optimization Under Category-Wise Split-Conformal Safety Constraints for Large Language Model Post-Training
- ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
- Watermarking: from Impossibility to Auditable Compliance
- A Conditional, Boundary-Aware, and Auditable Deletion-Effect Weighting Framework for MCDM with Positive Decision Matrices and a Fixed Set of Alternatives
BibTeX
@misc{hu2026faer,
title = {{FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training}},
author = {Miaobo Hu and Shuhao Hu and Xiaobo Guo and Xin Wang and Bokun Wang and Tianshu Fu and Daren Zha},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00385},
eprint = {2610.00385},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00385},
}
RIS
TY - UNPB
TI - FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training
AU - Miaobo Hu
AU - Shuhao Hu
AU - Xiaobo Guo
AU - Xin Wang
AU - Bokun Wang
AU - Tianshu Fu
AU - Daren Zha
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.00385
UR - https://arxiv.org/abs/2610.00385
AB - Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches 0.6476 ± 0.0139 over eight seeds (median 0.6481; paired 95% interval [+0.079,+0.122]) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.
ER -