Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
Abstract
Multi-constraint instruction following requires a model to respond to a query under many simultaneously active constraints. Even strong instruction-tuned models still routinely violate some of them. Existing approaches either augment supervision with sequence- or token-level RL rewards from external verifiers or learned graders, or use on-policy distillation (OPD) against a single full-context teacher whose probability mass becomes diluted as more constraints become simultaneously active. We propose CC-OPD (Counterfactual Constraint-Conditioned On-Policy Distillation), which inverts the standard supervision-generation direction in distillation. Rather than enriching the teacher with information beyond what the student sees, CC-OPD ablates each constraint from the teacher's conditioning in turn, and constructs the per-constraint signal from the resulting per-token probability differentials. The resulting per-token leave-one-out log-likelihood shifts are summed, clipped, and added to the vanilla OPD reward as a token-level shaping term. All shaping terms are obtained from the frozen teacher, without an external verifier during distillation, and the reward equals vanilla OPD wherever the aggregate shift is zero. Across two Qwen model pairs and seven benchmarks, CC-OPD achieves the highest average among all evaluated student-training methods. A 1.5B student trained with CC-OPD surpasses its own 7B RL-trained teacher on the MulDimIF benchmark.
Versions
Related
- Competence narrowing in AI-assisted oral English instruction: A quasi-experimental study of the spiral synergy framework
- UQ-ACE: Stability-Aware Counterfactual Controller Tuning via Bayesian SINDy
- A Counterfactual Framework for Estimating Infectious Disease Prevalence under Repeated Testing with Symptomatic and Contact-Tracing Components
- Counterfactual Certification for Sequential Unlearning with Long-Run Performance Guarantees
- Distributional Counterfactual Dependence in the Special Sciences
BibTeX
@misc{zheng2026counterfactual,
title = {{Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following}},
author = {Yanzhao Zheng and Yuanqiang Yu and Tianze Xu and Chao Ma and Zhentao Zhang and Jihuai Zhu and Baohua Dong and Hangcheng Zhu and Ruohui Huang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27421},
eprint = {2609.27421},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27421},
}
RIS
TY - UNPB
TI - Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
AU - Yanzhao Zheng
AU - Yuanqiang Yu
AU - Tianze Xu
AU - Chao Ma
AU - Zhentao Zhang
AU - Jihuai Zhu
AU - Baohua Dong
AU - Hangcheng Zhu
AU - Ruohui Huang
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.27421
UR - https://arxiv.org/abs/2609.27421
AB - Multi-constraint instruction following requires a model to respond to a query under many simultaneously active constraints. Even strong instruction-tuned models still routinely violate some of them. Existing approaches either augment supervision with sequence- or token-level RL rewards from external verifiers or learned graders, or use on-policy distillation (OPD) against a single full-context teacher whose probability mass becomes diluted as more constraints become simultaneously active. We propose CC-OPD (Counterfactual Constraint-Conditioned On-Policy Distillation), which inverts the standard supervision-generation direction in distillation. Rather than enriching the teacher with information beyond what the student sees, CC-OPD ablates each constraint from the teacher's conditioning in turn, and constructs the per-constraint signal from the resulting per-token probability differentials. The resulting per-token leave-one-out log-likelihood shifts are summed, clipped, and added to the vanilla OPD reward as a token-level shaping term. All shaping terms are obtained from the frozen teacher, without an external verifier during distillation, and the reward equals vanilla OPD wherever the aggregate shift is zero. Across two Qwen model pairs and seven benchmarks, CC-OPD achieves the highest average among all evaluated student-training methods. A 1.5B student trained with CC-OPD surpasses its own 7B RL-trained teacher on the MulDimIF benchmark.
ER -