Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning
- 1Arts et Métiers
Abstract
Sparse expert supervision can improve reinforcement learning, but deciding which
states are worth annotating remains difficult. Uncertainty identifies where a learner
is unsure, but does not directly identify where supervision will most strongly
change that learner. We introduce ALBERTA, a response-aware acquisition frame-
work that measures local policy and value changes produced by controlled expert
interventions, predicts those responses from pre-annotation signals, and selects
candidates in the resulting two-dimensional response space. A calibration set of
200 expert-labeled probes is used to learn response from policy entropy, historical
critic disagreement, and their interaction; Pareto selection then allocates a separate
121-state downstream budget. The response-aware branch therefore uses 321 ex-
pert labels in total, compared with 121 for the Random baseline. In a controlled
single-seed Atomic Chess experiment from a common RL10 branch point, the
Pareto branch achieves a higher mean game score than PPO-only at all four tested
evaluation temperatures, with smaller margins at higher temperatures, and shows
its largest margins over Random at lower temperatures. These results provide
evidence that learner response can be a useful acquisition target while exposing
substantial differences between policy and value responsiveness.
Related
- Network-Slice-AwareVNF Orchestration for Mission-Critical IoT in 6G Networks: A Multi-Objective Deep Reinforcement Learning Approach
- A hybrid reinforcement learning and Harris Hawks Optimization framework for multi-objective resource management in IIoT
- AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning
- Evolution of fairness in multi-objective reinforcement learning framework
- Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
BibTeX
@misc{koza2026learning,
title = {{Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning}},
author = {Tom Koza},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23416966},
url = {https://zenodo.org/records/23416966},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning
AU - Tom Koza
PY - 2026
DA - 2026/10/16
PB - Zenodo
DO - 10.5281/zenodo.23416966
UR - https://zenodo.org/records/23416966
AB - Sparse expert supervision can improve reinforcement learning, but deciding which states are worth annotating remains difficult. Uncertainty identifies where a learner is unsure, but does not directly identify where supervision will most strongly change that learner. We introduce ALBERTA, a response-aware acquisition frame- work that measures local policy and value changes produced by controlled expert interventions, predicts those responses from pre-annotation signals, and selects candidates in the resulting two-dimensional response space. A calibration set of 200 expert-labeled probes is used to learn response from policy entropy, historical critic disagreement, and their interaction; Pareto selection then allocates a separate 121-state downstream budget. The response-aware branch therefore uses 321 ex- pert labels in total, compared with 121 for the Random baseline. In a controlled single-seed Atomic Chess experiment from a common RL10 branch point, the Pareto branch achieves a higher mean game score than PPO-only at all four tested evaluation temperatures, with smaller margins at higher temperatures, and shows its largest margins over Random at lower temperatures. These results provide evidence that learner response can be a useful acquisition target while exposing substantial differences between policy and value responsiveness.
ER -