Does Agent Confidence Entropy Predict Task Difficulty? A Pre-registered, Provenance-Complete Test of Agentic Hyperparameter Tuning for Perturb-seq Response Prediction
- 1Carnegie Mellon University
Abstract
Multi-agent group-generation systems spend test-time compute uniformly across tasks. We ask whether the agents' own confidence-and-critique trace predicts how hard a task is, which would let the compute be allocated adaptively. Taking as testbed a five-agent orchestrator for Perturb-seq response prediction inspired by CellForge and operated via MassGen, we define four compute-free trace metrics — Agent Confidence Entropy (ACE), Critique Severity Dispersion (CSD), a convergence signal (ΔC), and a Winner Flip Rate (WFR) — and their composite, the Task Difficulty Index (TDI). We pre-register five hypotheses with gates: attainable error on Adamson 2016 (CRISPR interference) and Norman 2019 (CRISPR activation), the Architect agent's backbone-choice entropy, the rank correlation of each TDI component with held-out error, and cross-dataset transfer of a calibrated TDI. The design is 41 held-out K562 tasks (21 Adamson, 15 Norman singletons, 5 Norman doublets) drawn deterministically, a trainer sweep of 27 records per task and three fixed-length agentic lifecycle runs per task, all in one process. The LLM condition is a pinned roster of Anthropic Claude models (Haiku 4.5 for the four proposer roles, Sonnet 5.5 for the Validator) whose served model is recorded and asserted per call; a run with any rule-based fallback step is invalid. Every run records its git revision, resolved configuration, library versions, dataset SHA-256 digests, task plan and label contract. Gate outcomes: 4 of 5 gates pass; H5 fails (0 unevaluated) — the trainer sweep attains the pre-registered error on both screens (H1, H2), the Architect exercises its backbone choice (H3), and the confidence-drop component and the composite rank tasks by held-out error within both screens (H4); the Adamson-calibrated composite does not transfer to Norman at the pre-registered level (H5, ρ = 0.286 against > 0.4). The confidence-entropy point estimates run against the pre-registered direction in both screens (exploratory; no claim is made). The values reported in the v0.5.0 release of this work are superseded in full; the defects behind each are listed in the appendix.
Related
- Leptonic CP Conservation - Sealed Pre-Registered Prediction
- Exact Solution of the Cosmic Equation in Non-Equilibrium Dynamics: Cosmic Age, Time Variation of Physical Constants, Observational Difficulty, and a Falsifiable Prediction of Decelerating Expansion
- Rethinking hyperparameter optimization for efficient and explainable machine learning in civic health decision-making: empirical evidence from malaria severity prediction
- CLSP-REQA: a real-time quality-aware closed-loop seizure prediction framework with Mamba-BiLSTM and confidence-gated intervention
- Quantifying the Provenance-to-Function Gap in Antidiabetic Peptide Prediction: Homology-Aware Evaluation and the ADP-Hybrid Baseline
BibTeX
@misc{mo2026does,
title = {{Does Agent Confidence Entropy Predict Task Difficulty? A Pre-registered, Provenance-Complete Test of Agentic Hyperparameter Tuning for Perturb-seq Response Prediction}},
author = {Mang-yin Mo},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23422745},
url = {https://zenodo.org/records/23422745},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Does Agent Confidence Entropy Predict Task Difficulty? A Pre-registered, Provenance-Complete Test of Agentic Hyperparameter Tuning for Perturb-seq Response Prediction
AU - Mang-yin Mo
PY - 2026
DA - 2026/10/16
PB - Zenodo
DO - 10.5281/zenodo.23422745
UR - https://zenodo.org/records/23422745
AB - Multi-agent group-generation systems spend test-time compute uniformly across tasks. We ask whether the agents' own confidence-and-critique trace predicts how hard a task is, which would let the compute be allocated adaptively. Taking as testbed a five-agent orchestrator for Perturb-seq response prediction inspired by CellForge and operated via MassGen, we define four compute-free trace metrics — Agent Confidence Entropy (ACE), Critique Severity Dispersion (CSD), a convergence signal (ΔC), and a Winner Flip Rate (WFR) — and their composite, the Task Difficulty Index (TDI). We pre-register five hypotheses with gates: attainable error on Adamson 2016 (CRISPR interference) and Norman 2019 (CRISPR activation), the Architect agent's backbone-choice entropy, the rank correlation of each TDI component with held-out error, and cross-dataset transfer of a calibrated TDI. The design is 41 held-out K562 tasks (21 Adamson, 15 Norman singletons, 5 Norman doublets) drawn deterministically, a trainer sweep of 27 records per task and three fixed-length agentic lifecycle runs per task, all in one process. The LLM condition is a pinned roster of Anthropic Claude models (Haiku 4.5 for the four proposer roles, Sonnet 5.5 for the Validator) whose served model is recorded and asserted per call; a run with any rule-based fallback step is invalid. Every run records its git revision, resolved configuration, library versions, dataset SHA-256 digests, task plan and label contract. Gate outcomes: 4 of 5 gates pass; H5 fails (0 unevaluated) — the trainer sweep attains the pre-registered error on both screens (H1, H2), the Architect exercises its backbone choice (H3), and the confidence-drop component and the composite rank tasks by held-out error within both screens (H4); the Adamson-calibrated composite does not transfer to Norman at the pre-registered level (H5, ρ = 0.286 against > 0.4). The confidence-entropy point estimates run against the pre-registered direction in both screens (exploratory; no claim is made). The values reported in the v0.5.0 release of this work are superseded in full; the defects behind each are listed in the appendix.
ER -