Qualifying Software Changes Without Overstating Evidence: A controlled diagnostic study of evaluator failure and controller composition
Abstract
Software-change evidence can be genuine yet insufficient for the conclusion drawn from it. We examine this problem in a controlled diagnostic study of an author-directed evaluator-development effort assisted by artificial intelligence (AI) and a bounded comma-separated values (CSV) repair instance, BX01. Retrospectively selected counterexamples concern report origin, execution phase, program semantics, execution profile, event coverage and input domain. We distinguish incorrect local observations from valid observations used outside their scope. Two separately implemented controllers assess the same thirteen known transaction requests using shared admission and native-observation components. A fixed domain contains 584 inputs; complete goal verification requires all of them, whereas rejected, incomplete or faulty cases may supply fewer observations. Each controller returns two PASS, eight FAIL and three BLOCKED verdicts, with identical values in ten recorded assessment fields. Four changes pass all functional inputs but violate other transaction requirements. We report excluded implementations that meet the checked targets and descriptive timing; one timing difference reverses sign on replay. The study provides traceable diagnostic contrasts, counterexample-linked review questions and a bounded controller-composition check. It establishes neither a method-superiority result nor general correctness of the observers. Failure prevalence, independent-observer effectiveness, live-agent behavior and cross-project generalization remain untested.
Related
- Cryptanalysis of a controller-independent controlled bidirectional QSDC protocol
- The Influence of Sleep and Autonomic Recovery on Performance in World-Class Swimmers at Major Competitions: The Qualifying Trials, World Championships, and Olympic Games.
- Composition- and temperature-controlled architecture of pluronic F127–miltefosine mixed micelles
- Effects of Nut Consumption on Body Composition: A Systematic Review and Meta-Analysis of Randomized Controlled Trials with Subgroup Analyses by Nut Fatty-Acid Profile
- Effects of exercise interventions based on American College of Sports Medicine recommendations on physical function, body composition, and quality of life in older adults with sarcopenia: a meta-analysis of randomized controlled trials
BibTeX
@misc{liu2026qualifying,
title = {{Qualifying Software Changes Without Overstating Evidence: A controlled diagnostic study of evaluator failure and controller composition}},
author = {Huajun Liu},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23588320},
url = {https://zenodo.org/records/23588320},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Qualifying Software Changes Without Overstating Evidence: A controlled diagnostic study of evaluator failure and controller composition
AU - Huajun Liu
PY - 2026
DA - 2026/10/23
PB - Zenodo
DO - 10.5281/zenodo.23588320
UR - https://zenodo.org/records/23588320
AB - Software-change evidence can be genuine yet insufficient for the conclusion drawn from it. We examine this problem in a controlled diagnostic study of an author-directed evaluator-development effort assisted by artificial intelligence (AI) and a bounded comma-separated values (CSV) repair instance, BX01. Retrospectively selected counterexamples concern report origin, execution phase, program semantics, execution profile, event coverage and input domain. We distinguish incorrect local observations from valid observations used outside their scope. Two separately implemented controllers assess the same thirteen known transaction requests using shared admission and native-observation components. A fixed domain contains 584 inputs; complete goal verification requires all of them, whereas rejected, incomplete or faulty cases may supply fewer observations. Each controller returns two PASS, eight FAIL and three BLOCKED verdicts, with identical values in ten recorded assessment fields. Four changes pass all functional inputs but violate other transaction requirements. We report excluded implementations that meet the checked targets and descriptive timing; one timing difference reverses sign on replay. The study provides traceable diagnostic contrasts, counterexample-linked review questions and a bounded controller-composition check. It establishes neither a method-superiority result nor general correctness of the observers. Failure prevalence, independent-observer effectiveness, live-agent behavior and cross-project generalization remain untested.
ER -