Preregistered Adversarial Batteries: A Falsifiable Protocol for Security-Testing Autonomous AI Systems
- 1SQUIRL OS Technologies LLC
Abstract
Security testing of autonomous AI systems suffers from a credibility gap: penetration-test reports are private, vendor claims are selectively disclosed, and red-team outcomes are published only when favorable. This preprint introduces Preregistered Adversarial Batteries (PAB), a protocol importing preregistration from clinical-trial science into security operations: every falsifier (a specific attack with a predicted verdict) is specified and publicly committed before execution; adjudication is deterministic with no language model in the decision path; results are written to an append-only public ledger and DOI-stamped. Critical events are scored as hold-by-design verdicts: the predicted outcome is that the system fails closed and waits for a human decision, so a refusal counts as a pass.
Application to a self-healing AI infrastructure platform across six batteries reconstructing the declassified NSA BULLRUN-era tradecraft (operational faults, upstream context poisoning, cryptographic subversion, supply-chain interception, monitoring integrity, lawful compulsion): 30 falsifiers, 30 predicted verdicts, zero unexpected outcomes, six holds-by-design each adjudicated by the human operator through a verified channel. Three protocol imperfections (a missed execution window, a preregistration self-detection, a scoring amendment) are disclosed as misses in the public record, demonstrating the miss-disclosure mechanism functions. Limitations are stated plainly: self-adjudication, synthetic injections against a prototype system, and single-operator governance; the replication kit is released to answer them.
The contribution is not the pass rate — it is the format: a security test whose conclusions were committed before its data existed.
This software is a prototype and is provided for educational and research purposes only. It is not intended for production use, commercial deployment, or safety-critical environments. All systems are experimental and may contain defects.
Related
- From Candidate Hierarchy to Experimental Dependency Map: Results from Five Preregistered Adversarial Tests of Maintenance, Heredity, Learning, Agency and Self-Modelling.
- No Learning Under This Protocol: A Preregistered Validity Audit of a Downsized Frozen-Student Replication of Sherpa's Simulator
- A falsifiable framework for retrospective neuro-type (ADHD/ASD/AuDHD) attribution in historical figures: a Registered Report Stage 1 protocol
- Domain adaptation affects adversarial robustness in autonomous steering models
- An autonomous feedback protocol: responding to temperature and potential changes in energy converters
BibTeX
@misc{leoncalvin2026preregistered,
title = {{Preregistered Adversarial Batteries: A Falsifiable Protocol for Security-Testing Autonomous AI Systems}},
author = {Leon Calvin, II Long},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23422632},
url = {https://zenodo.org/records/23422632},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Preregistered Adversarial Batteries: A Falsifiable Protocol for Security-Testing Autonomous AI Systems
AU - Leon Calvin, II Long
PY - 2026
DA - 2026/10/16
PB - Zenodo
DO - 10.5281/zenodo.23422632
UR - https://zenodo.org/records/23422632
AB - Security testing of autonomous AI systems suffers from a credibility gap: penetration-test reports are private, vendor claims are selectively disclosed, and red-team outcomes are published only when favorable. This preprint introduces Preregistered Adversarial Batteries (PAB), a protocol importing preregistration from clinical-trial science into security operations: every falsifier (a specific attack with a predicted verdict) is specified and publicly committed before execution; adjudication is deterministic with no language model in the decision path; results are written to an append-only public ledger and DOI-stamped. Critical events are scored as hold-by-design verdicts: the predicted outcome is that the system fails closed and waits for a human decision, so a refusal counts as a pass. Application to a self-healing AI infrastructure platform across six batteries reconstructing the declassified NSA BULLRUN-era tradecraft (operational faults, upstream context poisoning, cryptographic subversion, supply-chain interception, monitoring integrity, lawful compulsion): 30 falsifiers, 30 predicted verdicts, zero unexpected outcomes, six holds-by-design each adjudicated by the human operator through a verified channel. Three protocol imperfections (a missed execution window, a preregistration self-detection, a scoring amendment) are disclosed as misses in the public record, demonstrating the miss-disclosure mechanism functions. Limitations are stated plainly: self-adjudication, synthetic injections against a prototype system, and single-operator governance; the replication kit is released to answer them. The contribution is not the pass rate — it is the format: a security test whose conclusions were committed before its data existed. This software is a prototype and is provided for educational and research purposes only. It is not intended for production use, commercial deployment, or safety-critical environments. All systems are experimental and may contain defects.
ER -