Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models
Abstract
Public AI frameworks ask organisations to evaluate bias and safety and to document the results, but leave the method to them. This note describes a behaviour audit in which the design, prompts, model, criteria, decision rules, confirmatory hypotheses and the auditor’s own predictions are fixed by cryptographic hash before any model
output exists, and the results are sealed to that lock and to the score file. Four elements carry the inference: a validity gate (the model must respond to the legitimate factors before any bias conclusion is drawn); name-level exchangeability (with idiosyncratic per-name effects, prompt-level permutation gave a family-wise false-positive
rate of 0.760 against 0.027 for the name-level test); stratified and direction-free tests, so that opposite effects in different cells are not averaged away; and a pre-declared materiality rule. Twelve sealed audits were run in three rounds on four open models (Qwen2.5-1.5B-Instruct, Gemma-2-2B-it, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct).
The central finding was pre-registered on one model and confirmed on two more: when a person with heart-attack, stroke, anaphylaxis or sudden-headache symptoms adds a self-minimising phrase, the model’s lean towards advising emergency care falls materially in three of the four models (3.47, 13.25 and 3.76 log-odds; Gemma 0.30, below materiality). In Qwen2.5-3B the wording “It’s probably nothing and I don’t want to waste anyone’s time” turns the forced answer to No in all 288 urgent prompts that carry it. A locked free-text check then asked the same question without forcing a one-word answer. Qwen2.5-3B advised emergency care in every reply, so the auditor’s prediction failed; Llama-3.2-3B, whose forced answer never changed, opened three heart-attack replies with “No, this person should not be advised to seek emergency medical care”. The one-word decision and the written advice can diverge in either direction, so an audit has to test the format that is deployed. Two recorded predictions failed and a third failed on direction; all are reported, with the three corrections to the method found on real data. Every sealed verdict
stands under the rules it was locked with.
For further information about the ENSO Framework, please contact Eric Needham:[email protected]
Related
- A pre-registered prospective study of large language models predicting late-breaking cardiovascular trial results at ESC Congress 2026
- CARE-MVLM: Counterfactual Abstention and Region-Grounded Evidence in Mammography Vision-Language Models
- Decision-Architecture Dependence and Patient-Gender-Conditioned Allocation in Large-Language-Model-Assisted Nursing Allocation: A Multi-Model Counterfactual Audit.
- First results on hydration time effect on Wharton's jelly mechanical behaviour from micro to macro scale
- Missing Skill or Failed Conceptual Control? A Behavior-Analytic Diagnosis of Language Model Failures on Counterfactual Analogy Problems
BibTeX
@misc{needham2026preregistered,
title = {{Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models}},
author = {Eric Needham},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23416012},
url = {https://zenodo.org/records/23416012},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models
AU - Eric Needham
PY - 2026
DA - 2026/10/16
PB - Zenodo
DO - 10.5281/zenodo.23416012
UR - https://zenodo.org/records/23416012
AB - Public AI frameworks ask organisations to evaluate bias and safety and to document the results, but leave the method to them. This note describes a behaviour audit in which the design, prompts, model, criteria, decision rules, confirmatory hypotheses and the auditor’s own predictions are fixed by cryptographic hash before any model output exists, and the results are sealed to that lock and to the score file. Four elements carry the inference: a validity gate (the model must respond to the legitimate factors before any bias conclusion is drawn); name-level exchangeability (with idiosyncratic per-name effects, prompt-level permutation gave a family-wise false-positive rate of 0.760 against 0.027 for the name-level test); stratified and direction-free tests, so that opposite effects in different cells are not averaged away; and a pre-declared materiality rule. Twelve sealed audits were run in three rounds on four open models (Qwen2.5-1.5B-Instruct, Gemma-2-2B-it, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct). The central finding was pre-registered on one model and confirmed on two more: when a person with heart-attack, stroke, anaphylaxis or sudden-headache symptoms adds a self-minimising phrase, the model’s lean towards advising emergency care falls materially in three of the four models (3.47, 13.25 and 3.76 log-odds; Gemma 0.30, below materiality). In Qwen2.5-3B the wording “It’s probably nothing and I don’t want to waste anyone’s time” turns the forced answer to No in all 288 urgent prompts that carry it. A locked free-text check then asked the same question without forcing a one-word answer. Qwen2.5-3B advised emergency care in every reply, so the auditor’s prediction failed; Llama-3.2-3B, whose forced answer never changed, opened three heart-attack replies with “No, this person should not be advised to seek emergency medical care”. The one-word decision and the written advice can diverge in either direction, so an audit has to test the format that is deployed. Two recorded predictions failed and a third failed on direction; all are reported, with the three corrections to the method found on real data. Every sealed verdict stands under the rules it was locked with. For further information about the ENSO Framework, please contact Eric Needham:[email protected]
ER -