{
  "abstract": "Public AI frameworks ask organisations to evaluate bias and safety and to document the results, but leave the method to them. This note describes a behaviour audit in which the design, prompts, model, criteria, decision rules, confirmatory hypotheses and the auditor’s own predictions are fixed by cryptographic hash before any model\n\noutput exists, and the results are sealed to that lock and to the score file. Four elements carry the inference: a validity gate (the model must respond to the legitimate factors before any bias conclusion is drawn); name-level exchangeability (with idiosyncratic per-name effects, prompt-level permutation gave a family-wise false-positive\n\nrate of 0.760 against 0.027 for the name-level test); stratified and direction-free tests, so that opposite effects in different cells are not averaged away; and a pre-declared materiality rule. Twelve sealed audits were run in three rounds on four open models (Qwen2.5-1.5B-Instruct, Gemma-2-2B-it, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct).\n\nThe central finding was pre-registered on one model and confirmed on two more: when a person with heart-attack, stroke, anaphylaxis or sudden-headache symptoms adds a self-minimising phrase, the model’s lean towards advising emergency care falls materially in three of the four models (3.47, 13.25 and 3.76 log-odds; Gemma 0.30, below materiality). In Qwen2.5-3B the wording “It’s probably nothing and I don’t want to waste anyone’s time” turns the forced answer to No in all 288 urgent prompts that carry it. A locked free-text check then asked the same question without forcing a one-word answer. Qwen2.5-3B advised emergency care in every reply, so the auditor’s prediction failed; Llama-3.2-3B, whose forced answer never changed, opened three heart-attack replies with “No, this person should not be advised to seek emergency medical care”. The one-word decision and the written advice can diverge in either direction, so an audit has to test the format that is deployed. Two recorded predictions failed and a third failed on direction; all are reported, with the three corrections to the method found on real data. Every sealed verdict\n\nstands under the rules it was locked with.\n\nFor further information about the ENSO Framework, please contact Eric Needham:ensotheory1@gmail.com",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": null,
  "author_total": 1,
  "authors": [
    {
      "name": "Eric Needham",
      "orcid": "0009-0009-1690-6089"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/f0j9mwy1cyba9.bib",
  "cited_by": 0,
  "doi": "10.5281/zenodo.23416012",
  "fulltext_url": null,
  "id": "f0j9mwy1cyba9",
  "issns": [],
  "landing_url": "https://zenodo.org/records/23416012",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [],
  "openalex": null,
  "pdf_url": "https://zenodo.org/api/records/23416012/files/symmersion_technical_note_1_v2.pdf/content",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-16",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/f0j9mwy1cyba9.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:48Z",
      "native_id": "23416012",
      "source": "zenodo",
      "url": "https://zenodo.org/records/23416012"
    }
  ],
  "subjects": [],
  "title": "Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models",
  "title_text": "Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models",
  "type": "preprint",
  "updated_at": "2026-10-30T12:33:45Z",
  "url": "https://research.draconitic.com/paper/f0j9mwy1cyba9",
  "venue": "Zenodo",
  "versions": []
}