Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models
Abstract
Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.
Related
- Verifier-guided discovery of exact high-order mimetic operators with large language models
- Inductive Grounded Arithmetic: Goodstein, the Hydra, and Γ0
- Provable circuit-level thresholds for non-abelian topological phases
- A counterexample to the circuit-cocircuit intersection conjecture
- Elementary modal logics and circuit complexity
BibTeX
@misc{vaidyanathan2026slaying,
title = {{Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models}},
author = {Sankaran Vaidyanathan and Rafal Urbaniak and Emily Bunnapradist and Michelangelo Naim and Daniel Waxman},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04017},
eprint = {2610.04017},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04017},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models
AU - Sankaran Vaidyanathan
AU - Rafal Urbaniak
AU - Emily Bunnapradist
AU - Michelangelo Naim
AU - Daniel Waxman
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.04017
UR - https://arxiv.org/abs/2610.04017
AB - Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.
ER -