Defending against deepfake speech attacks via feature dropout and dropin regularization in large audio language models.
6 affiliations
- 1Group on Language, Audio, & Music (GLAM), Imperial College London, London, United Kingdom.
- 2Chair of Health Informatics (CHI), TUM University Hospital, Munich, Germany.
- 3Shanghai Jiaotong University, Shanghai, China.
- 4Queen Mary University of London, London, United Kingdom.
- 5Munich Center for Machine Learning (MCML), München, Germany.
- 6Munich Data Science Institute (MDSI), München, Germany.
Abstract
Deepfakes have raised widespread concern owing to their threats to privacy, security, and societal trust, driving growing research interest in effective detection methods. Audio deepfake detection has moved from handcrafted-feature classifiers to end-to-end deep learning architectures and, more recently, to self-supervised speech foundation models and general-purpose speech recognizers such as Whisper. The use of instruction-following speech large language models (LLMs), of which Speech Audio Language Music Open Neural Network (SALMONN) is a representative example, remains comparatively less explored, and forms the setting of this study. We argue that large language models (LLMs), with their powerful representational and information retrieval capabilities, hold significant potential for this task-provided that relevant audio features are carefully selected and prioritized. To this end, we employ a speech LLM, specifically SALMONN, for deepfake audio detection, incorporating a two-stage training strategy in which Stage 1 injects acoustic-feature awareness into the Low-Rank Adaptation (LoRA) adapters via an auxiliary feature-prediction pretext task (referred to as feature dropin), and Stage 2 performs windowed random retention on the encoder token sequence (referred to as feature dropout) before the final real/fake decision. Our approach achieves an absolute accuracy improvement of 0.188 on the FakeOrReal benchmark and attains state-of-the-art performance on the Automatic Speaker Verification Spoofing and Countermeasures (ASVspoof) 2019 LA subset with an Equal Error Rate (EER) of 0.00496, representing a 33.0% relative improvement over prior methods. Furthermore, the model transfers non-trivially to an out-of-domain artificial intelligence (AI)-generated music dataset (M6), reaching accuracy comparable to a ResNet18 baseline reported in prior study, which we interpret as evidence of transferable audio representations rather than broad cross-domain robustness.
Related
- Credibility attacks do not enhance the impact of deepfake warnings
- Defending synthetic DNA orders against intra-order splitting-based obfuscation
- Text–relation joint modeling improves the robustness of Chinese social bot detection against paraphrasing attacks
- Gramian-Informed Framework for Edge Flow Network Analysis Against Nodal Attacks
- Secure Impulsive Control of Singular Markovian Jump Neural Networks Against Hybrid Attacks: A Hybrid Event-Triggered Mechanism
BibTeX
@article{li2026defending,
title = {{Defending against deepfake speech attacks via feature dropout and dropin regularization in large audio language models.}},
author = {Yupei Li and Yixiong Fang and Jiahao Xue and Manuel Milling and Björn W Schuller},
journal = {Frontiers in artificial intelligence},
year = {2026},
month = oct,
doi = {10.3389/frai.2026.1928592},
url = {https://europepmc.org/article/MED/42903335},
note = {CC BY},
}
RIS
TY - JOUR
TI - Defending against deepfake speech attacks via feature dropout and dropin regularization in large audio language models.
AU - Yupei Li
AU - Yixiong Fang
AU - Jiahao Xue
AU - Manuel Milling
AU - Björn W Schuller
PY - 2026
DA - 2026/10/06
JO - Frontiers in artificial intelligence
DO - 10.3389/frai.2026.1928592
UR - https://europepmc.org/article/MED/42903335
AB - Deepfakes have raised widespread concern owing to their threats to privacy, security, and societal trust, driving growing research interest in effective detection methods. Audio deepfake detection has moved from handcrafted-feature classifiers to end-to-end deep learning architectures and, more recently, to self-supervised speech foundation models and general-purpose speech recognizers such as Whisper. The use of instruction-following speech large language models (LLMs), of which Speech Audio Language Music Open Neural Network (SALMONN) is a representative example, remains comparatively less explored, and forms the setting of this study. We argue that large language models (LLMs), with their powerful representational and information retrieval capabilities, hold significant potential for this task-provided that relevant audio features are carefully selected and prioritized. To this end, we employ a speech LLM, specifically SALMONN, for deepfake audio detection, incorporating a two-stage training strategy in which Stage 1 injects acoustic-feature awareness into the Low-Rank Adaptation (LoRA) adapters via an auxiliary feature-prediction pretext task (referred to as feature dropin), and Stage 2 performs windowed random retention on the encoder token sequence (referred to as feature dropout) before the final real/fake decision. Our approach achieves an absolute accuracy improvement of 0.188 on the FakeOrReal benchmark and attains state-of-the-art performance on the Automatic Speaker Verification Spoofing and Countermeasures (ASVspoof) 2019 LA subset with an Equal Error Rate (EER) of 0.00496, representing a 33.0% relative improvement over prior methods. Furthermore, the model transfers non-trivially to an out-of-domain artificial intelligence (AI)-generated music dataset (M6), reaching accuracy comparable to a ResNet18 baseline reported in prior study, which we interpret as evidence of transferable audio representations rather than broad cross-domain robustness.
SN - 2624-8212
ER -