Do Audio Language Models Hear and Read Distinctive Features Alike?
Abstract
Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.
Related
- Can one hear the shape of a lattice random walk?
- Do we hear and read the same? Attention to detail in narrative storytelling while reading and listening with reduced distraction
- Dense Matrices Are Alike; Sparse Matrices Are Sparse in Their Own Way: A Structure-Adaptive Tile Cholesky Factorization
- Co-Adaptive Entropy Modeling and Rate-Distortion-Optimal Bit Allocation for Perceptual Audio Coding
- MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning
BibTeX
@misc{chen2026audio,
title = {{Do Audio Language Models Hear and Read Distinctive Features Alike?}},
author = {Yuanhao Chen and Peter Chin},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.30167},
eprint = {2609.30167},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.30167},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Do Audio Language Models Hear and Read Distinctive Features Alike?
AU - Yuanhao Chen
AU - Peter Chin
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2609.30167
UR - https://arxiv.org/abs/2609.30167
AB - Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.
ER -