MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning
Abstract
Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
Related
- DCRA: Diffusion-Conditioned Representation Alignment for Robust Time-Series Learning
- Dual-domain self-supervised feature alignment via spectral–spatial representation learning for deepfake anomaly detection
- ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training
- An Exact Poisson Mixture Representation of Siler Remaining Life Expectancy
- Causal Inference with Unobserved Confounding: A Mixture Learning Perspective
BibTeX
@misc{yamamoto2026micla,
title = {{MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning}},
author = {Kohei Yamamoto and Kosuke Okusa},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.27491},
eprint = {2610.27491},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.27491},
}
RIS
TY - UNPB
TI - MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning
AU - Kohei Yamamoto
AU - Kosuke Okusa
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.27491
UR - https://arxiv.org/abs/2610.27491
AB - Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
ER -