Title: Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder
Date: 8 October 2026
Abstract: Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XLSR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech–text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level.
In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic, speaker, language, and emotion representations and evaluating them on multilingual semantic retrieval, speaker verification, language identification, and speech emotion recognition.