AudioCALM Continuous Autoregressive Language Modeling for Universal Audio Generation

Abstract

Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text–audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.

1Method

A single causal Transformer autoregresses over fixed-size blocks of continuous audio latents. A flow-matching head replaces the softmax; a block-causal AR-Flow attention pattern enables streaming arbitrary-length output; an asymmetric speech-only residual expert captures the tighter alignment that speech alone demands.

Place architecture figure at assets/images/overview.png

2Zero-Shot Text-to-Speech Comparisons on LibriTTS

A reference utterance is prepended as the speaker prompt. AudioCALM is compared quantitatively against modality-specific TTS systems (F5-TTS, CosyVoice 3.0) and unified baselines (UniAudio, UniMoE-Audio, UniFlow-Audio, Ming-omni); full numbers are in the paper and summarized in the Results section below. Each card highlights the strongest baseline in each group — CosyVoice 3.0 (modality-specific) and Ming-omni (unified) — alongside Ground Truth and AudioCALM; the full baseline lineup is one click away under Compare baselines.

3Text-to-Sound Comparisons on AudioCaps

Held-out captions covering everyday acoustic events. AudioCALM is compared quantitatively against AudioLDM 2-Large, TangoFlux, Stable Audio Open, and the unified baselines (UniAudio, UniFlow-Audio, Ming-omni); full numbers are in the paper and summarized in the Results section below. Each card highlights the strongest baseline in each group — TangoFlux (modality-specific) and Ming-omni (unified) — alongside Ground Truth and AudioCALM; the full baseline lineup is one click away under Compare baselines.

4Text-to-Music Comparisons on Song-Describer

Held-out music captions spanning multiple genres. AudioCALM is compared quantitatively against MusicGen-Large, Stable Audio Open, and the unified baselines (UniAudio, UniMoE-Audio, UniFlow-Audio, Ming-omni); full numbers are in the paper and summarized in the Results section below. Each card highlights the strongest baseline in each group — Stable Audio Open (modality-specific) and UniMoE-Audio (unified) — alongside Ground Truth and AudioCALM; the full baseline lineup is one click away under Compare baselines.

5Results

Best per column in bold, second-best underlined. AudioCALM is a single model evaluated jointly across all three domains.

Metrics. WER (word error rate, lower is better) and SIM (cosine similarity between speaker embeddings of the generation and the reference, higher is better) measure speech intelligibility and speaker fidelity, respectively. FAD (Fréchet audio distance, lower is better) measures distributional fidelity. CLAP (text–audio cosine similarity, higher is better) measures prompt adherence. MOS is the subjective overall-quality score for speech (1–5, higher is better); MOS-Q and MOS-T are the subjective quality and text-relevance scores for sound and music (1–5, higher is better).

Text-to-speech

Text-to-sound and text-to-music

Ethics

AudioCALM supports zero-shot voice cloning and high-fidelity sound and music synthesis. Deployment requires provenance and consent safeguards against voice-cloning misuse and synthetic-media risks. All audio samples on this page are released solely for the purpose of academic peer review.