arXiv AI

Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

The paper introduces PlanAudio, an end‑to‑end framework that generates unified audio—combining speech and sound—directly from free‑form text prompts without intermediate preprocessing. PlanAudio uses latent planning to predict a continuous audio layout before synthesizing acoustic tokens, enabling coherent temporal and semantic interactions in the output. The authors build a large training corpus and a Free‑Form Unified Audio Generation Benchmark (FUAG), demonstrating that PlanAudio outperforms both pipeline and unified baselines on speech‑and‑sound tasks while remaining competitive with specialist models on speech and sound alone.

arXiv Computation and Language
Aug 28

FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

FireRedAudio is a 9‑billion‑parameter audio language model that separates continuous input representations for audio understanding and speech generation, enabling a single autoregressive LLM to perform tasks such as ASR, zero‑shot TTS, Instruct TTS, and semantic/acoustic speech editing. The model uses a dedicated Audio Encoder for recognition and a RedAE‑based pathway for generation, with the LLM directly generating text or conditioning a flow‑matching DiT to produce acoustic latents. Evaluations show competitive or leading performance in multilingual ASR, content‑accurate zero‑shot TTS, strong instruction following, and significant improvements in speech editing over prior work.

By Feiyu Shen, Fenglong Xie, Junjie Li, Kun Xie, Lei Xie, Xu Tang, Xuelong Geng, Yan Jia, Yao Hu, Yichen Han, Yichen Wu, Ziqi Dai, Junjie Chen, Kai Huang, Manzhen Wei, Yixuan Li
arXiv AI
Jun 18

Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

arXiv:2606. 19325v1 Announce Type: cross Abstract: Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings.

By Michael Finkelson, Daniel Segal, Eitan Richardson, Shahar Armon, Nani Goldring, Poriya Panet, Nir Zabari, Benjamin Brazowski, Or Patashnik, Yoav HaCohen
arXiv AI
Jun 2

MOSS-Audio Technical Report

arXiv:2606. 01802v1 Announce Type: cross Abstract: MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning.

By Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, Zhengyuan Lin, Ziqi Chen, Zhaoye Fei, Chenghao Liu, Jun Zhan, Kang Yu, Kexin Huang, Mingshu Chen, Qinyuan Cheng, Ruixiao Li, Shimin Li, Songlin Wang, Yang Gao, Yiyang Zhang, Xipeng Qiu
arXiv Computer Vision
Oct 9

SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

SepGen is a single model that extends a pretrained audio‑video generator to produce a video, its mixed soundtrack, and individual waveforms for each captioned sound source in one sampling run. It offers two modes: generation, where captions dictate the content of each stem (e.g., separate stems for each speaker in a dialogue), and separation, where the model extracts specified sources from a clean audio mix based on free‑text descriptions. The authors evaluate SepGen on text‑generated scenes, scenes rendered by other generators, and real recordings of speech, music, and sound effects, showing it outperforms language‑conditioned separators, especially on speech tasks.

By Aviad Dahan, Rajaei Khatib, Yonatan Bitton, Idan Szpektor, Lior Wolf, Raja Giryes
arXiv Computer Vision
Oct 27

Vorch-Omni: Multi-Task Orchestration of Sight and Sound

arXiv:2608.05803v2 Announce Type: replace Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...

By Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang