arXiv Machine Learning June 24, 2026 By Kartik Hegde, Rehana Mahfuz, Yinyi Guo, Erik Visser

Aligning Audio Captions with Human Preferences

Read the original on arXiv Machine Learning →

arXiv:2509. 14659v3 Announce Type: replace-cross Abstract: Current audio captioning relies on supervised learning with paired audio-caption data, which is costly to curate and may not reflect human preferences in real-world scenarios.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Jul 6

Claim-Level Rubric Rewards for Video Caption Reinforcement Learning

In this paper, we introduce Claim-Level Rubric Rewards (CuRe), a structured reward framework designed to address the reward-design bottleneck in reinforcement learning for dense video captioning. Existing reward designs generally fall into two categories: holistic response-level judgment across heterogeneous criteria, or alignment-based evaluation against reference captions.

reinforcement-learning safety

arXiv AI

Jul 1

Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

arXiv:2511. 16757v2 Announce Type: replace-cross Abstract: Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored.

By Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo, Yiwen Shao, Hao Zhang, Dong Yu

llms

arXiv Computation and Language

Sep 3

SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval

SonicCaps is a large-scale audio captioning dataset featuring approximately 15 million captions paired with 700,000 audio clips, created using the Qwen3-Omni multimodal language model. The dataset emphasizes diversity by generating around 24 captions per clip through structured prompt engineering and few-shot generation, covering main descriptions, rephrased variants, and semantic tags. Human evaluations rate SonicCaps higher than existing datasets, and training CLAP models on it improves audio retrieval and zero-shot classification across public and commercial benchmarks.

By Zineb Lahrichi, Marc Ferras, Ga\"el Richard, Geoffroy Peeters

llms multimodal benchmarks

arXiv Computation and Language

Sep 25

An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

The paper introduces an evaluation framework for structured audio captions that separates acoustic and semantic aspects, such as timestamped sound event descriptions. It covers five axes—tag sets, descriptions, reasoning, numeric measurements, and spectral profiles—using large language model judges for semantics and deterministic metrics for temporal and acoustic features. Controlled perturbations validate that the metrics are robust to paraphrases but sensitive to real semantic and acoustic errors.

By Liang-Yuan Wu, Sripathi Sridhar, Mark Cartwright, Magdalena Fuentes

llms

arXiv Machine Learning

Jun 8

Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation

arXiv:2606. 07387v1 Announce Type: new Abstract: State-of-the-art text-to-music generation systems rely on massive proprietary datasets and industrial-scale compute, making it impossible to disentangle architectural contributions from resource advantages.

By Yun-Chen Cheng, Tzu-Hung Huang, Chih-Pin Tan

benchmarks safety

arXiv Computer Vision

Aug 21

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

arXiv:2608. 18607v2 Announce Type: replace Abstract: Using reinforcement learning to post-train joint video-audio generation models requires a reward signal.

By Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Xu Hang, Yu-Gang Jiang, Zuxuan Wu

diffusion reinforcement-learning benchmarks