DuET is a Dual-Encoder Transformer that jointly learns note-level piano performance-to-score transcription and frame-level beat/downbeat tracking. It uses bidirectional cross‑attention between note and frame encoders and an autoregressive decoder that alternates between their representations to predict structured score attributes. The model introduces TimeRoPE and Time‑Pitch Rotary Positional Encoding (TPRoPE) to share a performance‑time reference and encode relative temporal and pitch relationships, achieving significant error reductions on ASAP and Gangqinpu datasets.
By Wei Zeng, Ye Wang
Piano2Pop is a music‑expansion task that converts piano MIDI into full instrumental pop productions. The method uses a pseudo‑supervised approach that conditions a pretrained flow‑matching model on piano rolls, trained on synthetic piano‑instrumental pairs derived from pop recordings. Experiments on 476 real‑world piano‑cover/pop pairs show that a pair‑supervised piano‑cover generator outperforms zero‑shot baselines, while function‑selective transcription (FST) reduces piano‑heavy outputs in pair‑free training and remains a promising alternative.
By Yun-Chen Cheng, Chih-Pin Tan, Tzu-Hung Huang, Yi-Hsuan Yang
arXiv:2606. 10627v1 Announce Type: cross Abstract: The quality of piano performance depends on nuanced timing, articulation, and dynamic control, but practice feedback is often summary-based and hard to act on.
By Kazuki Kawamura, Fujiki Nakamura, Hayato Nishioka, Momoko Shioki, Shinichi Furuya, Jun Rekimoto
arXiv:2606. 12282v1 Announce Type: cross Abstract: Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes.
By Dmitrii Gavrilev
arXiv:2604.15086v3 Announce Type: replace-cross
Abstract: Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-gra...
By Jianxuan Yang, Xinyue Guo, Zhi Cheng, Kai Wang, Lipan Zhang, Jinjie Hu, Qiang Ji, Yihua Cao, Yihao Meng, Zhaoyue Cui, Mengmei Liu, Meng Meng, Jian Luan
PVSync is a unified model that estimates audio‑visual offset and scores phoneme‑level articulation for lip‑sync tasks. It uses window‑level contrastive learning for synchronisation and a phoneme‑level articulation objective that aligns audio and video embeddings of the same viseme class across clips. The model achieves higher correlation with human rankings and better performance on offset recovery and viseme matching benchmarks compared to existing methods.
By Kevin Stephen, Varun Menon, Timo Mertens, Nikita Drobyshev
SpanSynth-Edit is a flow‑matching model that enables MIDI‑guided synthesis and editing of multi‑instrument audio mixtures using low‑frame‑rate scalar‑quantised latents. It encodes instrument‑labelled note lifecycles as unordered event sets, pools them into a conditioning vector per audio‑latent frame, and uses contextual audio for instrument‑specific timbre guidance. The model supports editing by resynthesising target regions from revised MIDI and demonstrates competitive performance on single‑ and multi‑instrument benchmarks, including within‑frame onset control.
By Sungkyun Chang, Keshav Bhandari, Simon Dixon, Emmanouil Benetos
Gangqinpu is a large, multi‑genre piano dataset that provides aligned audio, MIDI, and musical scores for 18,786 pieces, covering pop, classical, jazz, folk/traditional, rock, and blues. The dataset includes automatic note‑level alignments, beat/downbeat annotations, genre labels, and a binary classifier to filter non‑expressive performances. It is designed to reduce data leakage with ASAP splits and has shown cross‑dataset improvements in beat tracking and in‑domain gains for transcription and expressive performance rendering.
By Wei Zeng, Anvay Mathur, Ye Wang
AV-FlowEdit is a training‑free, inversion‑free framework that enables prompt‑driven audio‑visual editing using joint audio‑video diffusion models. It leverages source‑to‑target flow differences to guide edits while preserving non‑target structure and cross‑modal correspondence. The method includes a Cross‑Modal Event‑Aware Spectral Trajectory Preservation module that suppresses drift by combining flow saliency, aligned audio activity, multiscale low‑frequency displacement, and diffusion‑stage scheduling, achieving strong prompt alignment, temporal consistency, and competitive audio quality.
By Shaohua Pan, Xiao Liu
arXiv:2512. 02652v2 Announce Type: replace-cross Abstract: Existing methods for expressive music performance rendering, a conditional generation task that aims to generate a human-like performance from a symbolic score, rely on supervised learning over small labeled datasets, which limits scaling of both data volume and model size, despite the availability of vast unlabeled music, as in vision and language.
By Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang, Lin-Han Jia, Lan-Zhe Guo, Yu-Feng Li
InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.
By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
arXiv:2609.38748v1 Announce Type: new
Abstract: Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/...
By Hanmo Chen, Chengcheng Liu, Tianxiao Chen, Zheyu Zhang, Siming Zheng, Jinwei Chen, Xu Yang, Cheng Deng, Bo Li, Peng-tao Jiang