← Back to all news
arXiv Computer Vision By Qirong Guo, Ruijie Li, Zeyan Li, Sicheng Shu, Guanzhi Ding, Weihang Ding

FFMT: Frequency-Factorized Multi-Rate Tokenization for Music-to-Dance Generation

Read the original on arXiv Computer Vision →

The paper introduces Frequency-Factorized Multi-Rate Tokenization (FFMT) for music-to-dance generation, separating root-relative motion into a low-frequency choreographic scaffold and a high-frequency residual. FFMT assigns a lower token rate and smaller codebook to the low-frequency component, while using a higher token rate and larger codebook for the high-frequency part, enabling a plan-then-articulate autoregressive factorization. Experiments on AIST++ and FineDance demonstrate strong distributional fidelity, realistic diversity, and music–motion synchronization, with ablations confirming the benefits of frequency factorization, asymmetric temporal allocation, and low-to-high conditioning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

Hugging Face Trending Papers
Jun 29

OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models.

diffusionmultimodalbenchmarks
More like this →
arXiv AI
Sep 15

DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generation

DuoTok is a source‑aware dual‑track music tokenizer designed for vocal‑accompaniment generation. It first learns a semantic audio representation via self‑supervised pretraining, then refines source‑aware structure with feature‑replacement noise and multi‑task supervision (spectral reconstruction, source separation regularization, and an ASR head for lyric alignment). The encoder is frozen and hard‑routed codebooks for vocals and accompaniment are learned, while a diffusion decoder restores fine acoustic detail from the discrete tokens, achieving a favorable predictability‑fidelity trade‑off at ultra‑low bitrate across public benchmarks.

By Rui Lin, Zhiyue Wu, Jiahe Lei, Kangdi Wang, Weixiong Chen, Junyu Dai, Tao Jiang
diffusionnlpmultimodalbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 16

DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription

DuET is a Dual-Encoder Transformer that jointly learns note-level piano performance-to-score transcription and frame-level beat/downbeat tracking. It uses bidirectional cross‑attention between note and frame encoders and an autoregressive decoder that alternates between their representations to predict structured score attributes. The model introduces TimeRoPE and Time‑Pitch Rotary Positional Encoding (TPRoPE) to share a performance‑time reference and encode relative temporal and pitch relationships, achieving significant error reductions on ASAP and Gangqinpu datasets.

By Wei Zeng, Ye Wang
llms
More like this →