The paper introduces Frequency-Factorized Multi-Rate Tokenization (FFMT) for music-to-dance generation, separating root-relative motion into a low-frequency choreographic scaffold and a high-frequency residual. FFMT assigns a lower token rate and smaller codebook to the low-frequency component, while using a higher token rate and larger codebook for the high-frequency part, enabling a plan-then-articulate autoregressive factorization. Experiments on AIST++ and FineDance demonstrate strong distributional fidelity, realistic diversity, and music–motion synchronization, with ablations confirming the benefits of frequency factorization, asymmetric temporal allocation, and low-to-high conditioning.
arXiv:2604. 19532v3 Announce Type: replace-cross Abstract: Tokenizing music to fit the general framework of language models is a compelling challenge, especially considering the diverse symbolic structures in which music can be represented (e.
arXiv:2606. 22726v2 Announce Type: replace Abstract: Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics.
Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models.
DuoTok is a source‑aware dual‑track music tokenizer designed for vocal‑accompaniment generation. It first learns a semantic audio representation via self‑supervised pretraining, then refines source‑aware structure with feature‑replacement noise and multi‑task supervision (spectral reconstruction, source separation regularization, and an ASR head for lyric alignment). The encoder is frozen and hard‑routed codebooks for vocals and accompaniment are learned, while a diffusion decoder restores fine acoustic detail from the discrete tokens, achieving a favorable predictability‑fidelity trade‑off at ultra‑low bitrate across public benchmarks.
DuET is a Dual-Encoder Transformer that jointly learns note-level piano performance-to-score transcription and frame-level beat/downbeat tracking. It uses bidirectional cross‑attention between note and frame encoders and an autoregressive decoder that alternates between their representations to predict structured score attributes. The model introduces TimeRoPE and Time‑Pitch Rotary Positional Encoding (TPRoPE) to share a performance‑time reference and encode relative temporal and pitch relationships, achieving significant error reductions on ASAP and Gangqinpu datasets.
arXiv:2609.37400v1 Announce Type: new Abstract: Generating realistic 3D dance from music is a challenging task that requires accurate synchronization with musical rhythms while capturing the spatial...
arXiv:2607. 13978v1 Announce Type: cross Abstract: Music-driven dance generation aims to produce human motion that is both rhythmically synchronized and semantically consistent with music.
ChoreoStudio is a music‑driven group choreography system that explicitly plans what formations dancers should adopt, when to transition between them, and which dancers execute each transition. It uses a music disentangler to extract rhythmic and structural features, a frozen omni‑modal model to generate a global semantic script, and a Director module that outputs centroid‑relative formation points, transition gates, and mobility allocations. These high‑level plans guide a diffusion generator that produces beat‑aligned body motion, resulting in state‑of‑the‑art motion quality, group organization, and formation dynamics on the AIOZ‑GDance benchmark.
The paper introduces CPR, a piano rendering framework that combines continuous autoregressive modeling with local flow matching and full‑sequence refinement. It predicts continuous hidden states, generates 24 kHz acoustic latents, and upsamples to 48 kHz, while new techniques BREPA and MT‑RoPE enhance musical semantics and cross‑modal alignment.
arXiv:2606. 24307v1 Announce Type: cross Abstract: Interactive music and live performance relies on real-time human expression, but modern generative music AI remains largely absent from this domain due to its prohibitive inference latency and offline rendering paradigm.
arXiv:2609.27094v1 Announce Type: cross Abstract: Automatic tagging is a core task in Music Information Retrieval (MIR), yet most tagging systems exploit only audio. Live music performance is inheren...
The paper proposes the Effectiveness–Losslessness Framework to guide tokenization in domains beyond language, using predictive codelength as a criterion. It introduces two boundaries: the Fact–Token Boundary, where observable structure should be encoded into tokens, and the Token–State Boundary, where context‑dependent relations should remain for model state rather than being pre‑tokenized. Experiments on symbolic music show that making musical time explicit and applying tonal‑frame canonicalization improve predictive performance, while fixed pitch coordinates and reversible BPE can increase predictive code length, indicating that carrier compaction alone does not guarantee better predictions.