The paper proposes the Effectiveness–Losslessness Framework to guide tokenization in domains beyond language, using predictive codelength as a criterion. It introduces two boundaries: the Fact–Token Boundary, where observable structure should be encoded into tokens, and the Token–State Boundary, where context‑dependent relations should remain for model state rather than being pre‑tokenized. Experiments on symbolic music show that making musical time explicit and applying tonal‑frame canonicalization improve predictive performance, while fixed pitch coordinates and reversible BPE can increase predictive code length, indicating that carrier compaction alone does not guarantee better predictions.
By Yi Wang
The paper introduces Whole-Piece Training for Symbolic Music Language Models using Full-Horizon Compressed Recurrence (FHCR), which maintains the full temporal horizon of recurrent memory while compressing its key-value representation to fit GPU limits. An evaluation diagnostic, KV-Reset Context Utilization (KRCU), demonstrates that full-horizon models retain long-range context beyond local windows, whereas limiting recurrent memory weakens this dependence. FHCR thus preserves long-range context utilization while significantly reducing recurrent memory cost, enabling efficient whole-piece modeling.
By Yungang Yi, Weihua Li, Matthew Kuo, Catherine Shi, Quan Bai
arXiv:2608. 00576v1 Announce Type: cross Abstract: High-polyphony symbolic music is increasingly used in generation, analysis, and arrangement, yet many downstream tasks require bounded representations with fixed tracks or slots.
By Ziyue Kang, Nan Nan, Chenhao Lin, Xiaohong Guan
arXiv:2604. 19532v3 Announce Type: replace-cross Abstract: Tokenizing music to fit the general framework of language models is a compelling challenge, especially considering the diverse symbolic structures in which music can be represented (e.
By Lekai Qian, Haoyu Gu, Jingwei Zhao, Ziyu Wang
DuoTok is a source‑aware dual‑track music tokenizer designed for vocal‑accompaniment generation. It first learns a semantic audio representation via self‑supervised pretraining, then refines source‑aware structure with feature‑replacement noise and multi‑task supervision (spectral reconstruction, source separation regularization, and an ASR head for lyric alignment). The encoder is frozen and hard‑routed codebooks for vocals and accompaniment are learned, while a diffusion decoder restores fine acoustic detail from the discrete tokens, achieving a favorable predictability‑fidelity trade‑off at ultra‑low bitrate across public benchmarks.
By Rui Lin, Zhiyue Wu, Jiahe Lei, Kangdi Wang, Weixiong Chen, Junyu Dai, Tao Jiang
arXiv:2607. 14537v1 Announce Type: cross Abstract: Rich internal representations of musical structure are essential for music understanding tasks such as machine-assisted music co-writing, yet self-supervised approaches for symbolic music representation remain underexplored, particularly those that encode the hierarchical multiscale nature of musical structures.
By Scott H. Hawley
arXiv:2607. 11124v1 Announce Type: cross Abstract: Music creation is fundamentally a process of revision.
By Haoyu Gu, Lekai Qian, Haowu Zhou, Qi Liu, Shuai Wang
The paper investigates how large language models encode and use relational information among tokens across transformer layers. By analyzing activations from prompts that require inferring relationships among three cyclic tokens (months, hours, weekdays, musical notes), the authors find a consistent layerwise progression: intermediate layers capture pairwise relationships, while later layers encode the full three‑token relationship to predict the next token. They also identify geometrically structured token relationships that do not influence prediction, and show that constraining models to use only causally relevant joint representations improves next‑token accuracy.
By Gurbir Arora, Toni J. B. Liu, Jiajun Bao, Rapha\"el Sarfati, Christopher J. Earls
arXiv:2608. 03920v1 Announce Type: cross Abstract: Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space.
By Zixun Guo, Simon Dixon
arXiv:2412. 11449v2 Announce Type: replace-cross Abstract: We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.
By Prateek Verma
arXiv:2606. 30642v1 Announce Type: cross Abstract: Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts.
By Shun Lei, Huaicheng Zhang, Dapeng Wu, Yaoxun Xu, Lishi Zuo, Wei Tan, Hangting Chen, Guangzheng Li, Jianwei Yu, Zhiyong Wu, Dong Yu
arXiv:2607. 01849v1 Announce Type: cross Abstract: Musical performance involves executing a set of high-level musical instructions, yet recovering those instructions from the performance is a challenging inverse problem.
By Yewon Kim, Apurva Gandhi, David Chung, Graham Neubig, Chris Donahue