DeepMind Blog

Lyria 3 Pro: Create longer tracks in more

Introducing Lyria 3 Pro, which unlocks longer tracks with structural awareness. We’re also bringing Lyria to more Google products and surfaces.

Hugging Face Trending Papers
Jul 9

Whareformer: Learning to Track What is Where in Long Egocentric Videos

The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner.

arXiv AI
Sep 25

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

TrackEverything is a 3D point tracker that overcomes the trade‑off between sparse long‑horizon tracking and dense short‑clip tracking by representing videos as persistent 3D scene tracks in world coordinates. It introduces voxel‑based de‑duplication at sliding‑window boundaries, a two‑stage refinement process (endpoint refiner and lightweight trajectory refiner), and a 3D WAFT module that replaces memory‑heavy 4D correlation volumes with efficient feature sampling. The method can track all visible points in videos longer than 1000 frames using only 40 GB of GPU memory, outperforming existing dense trackers on short clips and matching sparse trackers on long sequences.

By Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley
arXiv Machine Learning
Jul 1

Visualizing High-Dimensional Graph Embeddings via Informed Multi-View Projections

arXiv:2606. 31119v1 Announce Type: new Abstract: Graphs are commonly visualized in 2D, where humans readily interpret spatial relationships, yet such layouts often distort higher-dimensional structure.

By Ya Ji (Khoury College of Computer Sciences, Northeastern University, Seattle), Xuefeng Li (Khoury College of Computer Sciences, Northeastern University, Seattle), Timo Brand (School of Computation, Information and Technology, Technical University of Munich, Heilbronn, Germany), Jacob Miller (School of Computation, Information and Technology, Technical University of Munich, Heilbronn, Germany), Peng Zhang (Khoury College of Computer Sciences, Northeastern University, Seattle), Stephen Kobourov (School of Computation, Information and Technology, Technical University of Munich, Heilbronn, Germany), Yifan Hu (Khoury College of Computer Sciences, Northeastern University, Seattle)
arXiv AI
Jun 30

LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training

arXiv:2606. 30642v1 Announce Type: cross Abstract: Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts.

By Shun Lei, Huaicheng Zhang, Dapeng Wu, Yaoxun Xu, Lishi Zuo, Wei Tan, Hangting Chen, Guangzheng Li, Jianwei Yu, Zhiyong Wu, Dong Yu