Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 21

VLA-Rolling: Fewer Denoising Steps, Better Control via Adaptive Action Reuse

arXiv:2610.25043v2 Announce Type: replace Abstract: Flow-matching vision-language-action (VLA) models generate an action chunk by iteratively denoising Gaussian noise, but typically execute only a sh...

By Sicheng He, Yi Pan, Qiuting Yu, Jiaming Huang, Qi Lu, Shuyan Wu, Hongyan Feng, Xuanyu Liu, Jun Zhang, Jie Zhang, Wenqi Zhang, Xuhong Zhang, Jianwei Yin

multimodal benchmarks

arXiv AI

Oct 21

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning

arXiv:2603.00889v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, largely enabled by supervised fine-tuning (SFT)- and reinforce...

By Xinyu Zhu, Yihao Feng, Yanchao Sun, Xianzhi Du, Pingzhi Li, Olli Saarikivi, Yun Zhu, Yu Meng

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Oct 21

Anchoring and Rescaling Attention for Semantically Coherent Inbetweening

arXiv:2603.17651v2 Announce Type: replace-cross Abstract: Generative inbetweening (GI) seeks to synthesize realistic intermediate frames between the first and last keyframes beyond mere interpolation...

By Tae Eun Choi, Sumin Shim, Junhyeok Kim, Seong Jae Hwang

benchmarks safety

arXiv AI

Oct 21

NVIDIA Spectrum-X: Hardware-Accelerated Multiplane Networking for Large-Scale AI Training

arXiv:2605.21187v2 Announce Type: replace-cross Abstract: As distributed model training scales to span hundreds of thousands of GPUs, scale-out networks face unprecedented performance and efficiency...

By Sajy Khashab, Albert Gran Alcoz, Alon Gal, Jacky Romano, Rani Abboud, Yonatan Piasetzky, Lior Maman, Amit Nishry, Barak Gafni, Omer Shabtai, Matty Kadosh, Dror Goldenberg, Gilad Shainer, Mark Silberstein

llms benchmarks

arXiv AI

Oct 21

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

arXiv:2609.35052v2 Announce Type: replace-cross Abstract: Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories...

By Hao Wang, Tao Yu, Liuzhou Zhang, HeXin Wang, Haopeng Jin, Yuxuan Zhou, Xinming Wang, Hongzhu Yi, Xinye Li, Yuanlei Wang, Ping Nie, Yan Huang, Yuxuan Zhang, Pengfei Zhou, Yanyan Zou, Wei Yang

robotics benchmarks

arXiv AI

Oct 21

EDISCO: Equivariant DIScrete Diffusion for Euclidean Combinatorial Optimization

arXiv:2610.04953v2 Announce Type: replace-cross Abstract: Euclidean combinatorial optimization problems (ECOPs), such as the Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (...

By Ruogu Chen, Jie Han

diffusion benchmarks safety

arXiv AI

Oct 21

Composing What Each Teacher Learned: Delta Distillation for Multi-Teacher On-Policy Learning

arXiv:2610.10460v2 Announce Type: replace-cross Abstract: Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollou...

By Yuanda Xu, Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard

efficiency benchmarks

arXiv AI

Oct 21

Muse Glimmer 30B: An On-Device Agentic Model

arXiv:2610.24628v2 Announce Type: replace-cross Abstract: We introduce Muse Glimmer, an open-source 30B parameter causal language model with perception encoder, purpose-built for autonomous agentic t...

By Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed, Juan C. Angeles Acuna, Alexis Audran-Reiss, Asaf Avrahamy, Yoram Bachrach, Can Balioglu, Amar Budhiraja, Onur \c{C}elebi, Gaurav Chaurasia, Joy Chen, Lynn Cherif, Lucia Cipolina-Kun, Gal Cohen, Timoth\'ee Darcet, Beto de Paola, Digant Desai, Sam Devlin, Derek Dunfield, Ivan Evtimov, Maryam Fazel-Zarandi, Rob Fergus, Jean-Christophe Gagnon-Audet, Grant Gardner, Bhavul Gauri, Paris Giampouras, Sujan Kumar Gonugondla, Antoine Grosnit, Young Han, Michael Hassid, Monuril Islam, Masoud Jalili Sabet, Songhao Jia, Narine Kokhlikyan, Svetlana Kurtsman, Sandra Lefdal, Chee Hau Leow, Noam Levi, Mike Lewis, Sanae Lotfi, Xiyang Luo, Despoina Magka, Saeed Mahloujifar, Neal Mangaokar, Ishita Mediratta, Alexander H. Miller, Hela Momand, Saba Nazir, Kelvin Niu, Mattia Opper, Parth Pathak, Alberto Pepe, Danielle Pintz, Mikhail Plekhanov, Sidhartha Reddy Potu, Anton Protopopov, Shengyi Qian, Shashank Rajput, Joseph Rocca, Scott Roy, Luca Sbordone, Amin Shakeri, Tatiana Shavrina, Michael Shvartsman, Ruan Silva, Anya Sims, Eric Michael Smith, Hugo Touvron, Mabel Ubong, Isabel Urrego, Yanzheng Xiang, Zhiyuan Yao, Alex Yarmula, He Ye, Jade Yu, Luke Zettlemoyer, Arman Zharmagambetov, Perpetua Fathima Ahlijah, Tremo Ahmed, Enrique Alfonseca, Pedram Bashiri, Nikolay Bashlykov, Amine Benhalloum, Aaron Berman, Rohith Bhatia, Siddharth Bhattacharya, Prashant Bhende, Chloe Bi, Shuchao Bi, Clement Anthonioz Blanc, Allen Bolourchi, Iva Simon-Bubalo, Pere-Llu\'is Huguet Cabot, Andrea Caciolai, Erica Cai, Xiang Tony Cao, Yue Cao, Andrew Caples, Jinzhao Chang, Lele Chen, Moya Chen, Weijie Chen, Wenhu Chen, Xiaobai Chen, Yinyin Chen, Zhengxing Chen, Wei Cheng, Pavel Chikulaev, Edward Chou, Sy Choudhury, Brian Clerkin, Marta R. Costa-juss\`a, Hanjun Dai, Curtis Davis, Myra Deng, Tiffany Siqi Deng, Dujian Ding, Hantian Ding, Shuangrui Ding, Jesse Dodge, Carl Domond, Shweta Doshi, Shirin Eghtesadi, Faith Eischen, Athyu Eleti, Jordi Armengol-Estape, Alex Fabbri, Han Fang, Paula Farani, Aiman Farooq, Darius Feher, Christoph Feichtenhofer, Amanda Felix, Romain Froger, Carl Chengyan Fu, Sean Gahagan, Yifan Gao, Itai Gat, Jannes Gladrow, Akashdeep Goel, Ahuva Goldstand, Bo He, David Heineman, Dylan Herman, Irene Hjelt, Joonas Hjelt, Alexander Holz, Rui Hou, Annie Hu, Yuan-Ting Hu, Bernie Huang, Jing Huang, Kun Huang, Ashley Hunter, Srinivasan Iyer, Somya Jain, Bei Jia, Michael Jiang, Song Jiang, Wenwen Jiang, Lin Jin, Jacob Kahn, Melanie Kambadur, Yingxin Kang, Justine Kao, Stephane Kasriel, Hossein Kavianihamedani, Madian Khabsa, Ameya Khandekar, Yash Khare, Gefen Kohavi, Kavya Kopparapu, Praveen Krishnan, Venu Kuchibhotla, Ananya Kumar, Leona Lan, Jack Lanchantin, Hunter Lang, Nikolay Pavlovich Laptev, Yuanzhi Li, Yunlu Li, Asad Liaqat, Johnson Lin, Runwei Lin, Zhichao Lin, Shaun Lindsay, Yaron Lipman, Frederick Liu, Mengchen Liu, Qiang Liu, Yinghai Lu, Yujie Lu, Christopher Luc, Faye Ma, Pingchuan Ma, Tengyu Ma, Wufei Ma, Xiaodong Ma, Damir Maksyutov, Josh Malone, Udeep Manchanda, Thomas Mann, Kalyani Marathe, Akhil Mathur, Roberto-Rafael Maura-Rivero, Ruslan Mavlyutov, Thomas Mesnard, Gregoire Mialon, Julian Michael, Sagar Miglani, Todor Mihaylov, Naureen Mitha, Archi Mitra, Leo Mok, Naila Murray, Muthukumaravel Muthuraman, Mergen Nachin, Nese Demir Nalci, Saaketh Narayan, Alyssa Newcomb, Peter Ney, Joycelyn Ng, Devin Norder, Maxwell Nye, Rajvardhan Oak, Cody Ohlsen, Artidoro Pagnoni, Yi Pan, Nilesh Pandey, Guan Pang, Richard Yuanzhe Pang, Mahesh Pasupuleti, Parth Patel, Yash Patel, Vladan Petrovic, Leslie St. Pierre, Norman Ponte, Yundi Qian, Zengyi Qin, Jack Rae, Ramya Raghavendra, Emmanuel Ramirez, Nikhila Ravi, Hongyu Ren, Riandy Riandy, Harrison Rudolph, Kalyan Saladi, Eduardo S\'anchez, Karthik Abinav Sankararaman, Alan Schelten, Thomas Scialom, Aman Shankar, Paresh Shenoy, Alana Shine, Hamid Shojanazeri, Mitesh Kumar Singh, Pawan Sisodiya, Ali Sol, Sten Sootla, DiJia Su, Suraj Subramanian, Brian Sun, Tony Sun, Zhiqing Sun, Natacha Supper, Jason Talpalar, Ellen Xiaoqing Tan, Liang Tan, Bu\u{g}ra Tekin, Jean-Christophe Testud, Parth Thakkar, Joseph Tighe, Ty Toledano, T Tse, Igor Tufanov, Alex Vaughan, Josh Walters, Alex Wang, Bethany Wang, Meng Wang, Rocky Wang, Weiyue Wang, Xiaolan Wang, Xin Wang, Zijian Wang, Bram Wasti, Quentin Wellens, Yeming Wen, Mary Williamson, Yuncheng Wu, Fei Xia, Tian Xie, Anlu Xing, David Xu, Jacob Xu, Puxin Xu, Zheng Xu, Shen Yan, Megan Yang, Yining Yang, Nick Yannopoulos, Kevin Yao, Michihiro Yasunaga, Haoxuan You, Michael You, Dingli Yu, Tianhe Yu, Yue Yu, Summer Yue, Xiang Yue, Shuwei Zeng, Hanwen Zha, Chen Zhang, Fred Zhang, Hongming Zhang, Kaiyu Zhang, Meng Zhang, Mingyang Zhang, Yi Zhang, Shengjia Zhao, Chao Zhou, Claire Zhou, Shuyan Zhou, Yanjun Zhou, Yaxuan Zhou, Jiacheng Zhu, Rosalie Zhu, Yun Zhu

llms agents efficiency benchmarks

arXiv AI

Oct 21

Policy Search in Code Space for Multi-Agent Pathfinding

arXiv:2610.25946v2 Announce Type: replace-cross Abstract: Developing coordination policies for complex multi-agent systems relies on either labor-intensive algorithm design or opaque task-specific re...

By Matthias Eder, Pantelis Emmanouil, Vivek Farias, Fabian Fischer, Adam Jozefiak, Tianyi Peng, Thomas Ulz, Raphael Watschinger, Fangyu Wu, Andrew Zheng

agents reinforcement-learning robotics efficiency benchmarks safety

arXiv Machine Learning

Oct 21

ORBIT: Autoregressive Time Series Generation via Next Frequency-Band Prediction

arXiv:2610.27000v1 Announce Type: new Abstract: Deep generative models provide an effective modern solution to alleviating the problem of data scarcity by synthesizing realistic time series. However,...

By Chunyi Hou, Xiangfei Qiu, Hanyin Cheng, Chenchen Sun, Yisheng An, Yang Shu, Jilin Hu, Bin Yang

llms efficiency benchmarks

arXiv AI

Oct 21

An Empirical Study of Development-Tuned Representation Sensitivity for Medical QA Data Selection

arXiv:2609.09684v2 Announce Type: replace-cross Abstract: We study medical question-answering (QA) data selection when a developer already has questions, labels, and auxiliary explanatory texts but c...

By Yuexin Wu, Dayou Yu, Vasile Rus

fine-tuning benchmarks

arXiv Machine Learning

Oct 21

Shaped SAEs: Expanding the Toolkit for Mechanistic Interpretability

arXiv:2610.27375v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) seek to decompose the dense activations of large language models (LLMs) into sparse, potentially interpretable features usin...

By Oveys Delafrooz Noroozi, Bhagyashree Puranik, Upamanyu Madhow

llms benchmarks safety

arXiv Machine Learning

Oct 21

BEACON: Bridging Multimodal Semantics and Asynchronous Propagation for Cloud System Anomaly Detection

arXiv:2610.27542v1 Announce Type: new Abstract: Modern large-scale cloud systems continuously generate massive and multimodal observability data, such as time-series metrics, log records, and system...

By Zhe Li, Tengxue Zhang, Yang Shu, Tung Kieu, Lunting Fan, Yingying Zhang, Bin Yang

multimodal benchmarks

arXiv Machine Learning

Oct 21

From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents

arXiv:2610.27623v1 Announce Type: new Abstract: Reinforcement learning for long-horizon language agents requires identifying whether intermediate actions make eventual task success more or less likel...

By Jiaxi Liu, Shixiong Kai, Zhentao Tang, Yang Li, Yuqi Cui, Mingxuan Yuan, Junchi Yan

agents reinforcement-learning benchmarks

arXiv Machine Learning

Oct 21

SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning

arXiv:2610.27812v1 Announce Type: new Abstract: As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by dir...

By Guanchen Li, Yixing Xu, Xuanwu Yin, Dong Li, Emad Barsoum

llms efficiency benchmarks

arXiv Machine Learning

Oct 21

Coincidence-Maximizing Customer Clustering Using Smart Meter Data

arXiv:2610.28178v1 Announce Type: new Abstract: Electricity distribution planning depends on peak-demand coincidence within customer groups, yet similarity-based clustering of smart-meter data does n...

By Ramiro H. G\'alvez, Gabriel Martos Venturini, Iv\'an Mondrzak

benchmarks

arXiv Machine Learning

Oct 21

Discount Yourself: Self-inclusive Freshness-weighted Gossip for Decentralised Learning under Client Outages

arXiv:2610.28545v1 Announce Type: new Abstract: Decentralised federated learning (DFL) trains personalised models on devices that average with their neighbours rather than with a central server. Real...

By Qiyuan Wang, Ke Xiao, Christos Anagnostopoulos

benchmarks

arXiv Machine Learning

Oct 21

MultiCard: Multiple Attribute Cardinality Estimation via Deep Learning

arXiv:2610.27019v1 Announce Type: cross Abstract: Cardinality estimation is essential for query optimization, but most existing learned estimators focus on a single predicate type, such as string LIK...

By Mobina Koochaknia, Maryam Ramezani, Mohammad Hasan Shafaat

llms efficiency benchmarks

arXiv Machine Learning

Oct 21

LOB-Impact Benchmark: Emergent Market Impact in Generative Limit Order Book Models

arXiv:2610.27100v1 Announce Type: cross Abstract: Realistic market impact is essential for generative limit order book (LOB) models to serve as world models for trading agents. If a simulator mispric...

By George Nigmatulin, Valentin Mohl, Sascha Frey, Kang Li, Aramis Marti-Shahandeh, Mihai Cucuringu, Anisoara Calinescu, Jakob Foerster, Stefan Zohren

agents benchmarks

arXiv Machine Learning

Oct 21

DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding

arXiv:2610.27379v1 Announce Type: cross Abstract: Block-diffusion speculative decoding accelerates large language model inference by drafting multiple tokens in parallel. However, one-step drafting i...

By Shreyas Kulkarni, Uttam Gurram, Madhav Kanda, Krishna Teja Chitty Venkata, Shichu Chen, Kartik Balasubramaniam, Supriyo Chakraborty, Sambit Sahu

llms diffusion efficiency benchmarks

Get the daily briefing