Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,530 stories · RSS feed

arXiv AI

Jun 10

Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling

arXiv:2605. 27286v2 Announce Type: replace-cross Abstract: Time series foundation models (TSFMs) are transforming the forecasting paradigm through large-scale cross-domain pretraining.

By Yiding Liu, Yifan Hu, Hongjie Xia, Peiyuan Liu, Hongzhou Chen, Xilin Dai, Zewei Dong, Jiang-Ming Yang

benchmarks safety

arXiv AI

Jun 10

Machine Learning Methods for Studying Latent Neural Activity Dynamics

arXiv:2606. 10530v1 Announce Type: cross Abstract: Recent developments in brain recording are driving a demand for machine learning tools capable of decoding the latent structure of large populations of neurons.

By Shufeng Kong, Fumei Deng, Xinyi Dong, Caihua Liu, Weiwei Chen, Yingheng Wang, Daniel Cao, Azahara Oliva, Antonio Fernandez-Ruiz, Carla Gomes

llms diffusion benchmarks

arXiv AI

Jun 10

Piper: A Programmable Distributed Training System

arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.

By Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang

benchmarks

arXiv AI

Jun 10

PatchSTG: Scalable Spatiotemporal Graph Transformers for Traffic Forecasting on Irregular Sensor Networks

arXiv:2606. 09872v1 Announce Type: cross Abstract: Traffic forecasting is a fundamental component of intelligent transportation systems, yet remains challenging in real-world settings due to irregular sensor distributions and the high computational cost of modeling large-scale spatiotemporal dependencies.

By Jichao Li, Xuanming Shi

llms benchmarks

arXiv AI

Jun 10

Dual-Branch Gated Fusion for Open-Set Audio Deepfake Source Tracing

arXiv:2606. 10223v1 Announce Type: cross Abstract: Attributing a synthetic utterance to its originating system remains an open challenge: closed-set models fail to reject unseen synthesizers and produce overconfident predictions.

By Awais Khan, Kutub Uddin, Khalid Malik

benchmarks

arXiv AI

Jun 10

Dmsh: A Multi-Agent Reinforcement Learning Framework for All-Quad Mesh Generation

arXiv:2606. 10601v1 Announce Type: cross Abstract: Generating high-quality meshes for arbitrary geometries remains a fundamental bottleneck in computational engineering, often demanding heuristic tuning and semi-manual workflows.

By Anirudh Kalyan, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabczuk, Somdatta Goswami, Sundararajan Natarajan

agents reinforcement-learning benchmarks

arXiv AI

Jun 10

Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners

arXiv:2511. 10234v3 Announce Type: replace-cross Abstract: While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations.

By Daniel Herbst, Lea Karbevska, Divyanshu Kumar, Akanksha Ahuja, Fatemeh Gholamzadeh Nasrabadi, Fabrizio Frasca

llms fine-tuning benchmarks

arXiv AI

Jun 10

Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problems

arXiv:2606. 10431v1 Announce Type: cross Abstract: Multi-task vehicle routing problems play a critical role in enhancing efficiency across various industries and service sectors.

By Shuangchun Gui, Zhiguang Cao, Wen Song, Yew-Soon Ong

rag benchmarks

arXiv Machine Learning

Jun 10

TRAPS: Therapeutic Response Analysis via Pathway-informed Stratification

arXiv:2606. 09898v1 Announce Type: new Abstract: Cancer treatment planning requires decisions across multiple clinical dimensions at once.

By Sujoy Banik, Sayantan Chakraborty, Boishakhi Das Toma, Zainab Ghafoor, Ushashi Bhattacharjee, Koushik Howlader, Tirtho Roy

benchmarks

arXiv Machine Learning

Jun 10

Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

arXiv:2606. 10528v1 Announce Type: new Abstract: Current reinforcement learning from human feedback (RLHF) methods primarily rely on scalar rewards from a trained reward model (RM).

By Guozheng Li, Xiyan Fu, Yiwen Guo

reinforcement-learning benchmarks

arXiv AI

Jun 10

AuRA: Internalizing Audio Understanding into LLMs as LoRA

arXiv:2606. 11033v1 Announce Type: cross Abstract: Recent efforts to extend large language models (LLMs) to speech inputs typically rely on cascaded ASR-LLM pipelines, end-to-end speech-language models, or bridge/distillation-based adaptation.

By Bo Cheng, Lei Shi, Zhanyu Ma, Yuan Wu, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

llms rag fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 10

Time series forecasting from partial observations via Non-negative Matrix Factorization

arXiv:2102. 05314v2 Announce Type: replace Abstract: In modern time series problems, one aims at forecasting multiple time series with possible missing and noisy values.

By Yohann de Castro (ICJ, PSPM, CERMICS UMR 9032, ECL, IUF), Luca Mencarelli (CERMICS UMR 9032)

llms benchmarks

arXiv Machine Learning

Jun 10

It\^o maps for any-step SDEs

arXiv:2606. 11156v1 Announce Type: cross Abstract: Recent one-step generative models accelerate sampling by learning deterministic flow maps of the underlying dynamics.

By Zhengkai Pan, Peter Potaptchik, Wenxi Yao, Michael S. Albergo, Jakiw Pidstrigach

efficiency benchmarks

arXiv AI

Jun 10

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning

arXiv:2606. 10989v1 Announce Type: new Abstract: Large language model unlearning aims to suppress designated undesirable knowledge while preserving benign capabilities.

By Bocheng Ju, Jianhua Wang, Chengliang Liu, Xiaolin Chang

llms fine-tuning benchmarks safety

arXiv AI

Jun 10

Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets

arXiv:2606. 10979v1 Announce Type: new Abstract: Many Markov decision processes (MDPs) in operations research have feasible actions that are state dependent and defined implicitly by various operational constraints.

By Yi Chen (Lucy), Rushuai Yang (Lucy), Qiang Chen (Lucy), Dongyan (Lucy), Huo

reinforcement-learning benchmarks

arXiv AI

Jun 10

ReflectiChain: Epistemic Grounding in LLM-Driven World Models for Supply Chain Resilience

arXiv:2606. 10359v1 Announce Type: new Abstract: AI agents in supply chains face a fundamental epistemic gap: large language models (LLMs) interpret policies but lack physical grounding, while reinforcement learning (RL) optimizes flows but is semantically blind to unstructured constraints.

By Jia Luo

llms agents diffusion reinforcement-learning benchmarks safety

arXiv AI

Jun 10

MMClima: A Framework for Multimodal Climate Science Data and Evaluation

arXiv:2606. 10194v1 Announce Type: cross Abstract: Climate change research increasingly requires AI systems that reason across text, dynamic visual content, and scientific figures, yet existing climate QA benchmarks are small, mostly textual, and cover a narrow range of models.

By Muhammad Umer Sheikh, Hassan Abid, Khawar Shehzad, Ufaq Khan, Muhammad Haris Khan

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jun 10

ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics

arXiv:2606. 10479v1 Announce Type: new Abstract: Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight.

By Shunkai Zhang, Haoran Zhang, Yun Luo, Qianjia Cheng, Haodi Lei, Yizhuo Li, Runzhe Zhan, Zhilin Wang, Bangjie Xu, Yucheng Su, Xinmiao Han, Xiaoye Qu, Dongrui Liu, Zhouchen Lin, Yu Qiao, Ning Ding, Yafu Li, Yu Cheng

llms benchmarks

arXiv AI

Jun 10

Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages

arXiv:2606. 10933v1 Announce Type: new Abstract: LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories.

By Aman Sharma, Sushrut Thorat, Paras Chopra

llms agents benchmarks

arXiv AI

Jun 10

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains

arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.

By Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin, Swasthi P Rao, Shikhhar Siingh, Houhan Lu, Nadia Bathaee, Sriharsha Hatwar, Paresh Dashore, Anmol Jain, Kshitij Tayal, Xiuzhu Lin, Anirban Das, Sambit Sahu, Shi-Xiong Zhang

llms agents benchmarks

Get the daily briefing