Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing

arXiv:2604.07747v2 Announce Type: replace Abstract: In reinforcement learning with verifiable rewards (RLVR), challenging math questions whose sampled groups remain uniformly incorrect yield no infor...

By Pei-Xi Xie, Che-Yu Lin, Cheng-Lin Yang

llms reinforcement-learning benchmarks

arXiv AI

Oct 15

OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking

arXiv:2605.03762v2 Announce Type: replace Abstract: Forecasting is a composite capability encompassing information gathering, evidence integration, situational assessment, and decision-making. Large...

By Yiding Ma, Chengyun Ruan, Kaibo Huang, Jin Tan, Zhongliang Yang, Linna Zhou

llms agents benchmarks

arXiv AI

Oct 15

VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

arXiv:2605.06068v2 Announce Type: replace Abstract: For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engin...

By Keisuke Kamahori, Shihang Li, Simon Peter, Baris Kasikci

llms agents benchmarks

arXiv AI

Oct 15

MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs

arXiv:2605.08374v4 Announce Type: replace Abstract: Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retr...

By Junwei Liao, Haoting Shi, Ruiwen Zhou, Jiaqian Wang, Shengtao Zhang, Wei Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Bo Tang, Weinan Zhang, Muning Wen

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

Argumentation-Based Person-Tailored Storytelling for Older Adults: A Reflective Hybrid AI Architecture

arXiv:2605.10531v2 Announce Type: replace Abstract: This work investigates knowledge-driven, large language model (LLM)-based storytelling for older adults. We present a reflective storytelling agent...

By Jayalakshmi Baskar, Vera C. Kaelin, Kaan Kilic, Helena Lindgren

llms agents benchmarks

arXiv AI

Oct 15

CoMIC: Collaborative Memory and Insights Circulation for Long-Horizon LLM Agents in Web Cloud-Edge Systems

arXiv:2606.00756v2 Announce Type: replace Abstract: LLM-based Web agents are increasingly deployed on edge servers to carry out long-horizon tasks on behalf of users with low latency. However, the li...

By Yannan Wang, Longli Yang, Zhen Liu, Abhishek Kumar, Carsten Maple

llms agents fine-tuning benchmarks

arXiv AI

Oct 15

Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models

arXiv:2608.01008v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) can memorize and disclose sensitive information, motivating machine unlearning to remove designated knowle...

By Junkai Lin, Junkai Chen, Siqi Hou, Yuhao He, Ruiqi Liu, Chenhan Jin, Shengze Xu, Tieyong Zeng, Xiangyuan Lan

llms multimodal benchmarks safety

arXiv AI

Oct 15

Composition without a Host: Carrying the Spatiotemporal-Composability Guarantee of Plugin-Based Agents across Processes

arXiv:2608.28553v4 Announce Type: replace Abstract: Plugin-based agents assemble capabilities at runtime, and the spatiotemporal-composability calculus proves a reversibility guarantee for this assem...

By Hanzhang Jia, Liheng Zeng, Hao Cheng, Yi Gao, Bo Ma, Guolong Xu

agents benchmarks

arXiv AI

Oct 15

Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

arXiv:2609.33196v2 Announce Type: replace Abstract: Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their...

By Haiquan Hu, Yuzhu Liang, Weicheng Tang, Yanzeng Li, Yao Shi, Li Xu, Tian Wang

llms benchmarks

arXiv AI

Oct 15

Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

arXiv:2610.10358v2 Announce Type: replace Abstract: As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pre...

By Junkai Chen, Yuhao He, Qianshan Wei, Junxiang You, Jingwen Shao, Junkai Lin, Zhongkai Yue, Xiaotian Ye, Zhengbo Jiao, Jiali Cheng, Zhijie Deng, Kening Zheng, Ruiqi Liu, Hadi Amiri, Yi Yu, Zhenan Sun, Qi Li, Ka-Ho Chow, Sijia Liu, Liang Wang, Jiaqi Li, Shu Wu

llms efficiency multimodal benchmarks safety

arXiv AI

Oct 15

FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence

arXiv:2402.03694v3 Announce Type: replace-cross Abstract: Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryp...

By Shinan Liu, Ted Shaowang, Gerry Wan, Jeewon Chae, Jonatas Marques, Sanjay Krishnan, Nick Feamster

benchmarks

arXiv AI

Oct 15

Randomly Wrong Signals: Bayesian Auction Design with ML Predictions

arXiv:2502.08792v4 Announce Type: replace-cross Abstract: Machine learning can help sellers predict buyers' willingness to pay, but its predictions may be plausible and yet entirely unrelated to a bu...

By Ilan Lobel, Humberto Moreira, Omar Mouchtaki

benchmarks safety

arXiv AI

Oct 15

Generative AI for Autonomous Driving: A Review

arXiv:2505.15863v2 Announce Type: replace-cross Abstract: Generative AI (GenAI) is rapidly advancing the field of Autonomous Driving (AD), extending beyond traditional applications in text, image, an...

By Katharina Winter, Abhishek Vivekanandan, Rupert Polley, Yinzhe Shen, Christian Schlauch, Mohamed-Khalil Bouzidi, Bojan Derajic, Natalie Grabowsky, Annajoyce Mariani, Dennis Rochau, Giovanni Lucente, Harsh Yadav, Firas Mualla, Adam Molin, Sebastian Bernhard, Christian Wirth, \"Omer \c{S}ahin Ta\c{s}, Nadja Klein, Fabian B. Flohr, Hanno Gottschalk

llms agents diffusion robotics benchmarks safety

arXiv AI

Oct 15

One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging

arXiv:2508.06163v2 Announce Type: replace-cross Abstract: Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task t...

By Yingfeng Luo, Dingyang Lin, Junxin Wang, Kaiyan Chang, Ziqiang Xu, Weiqiao Shan, Yinqiao Li, Tong Zheng, Bei Li, Anxiang Ma, Tong Xiao, Zhengtao Yu, Jingbo Zhu

llms fine-tuning efficiency multimodal benchmarks

arXiv AI

Oct 15

Frequency-Aware Model Parameter Explorer: A new attribution method for improving explainability

arXiv:2510.03245v3 Announce Type: replace-cross Abstract: State-of-the-art attribution methods such as AttEXplore rely on adversarial sample generation that applies an all-pass filter across the freq...

By Ali Yavari, Alireza Mohamadi, Elham Beydaghi, Philipp Seeb\"ock, Rainer A. Leitgeb

llms computer-vision benchmarks safety

arXiv AI

Oct 15

Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Hardware-Validated Benchmarking Study of Forward-Only Algorithms

arXiv:2511.01061v2 Announce Type: replace-cross Abstract: Backpropagation (BP) powers modern deep learning, but it must hold every activation in memory until a global backward sweep consumes it, and...

By Przemys{\l}aw Spyra, Witold Dzwinel

benchmarks

arXiv AI

Oct 15

Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos

arXiv:2601.16333v3 Announce Type: replace-cross Abstract: Foundation models are used for many real-world applications involving language understanding and generation from temporally-ordered multimoda...

By Aditya K Surikuchi, Raquel Fern\'andez, Sandro Pezzelle

multimodal benchmarks

arXiv AI

Oct 15

From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs

arXiv:2602.00628v3 Announce Type: replace-cross Abstract: We investigate the extent to which an LLM's hidden-state geometry can be recovered from its behavior in psycholinguistic experiments. Across...

By Louis Schiekiera, Max Zimmer, Christophe Roux, Sebastian Pokutta, Fritz G\"unther

llms benchmarks

arXiv AI

Oct 15

DCD: Decomposition-based Causal Discovery from Autocorrelated and Non-Stationary Temporal Data

arXiv:2602.01433v3 Announce Type: replace-cross Abstract: Multivariate time series in domains such as finance, climate science, and healthcare often exhibit long-term trends, seasonal patterns, and s...

By Muhammad Hasan Ferdous, Ian Stockwell, Md Osman Gani

benchmarks safety

arXiv AI

Oct 15

Formal Policy Enforcement for Agentic Systems

arXiv:2602.16708v4 Announce Type: replace-cross Abstract: As agentic computing systems grow increasingly popular, they raise serious security concerns due to their susceptibility to various attacks....

By Nils Palumbo, Sarthak Choudhary, Jihye Choi, Guy Amir, Prasad Chalasani, Isil Dillig, Somesh Jha

agents benchmarks

Get the daily briefing