Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

13,227 stories · RSS feed

arXiv AI

Aug 17

Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead

arXiv:2608. 13987v1 Announce Type: new Abstract: Nanbeige4.

By John T. Halloran

llms agents benchmarks

arXiv AI

Aug 17

Wrong but Useful: Trajectory Value Beyond Answer Correctness in Multi-Agent Messages

arXiv:2608. 14375v1 Announce Type: new Abstract: Multi-agent reasoning systems often use agreement, confidence, or automated scores to decide which messages should shape a final answer.

By Chih-Hsuan Yang, Anjir Ahmed Chowdhury, Cheng-Hau Yang, Weijian Zheng, Fernando Llorente, Xiaolong Ma, Xinyang Li, Eliu A. Huerta, Ian T. Foster, Rajeev Thakur

llms agents benchmarks

arXiv AI

Aug 17

AgentRewind: Recoverable Execution for Long-Horizon LLM Agents

arXiv:2608. 14380v1 Announce Type: new Abstract: Many real-world tasks require LLM agents to interact with their environments over long execution horizons.

By Yu Zhuang, Kefei Chen, Yitong Duan, Shuxin Zheng, Jian Li, Xu-Yao Zhang

llms agents benchmarks safety

arXiv AI

Aug 17

Shift Aware Transfer Learning with Adaptive Dual-Encoder Fusion for PM Forecasting in Data-Limited Environments

arXiv:2608. 14456v1 Announce Type: new Abstract: Short-horizon forecasting of fine particulate matter (PM2.

By Shahab Band, Hamed Mohammadi

fine-tuning benchmarks

arXiv AI

Aug 17

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

arXiv:2608. 13566v1 Announce Type: cross Abstract: Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.

By Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov, Mikhail Evtikhiev, Ana Terna, Rastislav Rabatin, Timur Kudashev, Timofey Bryksin, Arina Puchkova, Patrik Bartak, Egor Bogomolov, Sergey Titov

llms agents fine-tuning benchmarks

arXiv AI

Aug 17

BCMT: Blockwise Causal Memory Transformer

arXiv:2608. 13578v1 Announce Type: cross Abstract: Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length.

By Rachid Arezki

llms efficiency benchmarks

arXiv AI

Aug 17

Jais 2: A Family of Arabic-Centric Open Large Language Models

arXiv:2608. 13580v1 Announce Type: cross Abstract: Jais 2 is a family of Arabic-centric large language models developed jointly by MBZUAI, Cerebras, and Inception, designed to advance Arabic-centric language modeling, with strong performance across the Arabic and culturally grounded benchmarks evaluated in this report.

By Mohamed Anwar, Abed Alhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Sadallah, Gurpreet Gosal, Gokulakrishnan Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah AlBarri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, Sajid siddiki, Samta Kamboj, Onkar Pandit, Sunil Kumar Sahu, AbdelRahman Elbadawy, Amr Mohamed, Ahmad Chamma, Evan Dufraisse, Abdelaziz Bounhar, Dani Bouch, Hadi Abdine, Guokan Shang, Fajri Koto, Yuxia Wang, Zhuohan Xie, Ali Mekky, Rania Elbadry, Sarfraz Ahmad, Momina Ahsan, Omar El Herraoui, Daniil Orel, Hasan Iqbal, Kareem Elzeky, Mervat Abassy, Kareem Elozeiri, Saadeldine Eletter, Farah Atif, Nurdaulet Mukhituly, Haonan Li, Xudong Han, Aaryamonvikram Singh, Zainul Abedien Ahmed Quraishi, Neha Sengupta, Larry Murray, Avraham Sheinin, Joel Hestness, Natalia Vassilieva, Hector Xuguang Ren, Zhengzhong Liu, Michalis Vazirgiannis, Preslav Nakov

llms nlp efficiency benchmarks

arXiv AI

Aug 17

Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

arXiv:2608. 13624v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups.

By Zhe Liu

llms rag nlp multimodal benchmarks safety

arXiv AI

Aug 17

MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

arXiv:2608. 13690v1 Announce Type: cross Abstract: Medical image segmentation is still largely treated as a vision-only problem, although clinical interpretation often relies on textual knowledge of anatomy, location, appearance, and surrounding context.

By Rafi Ibn Sultan, Hui Zhu, Chengyin Li, Dongxiao Zhu

llms computer-vision multimodal benchmarks safety

arXiv AI

Aug 17

Agentic Transaction: Towards ACID-Compliant Agent Systems

arXiv:2608. 13900v1 Announce Type: cross Abstract: Large language model (LLM) agents are evolving from conversational assistants into autonomous systems that execute long-horizon tasks through reasoning, tool use, code generation, and workspace manipulation.

By Zhaoyan Sun, Xiaoxiao Wang, Guoliang Li

llms agents robotics benchmarks

arXiv AI

Aug 17

MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

arXiv:2608. 14015v1 Announce Type: cross Abstract: Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time.

By Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang

llms agents fine-tuning efficiency multimodal benchmarks

arXiv AI

Aug 17

P2Skill: Privacy Preserving Skill Distillation for Cloud-Local LLM Inference Systems

arXiv:2608. 14094v1 Announce Type: cross Abstract: Cloud-local LLM inference systems have the potential to use the reasoning capability of large cloud models while protecting sensitive user data on personal devices.

By Myunghoon Ryu, Geunpyo Park, Sungjoon Lee, XinYu Piao, Jong-Kook Kim

llms fine-tuning efficiency benchmarks safety

arXiv AI

Aug 17

SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation

arXiv:2608. 14138v1 Announce Type: cross Abstract: Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations.

By Jinsheng Quan, Jianhua Li, Siyi Xie, Xuanke Shi, Kewang Deng, Zukai Chen, Feifei Shao, Lei Yang, Quan Wang, Yawei Luo

multimodal benchmarks

arXiv AI

Aug 17

Multi-Objective Bayesian Optimization for Model Merging

arXiv:2608. 14264v1 Announce Type: cross Abstract: Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning.

By Utkarsh Agarwal, Vamshi Bonagiri, Raul Astudillo, Monojit Choudhury

llms fine-tuning benchmarks

arXiv AI

Aug 17

Reflex: Enabling Fast and Predictive Vision-Language-Action Models for Reaction-Critical Manipulation

arXiv:2608. 14379v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have recently achieved promising performance in robotic manipulation.

By Yuxuan Chen, Wanruo Zhang, Xiao Li

robotics multimodal benchmarks

arXiv AI

Aug 17

Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions

arXiv:2602. 06746v2 Announce Type: replace Abstract: We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy capable of generalising to arbitrary, possibly unseen tasks.

By Alessandro Abate, Giuseppe De Giacomo, Mathias Jackermeier, Jan Kret\'insk\'y, Maximilian Prokop, Christoph Weinhuber

rag agents reinforcement-learning benchmarks

arXiv AI

Aug 17

LLM-Advisor: An LLM Advisor for Cost-efficient Path Planning across Multiple Terrains

arXiv:2503. 01236v3 Announce Type: replace-cross Abstract: This paper addresses fixed-graph terrain-aware path refinement, in which a global planner is restricted to a predefined route space and may remain optimal within that space while missing lower-cost terrain corridors available in the native-resolution map.

By Ling Xiao, Toshihiko Yamasaki

llms multimodal benchmarks

arXiv AI

Aug 17

INFORM-CT: INtegrating LLMs and VLMs FOR Incidental Findings Management in Abdominal CT

arXiv:2512. 14732v3 Announce Type: replace-cross Abstract: Incidental findings in CT scans, though often benign, can have significant clinical implications and should be reported following established guidelines.

By Idan Tankel, Nir Mazor, Rafi Brada, Christina LeBedis, Guy ben-Yosef

llms agents computer-vision multimodal benchmarks

arXiv Machine Learning

Aug 17

Recent Advances in Deep Learning-Based Drug-Target Binding Affinity Prediction

arXiv:2608. 13797v1 Announce Type: new Abstract: Computational approaches to drug discovery involve multiple sub-problems, and among them, drug-target binding affinity prediction plays an important role.

By Jafin Khan, Md Hossain Shuvo

multimodal benchmarks safety

arXiv Machine Learning

Aug 17

HI-MeshGraphNets: Efficient and Accurate Mesh-based Physics Learning with Hierarchical Multi-scale Graph Neural Networks

arXiv:2608. 13827v1 Announce Type: new Abstract: Machine-learned physical surrogate models have become promising alternatives to mesh-based numerical solvers.

By SiHun Lee, Dong-Hyuk Park, Taesoo Bang, Seung-Hoon Kang

benchmarks

Get the daily briefing