Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 16

Retrieval-Driven Memory Reconsolidation for Long-Term LLM Agents

arXiv:2609.16053v1 Announce Type: cross Abstract: Long-term memory is essential for LLM-based agents operating over extended interactions. Existing memory systems primarily update memory when new inf...

By Yuanyi Song, Yukai Wang, Xinbei Ma, Zhihui Fu, Jianghao Lin, Weiwen Liu, Jun Wang, Huarong Deng, Yong Yu, Weinan Zhang

llms agents benchmarks

arXiv AI

Sep 16

ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglement

arXiv:2609.16284v1 Announce Type: cross Abstract: Query-conditioned vision--language models enable fine-grained interpretation by revealing how visual evidence changes with textual queries. However,...

By Yan Zhu, Yongbo Chen, Zhengming Ding, Rebecca Faust

llms multimodal benchmarks

arXiv AI

Sep 16

Integrating the Analytic Hierarchy Process with Large Language Models for Transparent Multi-Criteria Decision-Making

arXiv:2609.16779v1 Announce Type: new Abstract: LLMs are increasingly employed in a wide range of decision-making tasks. However, the opacity of their internal reasoning makes it difficult to validat...

By Han Zhiguang (IRIT-MELODI, UT3, IPAL), Farah Benamara (IRIT-MELODI, UT3, IPAL), Pascale Zarat\'e (IRIT, UT Capitole, IRIT-ADRIA)

llms benchmarks safety

arXiv AI

Sep 16

Causal Discovery via Transformed Low-Rank Quantile Surfaces

arXiv:2609.16931v1 Announce Type: cross Abstract: We propose Low-Rank Quantile Surfaces (LRQS), a bivariate causal model in which, in the causal direction, an unknown monotone transformation of the c...

By Ryo Kamimura, Thong Pham

fine-tuning benchmarks

arXiv AI

Sep 16

MUMINS: Metadata-conditioned Uncertainty-aware Medical Image Next-state Synthesis

arXiv:2609.17169v1 Announce Type: cross Abstract: Forecasting anatomical changes such as tumor growth and neurodegeneration is a challenging generative vision task. Morphological evolution is subtle...

By Anna Oliveras, Roger Mar\'i, Rafael Redondo, Oriol Guardi\`a, Cynthia Ifeyinwa Ugwu, Ana Tost, Bhalaji Nagarajan, Carolina Migliorelli, Vicent Ribas, Petia Radeva

diffusion benchmarks

arXiv Computer Vision

Sep 16

GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing

arXiv:2603.12264v2 Announce Type: replace Abstract: Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natur...

By Mingxin Liu, Ziqian Fan, Zhaokai Wang, Leyao Gu, Zirun Zhu, Yiguo He, Yuchen Yang, Changyao Tian, Xiangyu Zhao, Ning Liao, Shaofeng Zhang, Qibing Ren, Zhihang Zhong, Xuanhe Zhou, Junchi Yan, Xue Yang

multimodal benchmarks

arXiv Machine Learning

Sep 16

GeoCrossBench: Cross-Band Generalization for Remote Sensing

arXiv:2511.02831v2 Announce Type: replace Abstract: The data for remote sensing is constantly acquired, and new data comes from a growing number and diversity of satellites, while the vast majority o...

By Hakob Tamazyan, Ani Vanyan, Alvard Barseghyan, Anna Khosrovyan, Evan Shelhamer, Hrant Khachatrian

benchmarks

arXiv AI

Sep 16

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

arXiv:2609.16251v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engi...

By Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Qishi Zhan, Dehan Kong, Guohao Li, Kaixin Li

agents benchmarks

arXiv AI

Sep 16

Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance

arXiv:2609.16213v1 Announce Type: new Abstract: Artificial intelligence is reshaping biological research across an increasingly connected digital-to-physical workflow. General-purpose large language...

By Candace S. Y. Chan, Aris Karatzikos, Ilias Georgakopoulos-Soares

llms agents benchmarks safety

arXiv AI

Sep 16

AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

arXiv:2609.16519v1 Announce Type: new Abstract: Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that...

By Bernie Boscoe, Srinath Saikrishnan, Vikram Seenivasan, Jack Stark, Andrew Lizarraga, Morgan Himes, Jonathan Soriano, PJ Allen, Tuan Do

llms rag benchmarks safety data-engineering

arXiv AI

Sep 16

CoAdapt: An LLM-based Framework for Adaptive Collaborative Perception in IIoT Robotic Swarms

arXiv:2609.16852v1 Announce Type: new Abstract: Industrial IoT environments increasingly deploy autonomous mobile robots for tasks such as material handling, product assembly, or infrastructure inspe...

By Houssam Hajj Hassan (L2S), Antonia Maria Masucci (L2S), Lynda Zitoune (L2S), Salah-Eddine Elayoubi (L2S)

llms agents robotics benchmarks

arXiv AI

Sep 16

ThinkFlow: Self-Evolving Probabilistic Latent Memory for Lifelong Conversational Agents

arXiv:2609.17010v1 Announce Type: new Abstract: Lifelong conversational agents rely on memory systems to maintain deep, context-aware interactions with users. However, existing explicit textual memor...

By Cai Ke, Xin Liu, Han Zhang, Jiangyue Yan, Zike Yuan, Ling Deng, Yue Yu, Hui Wang, Ruifeng Xu

agents diffusion benchmarks safety

arXiv AI

Sep 16

The Immutable Past: Formalizing State Mutability and Conflict Resolution in Mutable RAG

arXiv:2609.16073v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) serves as the primary memory architecture for long-horizon autonomous agents. However, treating shared memory as...

By Hamed HaddadPajouh, Amir AmiriTabat

llms rag agents benchmarks

arXiv AI

Sep 16

AssemblyGrid v1: A Benchmark for Multi-Robot Production with Temporary Coalitions, Local Information, and Geometric Constraints

arXiv:2609.16075v1 Announce Type: cross Abstract: Flexible robotic production requires joint decisions on process progression, material routing, resource assignment, temporary cooperation, and simult...

By Fouad Bahrpeyma, David Heik, Dirk Reichelt

agents robotics benchmarks

arXiv AI

Sep 16

Coaching Qwen3 Coder 30B to Think Like a CodeClash Arena Agent

arXiv:2609.16096v1 Announce Type: cross Abstract: Large language model coding agents have recently become useful for software tasks, but weaker or open-weight agents still struggle to reliably interp...

By Ivy Ning Zhang

llms agents fine-tuning benchmarks

arXiv AI

Sep 16

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current ben...

By Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali, Bigyan Bhatta, Pramish Paudel, Francois Rameau, Danda Pani Paudel

llms multimodal benchmarks

arXiv AI

Sep 16

Where Should a Document Live: Context, Representations, or Parameters?

arXiv:2609.17346v1 Announce Type: cross Abstract: To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the co...

By Nathana\"el Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adri\`a de Gispert

llms fine-tuning benchmarks

arXiv Machine Learning

Sep 16

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

arXiv:2609.17184v1 Announce Type: new Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurren...

By SangLyul Cho, Langqing Cui, Sehoon Kim, Dongsu Han, Insu Han

llms benchmarks

arXiv AI

Sep 16

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

arXiv:2609.17394v1 Announce Type: cross Abstract: Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading...

By Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo

agents benchmarks

arXiv AI

Sep 16

FactorEngine: A Program-level Knowledge-Infused Factor Mining Framework for Quantitative Investment

arXiv:2603.16365v3 Announce Type: replace Abstract: We study alpha factor mining, the automated discovery of predictive signals from noisy, non-stationary market data-under a practical requirement th...

By Qinhong Lin, Ruitao Feng, Yinglun Feng, Zhenxin Huang, Yukun Chen, Zhongliang Yang, Linna Zhou, Binjie Fei, Jiaqi Liu, Yu Li

llms rag agents benchmarks safety

Get the daily briefing