Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 14

Metacognition in LLMs: Foundations, Progress, and Opportunities

arXiv:2607. 11881v1 Announce Type: cross Abstract: Metacognition is a foundational component of intelligence critical to effective learning, problem solving, decision-making, communication, and more.

By Gabrielle Kaili-May Liu, Areeb Gani, Jacqueline Lu, Jordan Thomas, Mark Steyvers, Arman Cohan

llms nlp benchmarks

arXiv AI

Jul 14

Multi-Scale Convolution with Optimal Transport Attention Effect on Multivariate Time Series

arXiv:2607. 10740v1 Announce Type: cross Abstract: The analysis of Multivariate Time Series (MTS) plays an important role in a lot of real-world practical applications, but it still remains some challenging problem about capturing multi-granularity structural patterns and suppressing noise appropriately.

By HaoChong Fu, Jian Xu

rag benchmarks

arXiv AI

Jul 14

FlowPainter: Inpainting Optical Flow via Confidence-Guided Completion

arXiv:2607. 10140v1 Announce Type: cross Abstract: Existing optical flow methods broadly follow two paradigms: iterative optimization and diffusion-based estimation.

By Yuang Meng, Chenyang Wu, Xianshun Liu, Chun-Le Guo, Zichen Liang, Lina Lei, Jie Liang, Hui Zeng, Chongyi Li, Lei Zhang

diffusion computer-vision benchmarks

arXiv Machine Learning

Jul 14

A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

arXiv:2607. 11873v1 Announce Type: cross Abstract: Institutions collect far more open-ended teaching-evaluation feedback than they read.

By Esteban U. Vega Barajas

llms rag benchmarks

arXiv Machine Learning

Jul 14

A multi-scale feature enhanced graph neural network for fluid dynamics prediction in complex geometries

arXiv:2607. 11672v1 Announce Type: new Abstract: Industrial design in fields such as vehicle and aerospace engineering often relies on large-scale numerical simulations to evaluate fluid dynamics performance, which can incur substantial computational costs.

By Li Xiao, Tianyu Li, Yiye Zou, Mingjie Zhang, Xiaogangd Deng

benchmarks

arXiv AI

Jul 14

SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data

arXiv:2602. 14687v2 Announce Type: replace-cross Abstract: Improving Sparse Autoencoders (SAEs) requires benchmarks that can precisely validate architectural innovations.

By David Chanin, Adri\`a Garriga-Alonso

llms benchmarks

arXiv AI

Jul 14

Cost of Reasoning in non-English Languages: A Case Study on Japanese

arXiv:2607. 10114v1 Announce Type: cross Abstract: Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant.

By Yuu Jinnai

llms benchmarks safety

arXiv AI

Jul 14

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

arXiv:2607. 10169v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities.

By Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 14

Verification of Adaptive Agentic Controllers through Finite Rule Revision

arXiv:2607. 09770v1 Announce Type: new Abstract: Industrial agentic AI systems increasingly exhibit a gap between prototype capability and production deployment.

By Roberto Garrone

agents benchmarks safety

arXiv AI

Jul 14

Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora

arXiv:2607. 10020v1 Announce Type: cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.

By Lars Henry Berge Olsen, Pierre Lison, Martin Jullum, Mark Anderson

benchmarks

arXiv AI

Jul 14

LegalFarePlan: A Label-Setting Framework for Fare-Transparent Urban Rail Route Planning under Non-Additive Fare Rules

arXiv:2607. 09755v1 Announce Type: new Abstract: Urban rail fare systems may be non-additive: the fare of a single paid journey from an origin to a destination can differ from the sum of fares over multiple legally separated journey legs.

By Tanghui Li

benchmarks

arXiv AI

Jul 14

PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains

arXiv:2508. 21787v3 Announce Type: replace-cross Abstract: Best-of-n sampling improves the accuracy of large language models (LLMs) and large reasoning models (LRMs) by generating multiple candidate solutions and selecting the one with the highest reward.

By Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema, Sohee Yang, Wai-Chung Kwan, Xuanli He, Wenda Li, Pasquale Minervini, Eleonora Giunchiglia, Shay B. Cohen

llms benchmarks

arXiv AI

Jul 14

SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptation

arXiv:2511. 19558v2 Announce Type: replace-cross Abstract: Text-to-image diffusion models can emit copyrighted, unsafe, or private content.

By Mohammed Talha Alam, Nada Saadi, Fahad Shamshad, Nils Lukas, Karthik Nandakumar, Fahkri Karray, Samuele Poppi

diffusion fine-tuning benchmarks safety

arXiv AI

Jul 14

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

arXiv:2607. 11523v1 Announce Type: cross Abstract: When should an intelligent assistant speak up without being asked?

By Gong Sitong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang

rag agents benchmarks safety

arXiv AI

Jul 14

SFO: Learning PDE Operators via Spectral Filtering

arXiv:2601. 17090v2 Announce Type: replace-cross Abstract: Partial differential equations (PDEs) govern complex systems, yet neural operators often struggle to efficiently capture the long-range, nonlocal interactions inherent in their solution maps.

By Noam Koren, Rafael Moschopoulos, Kira Radinsky, Elad Hazan

diffusion benchmarks

arXiv AI

Jul 14

The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

arXiv:2607. 11149v1 Announce Type: new Abstract: LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on disk, including logs, context snapshots, checkpoints, and debug traces.

By Chenglin Yu, Hongquan Gui, Ying Yu, Hongxia Yang, Ming Li

llms agents benchmarks

arXiv AI

Jul 14

Learning in Blocks: A Multi Agent Debate Assisted Personalized Adaptive Learning Framework for Language Learning

arXiv:2604. 22770v2 Announce Type: replace-cross Abstract: Most digital language learning curricula rely on discrete-item quizzes that test recall rather than applied conversational proficiency.

By Nicy Scaria, Silvester John Joseph Kennedy, Deepak Subramani

llms agents benchmarks

arXiv AI

Jul 14

Fail-Aware and Explainable Test Oracle Prediction

arXiv:2607. 11342v1 Announce Type: cross Abstract: Despite their central role in fault detection, test oracles remain challenging to construct effectively.

By Yue Zhao, Binish Tanveer, Jelena Zdravkovic

llms benchmarks

arXiv AI

Jul 14

CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series

arXiv:2607. 09880v1 Announce Type: cross Abstract: Clinical time series are central to patient monitoring, risk assessment, and clinical decision support.

By Frank Nie, Ethan B. Liu, Yuan Zhu, Loe Yan, Wei Fan, Jindong Han

nlp multimodal benchmarks

arXiv AI

Jul 14

BatteryLake: Agentic, Physics-Grounded Curation of Heterogeneous Battery Aging Data and Benchmarking

arXiv:2607. 09762v1 Announce Type: new Abstract: Public battery aging datasets are a critical asset for advanced health management, but their practical use is often limited by inconsistent formats, unclear schemas, and metadata scattered across repositories and publications.

By Tianwen Zhu, Hao Wang, Yonggang Wen

llms agents benchmarks

Get the daily briefing