Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 30

Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

arXiv:2606. 30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer?

By Wenlong Wang, Fergal Reid

llms benchmarks

arXiv AI

Jun 30

SAGA: Scene-Aware, Goal-Evolving Agents for Long-Horizon CivRealm Strategy Planning

arXiv:2606. 29932v1 Announce Type: new Abstract: Long-horizon strategic planning in complex strategy games demands concurrent reasoning across multiple decision domains under imperfect information and sparse reward.

By Tianyu Jin, Shuo Chen, Yida Wang, Liuyu Xiang, Yingzhuo Liu, Zhiyao Jiang, Yexin Li, Zhaofeng He

llms agents benchmarks

arXiv AI

Jun 30

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

arXiv:2606. 29464v1 Announce Type: cross Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets.

By Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon

llms rag efficiency multimodal benchmarks safety

arXiv AI

Jun 30

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

arXiv:2606. 29984v1 Announce Type: new Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs).

By Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 30

Reported Confidence in LLMs Tracks Commitment More Than Correctness

arXiv:2606. 29490v1 Announce Type: cross Abstract: Confidence is an estimate of the probability that a chosen answer is correct.

By Dharshan Kumaran

llms benchmarks

arXiv AI

Jun 30

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

arXiv:2606. 29887v1 Announce Type: new Abstract: In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies, rather than relying on predefined risk taxonomies.

By Jiacheng Zhang, Haoyu He, Sen Zhang, Shen Wang, Xiaolei Xu, Yuhao Sun, Meng Shen, Feng Liu

llms benchmarks safety

arXiv AI

Jun 30

One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models

arXiv:2606. 29600v1 Announce Type: cross Abstract: A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple visible and geometrically valid surfaces.

By Xiaohao Xu, Feng Xue, Xiang Li, Haowei Li, Shusheng Yang, Tianyi Zhang, Matthew Johnson-Roberson, Xiaonan Huang

llms computer-vision benchmarks

arXiv AI

Jun 30

CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

arXiv:2606. 29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading.

By Bo Qu, Mingguang Chen

llms agents benchmarks safety

arXiv AI

Jun 30

The CRISTAL Method: Neurosymbolic analysis from AI-synthesized world models

arXiv:2606. 29799v1 Announce Type: new Abstract: This project introduces the CRISTAL Method (Coherent Reliable Intentional Synthesis of Truthful Analysis Logic), a neurosymbolic framework for automating complex analysis workflows, with fundamental investment analysis as a primary use case.

By Rafael Kaufmann, Felix Neub\"urger, Michael Walters, Thomas Kopinski, Dimitrije Markovi\'c

llms agents benchmarks

arXiv AI

Jun 30

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

arXiv:2606. 29700v1 Announce Type: new Abstract: Planning often requires symbolic specifications that are both executable and verifiable.

By Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 30

ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control

arXiv:2606. 30362v1 Announce Type: cross Abstract: While current Behavior Foundation Models (BFMs) provide robust control priors for humanoids, they only execute pre-defined reference motions.

By Xiao Chen, Weishuai Zeng, Xiaojie Niu, Zirui Wang, Jianan Li, Huayi Wang, Furui Xu, Jiahe Chen, Weixiang Zhong, Lihe Ding, Kailin Li, Jiangmiao Pang, Tai Wang, Tianfan Xue, Jingbo Wang

rag benchmarks safety

arXiv AI

Jun 30

Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees

arXiv:2606. 29681v1 Announce Type: new Abstract: Probabilistic model checking for Markov decision processes (MDPs) provides quantitative guarantees, but often offers limited insight into why undesired outcomes occur.

By Ryohei Oura, Georgios Fainekos, Hideki Okamoto, Bardh Hoxha

reinforcement-learning benchmarks

arXiv AI

Jun 30

Beyond IID: How General Are Tabular Foundation Models, Really?

arXiv:2606. 30410v1 Announce Type: cross Abstract: Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry.

By Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzm\"uller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Ga\"el Varoquaux, Frank Hutter

benchmarks

arXiv AI

Jun 30

Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval

arXiv:2606. 30473v1 Announce Type: cross Abstract: We study retrieval over catalogs of structured metadata, where each record is a small schema whose fields answer different kinds of query.

By Aivin V. Solatorio, Olivier Dupriez, Rafael Macalaba

llms rag agents fine-tuning benchmarks

arXiv AI

Jun 30

Collective cooperation without individual fidelity in LLM agents

arXiv:2606. 30454v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making.

By Henrique Ferraz de Arruda, Carlos Gracia L\'azaro, Alberto Aleta, Yamir Moreno

llms agents benchmarks

arXiv AI

Jun 30

MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems

arXiv:2606. 30602v1 Announce Type: cross Abstract: Multi-agent systems (MAS) are increasingly used to automate complex, distributed workflows.

By Kunyang Li, Kyle Domico, Jonathan Gregory, Patrick McDaniel

llms agents benchmarks

arXiv AI

Jun 30

Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localization

arXiv:2606. 30576v1 Announce Type: cross Abstract: Cross-view object geo-localization (CVOGL) aims to locate a target object from a query view (e.

By Liyao Wang, Ruipu Wu, Haojun Xu, Lei Shi, Linjiang Huang, Si Liu

computer-vision robotics multimodal benchmarks

arXiv AI

Jun 30

MARS: A neurosymbolic approach for interpretable drug discovery

arXiv:2410. 05289v4 Announce Type: replace Abstract: Background: Neurosymbolic (NeSy) artificial intelligence describes the combination of logic or rule-based techniques with neural networks.

By Lauren Nicole DeLong, Yojana Gadiya, Paola Galdi, Jacques D. Fleuriot, Daniel Domingo-Fern\'andez

benchmarks safety

arXiv AI

Jun 30

Instance-Conditioned Adaptation for Large-scale Generalization of Neural Routing Solver

arXiv:2405. 01906v3 Announce Type: replace Abstract: In modern intelligent transportation systems (ITS), particularly in freight transportation and logistics, real-time route planning is crucial.

By Changliang Zhou, Xi Lin, Zhenkun Wang, Xialiang Tong, Mingxuan Yuan, Qingfu Zhang

efficiency benchmarks

arXiv AI

Jun 30

Data and Evaluation Closed-Loop for Model Capability Enhancement

arXiv:2606. 28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it only retrospectively, compressing samples, prompts, decoding, and scoring rules into one noisy score.

By Zhixuan Li, Jiangan Yuan, Han Xu

llms benchmarks

Get the daily briefing