Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv AI

Jun 4

ParetoPilot: Zero-Surrogate Offline Multi-Objective Optimization via Infer-Perturb-Guide Diffusion

arXiv:2606. 04468v1 Announce Type: cross Abstract: Offline multi-objective optimization (Offline MOO) aims to discover novel Pareto-optimal designs based on static datasets without expensive environment interactions.

By Ruiqing Sun, Sen Yang, Dawei Feng, Bo Ding, Yijie Wang, Huaimin Wang

diffusion benchmarks safety

arXiv AI

Jun 4

From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents

arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.

By Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Mingkai Zhang, Yanming Zhu

llms agents benchmarks safety

arXiv AI

Jun 4

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

arXiv:2606. 04460v1 Announce Type: cross Abstract: AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities.

By Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, Francisco De La Riega, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha, Jianhong Tu, Gabriel Han, Chenguang Wang, Jingxuan He, Wenbo Guo, Dawn Song

agents benchmarks

arXiv Machine Learning

Jun 4

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

arXiv:2606. 04485v1 Announce Type: new Abstract: Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states.

By Yuanrui Wang, Xingxuan Zhang, Han Yu, Mingchao Ming, Gang Ren, Hao Yuan, Li Mao, Yunjia Zhang, Chun Yuan, Peng Cui

nlp benchmarks

arXiv AI

Jun 4

Position: State-of-the-Art Claims Require State-of-the-Art Evidence

arXiv:2605. 17273v3 Announce Type: replace-cross Abstract: State-of-the-Art (SOTA) claims pervade Artificial Intelligence (AI) and Machine Learning (ML) research.

By YongKyung Oh

benchmarks

arXiv AI

Jun 4

Can Reasoning Path still be Effective as Input? Bridging Post-Reasoning to Chain-of-Thought Compression

arXiv:2510. 08647v2 Announce Type: replace-cross Abstract: Recent developments have enabled advanced reasoning in Large Language Models (LLMs) via long Chain-of-Thought (CoT), trading efficiency during inference for performance.

By Chengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Shengchao Liu, Guoxin Ma, Yu Lan, Cong Wang, Chao Shen

llms benchmarks

arXiv AI

Jun 4

SciDER: Scientific Data-centric End-to-end Researcher

arXiv:2603. 01421v3 Announce Type: replace Abstract: While large language models accelerate scientific discovery, existing agents face severe limitations in adaptability, domain generalization, and multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data.

By Ke Lin, Owais Aijaz, Yilin Lu, Yiyang Luo, Xuehang Guo, Preslav Nakov

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento

llms benchmarks

arXiv AI

Jun 4

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

arXiv:2601. 09719v3 Announce Type: replace-cross Abstract: Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning.

By Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, Kyungwoo Song

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 4

Training-Free Lexical-Dense Fusion for Conversational-Memory Retrieval

arXiv:2606. 04194v1 Announce Type: new Abstract: Retrieving the few past turns that answer a new query across long multi-session histories is the retrieval bottleneck behind long-term conversational memory (LoCoMo, LongMemEval).

By Christian Lysenst{\o}en

rag benchmarks safety

arXiv Machine Learning

Jun 4

Bayesian learning for the stochastic shortest path problem

arXiv:2606. 04845v1 Announce Type: cross Abstract: Sequential decision-making problems are often modelled as a Markov decision process (MDP).

By Chon Wai Ho, Sumeetpal S. Singh, Jiaqi Guo

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers

arXiv:2606. 04366v1 Announce Type: new Abstract: Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features.

By Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang, Congcong Zhu, Jingrun Chen

llms nlp benchmarks safety

arXiv AI

Jun 4

Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation

arXiv:2606. 04046v1 Announce Type: cross Abstract: In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control.

By Boyuan Xiao, Bohong Chen, Yumeng Li, Ji Feng, Yao-Xiang Ding, Kun Zhou

robotics multimodal benchmarks safety

arXiv AI

Jun 4

Adaptive Patching Is Harder Than It Looks For Time-Series Forecasting

arXiv:2606. 04074v1 Announce Type: cross Abstract: Adaptive patching is a recent and compelling proposal for time-series Transformers: allocate finer patches where the sequence looks locally informative.

By Federico Zucchi, Yi Xie, Chao Zhang, Keyuan Luo, Thomas Lampert, Ziyue Li

llms benchmarks safety

arXiv AI

Jun 4

Need to Know: Contextual-Integrity-Grounded Query Rewriting for Privacy-Conscious LLM Delegation

arXiv:2606. 04067v1 Announce Type: cross Abstract: As LLMs become increasingly woven into everyday workflows, user queries sent to cloud hosted LLMs routinely mix task-essential content with task non-essential sensitive disclosures, yet type based PII redaction is context agnostic and may raise two issues: over disclosing untyped sensitive context and over removing answer bearing spans.

By Xinyue Huang, Xiaochun Cao, Wenyuan Yang

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 4

RUBAS: Rubric-Based Reinforcement Learning for Agent Safety

arXiv:2606. 04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation.

By Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 4

Aligning Deep Implicit Preferences by Learning to Reason Defensively

arXiv:2510. 11194v3 Announce Type: replace Abstract: Personalized alignment is crucial for enabling Large Language Models (LLMs) to engage effectively in user-centric interactions.

By Peiming Li, Zhiyuan Hu, Yang Tang, Shiyu Li, Xi Chen

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 4

SymTRELLIS: Symmetry-Enforced Voxel Latents for 3D Generation

arXiv:2606. 04108v1 Announce Type: cross Abstract: Single-view 3D generative models have achieved impressive visual quality, yet they are not designed to satisfy structural or functional requirements, and in practice, often fall short.

By Guangda Ji, Qimin Chen, Qinchan Li, Mingrui Zhao, Kai Wang, Hao Zhang

robotics benchmarks

arXiv AI

Jun 4

Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?

arXiv:2506. 10912v4 Announce Type: replace Abstract: Toxicity remains a leading cause of early-stage drug development failure.

By Fei Lin, Ziyang Gong, Cong Wang, Tengchao Zhang, Yonglin Tian, Yining Jiang, Ji Dai, Chao Guo, Xiaotong Yu, Xue Yang, Gen Luo, Fei-Yue Wang

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 4

Adaptive Minds: Empowering Agents with LoRA-as-Tools

arXiv:2510. 15416v2 Announce Type: replace Abstract: We investigate a framework in which LoRA adapters are treated as callable tools that a base language model can dynamically select and invoke.

By Pavan C Shekar, Aswanth Krishnan

llms agents fine-tuning benchmarks

Get the daily briefing