Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

13,478 stories · RSS feed

arXiv AI

Aug 13

Deep Fictitious Play-Based Potential Differential Games for Learning Human-Like Interaction at Unsignalized Intersections

arXiv:2506. 12283v2 Announce Type: replace Abstract: Modeling vehicle interactions at unsignalized intersections is a challenging task due to the complexity of the underlying game-theoretic processes.

By Kehua Chen, Ryan Feng Lin, Shucheng Zhang, Yinhai Wang

benchmarks

arXiv AI

Aug 13

ExRole: From Team Trajectories to Executable Roles in Multi-Agent Language Models

arXiv:2608. 11949v1 Announce Type: new Abstract: Roles provide an interpretable interface for organizing language-model agents, yet most multi-agent systems treat them as hand-written prompt labels disconnected from learned behavior and parameter updates.

By Zhou Liu, Chaoyang Han, Zewei Pan, Zeli Su, Wentao Zhang

llms agents fine-tuning benchmarks

arXiv AI

Aug 13

Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversation

arXiv:2604. 03924v2 Announce Type: replace-cross Abstract: Goal-oriented conversational systems require making sequential decisions under uncertainty about the user's intent, where the algorithm must balance information acquisition and target commitment over multiple turns.

By Xinyi Ling, Ye Liu, Reza Averly, Xia Ning

llms benchmarks

arXiv AI

Aug 13

Reducing Per-Sample Interference in Stochastic Optimization

arXiv:2607. 16261v2 Announce Type: replace-cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv AI

Aug 13

Robust and Efficient Noisy-Label Time-Series Classification via Dynamic Time Warping Based Granular Ball Computing

arXiv:2608. 11704v1 Announce Type: cross Abstract: Dynamic Time Warping (DTW)-based Nearest-Neighbor (NN) classifiers are effective for time-series classification but are vulnerable to mislabeled training samples and require numerous DTW computations during inference.

By Ziqiang Li, Yun Liu, Gouhei Tanaka

benchmarks

arXiv AI

Aug 13

Domain-Aware Lightweight Spectral-Grouped Convolutions for Hyperspectral Fish Freshness Classification

arXiv:2608. 12227v1 Announce Type: cross Abstract: Hyperspectral imaging (HSI) offers nondestructive assessment of fish freshness by detecting biochemical alterations across spectral bands.

By Kazi Nabiul Alam, Pooneh Bagheri Zadeh, Akbar Sheikh-Akbari

llms benchmarks

arXiv Machine Learning

Aug 13

Long-Horizon Forecasting of Complete Financial Statements with Forma

arXiv:2608. 11327v1 Announce Type: new Abstract: Specialist training beats generalist scale when forecasting financial statements.

By Travis L. Johnson, Jiannan Jiang, Soumyabrata Chaudhuri, Yihao Chen, Lauren Falvey, Donal O'Cofaigh

llms benchmarks

arXiv AI

Aug 13

Uncertainty-Aware Probabilistic Constrained Clustering from Entangled Pairwise Supervision

arXiv:2608. 12027v1 Announce Type: cross Abstract: Pairwise constrained clustering typically relies on hard must-link/cannot-link labels, whereas realistic pairwise supervision may be real-valued and entangle intrinsic ambiguity, expert judgment, and stochastic corruption.

By Shaojie Zhang, Ke Chen

benchmarks

arXiv AI

Aug 13

When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use

arXiv:2608. 11715v1 Announce Type: cross Abstract: The reliability of Large Language Models (LLMs) for API calling degrades in multilingual settings.

By Siddharth Chauhan, Thomas Butler, Abhishek Singhania, Pankaj Porwal, Honey Gupta

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 13

Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

arXiv:2608. 11907v1 Announce Type: cross Abstract: As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge.

By Hao Zhang, Jiaxin Qi, Zhijiang Tang, Jianqiang Huang

llms multimodal benchmarks

arXiv Machine Learning

Aug 13

CosMAP: Contrastive Manifold Approximation and Projection for Dimensionality Reduction of Omics and Genealogical Data

arXiv:2608. 11269v1 Announce Type: cross Abstract: Omics datasets, particularly single-cell RNA sequencing data, are high-dimensional, sparse, noisy, and dominated by zero values, making faithful low-dimensional representation challenging.

By Fenosoa Randrianjatovo, Maya Saleh, Simon Girard, Amadou Barry

rag benchmarks

arXiv AI

Aug 13

Evaluating LLM Generated Detection Rules in Cybersecurity

arXiv:2509. 16749v1 Announce Type: cross Abstract: LLMs are increasingly pervasive in the security environment, with limited measures of their effectiveness, which limits trust and usefulness to security practitioners.

By Anna Bertiger, Bobby Filar, Aryan Luthra, Stefano Meschiari, Aiden Mitchell, Sam Scholten, Vivek Sharath

llms benchmarks

arXiv AI

Aug 13

How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

arXiv:2608. 12192v1 Announce Type: new Abstract: Foundation models for protein structure prediction remain unreliable on certain targets.

By Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics, Noam Ghenassia, Shikha Surana, Henry Moss, Paul Duckworth

reinforcement-learning benchmarks

arXiv AI

Aug 13

Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

arXiv:2608. 11727v1 Announce Type: new Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway.

By Zining Huang, Haoran Que, Hong Zeng, Ge Zhang, Zuo Wang, Jin Chen, Haodong Wang, Zhongfei Hou, Changxin Pu, Shen Yan, Wenhao Huang

agents benchmarks

arXiv AI

Aug 13

LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

arXiv:2608. 11847v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning.

By Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim

llms multimodal benchmarks safety

arXiv AI

Aug 13

AgenticTwin: An Agentic LLM Framework Integrated with Digital Twin for Anomaly Detection

arXiv:2608. 11679v1 Announce Type: new Abstract: Digital twins are increasingly used to monitor and simulate the behavior of cyber-physical systems.

By Touseef Hasan, Mounika Ghanta, Souvika Sarkar, Ujjwal Guin

llms agents benchmarks

arXiv Machine Learning

Aug 13

Dueling Deep Q-Learning for Intrusion Detection

arXiv:2608. 11291v1 Announce Type: cross Abstract: Intrusion detection systems (IDS) and automated systems for detecting and reporting cyber threats, are commonly handled via supervised machine learning methods.

By Logan Luna (Georgia Institute of Technology), Matthew P. Berkowitz (Embry-Riddle Aeronautical University), Laxima Niure Kandel (Embry-Riddle Aeronautical University), Sirio Jansen-S'anchez (Embry-Riddle Aeronautical University)

reinforcement-learning benchmarks safety

arXiv AI

Aug 13

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

arXiv:2608. 11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge.

By Karl Hanna, Chen Feng

llms benchmarks safety

arXiv AI

Aug 13

MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

arXiv:2608. 11749v1 Announce Type: cross Abstract: Multi-objective optimization (MOO) has demonstrated significant success in multi-task learning by mitigating task conflicts through gradient manipulation.

By Shiji Zhou, Kunlin Lyu, Lei Zhang, Ruodong Wang, Yifan Sun

llms robotics benchmarks

arXiv Machine Learning

Aug 13

Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks

arXiv:2608. 11815v1 Announce Type: new Abstract: Transfer-based adversarial attacks craft adversarial examples using surrogate models to mislead black-box victim models.

By Yaohua Liu, Yifan Guo, Jiaxin Gao

computer-vision benchmarks safety

Get the daily briefing