Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 10

Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies

arXiv:2607. 07957v1 Announce Type: new Abstract: Autism spectrum disorder (ASD) affects over 75 million individuals worldwide, yet scalable computational methods for remote behavioral screening remain limited.

By Raunak Mondal, Peter Washington

fine-tuning benchmarks

arXiv AI

Jul 10

SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

arXiv:2607. 08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.

By Adis Alihodzic, Selma Skopljakovic Hubljar

llms multimodal benchmarks

arXiv AI

Jul 10

Reinforcing the Generation Order of Multimodal Masked Diffusion Models

arXiv:2607. 08056v1 Announce Type: cross Abstract: Diffusion Language Models (DLMs) have recently achieved substantial progress in natural language generation tasks.

By Yidong Ouyang, Zhe Wang, Sourav Bhabesh, Dmitriy Bespalov

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 10

LEEVLA: Seeing What Matters in Latent Environment Evolution for Vision-Language-Action

arXiv:2607. 08182v1 Announce Type: cross Abstract: Vision-language-action (VLA) models aim to map multimodal inputs to robot actions.

By Qi Lyu, Baicheng Liu, Xudong Wang, Jiahua Dong, Lianqing Liu, Zhi Han

agents diffusion robotics multimodal benchmarks

arXiv AI

Jul 10

LEXIC: Lightweight Eye-tracking eXtension via Injected Complexity

arXiv:2607. 08152v1 Announce Type: cross Abstract: On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap: text-aware models using pretrained language models reach 56--63% AUROC, while gaze-only models operate at chance.

By Sumin Lee, Kyeonghun Kim, Subeen Lee, Jiwon Yang, Tien Nguyen, Ken Ying-Kai Liao, Nam-Joon Kim

llms benchmarks

arXiv Machine Learning

Jul 10

Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models

arXiv:2607. 08665v1 Announce Type: new Abstract: Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle.

By Teng-Ruei Chen

llms benchmarks

arXiv Machine Learning

Jul 10

Eigenvalue Calibration for Semantic Embeddings of Large Language Models

arXiv:2607. 08377v1 Announce Type: new Abstract: Uncertainty quantification is central to the reliable deployment of large language models (LLMs), and eigenvalues of semantic embeddings have recently emerged as a key tool in state-of-the-art methods.

By Sebastian G. Gruber, Nassim Walha, Francis Bach, Florian Buettner

llms rag benchmarks

arXiv AI

Jul 10

MobiDiff: Semantic-Aware Multi-Channel Discrete Diffusion for Human Mobility Data Generation

arXiv:2607. 08357v1 Announce Type: new Abstract: Human mobility data are essential for transportation optimization, urban planning, and resource allocation, yet real-world mobility data are costly to collect and difficult to share due to privacy concerns.

By Rongchao Xu, Lin Jiang, Dahai Yu, Ximiao Li, Taichi Liu, Desheng Zhang, Yuan Tian, Guang Wang

diffusion benchmarks safety

arXiv Machine Learning

Jul 10

Geometry-Aware Deep Congruence Networks for Manifold Learning in Cross-Subject Motor Imagery

arXiv:2511. 18940v3 Announce Type: replace Abstract: Cross-subject motor imagery decoding remains a fundamental challenge in EEG-based brain-computer interfaces due to substantial inter-subject variability.

By Sanjeev Manivannan, Chandra Shekar Lakshminarayan

benchmarks safety

arXiv Machine Learning

Jul 10

Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

arXiv:2607. 07748v1 Announce Type: new Abstract: Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia.

By Didula Samaraweera, Anjana Supun, Srinath Perera

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 10

Frequency-Domain Multi-Modality Transportation Modeling

arXiv:2607. 08475v1 Announce Type: new Abstract: Multi-modality transportation refers to urban systems composed of multiple transportation modes, such as traffic flow and public transit, whose dynamics are coupled by shared temporal patterns.

By Jiewen Deng, Hangchen Liu, Junchen Li, Boyuan Zhang, Renhe Jiang

benchmarks

arXiv AI

Jul 10

Peer-Predictive Self-Training for Language Model Reasoning

arXiv:2604. 13356v3 Announce Type: replace-cross Abstract: Mechanisms for continued self-improvement of language models without external supervision remain an open challenge.

By Shi Feng, Hanlin Zhang, Fan Nie, Sham Kakade, Yiling Chen

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 10

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

arXiv:2607. 08522v1 Announce Type: new Abstract: The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation.

By Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen

multimodal benchmarks

arXiv Machine Learning

Jul 10

DeepPySR -- A Symbolic Regression Framework with Dynamic Pruning, Pareto Selection, and Hierarchical Composition for Real-World Scientific Discovery

arXiv:2607. 08150v1 Announce Type: new Abstract: Symbolic regression (SR) discovers analytical equations from data, yielding glass-box models with directly interpretable formulas, unlike black-box methods that rely on unstable post-hoc tools such as SHAP or LIME.

By Fuling Chen, Kevin Vinsen, Phillip Melton, Rae-Chi Huang

efficiency benchmarks

arXiv AI

Jul 10

CoCo-Fed: A Unified Framework for Memory- and Communication-Efficient Federated Learning at the Wireless Edge

arXiv:2601. 00549v2 Announce Type: replace-cross Abstract: The deployment of large-scale neural networks within the Open Radio Access Network (O-RAN) architecture is pivotal for enabling native edge intelligence.

By Zhiheng Guo, Zhaoyang Liu, Zihan Cen, Chenyuan Feng, Xinghua Sun, Xiang Chen, Tony Q. S. Quek, Xijun Wang

efficiency benchmarks

arXiv AI

Jul 10

FedOPAL: One-Shot Federated Learning via Analytic Visual Prompt Tuning

arXiv:2607. 08368v1 Announce Type: new Abstract: With the widespread deployment of basic models in edge intelligence, communication bandwidth has become a core bottleneck restricting the scalability of federated learning.

By Lingyu Qiu, Daniela Annunziata, Stefano Izzo, Fabio Giampaolo, Francesco Piccialli

fine-tuning efficiency benchmarks

arXiv AI

Jul 10

Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

arXiv:2607. 08193v1 Announce Type: cross Abstract: Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills.

By Lorenzo Pant\`e, Andrea Fanti, Roberto Capobianco

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 10

Bridging Cognitive Neuroscience and Graph Intelligence: Hippocampus-Inspired Multi-View Hypergraph Learning for Web Finance Fraud

arXiv:2601. 11073v3 Announce Type: replace-cross Abstract: Online financial services constitute an essential component of contemporary web ecosystems, yet their openness introduces substantial exposure to fraud that harms vulnerable users and weakens trust in digital finance.

By Rongkun Cui, Nana Zhang, Kun Zhu, Qi Zhang

benchmarks safety

arXiv AI

Jul 10

Mechanistic Interpretability of LLM Jailbreaks via Internal Attribution Graphs

arXiv:2607. 07903v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit remarkable capabilities but remain highly vulnerable to adversarial prompts and jailbreak attacks.

By Anupam Wagle, Ifrat Ikhtear Uddin, Chaowei Zhang, Longwei Wang

llms benchmarks safety

arXiv AI

Jul 10

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

arXiv:2607. 07989v1 Announce Type: cross Abstract: Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures.

By Yufei Xia, Anjun Gao, Yueyang Quan, Zhuqing Liu, Minghong Fang

llms agents fine-tuning benchmarks

Get the daily briefing