Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv Machine Learning

Jun 29

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

arXiv:2606. 27886v1 Announce Type: new Abstract: Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts.

By Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

multimodal benchmarks

arXiv AI

Jun 29

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

arXiv:2603. 18558v2 Announce Type: replace-cross Abstract: Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows.

By Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

llms rag computer-vision nlp multimodal benchmarks

arXiv AI

Jun 29

When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models

arXiv:2602. 10179v2 Announce Type: replace-cross Abstract: Recent advances in large image editing models have shifted the paradigm from text-driven instructions to vision-prompt editing, where user intent is inferred directly from visual inputs such as marks, arrows, and visual-text prompts.

By Jiacheng Hou, Yining Sun, Ruochong Jin, Haochen Han, Fangming Liu, Wai Kin Victor Chan, Alex Jinpeng Wang

llms multimodal benchmarks safety

arXiv AI

Jun 29

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

arXiv:2510. 03243v3 Announce Type: replace-cross Abstract: Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable.

By Yiheng Tao, Yihe Zhang, Matthew Dearing, Xin Wang, Yuping Fan, Michael E. Papka, Zhiling Lan

llms efficiency benchmarks

arXiv AI

Jun 29

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

arXiv:2511. 03217v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information.

By Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius, Lasse Strothe, Andrii Lata, Jana Diesner

llms agents fine-tuning benchmarks

arXiv AI

Jun 29

SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game

arXiv:2606. 27397v1 Announce Type: cross Abstract: Evaluating LLM agents requires dynamic environments that go beyond static reasoning and zero-sum games.

By Yeqi Feng, Yuxin Chen, Tianxing He

llms agents benchmarks

arXiv Machine Learning

Jun 29

TeRoR: Decoupled Temporal Rotation with Relational Circular Region for Temporal Knowledge Graph Embedding

arXiv:2606. 27651v1 Announce Type: new Abstract: In recent years, with the emergence of Temporal Knowledge Graphs (TKGs), research on learning entity and relation representations in TKGs has attracted increasing attention, giving rise to a large number of TKG embedding methods.

By Peijia Xie, Yike Liu, Chao He, Huiling Zhu

rag benchmarks

arXiv AI

Jun 29

Applicability of memorization indicators for early spotting of overfitting while recalibrating sEMG-decoders on low sample sizes

arXiv:2606. 27855v1 Announce Type: cross Abstract: Deep learning models for surface electromyography (sEMG) can benefit substantially from subject-specific (re-)calibration, since no sufficiently large and diverse datasets are available to train fully generic decoders.

By Stephan J. Lehmler, Tobias Glasmachers, Ioannis Iossifidis

fine-tuning benchmarks

arXiv AI

Jun 29

Beyond Sparse Supervision: Diffusion-Guided Learning for Few-Shot Graph Fraud Detection

arXiv:2606. 28134v1 Announce Type: cross Abstract: Graph-based fraud detection is essential for safeguarding large-scale transaction systems, where undetected anomalies may lead to substantial financial losses and security risks.

By Liming Liu, Chao Hu, Mingfei Lu, Yiwei Ge, Xingle Li, Heyuan Shi

diffusion benchmarks

arXiv AI

Jun 29

Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning

arXiv:2603. 00374v2 Announce Type: replace Abstract: Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories.

By Austin A. Nguyen, Michael P. Wellman

reinforcement-learning benchmarks

arXiv AI

Jun 29

Agentic Hardware Design as Repository-Level Code Evolution

arXiv:2606. 28279v1 Announce Type: cross Abstract: We present HORIZON, a self-evolving agent framework that treats hardware design as repository-level code evolution.

By Cunxi Yu, Chenhui Deng, Nathaniel Pinckney, Brucek Khailany

agents benchmarks

arXiv AI

Jun 29

Toward Robust In-Context Segmentation via Concept Guidance

arXiv:2606. 28149v1 Announce Type: cross Abstract: In-context segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters.

By Zhigang Chen, Xiawu Zheng, Rongrong Ji

computer-vision benchmarks

arXiv AI

Jun 29

Energy-Structured Low-Rank Adaptation for Continual Learning

arXiv:2605. 27482v2 Announce Type: replace-cross Abstract: While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks.

By Longhua Li, Lei Qi, Qi Tian, Xin Geng

diffusion fine-tuning benchmarks

arXiv AI

Jun 29

From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection

arXiv:2606. 27973v1 Announce Type: cross Abstract: Speech-based cognitive impairment detection offers a noninvasive, accessible alternative to costly biomarker assays, yet transformer-based models remain clinically uninterpretable.

By Yasaman Haghbin, Sina Rashidi, Ali Zolnour, Fatemeh Taherinezhad, Ali Fartoot, Hossein Azadmaleki, James M Noble, Maryam Dadkhah, Maryam Zolnoori

llms multimodal benchmarks safety

arXiv AI

Jun 29

Grounded Iterative Language Planning: How Parameterized World Models Reduce Hallucination Propagation in LLM Agents

arXiv:2606. 27806v1 Announce Type: new Abstract: World models for language agents come in two useful forms.

By Xinyuan Song, Zekun Cai

llms agents benchmarks safety

arXiv AI

Jun 29

Position: The Term "Machine Unlearning" Is Overused in LLMs

arXiv:2606. 27379v1 Announce Type: cross Abstract: Large language models increasingly face demands to "forget" training data, knowledge, or behaviors due to regulatory deletion obligations, copyright/licensing disputes, and safety or product-policy requirements.

By Sangyeon Yoon, Yeachan Jun, Albert No

llms benchmarks safety

arXiv Machine Learning

Jun 29

Hyperellipsoid Density Sampling: Exploitative Sequences to Accelerate High-Dimensional Optimization

arXiv:2511. 07836v4 Announce Type: replace-cross Abstract: The curse of dimensionality remains a persistent challenge in modern optimization problems.

By Julian Soltes

benchmarks

arXiv AI

Jun 29

MultiHashFormer: Hash-based Generative Language Models

arXiv:2606. 28057v1 Announce Type: cross Abstract: Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size.

By Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras

llms rag benchmarks

arXiv AI

Jun 29

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

arXiv:2606. 26530v2 Announce Type: replace-cross Abstract: The Abstraction and Reasoning Corpus (ARC) contains tasks that require summarizing patterns from limited grid samples and predicting output grids.

By Yuxuan Yang, Feiyang Li, Yile Wang

llms fine-tuning benchmarks safety

arXiv AI

Jun 29

Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning

arXiv:2511. 20196v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) can inadvertently memorize privacy-sensitive information during training.

By Zhen Zeng, Leijiang Gu, Zhangling Duan, Feng Li, Cees G. M. Snoek, Meng Wang, Zenglin Shi

llms fine-tuning multimodal benchmarks safety

Get the daily briefing