Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 13

CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding

arXiv:2607. 09543v1 Announce Type: new Abstract: Self-supervised pretrained foundation models (FM) have shown early promise for non-invasive electroencephalogram (EEG) decoding applications.

By Gabriel Mahuas, Victoria Shevchenko, Ugo Tanielian, Yassir Bendou, Richard Gao

llms benchmarks

arXiv AI

Jul 13

SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliation

arXiv:2604. 22294v2 Announce Type: replace-cross Abstract: Systematic reviews -- which requires comprehensive evidence collection and synthesis from large document corpora in response to targeted research questions -- are foundational in finance, social sciences, and other technical fields.

By Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

llms rag agents benchmarks

arXiv AI

Jul 13

4DR360: State Reasoning for Joint 3D Detection and Occupancy Prediction in 4D Radar-Camera Full-Scene Perception

arXiv:2607. 09629v1 Announce Type: cross Abstract: Reliable autonomous driving requires full-scene perception that couples foreground objects with dense semantic layout.

By Xiaokai Bai, Lianqing Zheng, Runwei Guan, Songkai Wang, Siyuan Cao, Hui-liang Shen

agents robotics multimodal benchmarks

arXiv AI

Jul 13

PRecG: Legal Precedent Retrieval with Graph Neural Networks and Rhetorical Role Segmentation

arXiv:2607. 09094v1 Announce Type: cross Abstract: Legal precedent retrieval is a fundamental task in legal case preparation, planning, litigation strategy, and legal research.

By Devanshu Verma, Vasudha Bhatnagar, Vikas Kumar, Balaji Ganesan

rag computer-vision benchmarks

arXiv AI

Jul 13

Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization

arXiv:2602. 07764v2 Announce Type: replace-cross Abstract: Multi-objective reinforcement learning (MORL) seeks to train agents capable of balancing conflicting objectives.

By Tanmay Ambadkar, Sourav Panda, Shreyash Kale, Jonathan Dodge, Abhinav Verma

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 13

Forking-Sequences: Statistically and Computationally Efficient Multi-Horizon Forecasting with Reduced Volatility

arXiv:2510. 04487v5 Announce Type: replace Abstract: While accuracy is a critical requirement for time series forecasting, an equally important desideratum is reasonable forecast volatility across forecast creation dates (FCDs).

By Willa Potosnak, Malcolm Wolff, Mengfei Cao, Ruijun Ma, Tatiana Konstantinova, Dmitry Efimov, Michael W. Mahoney, Boris Oreshkin, Kin G. Olivares

llms benchmarks

arXiv AI

Jul 13

H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classification

arXiv:2511. 10260v2 Announce Type: replace-cross Abstract: Fine-Grained Visual Classification (FGVC) remains a challenging task due to subtle inter-class differences and large intra-class variations.

By Yongji Zhang, Siqi Li, Kuiyang Huang, Yue Gao, Yu Jiang

rag benchmarks

arXiv AI

Jul 13

Conceptual Networks for Cross-Linguistic Idiomatic Expressions:A Feature-Based Graph Approach

arXiv:2607. 09576v1 Announce Type: cross Abstract: We present an interpretable network-based framework for representing idiomatic and figurative meaning across eight typologically diverse languages, totaling 160 conventional expressions, the large majority of which are idiomatic.

By Kiran Pala, Punam Silu, Lixun Yu

llms rag benchmarks

arXiv AI

Jul 13

Parameter-Efficient Vision-Language Adaptation with Continuous Metadata Conditioning for Animal Re-Identification

arXiv:2607. 09443v1 Announce Type: cross Abstract: Long-term animal re-identification (ReID) must remain robust to gradual morphological evolution and seasonal appearance shifts.

By Anil Osman Tur, Tonje Knutsen Sordalen, Kim Tallaksen Halvorsen, Cigdem Beyan

llms rag multimodal benchmarks safety

arXiv AI

Jul 13

Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioning

arXiv:2504. 18902v3 Announce Type: replace-cross Abstract: In the forthcoming era of 6G networks, characterized by unprecedented data rates, ultra-low latency, and ubiquitous connectivity, effective management of Virtualized Network Functions (VNFs) is essential.

By Cyril Shih-Huan Hsu, Anestis Dalgkitsis, Paola Grosso, Chrysa Papagianni

llms reinforcement-learning benchmarks

arXiv AI

Jul 13

OpenProver: Agentic and Interactive Theorem Proving with Lean 4

arXiv:2607. 09217v1 Announce Type: new Abstract: In this system paper, we present OpenProver, an open-source system for LLM-driven automated theorem proving (ATP) with integrated Lean 4 formal verification.

By Mat\v{e}j Kripner, Milan Straka

llms agents benchmarks

arXiv AI

Jul 13

HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs

arXiv:2604. 20140v2 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks.

By Darsh Kachroo, Arjun Prasaath Anbazhagan, Adriana Caraeni, Brennan Lagasse, Kevin Zhu

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 13

Interference and Retention in Continual Learning

arXiv:2607. 09202v1 Announce Type: cross Abstract: Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation.

By Julius St\"ork

efficiency benchmarks

arXiv AI

Jul 13

The LLMbda Calculus: AI Agents, Conversations, and Information Flow

arXiv:2602. 20064v2 Announce Type: replace-cross Abstract: Large language models are increasingly deployed as agents: they plan, call tools, read untrusted data, and act on the results.

By Zac Garby, Andrew D. Gordon, David Sands

llms agents benchmarks

arXiv Machine Learning

Jul 13

TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems

arXiv:2607. 09528v1 Announce Type: new Abstract: The emergence of metaverse platforms has created virtual economies that introduce new challenges related to fraud, bot activity, and illicit financial behavior.

By Refat Ishrak Hemel, Ehsan Hallaji, Roozbeh Razavi-Far

multimodal benchmarks

arXiv Machine Learning

Jul 13

Federated Low-Rank Koopman Learning for Multivariate Time-Series Anomaly Detection in IoT Systems

arXiv:2607. 08978v1 Announce Type: new Abstract: Distributed IoT systems generate multivariate time-series streams for monitoring physical assets, servers, and embedded sensing platforms.

By Tung-Anh Nguyen, Van-Phuc Bui, Anh Tuyen Le, Kim Hue Ta, Minh Thuy Le, J. Andrew Zhang, Xiaojing Huang

efficiency benchmarks

arXiv Machine Learning

Jul 13

Accelerating Large Language Model Inference with Self-Supervised Early Exits

arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.

By Florian Valade

llms benchmarks

arXiv AI

Jul 13

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

arXiv:2605. 28215v2 Announce Type: replace Abstract: In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples.

By Carmen Quiles-Ram\'irez, Leticia L. Rodr\'iguez, Nicol\'as Martorell, Natalia D\'iaz-Rodr\'iguez

llms multimodal benchmarks

arXiv Machine Learning

Jul 13

Active rejection enables reliable generalization of universal machine-learning interatomic potentials

arXiv:2607. 09456v1 Announce Type: new Abstract: Universal machine learning interatomic potentials (uMLIPs) bridge quantum-mechanical accuracy and large-scale molecular dynamics, but the cost of high-accuracy calculations such as r$^2$SCAN limits training to datasets that remain small relative to the open materials space.

By Mingxiang Luo, Xinnan Mao, Lu Wang, Lei Bai, Feng Ding, Yuqiang Li

benchmarks

arXiv Machine Learning

Jul 13

SYNRARE: Synthetic Rare Disease EHR Generation for ML Benchmarking

arXiv:2607. 09404v1 Announce Type: new Abstract: Motivation: Rare disease (RD) diagnosis is frequently delayed due to the similarities in symptoms to common disease variants.

By Nicolai Dinh Khang Truong, Richard R\"ottger

benchmarks safety

Get the daily briefing