Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv AI

Jul 24

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

arXiv:2602. 19313v2 Announce Type: replace-cross Abstract: General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior.

By Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

llms robotics multimodal benchmarks

arXiv AI

Jul 24

PromptPack: Scaling LLM Annotation Agents for Online Recommendation

arXiv:2607. 20528v1 Announce Type: new Abstract: Online recommendation platforms increasingly use Large Language Models (LLMs) to extract structured features from ad creatives.

By Sebastian Koralewski, Merwan Barlier, Yulia Stolin, Bla\v{z} \v{S}krlj

llms agents efficiency benchmarks

arXiv AI

Jul 24

Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions

arXiv:2607. 20891v1 Announce Type: new Abstract: Deep Research agents extend LLM-based assistants into long-horizon workflows involving planning, retrieval, evidence synthesis, and report generation, yet their reliability in open information environments remains underexplored.

By Pengyu Zhu, Lijun Li, Longju Yang, Sen Su

llms agents benchmarks

arXiv AI

Jul 24

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

arXiv:2607. 21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations.

By Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon, Elisa Ricci

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance

arXiv:2511. 12997v2 Announce Type: replace Abstract: Multimodal LLM-powered agents have recently demonstrated impressive capabilities in web navigation, enabling agents to complete complex browsing tasks across diverse domains.

By Genglin Liu, Shijie Geng, Sha Li, Hejie Cui, Sarah Zhang, Xin Liu, Tianyi Liu

llms agents multimodal benchmarks

arXiv AI

Jul 24

NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers

arXiv:2601. 20174v3 Announce Type: replace-cross Abstract: Solving large-scale sparse linear systems originating from partial differential equations (PDEs) is a fundamental topic in high-performance scientific computing, where preconditioners are crucial.

By Alexander Benanti, Xi Han, Hong Qin

benchmarks

arXiv AI

Jul 24

AG-REPA: Causal Layer Selection for Representation Alignment in Audio Flow Matching

arXiv:2603. 01006v3 Announce Type: replace-cross Abstract: REPresentation Alignment (REPA) improves the training of generative flow models by aligning intermediate hidden states with pretrained teacher features, but its effectiveness in token-conditioned audio Flow Matching critically depends on the choice of supervised layers, which is typically made heuristically based on the depth.

By Pengfei Zhang, Tianxin Xie, Minghao Yang, Li Liu

diffusion benchmarks safety

arXiv AI

Jul 24

A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

arXiv:2607. 21274v1 Announce Type: cross Abstract: We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments.

By Katerina Papantoniou, Panagiotis Papadakos, Theodore Patkos, Dimitris Garefalakis, Nikos Vardakis, Dimitris Plexousakis

llms rag nlp benchmarks

arXiv AI

Jul 24

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

arXiv:2607. 20482v1 Announce Type: new Abstract: Recent advances in large language models have enabled web agents to autonomously execute complex tasks.

By Seungbin Yang, Chaewoon Ki, Dohyun Lee, Jaegul Choo, ChaeHun Park

llms agents benchmarks

arXiv AI

Jul 24

Confidently Deceptive: How Confidence Amplifies the Risk of LLM Deception

arXiv:2607. 20444v1 Announce Type: cross Abstract: Large language models (LLMs) can produce deceptive responses: outputs that mislead users in service of a contextually or experimentally induced goal.

By Ali Asad, Stephen Obadinma, Anshul Pattoo, Wenxuan Zhang, Xiaodan Zhu

llms fine-tuning benchmarks safety

arXiv AI

Jul 24

Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Sampling

arXiv:2607. 20791v1 Announce Type: new Abstract: High-temperature sampling is one of the primary mechanisms for increasing diversity in LLMs.

By Phillip Howard, Xin Su, Allen Roush, Manikandan Ravikiran, Amir Abdullah

llms benchmarks safety

arXiv AI

Jul 24

AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning

arXiv:2607. 21106v1 Announce Type: new Abstract: Effective memory is crucial for LLM agents, yet constructing it effectively remains challenging.

By Qinfeng Li, Yuntai Bao, Xinyan Yu, Hongze Chen, Wenqi Zhang, Xuhong Zhang

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jul 24

MSBraM: A Multi-scale Self-supervised Brain Foundation Model for Hierarchical EEG Dynamics Learning

arXiv:2607. 21402v1 Announce Type: new Abstract: Self-supervised foundation models have recently shown strong potential for electroencephalogram (EEG)-based analysis.

By Tao Zhou, Jing Han, Lingyu Shu, Zixing Zhang

nlp efficiency benchmarks

arXiv AI

Jul 24

An LLM-Driven Workflow for Automated Process Control Strategy Generation and Tuning from Dynamic Process Models

arXiv:2607. 21292v1 Announce Type: new Abstract: We present a structured large-language-model-driven workflow for automated multi-variable control design from dynamic process models.

By Ari Luna Rueda, Eike Cramer, Klaus Hellgardt, Mehmet Mercang\"oz

llms benchmarks

arXiv Machine Learning

Jul 24

Belief Propagation in LLM World Models: Measuring Strategic Information Bias with Prediction Markets

arXiv:2607. 20441v1 Announce Type: cross Abstract: Every information ecosystem produces beliefs that shape strategic decisions.

By Mykola Khandoga, Yevhen Kostiuk, Anton Polishko, Yurii Filipchuk, Kostiantyn Kozlov, Dmytro Zamriy, Artur Kiulian

llms benchmarks safety

arXiv AI

Jul 24

CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

arXiv:2607. 21155v1 Announce Type: cross Abstract: Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions.

By Hanseok Oh, Parishad BehnamGhader, Benno Krojer, Hyunji Lee, Paul Liang, Siva Reddy, Verna Dankers

llms rag nlp multimodal benchmarks

arXiv AI

Jul 24

Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances

arXiv:2511. 03354v2 Announce Type: replace-cross Abstract: Generative artificial intelligence (GenAI) is transforming bioinformatics by advancing genomics, proteomics, transcriptomics, structural biology, and drug discovery.

By Wasimul Karim, Riasad Alvi, Sayeem Been Zaman, Arefin Ittesafun Abian, Mohaimenul Azam Khan Raiaan, Saddam Mukta, Md Rafi Ur Rashid, Md Rafiqul Islam, Yakub Sebastian, Sami Azam

benchmarks safety

arXiv Machine Learning

Jul 24

Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation

arXiv:2607. 20560v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) systems retrieve and integrate external knowledge to ground large language model (LLM) outputs.

By Muntaser Syed, Marius Silaghi, Sheikh Abujar, Sharun Akter

llms rag benchmarks safety

arXiv Machine Learning

Jul 24

DART: A Degradation-Aware Recurrent Transformer for Archival Film Restoration

arXiv:2607. 21219v1 Announce Type: cross Abstract: Archival film restoration is a challenging problem because historical footage contains compound degradations such as scratches, dust, blur, noise, flicker, and photometric aging, while clean reference videos are unavailable.

By Miko{\l}aj Jastrz\k{e}bski, Wojciech Koz{\l}owski, Kamil Adamczewski

llms benchmarks

arXiv Machine Learning

Jul 24

Nipping the Butterfly Effect in the Bud: Self-Output Fine-Tuning for Autoregressive Weather Prediction

arXiv:2607. 21080v1 Announce Type: new Abstract: Long-horizon weather forecasting is a fundamental challenge in atmospheric science, for which autoregressive Deep Learning Weather Prediction (DLWP) has emerged as the primary paradigm.

By Yun-Ye Cai, Hsuan-Tien Lin

fine-tuning benchmarks

Get the daily briefing