Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv Machine Learning

Jun 3

Neutrino Fingerprints: Image-Based Encodings of IceCube Events for CNN Direction Reconstruction

arXiv:2606. 02788v1 Announce Type: cross Abstract: Reconstructing the direction of incoming neutrinos in the IceCube Neutrino Observatory is an important problem in astrophysics.

By Floriano Tori, Brecht Verbeken, Vincent Ginis

benchmarks

arXiv Machine Learning

Jun 3

CoughSense: Five-Class Respiratory Disease Classification via Whisper Encoder Fine-Tuning and Dual-Encoder Cross-Attention Fusion with Balanced Contrastive Learning

arXiv:2606. 02998v1 Announce Type: new Abstract: Automated cough analysis offers a path to low-cost respiratory screening, but most existing work stops at binary COVID-19 detection.

By Nikhil Vincent

fine-tuning multimodal benchmarks

arXiv AI

Jun 3

Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs

arXiv:2606. 02581v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) faces a fundamental three-way tension: deeper retrieval improves factual grounding but inflates token costs and end-to-end latency.

By Sanjay Mishra

llms rag benchmarks safety

arXiv Machine Learning

Jun 3

KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators

arXiv:2606. 02963v1 Announce Type: new Abstract: Production inference increasingly targets a heterogeneous mix of accelerators.

By Taras Sereda, Burak Bartan, Ankita Nayak, Tom St. John, Natalie Serrino, Zain Asgar

llms agents efficiency benchmarks

arXiv AI

Jun 3

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601. 08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment.

By Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi

llms agents multimodal benchmarks safety

arXiv AI

Jun 3

Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation

arXiv:2606. 03128v1 Announce Type: cross Abstract: Smart contracts face critical security challenges that require thorough auditing in decentralized web services.

By Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra

llms efficiency benchmarks safety

arXiv AI

Jun 3

AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following

arXiv:2606. 03116v1 Announce Type: cross Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation.

By Haitao Li, Tian Tan, Yuguang Yang, Shan Yang, Xie Chen

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 3

Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

arXiv:2510. 23216v4 Announce Type: replace Abstract: While several high profile video games have served as testbeds for Deep Reinforcement Learning (DRL), this technique has rarely been employed by the game industry for crafting authentic AI behaviors.

By Alessandro Sestini, Joakim Bergdahl, Jean-Philippe Barrette-LaPierre, Florian Fuchs, Brady Chen, Fabio Zinno, Michael Jones, Linus Gissl\'en

agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 3

RGMem: Renormalization Group-inspired Memory Evolution for Language Agents

arXiv:2510. 16392v3 Announce Type: replace Abstract: Personalized and continuous interactions are critical for LLM-based conversational agents, yet finite context windows and static parametric memory hinder the modeling of long-term, cross-session user states.

By Ao Tian, Yunfeng Lu, Xinxin Fan, Changhao Wang, Lanzhi Zhou, Yeyao Zhang, Yanfang Liu

llms rag agents nlp benchmarks

arXiv AI

Jun 3

Adaptive Latent Agentic Reasoning

arXiv:2606. 02871v1 Announce Type: cross Abstract: Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents.

By Dongwon Jung, Peng Shi, Yi Zhang, Junshan Zhang, Muhao Chen

llms agents benchmarks

arXiv AI

Jun 3

PHASE: Physiology-Aware Hyperspectral Reconstruction via Object-to-Human Domain Adaptation

arXiv:2511. 13020v2 Announce Type: replace-cross Abstract: Although hyperspectral imaging offers unparalleled non-invasive physiological insight, its bulky hardware, slow acquisition, and regulatory burden severely limit its clinical availability.

By Yufei Wen, Shuxing Zhong, Jingdan Kang, Yuting Zhang, Jintai Chen, Kaishun Wu

fine-tuning benchmarks safety

arXiv AI

Jun 3

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

arXiv:2512. 18552v3 Announce Type: replace-cross Abstract: While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.

By Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, Sida Wang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 3

Outsmarting the Chameleon: Counterfactual Decoupling for Tactical OOD Shifts in Live Streaming Risk Assessment

arXiv:2606. 02946v1 Announce Type: new Abstract: Live streaming has emerged as a primary medium for social interaction and digital commerce, yet it is increasingly plagued by sophisticated risks.

By Yiran Qiao, Jing Chen, Jiaqi Xu, Yang Liu, Qiwei Zhong, Xiang Ao

benchmarks safety

arXiv AI

Jun 3

Reasoning Structure of Large Language Models

arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.

By Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Fabian Farestam, Roger Wattenhofer

llms benchmarks

arXiv AI

Jun 3

Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models

arXiv:2601. 12247v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches.

By Miao Li, Hanyang Jiang, Sikai Cheng, Hengyu Fu, Yuhang Cai, Baihe Huang, Tinghan Ye, Xuanzhou Chen, Pascal Van Hentenryck

llms diffusion benchmarks

arXiv Machine Learning

Jun 3

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

arXiv:2606. 03234v1 Announce Type: new Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.

By Ziyue Wang, Aomufei Yuan, Yongfu Zhu, Shuai Dong, Wenpu Liu, Yiran Yao, Weichu Xie, Yuqi Xu, Caoyuan Ma, Wenqi Shao, Xiaoying Zhang, Nan Duan, Jiaqi Wang

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 3

Learning Temporal Causal Structure via Smooth Differentiable Optimization

arXiv:2606. 03227v1 Announce Type: new Abstract: Causal discovery with instantaneous effects in multivariate time series is challenging, as the instantaneous structure must be acyclic.

By Tong Zhao, Ce Guo, Wayne Luk, Emil Lupu, Ray Dipojjwal

benchmarks

arXiv AI

Jun 3

InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

arXiv:2602. 06960v3 Announce Type: replace-cross Abstract: Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects.

By Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, Yongliang Shen

nlp reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 3

Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigation

arXiv:2606. 03130v1 Announce Type: new Abstract: Small open-source code models that power IDE autocomplete still emit hallucinated Fill-in-the-Middle (FIM) completions: syntactically natural calls to methods, parameters, variables, and imports that do not exist in the surrounding project.

By Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg, Amabel Gale, Xiaoyu Liu, Pareesa Ameneh Golnari, Shengyu Fu

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 3

Fast Organic Crystal Structure Prediction with Unit Cell Flow Matching

arXiv:2606. 03199v1 Announce Type: new Abstract: Organic crystal structure prediction (CSP) is a requirement for computational modelling of organic solids, but traditionally costs several CPU-years per molecule.

By Alston Lo, Luka Mucko, Austin H. Cheng, Andy Cai, Alastair J. A. Price, Wojciech Matusik, Al\'an Aspuru-Guzik

diffusion benchmarks safety

Get the daily briefing