Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,030 stories · RSS feed

arXiv AI

Aug 11

Mitigating Gender Bias in English to Romanian Machine Translation

arXiv:2608. 08606v1 Announce Type: cross Abstract: Machine translation (MT) systems often fail to correctly translate gender, especially when converting from a gender-neutral language like English to a gendered target language such as Romanian.

By Ioana Grigore, Sergiu Nisioi

llms nlp fine-tuning benchmarks safety

arXiv AI

Aug 11

Fourier Self-Supervision for Fine-Grained Generalized Category Discovery

arXiv:2608. 08963v1 Announce Type: cross Abstract: Generalized Category Discovery aims to recognize known categories while identifying novel ones within unlabeled data.

By Sarah Rastegar, Mina Ghadimi Atigh, Pascal Mettes, Yuki M. Asano, Cees G. M. Snoek

diffusion benchmarks

arXiv AI

Aug 11

TRACE: TRajectory Attribution for Automated Context Engineering

arXiv:2608. 09153v1 Announce Type: new Abstract: Production AI agents fail when their context sources -- system prompts, knowledge bases, tool descriptions, and procedural skills -- contain errors or gaps.

By Yikai Zhao, Pradeep Kumar Misra, Saurabh Pandey

agents fine-tuning benchmarks

arXiv AI

Aug 11

ASPaeroFlow: Decomposition Heuristics for Joint Air Traffic Flow & Capacity Management

arXiv:2608. 09315v1 Announce Type: new Abstract: While mathematical models act as vital decision support systems for operational Air Traffic Flow and Capacity Management (ATFCM), existing approaches isolate Air Traffic Flow Management (ATFM) from Dynamic Airspace Configuration (DAC).

By Alexander Beiser, Markus Hecher, Nysret Musliu, Georg Trausmuth, Stefan Woltran

benchmarks

arXiv Machine Learning

Aug 11

Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

arXiv:2608. 08961v1 Announce Type: new Abstract: AI training's rising resource intensity is straining electricity supplies and carbon budgets, motivating systematic study of memory-efficient training on constrained hardware.

By Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri, Mehdi Hosseinzadeh, Reza Rawassizadeh

llms computer-vision multimodal benchmarks

arXiv AI

Aug 11

P$^{3}$: Joint Program-and-Proof Planning for Verified Code Generation

arXiv:2608. 09277v1 Announce Type: new Abstract: Verified code generation asks a large language model (LLM) to generate both an executable program and a machine-checkable proof that the program meets a formal specification, promising software that is correct by construction.

By Zenan Li, Ziran Yang, Peiyang Song, Zhaoyu Li, Kaiyu Yang

llms agents benchmarks

arXiv AI

Aug 11

Evaluation of Motivational Interviewing Counsellors with Task-Aware Multi-Stage LLM-Based Simulated Clients

arXiv:2608. 07499v1 Announce Type: cross Abstract: The development and benchmarking of Large Language Model (LLM)-based Motivational Interviewing (MI) counsellors now often rely on LLM-based simulated clients.

By Jiading Zhu, Xinyu Cindy Wang, Thomas Nguyen, Yan Qing Lee, Osnat C. Melamed, Peter Selby, Jonathan Rose

llms benchmarks

arXiv AI

Aug 11

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

arXiv:2608. 08029v1 Announce Type: cross Abstract: Khatri et al.

By Alizishaan Khatri, Dun Li Chan

llms benchmarks safety

arXiv AI

Aug 11

Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

arXiv:2608. 08307v1 Announce Type: cross Abstract: Medical Visual Question Answering (VQA) requires aligning subtle visual evidence, including lesion texture, boundary sharpness, and diffuse density changes, with clinical language.

By Yusra Tariq, Rakesh Chandra Joshi

nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 11

Open-World Hierarchical Perception: Taxonomic Abstraction over Class-Agnostic Proposals for the Safe Handling of Out-of-Vocabulary Road Objects

arXiv:2608. 07577v1 Announce Type: cross Abstract: A closed-set detector for autonomous driving must assign every object one of a fixed set of labels.

By Felix Schaller

agents computer-vision robotics benchmarks safety

arXiv Machine Learning

Aug 11

Bayesian Symbolic Regression with Entropic Reinforcement Learning

arXiv:2608. 09617v1 Announce Type: new Abstract: Symbolic regression is the problem of finding an algebraic expression describing a stochastic dependence of a target variable on a set of inputs.

By Oussama Boussif, Mohammed Mahfoud, Younesse Kaddar, Moksh Jain, Sida Li, Damiano Fornasiere, Xiaoyin Chen, Yoshua Bengio, Esmeralda S. Whitammer

reinforcement-learning benchmarks

arXiv AI

Aug 11

Calling the Bluff: Detecting Ever-Shifting Harmful Chat Dialogue via Ordered Reasoning Chain Regularization

arXiv:2608. 08451v1 Announce Type: cross Abstract: Harmful chat dialogues are ever-shifting through type-shifting and lexical evasion, yet we find they share invariant principles, i.

By Haojie Yu, Ziyou Jiang, Junjie Wang, Mingyang Li, Yuekai Huang, Jie Huang, Qing Wang

fine-tuning benchmarks

arXiv AI

Aug 11

Evidence-RL: Towards Evidence-intensive Visual Reasoning

arXiv:2608. 08021v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context.

By Haojie Huang, Xinlei Yu, Chengming Xu, Zhangquan Chen, Cheng Yang, Qingdong He, Yu Yang, Jiangning Zhang, Xiaobin Hu

llms reinforcement-learning multimodal benchmarks

arXiv AI

Aug 11

NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning

arXiv:2510. 18940v2 Announce Type: replace-cross Abstract: Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation.

By Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova

fine-tuning benchmarks

arXiv AI

Aug 11

Verication-driven closed-loop multi-agent large language modelframework for code-compliant structural design

arXiv:2608. 07978v1 Announce Type: cross Abstract: Multi-agent large language model(LLM)systems are applied to structural design,yet most use one-shot generation and cannot verify their output,leaving themill-suited to safety-critical tasks.

By Jianbin Luo, Weibin Lin, Yiran Lin, Qing Wei, Wei Guo

llms rag agents benchmarks safety

arXiv AI

Aug 11

Enhancing Knowledge Tracing through Leakage-Free and Recency-Aware Embeddings

arXiv:2508. 17092v2 Announce Type: replace-cross Abstract: Knowledge Tracing (KT) aims to predict a student's future performance based on their sequence of interactions with learning content.

By Yahya Badran, Christine Preisach

rag benchmarks

arXiv AI

Aug 11

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

arXiv:2608. 09555v1 Announce Type: new Abstract: External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks.

By Tianjun Pan, Yuan Li, Hongda Wang, Linbo Jin, Mengfei Song, Lei Gao, Qiming Shi, Shaokang Fu, Jiarong Zhao, Chengyu Wang, Chengfu Huo

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Aug 11

Matryoshka Language Model Suites

arXiv:2608. 09703v1 Announce Type: new Abstract: Training a language model suite classically requires training each model separately and serving them independently.

By Nathan Godey, Yoav Artzi

llms efficiency benchmarks

arXiv AI

Aug 11

Multi-Branch Policy Optimization for Multimodal Large Language Models

arXiv:2608. 07581v1 Announce Type: cross Abstract: Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response.

By Shuai Lyu, Yuning Gong, Ruiling Gao, Xiaoran Shang, Zhonghong Ou, Ping Zong, Yifan Zhu, Yuan Sun, Yang Qin, Peng Hu

llms reinforcement-learning multimodal benchmarks

arXiv AI

Aug 11

PRISM: A Predictive Protocol for Permutation Optimization via Landscape Diagnostics

arXiv:2608. 08344v1 Announce Type: cross Abstract: Permutation optimization arises whenever the components of a system are fixed but their ordering affects performance.

By Blessings Mambwe

benchmarks

Get the daily briefing