Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv AI

Jul 29

Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

arXiv:2607. 22205v2 Announce Type: replace-cross Abstract: Remote sensing multimodal large language models (RS-MLLMs) have improved general aerial-image understanding.

By Yuheng Zong, Minghua Wang, Xin Zhao, Zhi-Hui Zhan, Antonio Plaza, Jon Atli Benediktsson

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 29

LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation

arXiv:2607. 24780v1 Announce Type: new Abstract: Evaluating frontier LLMs is challenging: static benchmarks suffer from contamination and saturation -- leaving users unable to distinguish top models and developers blind to specific failure modes -- while human preference is subjective.

By Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

llms reinforcement-learning benchmarks

arXiv AI

Jul 29

Aletheia: An Offline-First Clinical Decision Support System for Differential Diagnosis in Low-Resource Healthcare Settings

arXiv:2607. 24814v1 Announce Type: new Abstract: Access to specialist clinical expertise remains severely limited across sub-Saharan Africa, where physician-to-patient ratios can fall below 1:25,000 in rural settings.

By Joseph Walusimbi, Ann Move Oguti, Abubakhari Sserwadda, Precious Boss Kasasira, Charles Brian Okoboi

llms fine-tuning benchmarks

arXiv AI

Jul 29

Crystalis: Progressive Nucleation and Semantic Annealing for Coordinated Multi-View Visualization Generation

arXiv:2607. 24766v1 Announce Type: new Abstract: Large language models (LLMs) can generate individual charts, but coordinated multi-view visualizations (CMVs), where views share data flows and cross-view interactions, remain out of reach.

By Dazhen Deng, Zhaoping He, Xin Qian, Xiaotong Wang, Zi Ying, Yingcai Wu

llms agents benchmarks

arXiv AI

Jul 29

CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

arXiv:2607. 24763v1 Announce Type: new Abstract: Masked diffusion language models (MDLMs) are advancing rapidly, yet the evaluation standards needed to reliably interpret their progress have not kept pace.

By Yash Shah, Abhijit Chakraborty, Vivek Gupta

llms diffusion benchmarks

arXiv AI

Jul 29

Steering topology distributions for unified generative design of architected metamaterials

arXiv:2607. 24777v1 Announce Type: new Abstract: Architected metamaterials derive their functions from structure, creating vast opportunities to program physical responses through topology design.

By Haolin Li, Yuyang Miao, Menglei Li, Jinshuai Bai, Liyuan Wang, Xin Liu, Bo Gao, Jiantao Liu, Danilo Mandic, Zahra Sharif Khodaei, M. H. Aliabadi, Weiqiu Chen

diffusion benchmarks

arXiv AI

Jul 29

Everyone is unique: Towards Behaviorally Heterogeneous Negotiation Dialogue Systems for Debt Collection

arXiv:2607. 25218v1 Announce Type: new Abstract: Debt collection is a critical negotiation task in the financial industry, with strong practical relevance and exceptional academic value as a behaviorally rich, high-stakes testbed for human-centered dialogue systems.

By Yuhang Yang, Kai Tang, Chao Ye, Haobo Wang, Qiqi Luo, Jinguang Zheng, Zhixin Zhang

llms agents benchmarks

arXiv AI

Jul 29

A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain

arXiv:2607. 25415v1 Announce Type: new Abstract: Production LLM agents are increasingly assembled from a frozen model wrapped in a harness: a prompt template, a tool set, a memory/retrieval layer, a planning strategy, and a verification policy.

By Debjyoti Paul

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 29

COVENANT: Natural-Language Workflow Compilation for Aligned Agent Execution

arXiv:2607. 25400v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly entrusted with natural-language workflow instructions (e.

By Jincheng Wang, Min Zheng, Tao Wei

llms agents benchmarks safety

arXiv AI

Jul 29

Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm

arXiv:2607. 25446v1 Announce Type: new Abstract: Multi-agent frameworks built on large language models (LLMs) routinely entangle three logically distinct concerns: who is on the team (organization), how members align (coordination), and which algorithm fuses their work (collaboration protocol).

By Huan Chen, Xiang Song, Jian Jin, Pan Ren, Liang-Jie Zhang

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 29

Localized Adaptation Reveals Distinct Learning Signatures in Transformers

arXiv:2607. 25663v1 Announce Type: new Abstract: Transformer adaptation is typically distributed across model depth, even when the intended change is narrow.

By Rebecca Ramnauth, Brian Scassellati

llms fine-tuning benchmarks

arXiv AI

Jul 29

Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

arXiv:2607. 25904v1 Announce Type: new Abstract: Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction.

By Chenrui Shi, Yuwei Wu, Yang Liu, Ruining Feng, Zirui Shang, Zhi Gao, Lifeng Fan, Che Sun

agents reinforcement-learning benchmarks

arXiv AI

Jul 29

A Cost-Effective Multimodal LLM Reasoning Framework for Question Answering over Irregular Clinical Time Series

arXiv:2607. 25947v1 Announce Type: new Abstract: Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications.

By Frank Nie, Ethan B Liu, Yuan Zhu, Wei Fan, Jindong Han

llms rag nlp efficiency multimodal benchmarks safety

arXiv AI

Jul 29

Measuring and Improving Behavioral Consistency in Large Language Models through Fact-Heuristic-Emotion State Enforcement

arXiv:2607. 24765v1 Announce Type: cross Abstract: Large language models (LLMs) can give different answers to the same decision problem across runs, and reverse a decision when their own prior answer returns as context.

By Gi-Hun Lee, Joong Yull Park

llms benchmarks

arXiv AI

Jul 29

Dual-Level Atomic and Coordination Geometry Learning for Crystal Property Prediction Using Graph Neural Networks

arXiv:2607. 24818v1 Announce Type: cross Abstract: Accurate prediction of crystal properties remains a key challenge in computational materials science.

By Sanjay Chakraborty

benchmarks

arXiv AI

Jul 29

Two Views, One Voice: Evidence-Grounded Conversational Music Recommendation

arXiv:2607. 24846v1 Announce Type: cross Abstract: Traditional conversational recommenders entangle retrieval and response generation within a single text interface, so exact entity cues fade as the dialogue's intent evolves, which compromises explanation credibility.

By Sungwook Yoo, Sewook Yoo

fine-tuning benchmarks

arXiv AI

Jul 29

Learning from 53.6K Real-World Developer Edits of AI-Generated Code

arXiv:2607. 25130v1 Announce Type: cross Abstract: Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant.

By Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, Valerie Chen

llms fine-tuning benchmarks

arXiv AI

Jul 29

TabRank: Chain-of-Thought Distillation for Table Re-Rankers

arXiv:2607. 25182v1 Announce Type: cross Abstract: The ability to retrieve relevant tables for answering questions is a key task for structured information retrieval.

By Adarsh Singh, Kushal Raj Bhandari, Jianxi Gao, Soham Dan, Vivek Gupta

llms rag efficiency benchmarks

arXiv AI

Jul 29

Physics-Informed Broad Learning System: An Efficient Backpropagation-Free Framework for Solving Partial Differential Equations

arXiv:2607. 25608v1 Announce Type: cross Abstract: Physics-informed neural networks (PINNs) have emerged as a powerful paradigm for solving partial differential equations (PDEs) by embedding governing physical laws into deep neural networks.

By Pinki Khatun, M. Sajid, Abhinav Jha, M. Tanveer

rag benchmarks

arXiv AI

Jul 29

KQFuzz: Knowledge-Guided Fuzzing for Quantum Libraries via Large Language Models

arXiv:2607. 25647v1 Announce Type: cross Abstract: As quantum computing continually improves, ensuring the reliability and correctness of quantum libraries has become increasingly critical.

By Fuyuan Xia, Qixin Zhang, Chenhao Ying, Haojin Zhu, Shuai Wang, Yuan Luo, Pingchuan Ma, Yuxuan Du

llms benchmarks

Get the daily briefing