Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,030 stories · RSS feed

arXiv AI

Aug 10

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.

By Guanzhi Deng, Haibo Wang, Kuan Wu, Xiangru Jian, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song

llms fine-tuning benchmarks

arXiv AI

Aug 10

EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided Revision

arXiv:2608. 07196v1 Announce Type: new Abstract: Many methods for automated multi-agent system design optimize prompts and topologies during an initial design stage and then deploy the resulting system unchanged on subsequent samples.

By Chao Fei, Qingyi Si, Kaihua Liang, Yanghua Xiao, Panos Kalnis, Hongcheng Guo

llms agents benchmarks

arXiv Machine Learning

Aug 10

Symbolic Graphics Programming with Large Language Models

arXiv:2509. 05208v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel at program synthesis, yet their ability to produce symbolic graphics programs (SGPs) that render into precise visual content remains underexplored.

By Yamei Chen, Haoquan Zhang, Yangyi Huang, Zeju Qiu, Kaipeng Zhang, Yandong Wen, Weiyang Liu

llms reinforcement-learning multimodal benchmarks

arXiv AI

Aug 10

Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models

arXiv:2608. 07243v1 Announce Type: new Abstract: Generative models are often evaluated through singular artifacts, whereas human creativity typically emerges through iterative generation, appraisal, and refinement.

By Rens Anderson, Tessa Verhoef, Amirhossein Zohrehvand

llms benchmarks

arXiv AI

Aug 10

Beyond the Black Box: Interpretable Models of Human Randomisation Failures

arXiv:2608. 07220v1 Announce Type: new Abstract: Mixed strategy equilibrium predicts i.

By Ngoc Linh Dao

benchmarks

arXiv AI

Aug 10

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

arXiv:2511. 07885v5 Announce Type: replace-cross Abstract: Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure.

By Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher R\'e

llms benchmarks

arXiv Machine Learning

Aug 10

CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM

arXiv:2504. 17584v2 Announce Type: replace-cross Abstract: Attention-FC Disaggregated (AFD) LLM inference systems offload memory-bound Attention operations to memory-rich accelerators (e.

By Qingyuan Liu, Liyan Chen, Haocheng Wang, Yanning Yang, Dong Du, Zhigang Mao, Naifeng Jing, Yubin Xia, Haibo Chen

llms efficiency benchmarks safety

arXiv AI

Aug 10

ED-CSP: Crystal Structure Prediction from Electron Diffraction

arXiv:2608. 06448v1 Announce Type: cross Abstract: Recovering a periodic 3D crystal structure from sparse, unindexed electron diffraction (ED) observations is a challenging generative inverse problem.

By Germain Poloudenny, Ya\"el Fr\'egier, Arnaud Demorti\`ere

benchmarks

arXiv AI

Aug 10

A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy

arXiv:2608. 07427v1 Announce Type: new Abstract: LLM inference accounts for over 90% of AI operational energy, scaling directly with input token count---a critical inefficiency for telecom network analytics and numerical time-series data analysis (NTSDA), where raw multivariate KPI windows from 4G/5G cell sites expand into thousands of floating-point tokens.

By Bhavika Jalli, Nikhil Korati Prasanna, Jayanta Choudhury

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 10

FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings

arXiv:2608. 07400v1 Announce Type: new Abstract: Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence.

By Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel, Fabian Woebbeking

rag nlp benchmarks

arXiv Machine Learning

Aug 10

Target-Weighted Neyman Allocation: Experimental Design for Heterogeneous Treatment Effects under Population Shift

arXiv:2608. 06512v1 Announce Type: new Abstract: Randomized experiments are often run in one population to guide decisions in another.

By Hoang Dang, Luan Pham, Minh Nguyen

benchmarks

arXiv Machine Learning

Aug 10

SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Prediction

arXiv:2608. 06441v1 Announce Type: new Abstract: Full-graph GNN training delivers high accuracy but scales poorly on multi-server clusters due to heavy, irregular inter-node embedding exchanges.

By Guofan Yu, Sitian Chen, Zhenheng Tang, Xiaowen Chu, Amelie Chi Zhou

rag benchmarks safety data-engineering

arXiv AI

Aug 10

Social World Models

arXiv:2509. 00559v3 Announce Type: replace Abstract: Humans intuitively navigate social interactions by simulating unspoken dynamics and reasoning about others' perspectives, even with limited information.

By Xuhui Zhou, Jiarui Liu, Akhila Yerukola, Hyunwoo Kim, Maarten Sap

llms agents benchmarks

arXiv Machine Learning

Aug 10

Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions

arXiv:2409. 18804v3 Announce Type: replace-cross Abstract: Denoising Diffusion Probabilistic Models (DDPM) are powerful state-of-the-art methods used to generate synthetic data from high-dimensional data distributions and are widely used for image, audio, and video generation as well as many more applications in science and beyond.

By Iskander Azangulov, George Deligiannidis, Judith Rousseau

diffusion benchmarks

arXiv AI

Aug 10

ADIAS: Automated Design of Interactive Agentic Systems

arXiv:2608. 06410v1 Announce Type: new Abstract: Automated agent design improves agent harnesses through iterative revision, evaluation, and feedback summarization.

By Lekang Jiang, Bohan Tang, Stephan Goetz, Yiwen Guo

agents nlp benchmarks

arXiv AI

Aug 10

Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

arXiv:2608. 06756v1 Announce Type: new Abstract: Vision-language models are increasingly serving as the reasoning core of embodied agents.

By Ying Chen, Weizhen Li, Zhe Hu, Zhenjiang Li, Rui Jiang, Zhifeng Gu, Lihuang Fang, Jiangping Liu, Lei Yi, Jie Chen

llms agents reinforcement-learning robotics efficiency multimodal benchmarks

arXiv AI

Aug 10

SABRE: Scalable and Automated Benchmarking of VLMs under Stress

arXiv:2608. 07435v1 Announce Type: cross Abstract: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify.

By Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou

llms multimodal benchmarks

arXiv AI

Aug 10

Not All Problems Are Best Modeled as MILP: A DSL-Centric Framework for Flexible and Accurate Optimization Modeling

arXiv:2608. 07040v1 Announce Type: new Abstract: Solving combinatorial optimization problems (COPs) requires not only efficient algorithms but also carefully crafted formulations.

By Shaofeng Zhang, Hongyuan Su, Qingwen Peng, Zefang Zong, Shengcai Liu, Ke Tang, Yong Li

llms benchmarks

arXiv Machine Learning

Aug 10

FedDOSE: Federated Learning Framework Decomposing Site Effects for Modeling Brain Dynamic Functional Connectivity

arXiv:2608. 07393v1 Announce Type: new Abstract: Functional Magnetic Resonance Imaging ( fMRI ) data are often pooled into collaborative multi-site consortia, as deep learning models for analyses require large datasets to generalize well.

By Deepank Girish, Yi Hao Chan, Yubin Zheng, Sukrit Gupta, Jagath C. Rajapakse

benchmarks safety

arXiv Machine Learning

Aug 10

Fast and Accurate: An Adaptive VLA Inference Framework through Environment-aware Model Selection

arXiv:2608. 06434v1 Announce Type: cross Abstract: Embodied intelligence demands both long-horizon reasoning and real-time closed-loop responsiveness.

By Yuewei Sun, Lang Qin, Zechuan Tian, Jingwen Li, Guiqin Wang, Shengzeng Huo, Wenxin Ren, Tao Fang, Xiaochen Zhang, Guanqing Deng, Xiang Wang, Xiaowen Dong, Qinghai Guo, Yuxin Ma

robotics efficiency multimodal benchmarks

Get the daily briefing