Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv AI

Jul 31

Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels

arXiv:2607. 26121v1 Announce Type: cross Abstract: Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.

By Xinyu Yang, Tianxing Chen, Honghao Su, Minxuan Wang, Chenze Yu, Zhangzheng Tu, Yue Chen, Yuxiao Huo, Lingfeng Zhang, Yan Huang, Yan Qin, Shaolong Zhu, Qiwei Liang, Hekun Tian, Shujia Liu, Guangyu Chen, Junhao Gong, Zixuan Li, Wenwei Lin, Zijian Lin, Wenxuan Zhu, Eric J Chen, Yue Yuan, Qize Yu, Jiaqi Liang, Haowen Yan, Hengfei Zhao, Weijie Wan, Zikun Xiao, Junyuan Tang, Baijun Chen, Kai-Chong Lei, Kaixuan Wang, Kailun Su, Zanxin Chen, Yao Mu, Renjing Xu, Chuqiao Lyu, Qi Xiong, Ping Luo, Wenbo Ding

agents robotics benchmarks safety

arXiv AI

Jul 31

From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence

arXiv:2607. 26903v1 Announce Type: new Abstract: The key bottleneck in embodied AI is not model architecture but data.

By Jia Luo

diffusion robotics benchmarks

arXiv Machine Learning

Jul 31

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

arXiv:2607. 28627v1 Announce Type: cross Abstract: Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints.

By Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem

llms rag efficiency multimodal benchmarks

arXiv AI

Jul 31

Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data

arXiv:2607. 27056v1 Announce Type: new Abstract: Personalized agents are increasingly applied to assist users across a wide range of tasks.

By Lingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou

llms agents benchmarks safety

arXiv Machine Learning

Jul 31

GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

arXiv:2607. 27694v1 Announce Type: cross Abstract: Low-bit quantization is essential for efficient LLM inference, and both rotation and fine-grained group quantization have shown individual promise.

By Sangjin Kim, Yuseon Choi, Byeongcheol Kim, Jungjun Oh, Hoi-jun Yoo

llms efficiency benchmarks

arXiv AI

Jul 31

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

arXiv:2606. 21848v2 Announce Type: replace-cross Abstract: Transformer architectures form the foundation of modern natural language processing, making it crucial to address the efficiency and scalability limitations of the standard QKV attention mechanism.

By Xin Gao, Xingming Xu

llms nlp efficiency benchmarks

arXiv Machine Learning

Jul 31

TopoFormer: Topology Meets Attention for Graph Learning

arXiv:2607. 28259v1 Announce Type: new Abstract: We introduce Topoformer, a lightweight and scalable framework for graph representation learning that encodes topological structure into attention-friendly sequences.

By Md Joshem Uddin, Astrit Tola, Cuneyt Gurcan Akcora, Baris Coskunuzer

llms rag benchmarks

arXiv AI

Jul 31

Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

arXiv:2510. 22170v3 Announce Type: replace Abstract: Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation.

By Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz-Ziv, Amirali Abdullah

llms benchmarks

arXiv AI

Jul 31

SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation

arXiv:2607. 26313v1 Announce Type: cross Abstract: Agentic systems act, so a defect in the evidence they retrieve becomes a wrong action with a currency cost.

By Gaston Besanson

agents benchmarks

arXiv Machine Learning

Jul 31

QAdapt: A Noise-Adaptive Neural Pre-Decoding Framework for Quantum Error Correction

arXiv:2607. 28422v1 Announce Type: new Abstract: Fault-tolerant quantum computing (FTQC) relies on quantum error correction to suppress physical errors and preserve logical information at scale.

By Ran Miao, Rui Luo, Xiaohan Shan, Xiaoming Sun

fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.

By Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang, Xin Zhang, Minghua Ma, Pengfei Gao, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

agents benchmarks

arXiv Machine Learning

Jul 31

MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers

arXiv:2607. 28589v1 Announce Type: cross Abstract: Post-training quantization (PTQ) has emerged as an effective solution for deploying Vision Transformers (ViTs) on resource-constrained devices.

By Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar Faruk

llms computer-vision efficiency benchmarks

arXiv Machine Learning

Jul 31

FinSMART: Financial Sentiment Analysis for Algorithmic Trading through Market-Aligned Reinforcement Learning

arXiv:2607. 28127v1 Announce Type: cross Abstract: Recent advances in Generative AI have substantially improved financial sentiment analysis through post-trained financial large language models (LLMs).

By Giorgos Iacovides, Wuyang Zhou, Danilo Mandic

llms nlp reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

(Towards) Scalable Reliable Automated Evaluation with Large Language Models

arXiv:2607. 28282v1 Announce Type: cross Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.

By Bertil Braun, Martin Forell

llms benchmarks

arXiv Machine Learning

Jul 31

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

arXiv:2607. 28418v1 Announce Type: cross Abstract: Pruning is a promising approach for improving the efficiency of LLMs.

By Haozhe Hu, Hao Wu, Peiran Yin, Chao Han, Yunpu Ma, Xiaoyu Shen

llms nlp efficiency benchmarks

arXiv AI

Jul 31

Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning

arXiv:2508. 16129v3 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have recently demonstrated remarkable reasoning abilities with reinforcement learning paradigm.

By Ruiqi Wu, Yuang Yao, Tengfei Ma, Chenran Zhang, Na Su, Tao Zhou, Geng Chen, Wen Fan, Yi Zhou

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jul 31

Graph Neural Network Force Fields for Spin Dynamics in Metallic Magnets

arXiv:2607. 28537v1 Announce Type: cross Abstract: Metallic magnets exhibit complex spin dynamics governed by electronically generated interactions.

By Ali Rayat, Yunhao Fan, Gia-Wei Chern

benchmarks

arXiv Machine Learning

Jul 31

THGFM: Dual-Branch Temporal Heterogeneous Graph Fusion Model

arXiv:2607. 27303v1 Announce Type: new Abstract: Temporal heterogeneous graphs offer a natural abstraction for dynamic relational systems in which diverse node and relation types co-exist and evolve over time.

By Yixin Peng, Diego Collarana, Er Jin, Stefan Decker

llms benchmarks

arXiv AI

Jul 31

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

arXiv:2607. 26735v1 Announce Type: cross Abstract: Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering.

By Xiaolong Liu, Junjian Li, Yuan Xiao, Jiaqi Deng, Dayong Ye, Tianqing Zhu, Huan Huo

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 31

Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance

arXiv:2607. 27283v1 Announce Type: new Abstract: Long-horizon benchmarks often show that agents fail more as tasks become longer.

By Chao Peng, Zhiheng Lyu, Peijie Dong, Hande Dong, Qiang Lin

agents benchmarks

Get the daily briefing