Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv Machine Learning

Jul 31

Bridging AI and Energy Forecasting: An Autonomous Workflow with Customized Toolkit

arXiv:2307. 07191v3 Announce Type: replace Abstract: Energy forecasting is crucial for the power grid, but fundamentally different from general time series analysis: it highly relies on covariates like meteorological factors, and its goals must align with actual power grid operations, such as risk assessment and system reliability.

By Zhixian Wang, Leandro Von Krannichfeldt, Qingsong Wen, Chaoli Zhang, Liang Sun, Shirui Pan, Yi Wang

llms agents benchmarks

arXiv Machine Learning

Jul 31

Region-adaptable retrieval of coastal biogeochemical parameters from near-surface hyperspectral remote sensing reflectance using physics-aware meta-learning

arXiv:2605. 05623v2 Announce Type: replace Abstract: Hyperspectral in situ sensing has shown promise in retrieving aquatic biogeochemical (BGC) parameters, such as total suspended solids, dissolved organic carbon, and total chlorophyll-a, for cost-effective monitoring of coastal water quality.

By Yiqing Guo, Nagur R. C. Cherukuru, Eric A. Lehmann, S. L. Kesav Unnithan, Tim J. Malthus, Gemma Kerrisk, Xiubin Qi, Faisal Islam, Tisham Dhar, Mark J. Doubell

fine-tuning benchmarks

arXiv Machine Learning

Jul 31

ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate

arXiv:2607. 03126v3 Announce Type: replace Abstract: Outcome-supervised reinforcement learning scales to verifiable reasoning tasks, but trajectory-level rewards assign the same outcome signal to all sampled tokens, overlooking their unequal contributions to the reasoning process.

By Zijun Xie, Yuyang You, Yongzhi Li, Enlei Gong, Quan Chen, Yanhua Cheng, Peng Jiang, Binbin Zheng, Xiaolong Liu, Zeyu Chen, Yadong Mu

reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

arXiv:2607. 25018v2 Announce Type: replace Abstract: Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one.

By Yifan Dou, Shikan Lian, Shibo Li

llms benchmarks

arXiv Machine Learning

Jul 31

Re-examining Granger Causality with Causal Bayesian Networks and Reichenbachs Principles

arXiv:2501. 02672v4 Announce Type: replace-cross Abstract: Granger causality (GC) is widely used to infer directed relationships in time-series data.

By S. A. Adedayo

benchmarks

arXiv Machine Learning

Jul 31

Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

arXiv:2508. 04227v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs), spanning predictive architectures to generative Multimodal Large Language Models (MLLMs), have revolutionized artificial intelligence through powerful cross-modal alignment and zero-shot generalization.

By Yuyang Liu, Qiuhe Hong, Linlan Huang, Alexandra Gomez-Villa, Dipam Goswami, Tiantian Peng, Xialei Liu, Joost van de Weijer, Yonghong Tian

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jul 31

MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding

arXiv:2511. 12449v3 Announce Type: replace-cross Abstract: Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding.

By Zhanheng Nie, Chenghan Fu, Daoze Zhang, Junxian Wu, Wanxian Guan, Pengjie Wang, Jian Xu, Bo Zheng

llms multimodal benchmarks safety data-engineering

arXiv Machine Learning

Jul 31

Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

arXiv:2603. 12055v3 Announce Type: replace-cross Abstract: Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion.

By Chiyuan He, Zihuan Qiu, Fanman Meng, Runtong Zhang, Linfeng Xu, Qingbo Wu, Hongliang Li

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 31

Predict before you train: Scaling Laws for particle physics foundation models

arXiv:2607. 23377v1 Announce Type: cross Abstract: The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be estimated before that compute is spent.

By Jan-Lucas Uslu, Benjamin Nachman, Christopher Re

llms fine-tuning benchmarks

arXiv AI

Jul 31

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

arXiv:2607. 25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases.

By Mateusz Koz{\l}owski

llms multimodal benchmarks

arXiv AI

Jul 31

IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval

arXiv:2607. 26072v1 Announce Type: cross Abstract: Long-term memory is becoming a core capability of LLM-based agents, but existing evaluations largely test conversational recall in open-domain or persona-grounded settings.

By Changyu Du, Alexander Vosseler, Filippo Mazza, Andr\'e Borrmann

llms agents benchmarks

arXiv AI

Jul 31

Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.

By Mingkuan Feng, Zhengqi Wen, Jianhua Tao

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 31

Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection

arXiv:2607. 26656v1 Announce Type: cross Abstract: Real-world vulnerabilities often span multiple functions, yet most learning-based detectors classify each function in isolation: on a sample of real CVEs, we find that 71.

By Yikun Li, Ting Zhang, Jiakun Liu, Jinfeng Jiang, Yuheng Yieh, Yixin Yang, Wen Bin Leow, Yide Yin, Yintong Huo, Eng Lieh Ouh, Lwin Khin Shar, David Lo

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

arXiv:2607. 27271v1 Announce Type: new Abstract: Code models are increasingly trained with execution feedback, but most training signals still stop at correctness.

By Huihao Jing, Haozhe Cui, Wenbin Hu, Shaojin Chen, Haochen Shi, Changxuan Fan, Yuxuan Liu, Hanyu Yang, Sirui Zhang, Ziyi Chen, Haoran Li, Yangqiu Song

agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.

By Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

llms agents efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Looped Transformers with Source-Centered State Evolution

arXiv:2607. 27656v1 Announce Type: new Abstract: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count.

By Bum Jun Kim, Kohei Hayashi, Shunsuke Kamiya, Masanori Koyama, Yusuke Iwasawa, Yutaka Matsuo

llms benchmarks safety

arXiv Machine Learning

Jul 31

ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendation

arXiv:2607. 27744v1 Announce Type: new Abstract: Modern recommendation models gain prediction quality by scaling feature-interaction and sequence modules, but production cost constraints cap how far systems can scale.

By Yuxin Chen, Liang Luo, Buyun Zhang, Jian Jiao, Boda Li, Haoyu Wang, Tongyi Tang, Ao Cai, Zijian Shen, Zhengkai Zhang, Wenyi Xie, Ryan Dick, Han Liu, Neng Shi, Bin Yu, Jianbo Xiao, Shuyao Bi, Hongtao Yu, Yuanwei Fang, Zhuoran Zhao, Sijia Chen, Yang Chen, Shuqi Yang, Qianru Li, Zikun Liu, Wei Ling, Sihan Zeng, Longhao Jin, Jiaxin Lu, Yinbin Ma, Jiawei Li, Yichen Ruan, Yong Ler Lee, Birmingham Guan, Zijian Li, Jianbo Sun, Zhengyu Zhang, Zeliang Chen, Xiaohan Wei, Yuchen Hao, GP Musumeci, Venkatesh Ranganathan, Yantao Yao, Chunqiang Tang, Wenlin Chen, Santanu Kolay, Ellie Dingqiao Wen

benchmarks

arXiv Machine Learning

Jul 31

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.

By Ken Ding

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 31

KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models

arXiv:2607. 28608v1 Announce Type: new Abstract: Clinical risk models routinely achieve strong aggregate performance while producing materially different error rates across patient subgroups.

By Sparsh Roy, Samuel Girmachew, Nishita Chavan

benchmarks safety

Get the daily briefing