Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv Machine Learning

Jun 26

RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

arXiv:2606. 26997v1 Announce Type: cross Abstract: Large language model (LLM) post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks.

By Rongjian Chen, Jianmin Hu, Kejiang Ye, Minxian Xu

llms reinforcement-learning benchmarks

arXiv AI

Jun 26

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

arXiv:2606. 27091v1 Announce Type: cross Abstract: LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data.

By Ryan Fetterman

llms fine-tuning benchmarks

arXiv AI

Jun 26

What We are Missing in Multimodal LLM Evaluation?

arXiv:2606. 26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.

By Po-han Li, Shenghui Chen, Sandeep Chinchali, Ufuk Topcu

llms multimodal benchmarks

arXiv AI

Jun 26

CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

arXiv:2606. 26216v1 Announce Type: cross Abstract: We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis.

By Jintao Huang, Fengqing Jiang, Radha Poovendran, Zhiqiang Lin

llms agents benchmarks

arXiv AI

Jun 26

Benchmarking Open-Weight Foundation Models for Global AI Technical Governance

arXiv:2606. 26099v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in artificial intelligence (AI) governance analysis across national and international organisations.

By Jason Hung

llms benchmarks safety

arXiv AI

Jun 26

EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting

arXiv:2606. 27277v1 Announce Type: new Abstract: Earth Observation (EO) forecasting aims to predict future Earth surface dynamics from satellite observations under changing meteorological conditions.

By Junwei Luo, Shuai Yuan, Zhenya Yang, Yansheng Li, Zhe Liu, Hengshuang Zhao

llms diffusion benchmarks

arXiv AI

Jun 26

Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

arXiv:2606. 26101v1 Announce Type: cross Abstract: Reliable evaluation of large language models should separate supported answering from unsupported guessing without conflating either with data contamination, prompt idiosyncrasy, or generic refusal behavior.

By Renwei Meng, Bowen Zhang, Jian Wang, Xican Wang, Haoyi Wu, Xuanyan Qiu, Shengan Yang

llms benchmarks

arXiv AI

Jun 26

Assert, don't describe: Linguistic features that shift LLM reasoning about animal welfare

arXiv:2606. 26104v1 Announce Type: cross Abstract: Animal-welfare advocates produce a lot of writing, and increasingly that writing trains the language models that millions of people then ask about animal welfare.

By Jasmine Brazilek, Harper Dunn

llms fine-tuning benchmarks

arXiv AI

Jun 26

MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG

arXiv:2606. 26793v1 Announce Type: cross Abstract: Multimodal agentic retrieval-augmented generation (RAG) systems expand the attack surface beyond prompt injection to include text poisoning, image injection, direct-query attacks, and orchestrator-level tool manipulation.

By Inderjeet Singh, Andr\'es Murillo, Motoyoshi Sekiya, Yuki Unno, Junichi Suga

rag agents robotics multimodal benchmarks safety

arXiv AI

Jun 26

Geometric Fairness-Aware Routing for Federated Edge Networks

arXiv:2606. 26125v1 Announce Type: cross Abstract: Emerging 6G and edge-intelligent networks require effective and balanced routing algorithms among varied and spatially distributed devices.

By Ratun Rahman

rag efficiency benchmarks safety

arXiv AI

Jun 26

Unsupervised Memory-Enhanced Video Transformers: Obstacle Detection for Autonomous Agricultural Rover

arXiv:2606. 26151v1 Announce Type: cross Abstract: While autonomous rovers have become indispensable to precision farming, achieving consistent operational safety remains a critical challenge.

By Th\'eo Biardeau (XLIM-ASALI, UFR SFA), Anne-Sophie Capelle-Laiz\'e (UP, XLIM-ASALI, XLIM-ASALI), Salwan Alwan (UFR SFA), David Helbert (UFR SFA)

llms agents computer-vision robotics benchmarks safety

arXiv AI

Jun 26

Narration-of-Thought: Inference-Time Scaffolding for Defeasible Ethical Reasoning in Large Language Models

arXiv:2606. 26366v1 Announce Type: new Abstract: Standard chain-of-thought on moral dilemmas exhibits two failure modes: stakeholder collapse (the trace names at most one party with a stake in the outcome) and uncertainty suppression (no explicit unknowns or hedges before committing to an action).

By Patrick Cooper, Alvaro Velasquez

llms agents fine-tuning benchmarks

arXiv AI

Jun 26

Learning Motion Feasibility from Point Clouds in Cluttered Environments

arXiv:2606. 26700v1 Announce Type: cross Abstract: Motion feasibility prediction plays a central role in robotics, particularly in task and motion planning and manipulation.

By Sajid Ansari, Arthi, Girish Varma, Antony Thomas

llms robotics benchmarks

arXiv AI

Jun 26

Disco-LoRA: Disentangled Composition of Content, Style, and Motion for Multi-concept Video Customization

arXiv:2606. 26668v1 Announce Type: cross Abstract: Video customization based on Text-to-Video (T2V) models aims to learn specific features from reference data to generate controllable videos.

By Xuancheng Xu, Gengyun Jia, Bing-Kun Bao

diffusion fine-tuning benchmarks

arXiv AI

Jun 26

TAVR-VLM: Risk-Conditioned Causal Grounding for Hallucination-Resistant Report Generation

arXiv:2606. 26874v1 Announce Type: new Abstract: Transcatheter Aortic Valve Replacement (TAVR) planning requires meticulous multimodal reasoning.

By Zhixiang Lu, Xiwei Liu, Sifan Song, Changkai Ji, Anh Nguyen, Jionglong Su, Imran Razzak, Jinfeng Wang

llms multimodal benchmarks safety

arXiv AI

Jun 26

Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge

arXiv:2606. 26511v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) gives agents access to accumulated knowledge, but has no model of time.

By Neeraj Yadav

llms rag agents benchmarks

arXiv AI

Jun 26

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

arXiv:2606. 26947v1 Announce Type: cross Abstract: While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task.

By Wenwang Huang, Yusen Fu, Junjie Wang, Mengfei Huang, Yulin Li, Gan Liu, Jing Cai, Yancheng He, Zhuotao Tian

diffusion fine-tuning benchmarks

arXiv AI

Jun 26

In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

arXiv:2606. 26981v1 Announce Type: cross Abstract: Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism.

By Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li, Liang Lin, Ziliang Chen

llms benchmarks safety

arXiv AI

Jun 26

VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

arXiv:2603. 01195v2 Announce Type: replace-cross Abstract: The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning.

By Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu

fine-tuning multimodal benchmarks

arXiv AI

Jun 26

Reconstruction Alignment Improves Unified Multimodal Models

arXiv:2509. 07295v4 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture.

By Ji Xie, Trevor Darrell, Luke Zettlemoyer, XuDong Wang

rag diffusion multimodal benchmarks safety

Get the daily briefing