Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 21

Fine-Tuning Small Language Models for Reliable VASP INCAR Generation

arXiv:2608.05387v1 Announce Type: cross Abstract: Language models can prepare VASP INCAR files from natural-language requests, but so far only large proprietary cloud models come close to handling th...

By Xinyue Zhang, Jixiang Li, Bin Shao, Baishun Yang, Zhiyang Liu, Weichao Wang

llms fine-tuning efficiency benchmarks safety

arXiv AI

Oct 21

Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation

arXiv:2610.27515v1 Announce Type: cross Abstract: Pretrained Vision-Language Models (PVLMs) achieve strong zero-shot recognition but often suffer performance degradation under target-domain shift. Un...

By Hao Yan, Guanghui Wang

llms rag multimodal benchmarks

arXiv AI

Oct 21

Corpus Quality Improves, Retrieval Utility Does Not: An Ablation of Preprocessing and Refinement in a Speech-to-QA Pipeline

arXiv:2610.27652v1 Announce Type: cross Abstract: Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it...

By Wonsup Shin, Jingu Kim

llms benchmarks

arXiv AI

Oct 21

VLA-Retain: Benchmarking Cross-Dimension Effects of Vision Tuning in VLAs

arXiv:2610.27698v1 Announce Type: cross Abstract: Vision-language-action (VLA) models transfer knowledge from large-scale pretraining into robot control, but their manipulation performance remains se...

By Borui Xu, Han Chen, Bingsheng He, Ming Wu, Jason Zeng, Michael Heinrich

robotics fine-tuning multimodal benchmarks

arXiv AI

Oct 21

Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation

arXiv:2610.27868v1 Announce Type: cross Abstract: Code translation converts source code from one programming language to another while preserving its functionality. Although Large Language Models (LL...

By Yali Du, Yunji Zhang, Ming Li

llms benchmarks safety

arXiv Computer Vision

Oct 21

LEVI: Benchmarking Visual Identity Fidelity with Verifiable Low-Entropy Constructs

arXiv:2610.25716v2 Announce Type: replace Abstract: Modern generative models can produce visually plausible images while violating exact identity constraints. Evaluating such failures requires distin...

By Xiao Ma, Zhonghao Zhan, Hamed Haddadi, Qi Chen

diffusion benchmarks

arXiv AI

Oct 21

On-Policy Transition Distillation

arXiv:2610.28048v1 Announce Type: cross Abstract: On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on stud...

By Runyang You, Yongqi Li, Tiezheng Yu, Wenjie Li

efficiency benchmarks

arXiv AI

Oct 21

Assessing Representation Semantics Under Language Pretraining in Fine-Grained Classification

arXiv:2610.28051v1 Announce Type: cross Abstract: Understanding how supervision signals shape semantic visual representations remains challenging, particularly in fine-grained, low-data settings. Sta...

By Md Raisul Kibria, S\'ebastien Lafond

benchmarks

arXiv AI

Oct 21

CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models

arXiv:2610.28356v1 Announce Type: cross Abstract: Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many exis...

By Marc Chapus, Carlos Crispim-Junior, V\'eronique Eglin, Atilla Baskurt

llms multimodal benchmarks safety

arXiv AI

Oct 21

The Instruction Tax on Factual Recall: Base Models Can Beat Their Instruction-Tuned Counterparts

arXiv:2610.28407v1 Announce Type: cross Abstract: We present our solution to the AKBC LM-KBC 2026 Shared Task, where we achieved a score of 0.6836 on the official leaderboard, more than double the ba...

By Christophe Khalil, Aya Mourad

llms fine-tuning benchmarks

arXiv AI

Oct 21

WorldSR: Harnessing World Knowledge Search for Grounded Image Super-Resolution

arXiv:2610.28423v1 Announce Type: cross Abstract: Generative-prior-based super-resolution (SR) methods leverage implicit knowledge learned from large-scale models to enhance low-resolution inputs. Ho...

By Junyuan Deng, Xinyi Wu, Zhenyao Wu, Song Wang, Yu Wang, Congchao Zhu

benchmarks safety

arXiv AI

Oct 21

After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents

arXiv:2610.28425v1 Announce Type: cross Abstract: Modern LLM agents persist state across sessions via two channels: long-lived memory files silently prepended to the system prompt, and scheduled task...

By Zhiyue Xu, Xukai Wang, Zhiyuan Huang, Mingjie Zhan, Jian Zhao, Xu-Yao Zhang

llms agents benchmarks

arXiv AI

Oct 21

MindView: Benchmarking Generative Multi-View Spatial Intelligence

arXiv:2610.28622v1 Announce Type: cross Abstract: Spatial intelligence has recently been assessed through generative tasks such as spatially grounded image editing, yet language-guided generation of...

By Yanbo Wang, Siqi Luo, Siyu Zhang, Chenming Zhu, Wenzhao Zheng, Qian Yu, Juncheng Yan, Bin Fu, Tai Wang, Yu Qiao, Yihao Liu, Jiwen Lu

fine-tuning benchmarks

arXiv AI

Oct 21

How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

arXiv:2508.01300v2 Announce Type: replace Abstract: AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Mode...

By Ma'ayan Armony, Albert Mero\~no-Pe\~nuela, Gerard Canal

llms agents nlp benchmarks

arXiv AI

Oct 21

Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control

arXiv:2508.20784v3 Announce Type: replace Abstract: Bus bunching remains a challenge for urban transit due to stochastic traffic and passenger demand. Traditional solutions rely on multi-agent reinfo...

By Yifan Zhang, Liang Zheng, Qifan Zhang

agents reinforcement-learning benchmarks

arXiv AI

Oct 21

The Reward Model Selection Crisis in Personalized Alignment

arXiv:2512.23067v3 Announce Type: replace Abstract: Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption th...

By Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Oct 21

Agentic vs. Parametric World Models: Hybrid Planning for Reliable Language Agents

arXiv:2606.27806v4 Announce Type: replace Abstract: Language agents, such as ReAct, Reflexion, and RAP, implicitly predict how the world will change as they plan and act, using imagined state transit...

By Xinyuan Song, Zekun Cai

llms agents benchmarks

arXiv AI

Oct 21

IndustryForge-27B: A Domain-Enhanced Multimodal Foundation Model for Industrial CAD

arXiv:2607.28050v2 Announce Type: replace Abstract: Automating industrial CAD design and manufacturing places distinctive demands on multimodal foundation models: the model must see engineering drawi...

By Nianchen Deng, Jiaxin Ai, Tao Hu, Shu Zou, Yurui Dong, Siqi Li, Xinyu Cai, Xuemeng Yang, Licheng Wen, Hongbin Zhou, Hairong Zhang, Pinlong Cai, Botian Shi

llms agents fine-tuning multimodal benchmarks

arXiv AI

Oct 21

Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction

arXiv:2610.19432v2 Announce Type: replace Abstract: Comparing financial disclosures across languages requires identifying shared accounting concepts and recovering all numerical facts that express th...

By Lingfei Qian, Yan Wang, Xueqing Peng, Polydoros Giannouris, Nadhem Zmandar, Yi Han, Yanguang Liu, Sophia Ananiadou, V\'ictor Guti\'errez-Basulto

llms benchmarks

arXiv AI

Oct 21

Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants

arXiv:2610.20022v2 Announce Type: replace Abstract: Memory systems for persistent personal agents must move beyond isolated memories, maintaining both personal memory about whom they serve and proced...

By Maosheng Qin, Qi Zhang, Mingxuan Xia, Jinglei Chen, Jiexiang Wang, Sheng Guo, Cheng Peng, Haobo Wang

llms agents benchmarks

Get the daily briefing