Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 21

LEAF: LUT-aware Efficient Attention on FPGAs

arXiv:2610.27078v1 Announce Type: new Abstract: We introduce Lookup-table (LUT)-aware Efficient Attention on FPGAs (LEAF). LEAF is the first nanosecond-scale LUT-aware transformer for FPGAs. Deployin...

By Arianna Cox, Chang Sun, Lauri Laatu, Benedikt Maier, Aadeesh Sharma, Leo Rozanov, Wayne Luk, Maria Spiropulu, Alexander Tapper, Zhiqiang Que
arXiv Computer Vision
Oct 21

PURE: Refining Feed-forward 3D Reconstruction with Unlabeled Post-training

PURE is an unlabeled post‑training refinement framework that enhances feed‑forward 3D reconstruction models by improving depth and camera estimation without ground‑truth supervision. It distills structural knowledge from state‑of‑the‑art 2D priors into 4D architectures, using relative depth constraints, affine‑invariant geometry distillation, and pose refinement via feature correspondences. Experiments on synthetic and real‑world dynamic benchmarks show consistent improvements across multiple state‑of‑the‑art models, effectively mitigating domain shifts and unlocking their architectural limits in challenging in‑the‑wild scenes.

By Yaolong Zhu, Peng Li, Jiahao Lu, Chengfeng Zhao, Mengfei Li, Chao Wang, Wenyue Chen, Zekai Gu, Chengwei Ren, Ronggang Wang, Yuan Liu
arXiv Computation and Language
Oct 21

OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants

OptionWorld is a new benchmark that tracks how large language models handle long‑term personal assistant tasks, focusing on information gathering, decision making, actions, and outcomes over simulated daily life scenarios. The study finds that in 35.5% of failed cases the key facts are still present in the model’s last input, and identifies three main failure modes: missing or uncombined information, unused action opportunities, and unnecessary restrictions imposed by the agent. Follow‑up experiments show that providing missing facts, adding action reminders, and correcting erroneous restrictions can improve task success, highlighting gaps in state maintenance, deadline adherence, and assumption handling.

By Ruoyu Zhang, Zhaocen Liu, Weiyi Tian, Melissa Feng, Shilong Liu
arXiv Computation and Language
Oct 21

NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification

NarraSpace is a multimodal framework that links words and speech times to specific picture elements in picture‑description dementia classification tasks. It retrieves utterances for predefined scene events, summarizes coverage and temporal organization, and uses a frozen language model to contrast event text between Alzheimer’s disease and healthy controls, incorporating transcript surprisal and coverage metrics. On the Alzheimer’s Dementia Recognition through Spontaneous Speech benchmark, a support vector classifier built on these features achieves about 81% accuracy and 91% AUC, while the system preserves source records that map event scores back to utterances, speech intervals, and lexical matches.

By Qingming Li, Ruiyan Sun, Satoshi Nakamura
arXiv AI
Oct 21

PROVISO: Provenance-Informed Action Selection for Secure LLM Agents

PROVISO is a controller that selects which action an LLM agent should commit after the agent proposes an action and before execution. It encodes trusted and external observations, the proposed action, and an alternative into provenance-typed memory, uses role‑specific queries to relate them, and commits the action with the highest calibrated value learned from outcomes. In four agent security tasks, PROVISO outperforms all baselines in both security and utility, improving F1 by 13.84 points on SkillsBench and 7.07 points on ToolSafe, and achieving 88–99 F1 on three additional skill detection benchmarks.

By Bo Yan, Qi Zhang, Ziwen Kan, Fnu Suya, Song Wang
arXiv Machine Learning
Oct 21

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

arXiv:2610.27444v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) attaches small low-rank matrices to a frozen backbone, and public repositories host many LoRAs per backbone, each for a di...

By Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane, W. K. Alwis, Jason Mars, Udaya Sampath K. Perera Miriya Thanthrige, Ranga Rodrigo
arXiv Machine Learning
Oct 21

Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization

arXiv:2505.23442v2 Announce Type: replace Abstract: Knowledge graphs (KGs) provide structured representations of entities and relations, and underpin data management and Web applications such as sear...

By Linyu Li, Zhi Jin, Yuanpeng He, Dongming Jin, Haoran Duan, Zhengwei Tao, Xuan Zhang, Jiandong Li, Huanyao Zhang, Gadeng Luosang, Nyima Tashi
arXiv Machine Learning
Oct 21

STRABLE: Benchmarking Tabular Machine Learning with Strings

arXiv:2605.12292v2 Announce Type: replace Abstract: Benchmarking tabular learning has revealed the benefit of dedicated architectures, pushing the state of the art. But real-world tables often contai...

By Gioia Blayer, Myung Jun Kim, F\'elix Lefebvre, Lennart Purucker, Alan Arazi, Eilam Shapira, Roi Reichart, Frank Hutter, Marine Le Morvan, David Holzm\"uller, Ga\"el Varoquaux