← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning

Oct 27

On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution

arXiv:2610.32748v1 Announce Type: new Abstract: The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of hybrid deep generative models (hybrid-DG...

By Yubo Ye, Maryam Toloubidokhti, Sumeet Vadhavkar, Xiajun Jiang, Huafeng Liu, Linwei Wang

benchmarks safety

arXiv Machine Learning

Oct 27

A Dynamic Inventory Rationing Model for Last-Mile Delivery of Medication in Liberia's Community Health Program

arXiv:2610.32051v1 Announce Type: cross Abstract: We study the dynamic allocation of scarce medications in Liberia's community health program. A central warehouse supplies a health facility and the C...

By Baris Ata, Yufan Chen, Robert Montgomery

benchmarks safety

arXiv Machine Learning

Oct 27

Tabular Foundation Models for Credit Risk Prediction: A Dedicated Review and Empirical Comparison

arXiv:2605.18147v3 Announce Type: replace Abstract: Predictive models are central to credit risk management, where the accuracy of default-probability and loss estimates affects the profitability of...

By Bart Baesens, Andreas Goethals, Stefan Lessmann, Simon De Vos, Cristi\'an Bravo, David Martens, Victor Medina-Olivares, Christophe Mues, Maria Oskarsd\'ottir, Seppe vanden Broucke, Tony Van Gestel, Tim Verdonck, Wouter Verbeke

llms benchmarks

arXiv Machine Learning

Oct 27

How do standard time series forecasting benchmarks translate to finance? A broad study of risk adjusted performance

arXiv:2603.01820v2 Announce Type: replace-cross Abstract: Forecasting financial asset returns is challenging due to low signal-to-noise ratios, non-stationarity and heavy-tailed distributions. We ben...

By Adir Saly-Kaufmann, Kieran Wood, Jan-Peter Calliess, Stefan Zohren

llms benchmarks

arXiv Computation and Language

Oct 27

Words Are the Voice of the Mind: Depression Severity Estimation from Social Traces via Mental-Guided Evidence Reasoning

arXiv:2610.33586v1 Announce Type: new Abstract: Depression severity estimation is crucial for mental health assessment and intervention, yet traditional approaches depend on clinical interviews or se...

By Nada Xu, Xueqin Chen, Qiang Gao, Hongru Liang, Wenqiang Lei

llms benchmarks safety

arXiv Computation and Language

Oct 27

Schema-Free Text-to-SQL: Learning Enterprise Database Structure from Query Logs

arXiv:2610.33946v1 Announce Type: new Abstract: Text-to-SQL systems achieve remarkable accuracy on academic benchmarks, yet performance drops dramatically in enterprise deployments. This gap stems fr...

By \"Omer Faruk Akg\"ul, Nikhil Kapahi, Prajit Reddy Muppidi

rag benchmarks data-engineering

arXiv Computation and Language

Oct 27

Does the Outfit Change the Price? Context Halo in Vision-Language Model Valuation of an Identical Low-Cost Necklace

arXiv:2610.34591v1 Announce Type: new Abstract: A chunky gold-tone chain necklace bought online for 2.43 USD was photographed four ways: worn with formal attire against a wood-panelled wall, worn at...

By Brianne Lee

llms multimodal benchmarks

arXiv Computation and Language

Oct 27

iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

arXiv:2610.34735v1 Announce Type: new Abstract: An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise with...

By Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols, Eldon Schoop

agents benchmarks

arXiv Computation and Language

Oct 27

When the Persona Contradicts the Line: Diagnosing Plan-to-Synthesis Failures in Chain-of-Thought TTS

arXiv:2610.35008v1 Announce Type: new Abstract: Chain-of-thought TTS systems rely on language-model planners to translate persona and context into vocal delivery, but it is unclear where this intent...

By Joonyong Park, Jerry Li

arXiv Computation and Language

Oct 27

SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning

arXiv:2610.35042v1 Announce Type: new Abstract: Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have a...

By Haotian Tan, Hiroki Ouchi, Sakriani Sakti

llms reinforcement-learning benchmarks

arXiv Computation and Language

Oct 27

Ask Before You Answer: A Question-Driven Evidence Reasoning Architecture (QERA) for In-Context Learning

arXiv:2610.35083v1 Announce Type: new Abstract: AI assistants are increasingly asked to reason over a user's own web traces: group chats, community threads, activity logs and revision histories. On C...

By Jinhu Qi, Minda Hu, Wentao Zhang, Zixing Song, Irwin King

llms benchmarks

arXiv Computation and Language

Oct 27

Do Language Models Understand Igbo Culture? Beyond Multiple Choice

arXiv:2610.35168v1 Announce Type: new Abstract: The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use th...

By Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad

llms benchmarks safety

arXiv Computation and Language

Oct 27

ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation

arXiv:2610.35416v1 Announce Type: new Abstract: Text-in-image translation aims to translate all textual content embedded in an image. In Text Image Translation (TIT), the output is translated text; I...

By \v{S}\'arka Uramov\'a, Petar Kirilov Petrov, Josef Jon, Ond\v{r}ej Bojar, Michal Nov\'ak

llms computer-vision nlp multimodal benchmarks

arXiv Computation and Language

Oct 27

MANTIS: Multi-Axis News Transformations for Instance-Wise Stress Testing and Worst-Case Training in Fake News Detection

arXiv:2610.35475v1 Announce Type: new Abstract: Automated fake news detection is vital on social media, where fake news is easily rewritten to fool detectors. However, existing approaches typically r...

By Dexin Liu, Fan Yang, Zimu Wang, Xunfei Zhu, Bolin Ren, Bingyan Zhang, Wei Wang, Qi Chen

benchmarks safety

arXiv Computation and Language

Oct 27

EXCISE: Locating and Excising Distracting Sentences in Medical Question Answering

arXiv:2610.35558v1 Announce Type: new Abstract: Language models are increasingly considered for clinical decision support, where their answers should be grounded in clinical evidence and remain robus...

By Yerim Hong, Sohyun Son, Heeseung Chung, Mujeen Sung

llms nlp benchmarks

arXiv Computation and Language

Oct 27

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks

arXiv:2610.35671v1 Announce Type: new Abstract: As frontier language models grow increasingly capable and autonomous, independent third-party evaluations are critical for characterizing their systemi...

By Yao Huang, Zhiyuan Zeng, Yu Wang, Zhitong Zhu, Dingcheng Duan, Hao Li, Shangzhi Xu, Yifan Wang, Jinzhe Tu, Shukun Xiong, Yihe Zang, Han Yan, Hongchang Yang, Fuyu Gu, Wei Xu, Yinpeng Dong, Xiaojian Li

llms agents robotics benchmarks safety

arXiv Computation and Language

Oct 27

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

arXiv:2610.35698v1 Announce Type: new Abstract: For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still...

By Harine Choi (Texas A&M Transportation Institute, Texas A&M University System), Chenchen Kuai (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Zihao Li (Department of Civil, Construction and Environmental Engineering, Marquette University), Jiwan Jiang (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Yunlong Zhang (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Zhiwen Fan (Department of Electrical and Computer Engineering, Texas A&M University), Zhengzhong Tu (Department of Computer Science and Engineering, Texas A&M University), Yang Zhou (Zachry Department of Civil and Environmental Engineering, Texas A&M University)

llms multimodal benchmarks

arXiv Computation and Language

Oct 27

SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy

arXiv:2610.35743v1 Announce Type: new Abstract: Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but...

By Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Jialin Song, Qiyao Ma, Chengquan Guo, Carolyn Buractaon, Christopher White, Jianfeng Gao

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Oct 27

Can Language Models Learn While They Think? Testing Gradient Adaptation Across Architectures

arXiv:2610.33025v1 Announce Type: cross Abstract: Letting a language model take gradient steps on the text it is reading, known as dynamic evaluation, lowers its loss on the rest of the document. Its...

By Surya Saka

llms diffusion benchmarks

arXiv Computation and Language

Oct 27

LLMs Can Tackle High Energy Theoretical Physics, yet Gaps Remain: Part I, Quantum Field Theory

arXiv:2610.33134v1 Announce Type: cross Abstract: Beyond math and code, Large language models (LLMs) are beginning to tackle problems that have traditionally required substantial training for theoret...

By Tong Liu, Bohan Li, Gongrui Cheng, Zixiong Yu, Yulan He, Volker Tresp

llms benchmarks