Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 21

ShlokaDhara: A Multi-Speaker Sanskrit Speech Corpus for Learning Meter-Aware Prosody in Text-to-Recitation

arXiv:2610.26929v1 Announce Type: cross Abstract: Sanskrit verse recitation is governed by $\textit{Chandas}$, a strict prosodic framework defined by rhythmic $\textit{Laghu}$ (light) and $\textit{Gu...

By Harshal Kausadikar, Onkar Kishor Susladkar, Prathmesh Santosh Choudhari, Vedangi Deepak Deshpande, Ismini Lorentzu, Sparsh Mittal
arXiv AI
Oct 21

Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment

The paper introduces AuthorizationBench, a benchmark designed to evaluate whether large language model (LLM) agents act within the scope of user authorization when acting on their behalf. The benchmark comprises 286 executable tasks across ten application domains, covering situational triggers, presence or absence of direct harmful consequences, and varying reasoning effort. In experiments with four models and multiple agent harnesses, agents violated authorization in 19.2% of trajectories, with rates ranging from 10.5% to 26.6% across configurations, and increased reasoning effort did not substantially reduce these violations.

By Zhiping Zhang, Franklin Mingzhe Li, Zihao Huang, Tianshi Li
arXiv Computation and Language
Oct 21

When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

arXiv:2610.24993v2 Announce Type: replace Abstract: The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional accept...

By Yongchao Long, Chunrui Zheng, Yuxuan Zhou, Guangkun Nie, Tian-Chi Ma, Chao Pang, Shuohuan Wang, Yuchen Ding, Xianbin Wen, Dianhai Yu, Junyuan Shang, Yuxi Zhou, Shenda Hong
arXiv Computation and Language
Oct 21

Explore More, Validate Less: Efficient Reflective Prompt Optimization

The paper introduces a two‑stage framework for reflective prompt optimization that separates candidate generation from validation. In the construction stage, multiple parents are sampled and their training responses reused across parallel reflection branches, with each revision screened on a small minibatch to allow locally improved candidates to propagate without full validation. In the exploration stage, a fixed full‑validation budget is used to select the best prompt from a frozen candidate pool, enabling independent control of pool size and validation effort. Experiments on five benchmarks with Qwen3.5‑9B and Llama‑3.1‑8B‑Instruct show that retaining 100 candidates and performing 20 full validations can improve performance by up to 11.22 percentage points over the initial solver, and learned selection consistently outperforms the initial solver across all comparisons.

By Xiaofan Zhou, Lu Cheng
arXiv AI
Oct 21

Fix, Not Just Resample: Bounded Compiler-Guided Repair in Lean 4

The paper presents a bounded, online step-level repair module for Lean 4 that uses compiler error messages to fix rejected tactics instead of discarding them. By sampling up to $K$ candidate tactics and allowing at most $R$ repairs per step, the method improves solve rates over a sampling-only baseline, achieving up to a 32% relative gain and solving 82 additional theorems on a 1,990-theorem subset of the LeanDojo benchmark. The approach trades extra runtime for reachable proofs, with most benefit coming from early repair attempts and shallow tactic failures, and it successfully replicates on the Lean Workbook benchmark, nearly doubling its solve rate without regressions.

By Haotian Deng, Chris Farber, Sonal Narasimha, Samskrithi Sivakumar, Yexiang Xue
arXiv AI
Oct 21

Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing

The paper investigates how language models fail to detect harmful intent after polarity-altering transformations, a vulnerability called polarity blindness. It examines grammatical negation, antonym replacement, and synonym replacement as controls, and tests the impact of adversarial wrappers. Across 260 prompts, 10 LLMs, and three languages, polarity blindness is widespread, with the most vulnerable model scoring an average harm level of 4.12 on a 0‑5 scale.

By Diego Miguel Lozano, Erik Derner, Juan Antonio P\'erez-Ortiz, Nuria Oliver
arXiv AI
Oct 21

Benchmarking Arabic Dialects for Math Problems

arXiv:2610.28607v1 Announce Type: cross Abstract: Mathematical reasoning remains underexplored for low-resource languages, including Dialectal Arabic. To bridge this gap, we introduce the DAR-MGSM (D...

By Noor Abo Mokh, Teresa Lynn, Nizar Habash, Ahmed Elshabrawy, Sara Alhajeri, Alaa Elsetohy, Preslav Nakov, Alham Fikri Aji
arXiv Computation and Language
Oct 21

WaveCommitments: Private and Verifiable Speech-Agent Marketplaces

WaveCommitments is an open‑source framework that enables private and verifiable marketplaces for autonomous speech agents. It combines confidential asset valuation using Data Shapley within a trusted execution environment with multivariate polynomial commitments, allowing selective openings and tamper‑evident provenance. Experiments on three datasets and state‑of‑the‑art encoders, including Whisper, show up to 20× faster benchmarks and 39× smaller proofs, while also revealing how speaker value varies across tasks and encoders.

By Oguzhan Baser, Kaan Kale, Sriram Vishwanath, Sandeep Chinchali
arXiv AI
Oct 21

HierDTA: Hierarchical Structure-Aware Representation Learning for Drug--Target Affinity Prediction

HierDTA is a hierarchical structure‑aware framework for drug–target affinity prediction that represents drugs as atom–motif heterogeneous graphs and proteins as residue interaction graphs. It integrates atom and residue representations via cross‑attention, supplemented by molecule fingerprints and pretrained ESM‑2 embeddings, achieving superior performance on the Davis and KIBA datasets and strong generalization in cold‑start scenarios. Ablation studies and interpretability analyses confirm the contributions of its hierarchical augmentation and interaction components, and case studies demonstrate early enrichment of candidate bioactive compounds.

By Huangchao Xu, Baohua Zhang, Qian Liu, Zhong Jin, Yi Du