Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 18

Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.

By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
arXiv Computation and Language
Sep 18

Towards Proactive Detection of User-Side Implicit Conflicts in Human-LLM Dialogue

The paper introduces UC-Bench, a human‑annotated benchmark for detecting user‑side implicit conflicts in Human‑LLM dialogue, a problem largely overlooked compared to LLM‑side conflicts. Experiments show current LLMs struggle with these conflicts, especially when they stem from implicit incompatibilities in dialogue history. To address this, the authors propose SynUC, a constraint‑guided data synthesis method that generates a new training set, UC‑Data, which improves performance of lightweight LLMs on UC‑Bench compared to larger general‑purpose models and existing synthesis approaches.

By Jinqiang Wang, Tao Zhu, Huansheng Ning
arXiv Computation and Language
Sep 18

Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

The paper introduces the Profiling, Investigation, and Judgment (PIJ) benchmark, which contains 2,500 real homicide cases from five countries to evaluate large language models (LLMs) on pre‑arrest criminal investigation tasks. It assesses LLMs across criminal profiling, crime process reconstruction, and sentence prediction, revealing that performance drops as tasks require more implicit reasoning about unknown suspect profiles. The study finds that LLMs lag behind human experts, especially on inferential tasks like motivation and victim‑offender relationships, and exhibit biases in gender, age, and motive attribution.

By Yutong Yao, Yanjie Cao, Guanhua Chen, Xu Yang, Junchao Wu, Zeyu Wu, Lidia S. Chao, Derek F. Wong
arXiv Computer Vision
Sep 18

PerSeM: Persistent Semantic Memory for Long-Horizon Open-Vocabulary UAV Mapping

PerSeM is a training‑free framework that builds a persistent semantic memory for long‑horizon UAV mapping by associating frame‑wise segmentation results with world‑space voxels and refining them through spatial refinement, trust‑aware replay, and context‑guided verification. Experiments on Forest and UAVScenes benchmarks show that this persistent 3D memory improves semantic correctness and temporal stability compared to frame‑wise predictions, especially in semantically difficult and temporally unstable regions. The method achieves these gains without retraining or additional neural‑network inference.

By Saurbh Singh Jamwal, Ganesh Ramakrishnan
arXiv Computer Vision
Sep 18

FlowSGS: Improving Flow Matching Priors for Inverse Imaging with Stochastic Interpolants

FlowSGS introduces a flow-based posterior sampling method that combines Split Gibbs Sampling (SGS) with Langevin dynamics for the likelihood step and Stochastic Interpolants (SI) for the prior step. By integrating a pretrained flow model into the prior step via SI's reverse-time SDE and a novel timestep correction, FlowSGS reduces the number of network evaluations compared to plug‑and‑play diffusion samplers. Experiments demonstrate state‑of‑the‑art performance on various inverse problems, including the first flow‑based solution to a nonlinear inverse problem (Fourier phase retrieval).

By Tianao Li, Xinhui Qian, Emma Alexander
arXiv Machine Learning
Sep 18

Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning

Uni-LaDiR (Unified Latent Diffusion Reasoner) is a new framework that unifies multimodal reasoning by mapping teacher reasoning steps from different modalities into a shared latent space of thought tokens. It employs a diffusion model to predict the next block of thought tokens, jointly training the encoder and reasoner with shared weights to ensure tokens are both useful and predictable. The approach achieves relative gains of 7.3% on visual reasoning benchmarks and 6.1% on robot manipulation tasks compared to the strongest baselines.

By Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Yian Ma, Lianhui Qin
arXiv AI
Sep 18

Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies

The paper introduces SecTB-RTL, an auditable framework for evaluating AI-generated RTL verification plans against 31 tasks and 124 hardware‑security regressions. In a confirmatory run, the AI model’s responses were rejected by the provider’s schema, and after a schema‑only repair, only nine of 1,857 accepted responses passed the production semantic validator, revealing a mismatch between generation and execution rules. The study demonstrates that schema acceptance does not guarantee execution validity and provides a benchmark, failure‑preserving contract, incident provenance, and governance controls to prevent misreporting of infrastructure behavior as model behavior.

By Hang Xiao, Chuhong Xu, Kainan Zhou, Gangzhen Qian, Lu Yi
arXiv Computer Vision
Sep 18

Open ultrasound foundation model for robust segmentation and clinical measurement across heterogeneous settings

The paper introduces SonoCorpus, an open dataset of 456,963 ultrasound images with 1,626,085 expert masks from 53 public sources across 24 clinical applications and 17 countries, and SonoBase, an interactive segmentation foundation model pretrained on this data. SonoBase outperforms existing models (SAM2, MedSAM2, MedSAM3) on fifteen diverse evaluation datasets, matching specialist models and achieving clinically relevant accuracy for metrics such as ejection fraction, fetal head circumference, and gestational age. The authors provide full reproducibility resources, including checkpoints, optimizer states, and starter code, to enable community adoption and further development.

By Chao Qin, Fahad Shahbaz Khan, Salman Khan, Sarim Ather, Siddiq Anwar, Rao Muhammad Anwer, Shadab Khan
arXiv Computation and Language
Sep 18

PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents

PersonalAI 2.0 (PAI‑2) is a new framework that improves large language model (LLM) systems by integrating external knowledge graphs through a dynamic, multistage query‑processing pipeline. It performs adaptive, iterative information search guided by extracted entities, matched graph vertices, and generated clue‑queries, and has been evaluated on five benchmarks (Natural Questions, TriviaQA, HotpotQA, 2WikiMultihopQA, and MuSiQue). The results show a 9% average gain in factual correctness over comparable methods, an 18% boost from the search‑plan enhancement mechanism, and state‑of‑the‑art performance on the MINE‑1 benchmark, indicating that PAI‑2 effectively reduces hallucinations and increases precision for personalized AI applications.

By Mikhail Menschikov, Matvey Iskornev, Alexander Kharitonov, Alina Bogdanova, Mikhail Belkin, Ekaterina Lisitsyna, Artyom Sosedka, Victoria Dochkina, Ruslan Kostoev, Ilia Perepechkin, Evgeny Burnaev
Hugging Face Trending Papers
Sep 17

Large Language Models as Falsifiers for Cyber-Physical Systems

The paper introduces LLM-Falsifier, a method that uses large language models to find counterexamples in cyber‑physical systems by minimizing Signal Temporal Logic robustness. By providing the LLM with natural‑language context, trajectory outputs, and critical‑time witnesses, the approach achieves smarter, more sample‑efficient searches. On ARCH‑COMP benchmarks, LLM‑Falsifier outperforms existing tools across 14 of 21 specifications in terms of simulations needed to locate a counterexample.

Hugging Face Trending Papers
Sep 17

Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

The paper investigates test‑time adaptation for medical image segmentation, showing that a fixed adaptation horizon can harm many individual cases. It introduces prediction fragmentation—a measure of disagreement between the source model and the adapted mask—to predict harmful adaptation without extra labels or backward passes. Using a case‑level router based on this metric, the authors reduce harmful adaptation on cardiac MRI from 58.7% to 20% while maintaining accuracy.

Hugging Face Trending Papers
Sep 17

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Chronicle introduces a method called cut‑point replay to make regression testing of large language model (LLM) agents reproducible. It records an agent run at non‑deterministic boundaries as immutable envelopes and replays selected boundaries while executing the rest live, turning recorded failures into continuous‑integration tests. Benchmarks show minimal overhead, bit‑stable full replay, and effective detection of faulty code changes in a mutation study.

Hugging Face Trending Papers
Sep 17

DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering

DocAttriBench (DAB) is a large-scale benchmark that provides fine-grained, element-level source attribution for Document Visual Question Answering (VQA). It introduces MAPPET, a Mask-based Perplexity-Derived Attribution method that uses document layout and language modeling to identify the most informative layout element for each answer. The benchmark contains 237k documents and 296k question-answer pairs with grounding annotations, and it evaluates multimodal LLMs on answer accuracy, attribution accuracy, and overall answer quality.

Hugging Face Trending Papers
Sep 17

When Do Language-Grounded Explanations Help? A Graph-Bottleneck for Farm Monitoring Interpretable Sheep Facial Pain

The paper investigates whether language‑grounded explanations improve trust in automated sheep pain detection from facial expressions. It finds that attention‑based explanations tied to the Sheep Pain Facial Expression Scale (SPFES) are largely ineffective, prompting the authors to replace the appearance bypass with a concept bottleneck that uses SPFES concept scores supervised by per‑region annotations. This new architecture slightly reduces performance but yields more semantically meaningful concepts, recovering minority pain states and revealing ear‑and‑eye severity orderings without explicit severity supervision, and highlights the pitfalls of pooled concept accuracy under clinical imbalance.

Hugging Face Trending Papers
Sep 17

COMPASS: Ordered Clustered Routing at 100K Scale

COMPASS is a new algorithm for the Ordered Clustered Traveling Salesman Problem (OCTSP) that combines search with learning‑accelerated routing by orchestrating parallel sub‑solvers. It exploits the clustered structure to achieve exact solutions in time exponential in cluster size rather than instance size, and its solutions improve continuously with more compute. Empirically, COMPASS outperforms existing methods and scales to 100K synthetic nodes and 28.5K real e‑commerce nodes, representing the largest reported routing solution over asymmetric distances—9× beyond established ATSP benchmarks.

Hugging Face Trending Papers
Sep 17

AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images

AgriScope is a unified pixel‑grounded multimodal framework designed for agricultural image understanding, supporting image‑level, region‑level, and pixel‑level tasks such as grounded caption generation, referring expression segmentation, and multi‑turn multimodal interaction. It incorporates biologically specialized semantic representations with dense spatial grounding through biological‑semantic encoding, dense spatial representations, and pixel decoding. The authors also introduce AgriGround, a large‑scale dataset of over 500K images and 11M instruction‑following samples, created via an automatic annotation pipeline that combines caption generation, phrase‑level grounding, segmentation mask generation, and task‑oriented instruction synthesis to provide densely grounded supervision for agricultural vision‑language learning.

Hugging Face Trending Papers
Sep 17

To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals

The paper introduces SwitchSD, an adaptive framework that treats copying as a latent control signal in large language model decoding. By training lightweight probes on internal representations, SwitchSD accurately detects genuine copy intent (AUC > 0.99) and dynamically switches between neural drafting and context-based copying. Experiments on Llama and Qwen models show up to 15 % throughput gains over state‑of‑the‑art baselines such as EAGLE3.

Hugging Face Trending Papers
Sep 17

Think Thrice Before Reranking: Multi-perspective Evidence and Reasoning Integration for Text Reranking

MERIT‑Rank introduces a multi‑trajectory reasoning framework for text reranking, combining complementary reasoning paths to enhance robustness against errors. It defines a Multi‑Trajectory Reasoning Space (MTRS) and a joint reranker that merges these perspectives into a single ranking decision. The Progressive Rank Policy Optimization (PRPO) training scheme stabilizes reasoning trajectories and progressively improves ranking quality, yielding superior performance on both reasoning‑intensive and traditional retrieval benchmarks, with a 4B model outperforming larger rerankers on BRIGHT.

Hugging Face Trending Papers
Sep 17

Solving Minimum Span Antibandwidth and Cyclic Antibandwidth Labeling Problems

The paper introduces the Minimum Span Antibandwidth and Cyclic Antibandwidth Labeling (MSABL/MSCABL) problems, which fix a minimum (cyclic) distance between labels of adjacent vertices and aim to minimize the overall label span. A unified Boolean Satisfiability (SAT) framework is developed, formulating the problems as a sequence of decision problems and exploiting monotonicity to accelerate search. Two SAT solving strategies—parallel and incremental—are evaluated on benchmark instances, showing that SAT-based approaches are highly competitive with commercial solvers, especially for MSCABL.