Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Computation and Language
Sep 11

Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue

The paper introduces Inverse Turing Bench, a benchmark designed to assess how well language models can distinguish between human-only and human-AI dialogues in multi-turn text. It provides paired dialogue transcripts and evaluates models on correctly identifying the type of conversation. Preliminary results show GPTZero, Claude Opus-4.6, and GPT-5.5 achieving the highest accuracies of 89.41%, 77.92%, and 75.94% respectively, highlighting both the strengths and limitations of statistical versus semantic detection approaches.

By William Hager, Ishika Rathi, Masum Hasan, Cameron Jones
arXiv Computation and Language
Sep 11

On the Impact of Anonymization on the Performance of Large Language Models

The paper systematically studies how anonymizing input data affects large language models (LLMs). Five prominent LLMs were evaluated on eleven benchmarks, comparing performance on original versus pseudonymized inputs. Results show that anonymization generally degrades performance, with larger drops for more capable models and task-dependent effects; reversible anonymization preserves entity uniqueness better than irreversible redaction, and prompting about anonymization offers no benefit.

By Tobias Deu{\ss}er, Max Hahnb\"uck, Lorenz Sparrenberg, Tobias Uelwer, Christian Bauckhage, Rafet Sifa
arXiv Computer Vision
Sep 11

MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery

MHE-Former introduces a Transformer-based multi-hypothesis framework for monocular 3D hand and body mesh recovery, addressing occlusion and ambiguity by maximizing entropy during exploration. The method generates diverse, plausible mesh hypotheses efficiently and includes a context-aware hypothesis selection process that leverages visual language models and natural language intent. Experiments show state‑of‑the‑art accuracy and diversity, and a user preference study confirms the practicality of the selection mechanism.

By Boshu Jia, Rongyu Chen, Linlin Yang, Zihao Liu, Yingjie Chen, Zhongqun Zhang, Zhulin Tao, Shaohui Lin, Xiaoyu Wu, Libiao Jin, Baochang Zhang, Angela Yao
arXiv Computer Vision
Sep 11

TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs

TrajFusionNet+ is a transformer-based model that predicts pedestrian crossing intention by fusing sequential trajectory data, visual trajectory overlays, and graph-based scene context. It extends the earlier TrajFusionNet with three attention modules—Sequence, Visual, and Graph—to capture temporal, visual, and relational cues. The model outperforms state‑of‑the‑art methods on the PIE and JAAD datasets and shows better generalization under a joint‑training, separate‑evaluation protocol.

By Fran\c{c}ois G. Landry, Moulay A. Akhloufi
arXiv Computer Vision
Sep 11

FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation

FreeFlow is a hierarchical transformer for optical flow estimation that eliminates traditional flow-specific inductive biases such as correlation volumes, feature warping, and iterative refinement. It relies on a single feed-forward encoder–decoder architecture that integrates window attention for local processing, shifted-window attention for cross-window communication, and global attention at reduced resolution. This design allows the model to scale naturally with capacity, achieving state-of-the-art performance on Sintel, KITTI-2015, and Spring benchmarks while remaining memory efficient at 1080p inference.

By Vladislav Bargatin, Alexander Yakovenko, Khaled Abud, Dmitriy Vatolin
arXiv Computer Vision
Sep 11

MMGait: Benchmarking and Unifying Gait Recognition across Heterogeneous Modalities

MMGait is a large‑scale multi‑sensor benchmark that aligns visible, infrared, depth, LiDAR, and radar observations at the sequence level, enabling evaluation of single‑modal, cross‑modal, and multi‑modal gait recognition. The study shows that modality rankings shift with probe conditions, cross‑modal alignment remains challenging, and fusion can yield complementary gains. To address the scalability issue of training separate experts, the authors propose Omni‑Modal Gait Recognition and its implementation, OmniGait++, which unifies all recognition settings within a shared identity space using modality‑specific front ends, a shared encoder, and an anchor‑guided fusion module. whyItMatters":"MMGait provides a common testbed for heterogeneous gait sensing and demonstrates that unified recognition across varying modality availability is feasible, offering a scalable alternative to task‑specific experts."

By Saihui Hou, Chenye Wang, Qingyuan Cai, Aoqi Li, Yongzhen Huang
arXiv Computer Vision
Sep 11

Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

The paper introduces Caption‑once, Frames‑on‑Demand (CFD), a budget‑aware edge‑cloud framework for long‑video understanding. CFD first runs a single offline captioning pass on the edge to build a dual‑track narrative index—an event‑level story skeleton and a clip‑level micro‑log—that is cached for future queries. At query time, a cloud‑side MLLM uses a Visual‑Need Router to decide whether to retrieve keyframes for perceptual questions, thereby limiting visual processing while preserving temporal structure in language space.

By Weitong Cai, Hang Zhang, Yukai Huang, Yiqiao Xie, Shan Gao, Jiankang Deng, Songcen Xu, Jifei Song, Zhensong Zhang
arXiv Computation and Language
Sep 11

Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment

The study examines whether financial sentiment tools that are validated against human labels also reliably predict market outcomes. Using a large corpus of securities class action messages linked to abnormal stock returns, the authors compare five sentiment instruments—VADER, Loughran‑McDonald, FinBERT, Twitter‑RoBERTa, and an LLM annotator—within a single pipeline. Results show that the alignment between human agreement and sentiment scores varies with sampling strategy and time horizon: conventional sampling favors same‑day associations, while fixed‑n panels yield similar correlations for both same‑day and one‑day‑ahead predictions, yet overall predictive rankings remain weak.

By AS Aravinthkakshan, Laven Srivastava, Harsh Nandwani
arXiv AI
Sep 11

Safe to Stop? Risk-Constrained Stopping for Sequential Clinical Diagnosis Agents

The paper introduces Cros, a risk‑constrained stopping layer for sequential clinical diagnosis agents that determines when to stop testing and make a diagnosis. Cros combines state‑wise error ranking, policy design on disjoint development splits, and exact tests of selective diagnostic error to provide finite‑sample guarantees. On a MIMIC‑derived abdominal‑pain benchmark, Cros achieves higher state‑error AUROC and lower selective error rates compared to baseline stopping methods, though its performance varies across development resplits.

By Yuexin Wu, Vasile Rus
arXiv AI
Sep 11

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

The paper introduces Trimmed Logit-Gap SFT (TrimSFT), a token-level reweighting strategy that adjusts supervised fine-tuning loss based on the logit gap between the correct token and its strongest competitor. TrimSFT trims supervision from tokens that are either already mastered (large logit gap) or poorly supported (small or negative logit gap), focusing learning on tokens with intermediate logit gaps. Experiments on six base models across five mathematical reasoning benchmarks show that TrimSFT consistently outperforms standard SFT, achieving the best average performance on five of six models and up to +26.9 points on MATH500.

By Yaning Jia, Chunhui Zhang, Wenxuan Xu, Xingjian Diao, Xiaoyuan Wang, Soroush Vosoughi
arXiv AI
Sep 11

LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

LexAgentHallu is a new benchmark that profiles hallucinations in legal agents across multi-step interactions. It contains 3,414 instances spanning 17 legal categories and 6 task types, each annotated with a dual-layer taxonomy of 7 high-level and 27 fine-grained hallucination categories. The benchmark introduces fine-grained metrics to quantify and localize failures along an agent’s execution path, revealing patterns such as the Right-Answer-Wrong-Reason effect and clustered hallucination subclasses.

By Yujin Zhou, Mingxuan Zheng, Chuxue Cao, Huang Yidan, Jiale Chen, Yike Guo, Sirui Han
arXiv AI
Sep 11

Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward

The paper introduces the concept of proof‑carrying cognition, aiming to close the verification gap in language‑model reasoning by using reality‑settled rewards. It presents a theoretical framework linking verifier‑gold correlation to compute‑capability trade‑offs, demonstrates that unsound verifiers degrade under best‑of‑N selection while sound verifiers improve, and proposes a new benchmark metric, Soundness‑under‑Pressure, for evaluating reality‑settled reasoning systems.

By Eshwar Reddy M, Sourav Karmakar
arXiv AI
Sep 11

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.

By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Sep 11

Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States

The paper introduces a reference‑based bias detection method that audits hidden‑state representations of language models by encoding sentences as similarities to a fixed set of anchor sentences. This relative representation allows comparison across model variants, such as before and after fine‑tuning, and yields a metric called Representational Bias Shift (ΔB). ΔB correlates strongly with output‑level bias changes, can detect bias‑increasing checkpoints with high ROC AUC, and is computationally efficient, requiring only a few minutes and far less compute than traditional benchmarks.

By Marek Jeli\'nski, Jan Dubi\'nski, Maciej Chrabaszcz, Sebastian Cygert
arXiv AI
Sep 11

JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

JarvisGUI is a new benchmark that tests GUI agents on cross-device workflows involving Android, Windows, and Ubuntu, requiring transfer of intermediate results and coordination across heterogeneous platforms. It formulates tasks as input-output transformations under a lightweight type system, enabling automatic composition of multi-step, cross-device workflows and dynamic evaluation within a unified framework. The benchmark reveals that state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a critical capability gap invisible to existing benchmarks.

By Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
arXiv AI
Sep 11

Compute-Bounded Security Assurance - Coverage, Verification, and Response under Resource Constraints

The paper introduces a resource‑constrained framework that distinguishes between repeated success, unique coverage, accepted evidence, and operational protection in security assurance tasks. It derives formulas for coverage under repeated, conditionally independent attempts, shows that pairwise outcome correlation does not necessarily limit coverage, and links coverage to evidence checking, scoring, resource accounting, and response modeling. The authors propose a defensive architecture separating evidence analysis, adjudication, and operational authority, and outline an evaluation protocol with held‑out tasks and uncertainty reporting.

By Jithin VG, Ditto PS
arXiv AI
Sep 11

Reliable Near-Field Multi-User Positioning Informed by Two-Stage MUSIC

The paper introduces MUSIC-Net, an end-to-end deep learning framework for near-field multi-user positioning that incorporates a two-stage MUSIC algorithm to isolate line-of-sight signal components and estimate surrogate distances. By embedding these MUSIC-derived objects into training, the method bypasses separate parameter estimation and path/source association, directly recovering user positions even in mixed LoS/NLoS multipath scenarios. Additionally, the authors employ split conformal prediction to provide statistically guaranteed confidence sets for each user’s position, achieving lower mean positioning error and tighter prediction regions compared to existing benchmarks.

By Jiaying Li, Haifeng Wen, Changsheng You, Yuanwei Liu, Hong Xing
arXiv AI
Sep 11

Cascading Gradient Inversion via LT-Code Inspired Peeling in Federated Learning

The paper introduces a new gradient inversion attack for federated learning that leverages concepts from erasure‑correcting codes to recover entire training batches and their labels from a single FedSGD round. Unlike previous analytic attacks, this method can exactly reconstruct batches of up to 128 samples on ImageNet and achieves over 90% recovery even when the attacker actively manipulates the model. The study demonstrates that federated learning’s privacy leakage is far greater than previously estimated.

By Saeed Shariati, Mohsen Alambardar Meybodi
arXiv AI
Sep 11

OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis

OmniMed‑FL is a multimodal federated learning framework that fuses chest radiographs and synthetic patient notes to classify five clinical conditions. The study benchmarks eight fusion strategies, three initializations, and four missing‑text imputation rules across 3–20 hospital clients under non‑IID Dirichlet partitioning, showing that federated approaches (FedAvg, FedProx, SCAFFOLD‑AdamW) outperform local‑only training. Multimodal fusion consistently improves performance, achieving macro‑F1 scores up to 0.956 on the synthetic corpus and 0.906 on the radiograph corpus.

By Ayush Debnath, Ruelia Saha, Sudip Misra