Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 7

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

The paper introduces QRAKEN, a training‑free, ontology‑agnostic neurosymbolic pipeline that grounds natural‑language queries to RDF knowledge graphs using empirical graph evidence instead of schema expectations. QRAKEN distills a compact TTQL representation offline, then uses it online to guide large language models, providing deterministic checks and an iterative refinement loop. On the CK25 benchmark, QRAKEN achieves a strict F1 of 0.643–0.652 with GPT‑4.1 mini and GPT‑5.4, outperforming state‑of‑the‑art systems and demonstrating the effectiveness of empirical pattern distillation over schema‑only approaches.

By Remo Grillo, Lukas Klic, Giovanni Colavizza
arXiv AI
Oct 7

Contrastive Learning for Aspect Representation towards Explainable Recommendation

The paper introduces CLARER, a recommendation model that fuses aspect features extracted from textual reviews with rating data to enhance recommendation accuracy and explainability. It learns user and item representations by combining rating-based features via an MLP and aspect-based features via a transformer encoder with contrastive learning. A transformer decoder then generates explanations using the combined representations, and experiments on three benchmark datasets show superior performance over baseline methods in both recommendation accuracy and explanation generation.

By Emrul Hasan, Chen Ding
arXiv AI
Oct 7

Tool-calling retrieval versus vector RAG for a small Greek--English knowledge base: accuracy and robustness to how users type Greek

The study compares two methods for grounding assistants in a small Greek–English agricultural knowledge base: tool‑calling retrieval via a live data interface and vector retrieval‑augmented generation (RAG). Using the KyGround benchmark of 198 questions, vector RAG achieved 95.3% accuracy on canonical Greek questions, outperforming the tool agent’s 71.6% and revealing that the tool agent’s failures stem from literal searches that miss non‑verbatim matches. The results show that search tolerance to user typing variations—such as accents, capitalization, and Greeklish transliterations—is crucial for reliable community knowledge interfaces.

By Nikolaos D. Tantaroudas, Ilias Karachalios, Andrew J. McCracken
arXiv AI
Oct 7

Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution under Analysis Budgets

The paper introduces a cost‑aware Hierarchical Multi‑Agent System (HMAS) for ransomware detection and family attribution that prioritizes static analysis and escalates to dynamic and memory analysis only when necessary, thereby reducing analysis time and resource usage. In experiments on 12,439 samples from 16 ransomware families, the deterministic HMAS achieved high F1 scores (0.93) while resolving nearly 58% of cases with static evidence alone and cutting average internal analysis cost by 44.6% compared to exhaustive methods. The system also records a complete provenance trace for each decision and includes optional local LLM review for limited verdict adjustment.

By Mubashar Iqbal, Asifullah Khan, Hifsa Asif, Saddam Hussain Khan, Umme Zahoora, Asifullah Khan
arXiv Machine Learning
Oct 7

When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO

The paper introduces Guidance-Augmented GRPO (GA‑GRPO), a theoretical framework that unifies several external‑guidance methods for reinforcement learning with verifiable rewards (RLVR) used to improve large language model reasoning. GA‑GRPO treats external guidance as a stochastic operator that rewrites the question distribution, yielding a biased on‑policy policy‑gradient estimator whose bias is bounded by the total‑variation guidance divergence. The authors prove convergence rates, derive an optimal guidance‑weight formula, and validate their predictions experimentally on a math‑reasoning benchmark, showing that the optimal‑weight GA‑GRPO outperforms existing methods while saving GPU time.

By Sofia Torres, Gabriel Almeida, Carter Adams, Camila Rocha
arXiv Machine Learning
Oct 7

Fed-BRDECS: Privacy-Preserving and Heterogeneity-Aware Federated Deep Embedded Clustering

Fed-BRDECS is a federated deep embedded clustering framework that protects client privacy by replacing globally normalized clustering objectives with a locally computable sample‑stability loss, eliminating the need to share sensitive soft‑assignment distributions. It addresses non‑IID client data through prediction‑balanced sampling, which oversamples rare predicted clusters, and centroid‑level restarting, which refreshes biased or inactive centroids. Experiments on image, text, and time‑series datasets demonstrate that Fed‑BRDECS outperforms existing federated and deep clustering methods while adding no inference‑time cost for anomaly detection.

By Haemin Park, Diego Klabjan, Martin W. Braun, Xiuqi Li, Balakrishnan Ananthanarayanan
arXiv Machine Learning
Oct 7

Improving Synthetic Data Generation for Argument Mining via Adversarial Reinforcement Learning

The paper introduces an adversarial reinforcement learning framework to generate synthetic data for Argument Mining (AM). By jointly training a generator and a discriminator, the system produces structured AM instances that are both accurate and diverse. Experiments show consistent performance gains on three benchmark datasets in both full-data and low-resource scenarios.

By Zhijun Zhang, Qianlong Wang, Keyang Ding, Genan Dai, Bowen Zhang, Bin Liang, Ruifeng Xu, Yongsheng Liang
arXiv Machine Learning
Oct 7

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

The paper investigates how small language models acquire abstract reasoning skills on the ARC‑TGI benchmark, which groups grid‑transformation tasks into controllable families and allows resampling, spatial shifts, and cross‑benchmark transfer. Over 1,000 supervised fine‑tuning runs across decoder‑only, encoder‑decoder, and mixture‑of‑experts families, the study finds that high in‑distribution accuracy is possible but depends heavily on optimization and is uneven across task families. Performance drops sharply outside the training distribution, and gains from larger training sets or additional in‑context examples vary by model family; attention diagnostics reveal distinct patterns but do not explain causal mechanisms.

By Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei, Sahar Vahdati
arXiv Machine Learning
Oct 7

Artifact removal improves electrodermal waveforms but not downstream classification in a virtual-reality balance task

The study examined whether artifact removal improves electrodermal activity (EDA) classification in a virtual‑reality balance task. A residual gating network effectively identified artifacts on a benchmark dataset (median AUROC 0.94) and reduced waveform error by 17.8%, yet it did not enhance classification accuracy in the VR recordings, with changes ranging from –1.35 to +0.93 percentage points. The authors conclude that preprocessing should be evaluated based on its impact on downstream decisions rather than waveform quality alone.

By Haochen Chai, Qixu Zhu, Siyao Li, Fangfang Jiang
arXiv Computation and Language
Oct 7

AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking

AVMeme Exam is a human‑curated benchmark featuring over a thousand iconic Internet audio‑visual clips—including speech, songs, music, and sound effects—each paired with a unique Q&A that probes understanding from surface content to context, emotion, usage, and world knowledge. The benchmark also provides metadata such as original year, transcript, summary, and sensitivity. Evaluations of state‑of‑the‑art multimodal large language models (MLLMs) and human participants reveal that current models perform poorly on textless music and sound effects and struggle to think in cultural and contextual terms compared to surface content.

By Xilin Jiang, Qiaolin Wang, Junkai Wu, Xiaomin He, Zhongweiyang Xu, Yinghao Ma, Minshuo Piao, Kaiyi Yang, Xiuwen Zheng, Riki Shimizu, Yicong Chen, Arsalan Firoozi, Gavin Mischler, Sukru Samet Dindar, Richard Antonello, Linyang He, Tsun-An Hsieh, Xulin Fan, Yulun Wu, Yuesheng Ma, Chaitanya Amballa, Weixiong Chen, Jiarui Hai, Ruisi Li, Vishal Choudhari, Cong Han, Yinghao Aaron Li, Adeen Flinker, Mounya Elhilali, Emmanouil Benetos, Mark Hasegawa-Johnson, Romit Roy Choudhury, Nima Mesgarani
arXiv Computer Vision
Oct 7

Compositional Concept Erasure in Text-to-Image Diffusion Models via Hierarchically Grounded Semantic Surgery

The paper introduces Hierarchically Grounded Semantic Surgery (HGSS), a training‑free method for removing specific concepts from text‑to‑image diffusion models without altering model weights. HGSS uses hierarchical span grounding to accurately identify target spans and dynamic attribute binding to refine text conditioning during early denoising, thereby preventing routing failures on compositional prompts. Experiments on SEE and UnlearnCanvas show that HGSS significantly reduces hierarchical evasion and attribute leakage, outperforming existing erasure techniques.

By Chen Dai, Ganyu Zou, Nathan Self, Kevin Piper, Ramachandra Rao Seethiraju, Karthik Shyamsunder, Chang-Tien Lu, Naren Ramakrishnan
arXiv Computer Vision
Oct 7

Event Detection in Table Tennis Videos using 2D Keypoints

The paper introduces EventNet, a two‑stage pipeline that uses 2D keypoints of players, table corners, and the ball to detect key events in table tennis videos. First, a keypoint transformer condenses the pose and ball information into a robust representation; second, a transformer encoder predicts how close each frame is to the next and previous ball‑racket contact using a novel temporal cosine‑like target signal. Experiments on Latte‑MV and TTHQ datasets show high accuracy, with an F1 score of 91.16% and a mean frame deviation of 0.42 on Latte‑MV, and 73.08% / 1.16 on TTHQ.

By Rainer Lienhart, Daniel Kienzle, Shin'ichi Satoh, Anastasiia Bilinska
arXiv Computer Vision
Oct 7

Con-DSO: Learning Short-Horizon Consistency Priors for RGB-D Direct Sparse Odometry

Con-DSO introduces a consistency-aware RGB‑D direct sparse odometry framework that learns pixel‑level photometric and geometric uncertainty from adjacent RGB‑D frame pairs. The network predicts uncertainties that are converted into pairwise quality scores, guiding support‑pixel selection and forming a host‑side quality prior for keyframe tracking. Experiments on five public benchmarks show that this approach reduces absolute trajectory error by over 20% on ICL‑NUIM and by 50–80% on other datasets, improving robustness in challenging environments.

By Haolan Zhang, Thanh Nguyen Canh, Chenghao Li, Ziyan Gao, Xiongwen Jiang, Nak Young Chong
arXiv AI
Oct 7

Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

The paper evaluates the impact of adding a persistent memory tier to a multi‑agent large language model inference system. While decomposing long‑context inference across agents reduces the peak KV cache usage from about 35 MiB to 14.3 MiB per query, the persistent tier adds only a modest 0.368 MiB to the cache and shows no measurable accuracy improvement across eight dataset pairs. The authors argue that the lack of benefit is structural, as single‑question benchmarks do not provide informative recall opportunities, and they outline conditions and detection procedures for agent‑memory ablations.

By Hochan Son, Kyungdoe Han, Jaehan Koh, Xiaowu Dai, Wenlu Xu, Guang Cheng
arXiv Machine Learning
Oct 7

An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing

The paper introduces an information‑theoretic framework for evaluating knowledge tracing (KT) models that moves beyond aggregate metrics like AUC and accuracy. By applying Context Tree Weighting to item‑response histories, the authors create a causal uncertainty coordinate that separates observable uncertainty from unobserved local irreducible uncertainty (LIU). This approach allows model performance to be assessed across different entropy bands, revealing that modern KT models improve mainly in high‑entropy regions while also highlighting areas where apparent gains may be due to noise.

By Houru Jiang, Zixi Wang, Tengteng Cheng, Xueyi Li, Mingliang Hou, Jiaqi Zheng, Renqiang Luo, Teng Guo, Zitao Liu
arXiv Machine Learning
Oct 7

Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints

The paper introduces HARP, a hard-budget allocation method for evaluating large language models (LLMs) in multi-turn interactions where the time-to-event is partially observed due to resource limits. HARP guarantees that the total computational budget is never exceeded, reallocates unused budget, and provides lower predictive bounds (LPBs) with finite-sample coverage and unbiased metric estimates. Experiments on tasks such as jailbreaks, toxic content, and hallucinations demonstrate that HARP achieves near-nominal coverage with low variance while respecting the fixed budget.

By Shai Feldman, Yaniv Romano
arXiv Machine Learning
Oct 7

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

MASKerade is a new dense‑to‑MoE training method that learns experts as sparse subnetworks of a frozen pretrained feed‑forward network (FFN) using binary masks. A token‑level router selects which masked FFNs to execute, and both the router and mask scores are jointly optimized while the underlying FFN weights remain unchanged. In experiments on five vision‑language benchmarks with Qwen and Gemma backbones, a configuration of four 2:4 experts with top‑2 routing achieved the best performance among compared baselines, demonstrating that mask learning over frozen weights is a practical alternative for constructing token‑routed MoE experts.

By Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu
arXiv Machine Learning
Oct 7

Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models

The paper introduces Mask Fine‑Tuning (MFT), a new approach for adapting Vision‑Language Models that avoids modifying backbone weights. MFT learns masks to selectively route information through existing pretrained connections, dynamically uncovering subnetworks that better align with downstream tasks. Experiments demonstrate that MFT consistently outperforms both Full Fine‑Tuning and Parameter‑Efficient Fine‑Tuning across multiple benchmarks, while also offering insights into how pretrained VLMs reorganize their internal pathways during adaptation.

By Mingyuan Zhang, Yue Bai, Yifan Wang, Yiyang Huang, Yun Fu
arXiv Computation and Language
Oct 7

Structured but Silent: Probing Capability Requirements in LLM Hidden States

The paper investigates whether large language models (LLMs) can infer the capability requirements of a user query before generating a response. Using the TACIT framework, the authors decompose these requirements into eight classes across three axes—Source, Transformation, and World Effect—and train linear probes on hidden states from four open-weight LLM families. Their results show that these capability structures are linearly decodable with high accuracy, yet the models struggle to express the same information in natural language, a phenomenon termed "structured but silent."

By Kyojun Choo, Minsoo Song, Yunju Kang, Chanjun Park
arXiv Computation and Language
Oct 7

The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception

The paper introduces EmoNet‑Face‑HQ, a fine‑grained emotion recognition benchmark that uses generated portraits and a 40‑category taxonomy to evaluate vision‑language models (VLMs). It finds that VLMs perform poorly when asked to generate responses but can match or surpass a fine‑tuned model (Empathic‑Insight‑Face) when their logits are read directly as binary queries. The study shows that the benchmark’s difficulty lies in the readout process rather than in perception, and that graded probability outputs yield better performance than simple yes/no questions.

By Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth Andr\'e