Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
1d ago

Agents, Think Only When it Helps

The paper investigates the role of thought supervision in distilling large language model agents. By independently perturbing teacher thoughts and actions during distillation, the authors find that removing or altering thoughts has little impact on student performance across various benchmarks and model scales. However, thought supervision becomes important when the action does not reveal the information contained in the thought, suggesting that including thoughts in the training data may be unnecessary for many cases.

By Min-Seon Kim, Seungdong Yoa, Sanghyu Yoon, Ye Seul Sim, Suhee Yoon, Dongmin Kim, Jung-Eun Kim
arXiv AI
1d ago

Where Search Agents Go Off the Rails: Graph-Grounded Step-Level Verification

The paper introduces GraphRail, a verification framework for search-augmented reasoning systems that checks whether evidence covers the question, reasoning addresses the question, and reasoning is grounded in the evidence. GraphRail represents these components as structured claims and compares them without relying on a generative judge or gold answer. Experiments on three multi‑hop QA benchmarks and three reasoning frameworks show that GraphRail improves error recall over generative verifiers and remains stable under stochastic extraction, while also providing diagnoses that can guide online reasoning to detect and correct intermediate errors.

By Hyeseo Jeon, Seonho Yoo, Jay-Yoon Lee
arXiv Machine Learning
1d ago

Alleviating Limited Effectiveness and Multi-Class Collapse in Deep Graph-level Clustering

The paper introduces S$^3$GLC, a deep graph-level clustering framework that combines global topological features—derived via a Spectral Signature based on the Hoffman-Wielandt inequality—with local neighborhood information to improve clustering of graph datasets. It also incorporates a joint manifold-preserving sparse regression module to fuse these representations while reducing noise, and a balanced clustering objective using optimal transport to prevent clustering collapse, especially in imbalanced multi-class settings. Experiments on ten real-world datasets, including REDDIT-12K and OGBG-PPA, show that S$^3$GLC outperforms fifteen state-of-the-art baselines.

By Xiaoyang Lin, Renchi Yang, Yuqi Chen, Xin Huang
arXiv Machine Learning
1d ago

An Interaction-Aware Decision Framework for FDIC-Assisted Distressed Bank Acquisitions: Empirical Evidence from the FDIC Failed-Bank Panel and the SVB-First Citizens Case

The paper introduces the Spider Network Filtering Framework, an interaction‑aware decision model for FDIC‑assisted distressed bank acquisitions. It encodes dependencies among five dimensions—acquisition value, regulatory protection, risk, buyer strength, and execution difficulty—into four interpretable rules, producing graph‑inspired features for a logistic model and a Pursue/Caution/Avoid decision rule. Evaluated on 55 FDIC‑assisted acquisitions from 2008‑2024, the Spider‑enhanced model achieves a mean AUC of 0.594, slightly lower than logistic (0.615) and ridge (0.632) regressions, and permutation tests show only weak evidence of performance above chance.

By Yousef Nejatbakhsh, Malihe Aliasgari, Parsa Safaee, Sina Shokoohyar
arXiv Machine Learning
1d ago

SecureGARD: Dynamic Ontology-Guided Governance for Secure Tool-Using LLM Agents

SecureGARD is a dynamic ontology‑guided governance framework designed to secure tool‑using large language model agents. It grounds heterogeneous tool calls in a shared semantic representation, using an executable ontology combined with episode‑level task state to make deterministic ready, waiting, and forbidden decisions. The system evolves its ontology offline from verified training executions, ensuring logical consistency and protecting entailments, and achieves a 0% attack success rate across multiple benchmarks while improving task utility under attack.

By Guoqin Xiong, Shengtao Wen, Xiang Chen, Xinyi Zeng, Zhaoxin Fan, Wei Ye, Xiaofeng Cao, Sheng-Jun Huang
arXiv Machine Learning
1d ago

Adaptive Multi Mode: Budget-Aware Routing of Optimizers By Online Landscape Probes

The paper introduces Adaptive Multi Mode (AMM), a budget‑aware router that selects among several optimization engines—quasi‑Newton, trust‑region, coordinate, multistart, and evolution‑strategy—based on online probes of the problem landscape. Each probe evaluates features such as curvature, plateaus, separability, kinks, basins, and fractal structure, and is assigned a validity range and cost relative to the budget. In extensive pre‑registered tests on the bbob benchmark, AMM achieved the highest target share in 382 of 600 function‑dimension‑budget cells, outperforming up to seven competitors and maintaining strong performance across transformations and constraints.

By Benjamin Jacob Bodner
arXiv Computation and Language
1d ago

CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.

By Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
arXiv Computation and Language
1d ago

Adapting Qwen3 to Tajik from 0.6B to 8B: Lower Text Loss Did Not Mean Better Understanding

The paper reports on adapting the Qwen3 language model to Tajik using small LoRA adapters across sizes from 0.6B to 8B. While the adapters consistently lowered Tajik text loss by 0.60 to 1.45 nats per token, they did not improve Tajik reading comprehension on the Belebele benchmark; in some cases accuracy even dropped. A more extensive 8B training pipeline, including 375 million tokens of raw Tajik text and supervised fine‑tuning, produced a new model (e3) that outperformed Qwen3‑8B on Belebele by 11.4 points, though it slightly degraded English MMLU performance.

By Tohir Saidzoda, Madina Abdusalomzoda, Nasimjon Rajabov, Akhrorkhon Ikromov, Komron Giyosiev, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
arXiv Computation and Language
1d ago

LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

The paper introduces FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, designed to test large language models’ (LLMs) temporal self‑knowledge in financial backtesting. Each fact is paired with four prompts that probe the model’s recall, prediction, and self‑reported knowledge, with controls ensuring chance accuracy near 0.27. Results reveal that most frontier models understate their knowledge, with GPT‑5 correctly answering 76% of facts yet claiming to know only 38%, and that declared dates largely eliminate prior‑knowledge claims without affecting accuracy.

By Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao, Jiarui Chen, Zehan Li, Jimin Huang, Alejandro Lopez-Lira, Yuehua Tang, Sophia Ananiadou, Jian-Yun Nie