Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 21

Dual-Interest Sequential Product Recommendation With Multi-Granular SSM

The paper introduces DSRec, a dual‑interest sequential recommendation model that separates item representations into long‑term and short‑term semantic contexts. Long‑term embeddings capture stable preferences through historical aggregation, while short‑term embeddings focus on local session intent modulated by inter‑click time intervals. Each branch is processed by a distinct State Space Model— a full‑sequence Mamba for long‑term modeling and a time‑modulated SSM for short‑term dynamics— and a residual cross‑fusion mechanism aligns the two granularities while preserving their independence. Experiments on public benchmarks show that DSRec outperforms state‑of‑the‑art methods.

By Shuiying Liao, P. Y. Mok
arXiv AI
Sep 21

Beyond Accuracy: Centroid-Guided Contrastive Loss for Structured Fraudulent Job Posting Detection

The paper introduces Centroid‑Guided Contrastive Loss (CGCL), a novel loss function that combines classification with dense clustering to improve fraudulent job posting detection. CGCL reshapes the latent space using a centroid‑driven top‑k push‑and‑pull mechanism, ensuring accurate decision boundaries while preserving compact clustering of similar fake posts. Experiments on the EMSCAD benchmark show that CGCL achieves state‑of‑the‑art performance.

By Syed Ali Ahmed (National University of Computer and Emerging Sciences, Karachi, Pakistan), Malaika Raza (National University of Computer and Emerging Sciences, Karachi, Pakistan), Muhammad Shoaib Siddiqui (Islamic University of Madinah, Madinah, Saudi Arabia), Muhammad Rafi (National University of Computer and Emerging Sciences, Karachi, Pakistan)
arXiv AI
Sep 21

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

arXiv:2609.22068v1 Announce Type: new Abstract: Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich sourc...

By Bowen Ye, Lei Li, Shicheng Li, Zihao Yue, Linghao Zhang, Hanglong Lv, Yuanxin Liu, Wenhan Ma, Hao Tian, Rang Li, Jinhao Dong, Yikai Zhao, Xiangwei Deng, Hailin Zhang, Liang Zhao, Qi Liu, Lingpeng Kong, Tong Yang, Fuli Luo
arXiv AI
Sep 21

Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Designer‑RSI presents a continual adaptation framework that lets a frozen frontier model operate professional design software while an external procedural memory learns natural‑language design skills from user traffic. Over five rounds on 1,406 real briefs and 1,869 graded trajectories, the memory grew from 76 to 139 skills, boosting execution success from 72.7% to 99.3% and improving win rates on four design benchmarks. The study shows that widening and deepening the memory, especially together, significantly outperforms a no‑skill baseline.

By Hongyang Du, Lan Yan, Christian Flores, Asim Kadav
arXiv AI
Sep 21

Scaling Discovery through Test-Time Communication

The paper demonstrates that test‑time communication among agents can significantly outperform independent parallel attempts on complex tasks. In experiments on the ARC‑AGI‑3 benchmark, a team of $k$ communicating agents matched the success rate of $4k$ independent agents, with the advantage growing as the team size increased. The study also shows that communication enables solving tasks that no single agent can solve, and that these benefits transfer to research‑oriented problems such as polyomino packing and MNIST classifier compression, where communicating agents surpassed prior best scores.

By Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos
arXiv AI
Sep 21

Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation

The paper introduces the COmmunity-centered Peer Engaged Support (COPES) dataset and a three‑axis evaluation framework to gauge how well Large Language Models (LLMs) align with community perspectives on mental‑health support queries. Experiments show that fine‑tuning LLMs on COPES improves strategy alignment and emotion‑tone alignment by over 50% for general‑purpose models, yet these gains are uneven across subreddits and coping strategies. The study also finds that post‑training shifts the model’s recommendations toward problem‑focused advice while reducing emotion‑focused responses, indicating persistent disparities in performance across different communities and needs.

By Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar, Munmun De Choudhury
arXiv AI
Sep 21

FOCAL-VLA: Subtask-Guided Geometry Distillation and Implicit World Modeling for Vision-Language-Action Models

FOCAL‑VLA is a framework that improves vision‑language‑action models by combining subtask‑guided geometry distillation with implicit world modeling. It transfers geometric knowledge from VGGT to focus on subtask‑relevant image regions and uses Track4World features to capture future 3D evolution, guiding action generation without running these models at inference time. Experiments demonstrate that FOCAL‑VLA outperforms baselines on both simulation benchmarks and real‑world manipulation tasks.

By Zhiyuan Gao, Di Wen, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Sch\"afer, Kunyu Peng, Michael Beetz
arXiv AI
Sep 21

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

CESBench is a new benchmark for evaluating large language models on cryptographic engineering security for IoT devices, comprising 380 expert‑written items across six sub‑domains such as side‑channel, fault injection, and implementation. The benchmark includes four task types—multiple‑choice, judgment, scenario, and code—each designed to test different competences, with automatic scoring for the first two and LLM‑based judging for the latter two. Evaluation of 11 open‑weight and proprietary LLMs shows strong performance on multiple‑choice and code tasks but weaker results on judgment and scenario tasks, highlighting gaps in justifying security verdicts.

By Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang, Yaoling Ding, Liehuang Zhu
arXiv AI
Sep 21

From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers

The paper introduces a Situation–Internal state–Behavior Persona method to improve large language models’ ability to impersonate real individuals in social media contexts. It also proposes an evaluation protocol that supplies LLM evaluators with reference information about the target individual. Experiments on a new dataset of social media replies show the method surpasses state‑of‑the‑art in‑context learning baselines, and the protocol correlates moderately with human judgments, while additional tests on fictional characters confirm broader applicability.

By Ji-Lun Peng, Yi-Zhen Zhang, Chun-Nan Chou, Yun-Nung Chen
arXiv AI
Sep 21

Knowledge-Graph-Augmented Chronos-2 for HEC-RAS Surrogate Forecasting

The paper introduces KG‑Chronos‑2, a surrogate forecasting model that augments a frozen Chronos‑2 time‑series predictor with knowledge‑graph‑conditioned retrieval and correction mechanisms for HEC‑RAS water‑surface elevation (WSE) prediction. In a benchmark involving 64 24‑hour windows across 4,675 cross sections, KG‑Chronos‑2 outperforms persistence, residual LSTM, GeoFNO, and hydraulic DCRNN‑style models, achieving a 0.246970 event‑balanced RMSE and reducing error by 14.13% relative to frozen Chronos‑2. The model also attains the lowest active‑window and final‑lead RMSE among the evaluated systems, demonstrating the benefit of coupling a frozen temporal predictor with project‑specific knowledge for warm‑start HEC‑RAS forecasting.

By Edward Holmberg, Elias Ioup, Mahdi Abdelguerfi
arXiv AI
Sep 21

AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

AgentVidBench is a new multi‑hop video question‑answering benchmark designed to evaluate spatial, temporal, and causal reasoning in multimodal large language models (MLLMs). Unlike existing tests that focus on simple scene queries or global summaries, AgentVidBench includes step‑by‑step solution traces to assess whether agents gather the necessary evidence to justify their answers. Experiments with 12 MLLMs show limited single‑turn performance, but integrating these models into agentic workflows improves both accuracy and trajectory scores, establishing AgentVidBench as a comprehensive testbed for future research on agentic video understanding.

By Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee
arXiv AI
Sep 21

Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction

LoG-VGGT is a memory‑efficient framework for long‑sequence 3D reconstruction that balances local temporal modeling with global camera consistency. It uses cross‑window attention in a small subset of transformer blocks to propagate information across adjacent temporal windows while keeping memory usage bounded. A global camera consistency refinement module further improves long‑horizon pose stability by enforcing scene‑level constraints through cross‑attention between camera and compact register tokens, leading to better depth accuracy and robust camera pose estimation on multiple benchmarks.

By Jingke Zhou, Chenhang Ma, Zhizhou Zhong, Mingkai Liu, Zhuang Zhou, Yicheng ji, Binghua Su, Bo Cai, Xianliang Huang