Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows

arXiv:2610.17305v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently improved Text-to-SQL generation, but their computational cost can hinder deployment in resource-constrained...

By Bilel Abderrahmane Benziane, Xavier Schepler

llms benchmarks data-engineering

arXiv AI

Oct 15

KNN Implementation Details Can Dramatically Change Performance: An Example from Cover Trees

arXiv:2610.16940v1 Announce Type: cross Abstract: Nearest neighbor search algorithms are widely used and have a long history, but we find that key implementation details are seldom reviewed in articl...

By Amol Khanna, Edward Raff

benchmarks

arXiv AI

Oct 15

A Scaled-Up Empirical Study of Syntactic Language Models

arXiv:2610.16837v1 Announce Type: cross Abstract: Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalizati...

By Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu

llms benchmarks

arXiv AI

Oct 15

Cowork-Gym: Synthesizing Interactive Work Environments at Scale

arXiv:2610.17670v1 Announce Type: new Abstract: Cowork agents navigate dependencies across tools and files while adapting to changing user requirements and edits to shared workspaces. Scaling such en...

By Anni Zou, Cheng Chen, Jiaji Deng, Weikang Zhou, Xinji Mai, Yue Wu, Jianing Lin, Lipeng Xie, Lixiong Qin, Qingyue Zhao, Que Shen, Shuchang Tao, Xuchen Pan, Yaliang Li, Yunpeng Zhai, Yushuo Chen, Zhihai Wang, Zhuo Zhang, Zhaoyang Liu, Bolin Ding, Jingren Zhou

agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Oct 15

Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution

arXiv:2610.17816v1 Announce Type: new Abstract: Edge-cloud collaboration combines the lower inference overhead and greater data locality of edge-side execution with the stronger capabilities of cloud...

By Pengcheng Huang, Zhenghao Liu, Shixin Ren, Shi Yu, Sijia Yao, Yukun Yan, Chaojun Xiao, Maosong Sun

llms agents benchmarks safety

arXiv AI

Oct 15

STAMP as a topology-aware molecular language for fragment-level learning and editing

arXiv:2610.18438v1 Announce Type: cross Abstract: Molecular fragments provide chemically meaningful units, but existing molecular languages rarely make their locations and connections explicit for ed...

By Yuxuan Ren, Jiatong Li, Ziqiao Meng, Yatao Bian

llms benchmarks

arXiv AI

Oct 15

LLM Agents for Automated Post-Training via Structured Recipe Optimization

arXiv:2610.18572v1 Announce Type: new Abstract: Automated post-training transforms pre-trained large language models (LLMs) into models capable of handling diverse domains by automatically constructi...

By Ruohong Liu, Bohan Tang, Lekang Jiang, Jack Umenberger, Yiwen Guo

llms agents benchmarks

arXiv Machine Learning

Oct 15

TEX-MEX: Similarity-Guided Temporal Contrastive Learning for Time Series

arXiv:2610.19557v1 Announce Type: new Abstract: Contrastive learning for time series depends on deciding which pairs of inputs should be treated as similar and dissimilar. Because few augmentations r...

By Aditya Radhakrishnan, Maxwell A. Xu, Wanting Mao, Santosh Kumar, Harish Haresamudram, James M. Rehg

benchmarks

arXiv AI

Oct 15

Subgraph-Aware Training of Language Models for Knowledge Graph Completion Using Structure-Aware Contrastive Learning

arXiv:2407.12703v5 Announce Type: cross Abstract: Fine-tuning pre-trained language models (PLMs) has recently shown a potential to improve knowledge graph completion (KGC). However, most PLM-based me...

By Youmin Ko, Hyemin Yang, Taeuk Kim, Hyunjoon Kim

llms fine-tuning benchmarks safety

arXiv AI

Oct 15

GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

arXiv:2610.13995v1 Announce Type: cross Abstract: Post-training large language models (LLMs) on two-player zero-sum games provides a verifiable reward, and the gains have been shown to transfer to ge...

By Weixian Waylon Li, Jiaxin Zhang, Han Zhou, Chengzu Li, Ke Wang, Fengxiang He, Yiwen Guo

llms reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Oct 15

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

arXiv:2610.19975v1 Announce Type: new Abstract: Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonst...

By Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu, Yun Luo, Yafu Li, Yu Cheng

benchmarks

arXiv Machine Learning

Oct 15

Safe-FCL: Continual Risk Awareness and Fine-Grained Safety Aggregation for Federated Continual Learning of LLMs

arXiv:2610.19968v1 Announce Type: new Abstract: Federated fine-tuning enables large language models (LLMs) to learn collaboratively from distributed client data, but malicious client updates can comp...

By Shuaijie Ma, Yongkang Hu, Yu Cheng, Jiuan Zhou, Yi Xie, Yushuo Zhang, Zhaoxia Yin

llms fine-tuning benchmarks safety

arXiv AI

Oct 15

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

arXiv:2610.14071v1 Announce Type: cross Abstract: As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue h...

By Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi, Ulrich A\"ivodji, Spandana Gella, Adriana Romero-Soriano, Golnoosh Farnadi

llms agents benchmarks safety

arXiv Machine Learning

Oct 15

MorphingNet: Symbolic Regression Where the Network Is the Formula

arXiv:2610.19942v1 Announce Type: new Abstract: We introduce MorphingNet, a network for symbolic regression (SR) whose units change their function continuously. Instead of searching over expressions...

By Jincheng Gao, Jiahe Yi, Shuo Sun

benchmarks

arXiv AI

Oct 15

Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework

arXiv:2610.17089v1 Announce Type: cross Abstract: Existing embodied navigation tasks commonly assume that traversability and object states remain fixed during an episode. Yet reaching a goal may requ...

By Delun Li, Xueyao Wei, Rui Chen, Yifei Wang, Junzhi Yu, Wenjie Song, Xingyu Chen

llms agents robotics multimodal benchmarks

arXiv AI

Oct 15

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

arXiv:2610.16155v1 Announce Type: new Abstract: Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlookin...

By Yutao Xie, Jingbo Yang, Guanyu Yao, Xiaoyu Yang, Duo Ding, Shiyu Chang

llms agents nlp benchmarks

arXiv AI

Oct 15

MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging Knowledge

arXiv:2610.06170v2 Announce Type: replace Abstract: Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multipl...

By Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique

llms benchmarks

arXiv AI

Oct 15

Harness^2: Recursive Agent Harnessing for an Open World

arXiv:2610.14224v1 Announce Type: new Abstract: LLM agents' capabilities depend not only on their underlying models but also on their execution harnesses. Recursive self-improvement (RSI) offers a pr...

By Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister

llms agents benchmarks

arXiv AI

Oct 15

Simulating Student-Tutor Interactions with LLMs for Multilingual Math Feedback

arXiv:2506.04920v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated the ability to generate formative feedback and instructional hints in English, making them inc...

By Junior Cedric Tonga, KV Aditya Srivatsa, Kaushal Kumar Maurya, Fajri Koto, Ekaterina Kochmar

llms benchmarks

arXiv AI

Oct 15

VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning

arXiv:2610.14939v1 Announce Type: new Abstract: GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individu...

By Jiahe Feng

reinforcement-learning efficiency multimodal benchmarks

Get the daily briefing