Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 27

T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

The paper introduces T2D, a framework that prunes and adapts a target large language model into a draft model optimized for both inference latency and alignment with the target model’s outputs. By formulating draft model design as an optimization problem, T2D achieves up to 92% pruning while maintaining high speculative decoding performance across multiple target model families and domains. The resulting draft models can also be integrated with other speculative decoding techniques to yield additional gains.

By Shahriar Kabir Nahin, Xiao Ye, Muhao Chen, Ben Zhou, Anshuman Chhabra
arXiv AI
Oct 27

Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

arXiv:2604.08362v3 Announce Type: replace-cross Abstract: The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmark...

By Jiawei Chen, Ruoxi Xu, Boxi Cao, Ruotong Pan, Yunfei Zhang, Yifei Hu, Yong Du, Tingting Gao, Han Li, Yaojie Lu, Yingfei Sun, Xianpei Han, Le Sun, Xiangyu Wu, Hongyu Lin