Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
5d ago

TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research

TSFLab is a self‑improving, agent‑agnostic workbench designed to streamline time‑series forecasting research. It equips coding agents with shared resources and tools, enabling low‑cost experimentation, reproducibility, and continuous maintenance of research artifacts. Experiments show that TSFLab boosts task completion rates by 20‑163% and reduces token usage and development time significantly compared to baseline approaches.

By ChengAo Shen, Yanjun Zhao, Haoxin Liu, Bonnie Chen, Miles Ma, Dongjin Song, Hanghang Tong, Qingsong Wen, Jingrui He, Jingchao Ni
arXiv AI
5d ago

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

arXiv:2610.40735v1 Announce Type: cross Abstract: Evaluating approaches that improve the security of LLM code generation requires datasets that jointly assess security and functionality and remain ad...

By Zacharie Chenail-Larcher, Daniele Lunghi, Melissa Tessa, Quentin Sti\'evenart, Jacques Klein, Tegawend\'e F. Bissyand\'e, Florent Avellaneda, Naouel Moha
arXiv Computation and Language
5d ago

Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

The paper introduces BILAL-BENCH, a 240-question benchmark designed to evaluate open-ended Islamic-source fabrication in Bengali. It assesses models on Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial prompts, using a rubric that separates source errors from assertiveness. Seven open-weight models were tested, revealing that most confidently fabricate on 79.3–95.3% of source-production questions, with hadith provenance being the weakest area and no clear improvement with larger model size.

By Sabah Nushra, Md Kishor Morol
arXiv AI
5d ago

P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior

P-ContextBench is a benchmark designed to evaluate how well large language models understand and safely use personal context. It creates realistic, life‑story‑grounded digital humans from U.S. demographic seeds, embedding traits, routines, goals, and constraints in fragmented, evolving synthetic mobile‑app records. The benchmark tests five capabilities—fact recovery, temporal tracking, pattern inference, and safe handling of conflicting requests—using a multi‑stage validation pipeline and 4,810 verified test cases.

By Grant Nasir Brown, Abdul Basir Bari, Anmol Tadikonda, Isabel Song