arXiv AI
Jun 2
TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
arXiv:2606. 01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains.
TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoning
arXiv:2606. 03629v1 Announce Type: new Abstract: Assessing the quality of time series (TS) data is fundamental yet inherently challenging due to the multifaceted nature of quality dimensions.
Related stories
arXiv AI
Aug 19
TSQueryBench: LLM-as-a-Judge for Time Series Explanations
TSQueryBench is a synthetic benchmark comprising 500 time‑series instances and 10 query types, each paired with correct, partially correct, and incorrect natural‑language explanations. The study evaluates six large language models on explanation generation, ranking, scoring, and anomaly detection, revealing that models often fail to generate numerically correct explanations yet can reliably identify or score correct ones. These findings suggest that rubric‑guided LLM evaluation is more dependable than generation for numerically grounded time‑series reasoning.
arXiv AI
Jun 11
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
arXiv AI
Jun 2
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
arXiv:2606. 02109v1 Announce Type: new Abstract: Enterprise AI systems that translate natural language into SQL queries and orchestrate multi-step agentic reasoning pipelines require evaluation approaches fundamentally different from academic benchmarks.
arXiv AI
Aug 3
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.
arXiv AI
4d ago
PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators
PADM'E is a method for synthesizing preference‑aligned data to meta‑evaluate language‑model (LM) evaluators of agentic behaviors. It reframes meta‑evaluation as a preference judgment problem, generating criterion‑based data with small LMs and no human involvement. In a prototype, PADM'E produced 1,000 samples across four domains and three criteria, and human validation showed agreement with human judgment rising from 73% to 85% compared to a naive baseline.
arXiv AI
Jul 14
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?
arXiv AI
Aug 7
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
arXiv:2608. 06329v1 Announce Type: cross Abstract: Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed.
arXiv AI
Jul 3
AgenticDataBench: A Comprehensive Benchmark for Data Agents
arXiv:2607. 01647v1 Announce Type: cross Abstract: Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society.
arXiv AI
Jul 16
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
arXiv:2603. 00546v2 Announce Type: replace Abstract: Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains.
arXiv AI
Jun 11
A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models
arXiv:2509. 11575v3 Announce Type: replace Abstract: Time series reasoning treats time as a first-class axis and incorporates intermediate evidence directly into the answer.
arXiv AI
Jun 10
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.