← Back to all news
arXiv AI By Charin Polpanumas

Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Read the original on arXiv AI →

Vendster is an evaluation arena that studies how the bias and prediction‑interval width of time‑series forecasters influence the outcomes of buyer‑vendor discount negotiations conducted by large language model (LLM) agents. In its reference instance, Vendster‑M5, forecasts generated by LLMs increased sales target attainment by up to 17 percentage points, yet traditional forecast accuracy metrics did not predict negotiation success. Controlled experiments revealed that under‑prediction bias significantly improves outcomes for agents that react strongly to forecasts, while interval width has no notable effect when bias is zero.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 15

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

GIFT‑Eval‑CTX is a new benchmark for contextual forecasting that tests whether models can use verified future information to produce accurate time‑series predictions. It builds on GIFT‑Eval and contains 10,000 instances across 20 datasets, each annotated with macro‑level and micro‑level effects derived from actual future events. The benchmark evaluates traditional TSFMs, advanced LLMs, and agentic systems, showing that structured effects can substantially lower MASE for models like GPT‑5.6 Sol and Claude Opus 4.8, with MoiraiAgent achieving the best overall performance.

By Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 19

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

AlphaBet is a reinforcement learning framework that trains a single large language model to search for evidence, reason about it, and report probabilities for unresolved events across multiple pre-resolution dates. By scoring each dated forecast with a proper scoring rule, AlphaBet uses one resolved event to supervise decisions under varying information states, decomposing forecasting regret into evidence-acquisition and probability-reporting gaps. Experiments on GLM-4.5-Air show that AlphaBet improves Brier scores on Forecast-Dojo and FutureX, attributing gains to multi-date training and better information acquisition and probability reporting.

By Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
llmsagentsreinforcement-learning
More like this →
arXiv AI
Sep 25

When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing

The paper investigates when forecasting agents should employ different behaviors—retrieval, reasoning, deferring to market priors, or using historical analogs—on binary forecasting tasks. It finds that the optimal mechanism depends on the data source, with structured analogs excelling for some processes and market or conservative baselines for others. The authors propose ReliabilityRoute, a rule‑based system that steers agent behavior using reliability features, achieving competitive performance across multiple LLM versions while highlighting that more reasoning is not always better.

By Yufeng Wang
llmsagents
More like this →
arXiv Computation and Language
Sep 30

Can Language Models Learn to Forecast Stock Prices

arXiv:2609.36914v1 Announce Type: new Abstract: Post-training has been shown to significantly improve language models' performance on tasks with verifiable outcomes, including mathematical reasoning,...

By Jiacheng Guo, Suozhi Huang, Shuzhen Li, Yunlong Gao, Zerui Cheng, Jason Ge, Shushu Liang, Zihao Li, Hao Lu, Ming Yin, Shilong Liu, Jiashuo Liu, Xu Kuang, Mengdi Wang
llmsagentsreinforcement-learningfine-tuningbenchmarks
More like this →