The paper introduces Research Attention Prediction (RAP), a benchmark designed to evaluate large language model (LLM) agents’ ability to infer changes across entire research fields from limited evidence gathered during online scholarly searches. RAP is built on a local search environment comprising 356,357 dated arXiv papers and includes two evaluation sets: a Core set of 2,160 questions covering 54 AI/ML fields and an Extended set spanning 269 fields. Experiments show that while search improves agents’ recognition of recent changes, their forecasting of future shifts remains limited or even negative, indicating that LLM agents can detect some recent developments but struggle to reliably anticipate future research activity.
By Yingqian Wu, Jingcong Liang, Shengbin Yue, Siyuan Wang, Zhenfei Yin, Junchi Yu, Zhongyu Wei
AlphaBet is a reinforcement learning framework that trains a single large language model to search for evidence, reason about it, and report probabilities for unresolved events across multiple pre-resolution dates. By scoring each dated forecast with a proper scoring rule, AlphaBet uses one resolved event to supervise decisions under varying information states, decomposing forecasting regret into evidence-acquisition and probability-reporting gaps. Experiments on GLM-4.5-Air show that AlphaBet improves Brier scores on Forecast-Dojo and FutureX, attributing gains to multi-date training and better information acquisition and probability reporting.
By Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
The study investigates how cognitive biases influence large language model (LLM) agents that use search. By controlling search conditions across 10 models and 1,000 synthetic search scenarios, the authors examine how the order of evidence, prior work records, and initial hypotheses affect retrieval, document opening, and final answers. Findings indicate that initial evidence order during active search has a stronger impact on outcomes than when all evidence is presented at once, and that dead‑end histories reduce strategy switching beyond what context length alone would explain.
By Yanru Xiao, Yingwei Yu
arXiv:2609.00747v1 Announce Type: new
Abstract: Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they...
By Fenghai Li, Zihan Tang, Haofei Yu, Yining Zhao, Jiaxuan You
The paper introduces an agentic forecasting environment built on 2,100+ resolved Polymarket questions, where a language model (Qwen3.5-35B-A3B) learns to gather evidence during rollout via web search, page reading, and financial time series, all filtered to avoid post‑cutoff leaks. Training with single‑epoch GRPO and a Brier‑score reward improves calibration by 30‑40% and reduces search attempts, while the trained policy outperforms four frontier models in evidence‑based forecasting, achieving lower soft‑Brier scores at roughly 5% of the inference cost. The authors release the environment, dataset, and per‑rollout records as a reusable harness for temporal forecasting agents.
By Yusuf Afifi, Artur Kiulian, Anton Polishko, Mykola Khandoga, Hamudi Naanaa, Alina Krasnobrizha
BridgeMem is a new method for temporal knowledge graph forecasting that focuses on pair‑specific transition evidence, adding a residual correction to the log scores of a frozen full‑vocabulary forecaster. It retrieves and encodes prior events between a query actor and candidate, converting them into a likelihood‑ratio correction via a support‑adaptive empirical‑Bayes reader. Across five benchmarks, BridgeMem outperforms nine baselines from 2021–2026, improving filtered MRR and Hits@{1,3,10} metrics by up to 0.0216.
By Zeyan Li, Libing Chen, Shengda Zhuo, Yin Tang, Jianfeng Xu
arXiv:2605.16358v2 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remain...
By Mingtian Tan, Mihir Parmar, Palash Goyal, Chun-Liang Li, Nanyun Peng, Thomas Hartvigsen, Jinsung Yoon, Tomas Pfister
arXiv:2610.18602v1 Announce Type: new
Abstract: Point-in-time (PIT) agents can recall post-cutoff facts even when their external evidence respects a historical cutoff. We examine whether this access...
By Rongtian Ye
The paper introduces a paired benchmark to detect hindsight bias in clinical language models by comparing model responses to questions posed at a clinically relevant cutoff versus the full timeline. It uses 171 case reports (40 sepsis, 131 GLP‑1/diabetes) with both human‑annotated and LLM‑generated time‑series data, evaluating accuracy, hindsight trap rate, answer instability rate, and hindsight bias rate. Results show that exposing models to the full timeline consistently increases hindsight bias, while truncating the timeline mitigates bias without sacrificing accuracy.
By Misaki Matsuura, Sayantan Kumar, Ojas Kadam, Jeremy C. Weiss
The paper introduces HeadEntropy, a training‑free method that predicts the correctness of large language model (LLM) answers by measuring how stable each attention head’s pattern is to further gradient updates. By linking the trace of the softmax Jacobian to 2‑Renyi entropy, the authors show that attention spread correlates with gradient stability, enabling accurate hallucination detection without reference annotations. Across five instruction‑tuned LLMs and five diverse datasets—including medicine, multi‑hop reasoning, and mathematics—HeadEntropy achieves a 0.736 AUROC, outperforming other training‑free baselines and matching hidden‑state probes while incurring less than 1% of inference cost.
By Sophie Ostmeier, Brian Axelrod, Maya Varma, Asad Aali, Yabin Zhang, Magdalini Paschali, Sanmi Koyejo, Curtis Langlotz, Akshay Chaudhari
The paper introduces ICD-Deepresearch, a workflow that combines foundation models for electronic health records (EHR) and language models with medical search and ICD dictionaries to forecast future ICD codes for upcoming clinical encounters. It evaluates candidate code transitions by linking patient evidence, external clinical relations, and exact code semantics within a fixed top‑K budget, using SparseEHR for initial priors, GPT‑5 for complementary forecasts, and a final selection step that validates, deduplicates, and ranks candidates. The method achieves patient‑averaged precision/recall of 24.60/35.09% on MIMIC‑III and 25.14/48.32% on MIMIC‑IV, with physicians rating 51–68% of its retrieved documents as useful, outperforming standalone GPT‑5 web search and Medical Deep Research.
By Junda Wang, Meysam Ghaffari, Akshat Choube, Mohsen Sharifi Renani, Hong Yu, Carlos Morato
The paper introduces an information‑matched, leakage‑safe auditing protocol for evaluating conversational emotion‑shift forecasting models. It addresses common distortions such as unequal feature richness, noisy current‑emotion labels, and differing hyperparameter budgets by using explicit commitment points, causality tests, balanced configuration searches, and seed‑dialogue‑aware inference. Applying the protocol to six causal model families across four corpora shows that a deployable train‑only current‑emotion classifier improves over a predicted‑label baseline, especially on DailyDialog, while matched‑gold conditions reveal baseline competitiveness on IEMOCAP and stronger performance on MELD.
By Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan