← Back to all news
arXiv Computation and Language By Anirudh Phukan, Tanya Goyal, Danish Pruthi

BlindToChange: AI Agents Can Find Changes When Asked But Do They Look When Needed?

Read the original on arXiv Computation and Language →

BlindToChange is a new benchmark that tests whether AI agents can proactively discover recent real‑world events that affect their responses, even when those events are not explicitly mentioned in user requests. The benchmark uses automatically extracted events from Wikidata to create recommendation tasks where an obvious candidate is no longer applicable due to a recent change. Experiments show a large performance gap—up to 27.5 points—when agents fail to verify recent changes, revealing that many failures stem from not directly searching for updated facts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 2

Revision-Aware Independent Agent Graphs for Dynamic Reasoning

The paper introduces Revision‑Aware Independent Agent Graphs (RIAG) to address dynamic task routing, where an event stream continually revises task bindings and a system must select the correct document version at query time. By repurposing six benchmarks into over 31,000 dynamic episodes, the authors demonstrate that RIAG balances recomputation and reuse, achieving 54.24 % joint routing‑and‑answer accuracy with only 0.62 calls per query—substantially better than the strongest baseline. The study highlights the trade‑off between stale conclusions and wasted work in dynamic reasoning settings.

By Yan Luo, Selim-Antoine Lali, Jeremy Moebel, Iliass Khoutaibi, Ahmadou Aidara, Mengyu Wang
agentsbenchmarks
More like this →
arXiv Machine Learning
4d ago

Benchmarking Forecasting Agents on Yesterday's Tomorrows

The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.

By Yuxuan Wang, Yishu Wang, Jiaqi Deng
agentsbenchmarks
More like this →