PAYWORTHY is a new benchmark that tests whether large language models can predict the success of an AI agent on freelance jobs before the agent attempts them. The dataset contains 1,494 real job postings sampled from 18.8 million postings on a freelancing platform, and 18 models are asked to forecast the likelihood of success. When the agent’s limitations are explicitly described, models achieve AUC scores between 0.66 and 0.86; when the agent is described generically, the models perform near random (AUC 0.45–0.56).
By Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo, Andrew Rabinovich, Zhao Chen
The paper "rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment" investigates whether advanced language models can act as research agents that propose and rank minimum viable experiments (MVEs) for recent NLP papers. Six frontier models were given seven papers and tasked with forecasting experiment outcomes and ranking them; a separate executor ran all MVEs and auditors evaluated the results. Of 415 MVEs, 194 were deemed sound, valuable, and new, but only 73 were judged beyond the seed paper and non‑obvious; while agents’ first choices matched auditors’ top picks more often than random, they did not outperform simple heuristics, and team models added cost without clear quality gains.
"whyItMatters":"The study provides a dataset and evaluation harness for assessing AI‑driven research planning, highlighting both the potential and current limitations of using language models as research agents."
arXiv:2609.27051v1 Announce Type: new
Abstract: Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retir...
By Bo Qu, Mingguang Chen, Licheng Wang
arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
By Jiacheng Ding, Cong Guo, Jason Xu
Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent shoul...
The paper introduces World Reward Models (WRMs), a new approach for machine‑learning agents that selects the best candidate program directly from execution results instead of predicting intermediate outcomes. WRMs, exemplified by the ForeCritic model, are trained on thousands of executed program pairs and achieve a 79.4% success rate in choosing the better program, outperforming general‑purpose LLM judges. When integrated into systems like AIDE, ML‑Master, and ForeAgent, ForeCritic reduces mean task time by 4.4–7.5× while maintaining or improving overall performance.
By Yuxiao Lee, Zherui Yang, Fan Liu, Hao Liu
Vendster is an evaluation arena that studies how the bias and prediction‑interval width of time‑series forecasters influence the outcomes of buyer‑vendor discount negotiations conducted by large language model (LLM) agents. In its reference instance, Vendster‑M5, forecasts generated by LLMs increased sales target attainment by up to 17 percentage points, yet traditional forecast accuracy metrics did not predict negotiation success. Controlled experiments revealed that under‑prediction bias significantly improves outcomes for agents that react strongly to forecasts, while interval width has no notable effect when bias is zero.
By Charin Polpanumas
arXiv:2609.37493v1 Announce Type: cross
Abstract: Serving an answer from a large language model requires deciding when to abstain, yet a verifier's ranking accuracy alone does not determine the error...
By Dongyub Jude Lee, Jungseob Lee, Chanjun Park, Hyeonseok Moon, Heuiseok Lim
The study examines how the choice of large language model (LLM) used to write job application cover letters influences hiring decisions, independent of the applicant’s qualifications. Using a multi‑model audit with eleven writer agents and nine evaluator agents across ten job descriptions, the authors find a consistent ranking of writer models—locally hosted models are disadvantaged and evaluators often favor their own outputs. The hierarchy has tangible effects, with up to 1.8 of 25 qualified candidates being rejected solely due to model mismatch, and the patterns are explained by shared stylistic cues that both writers and evaluators reward or penalise.
By Nicol\`o Pagan, Michael Andreas Ebling, Stefania Ionescu, Anik\'o Hann\'ak
arXiv:2610.14180v1 Announce Type: new
Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and...
By Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma
The paper introduces $ au^ au$-Bench, a benchmark that turns the construction of AI agents into a measurable task. In this environment a developer agent receives real business records, client requirements, a production API, an existing codebase, and constraints on cost and models, and must deliver a complete customer‑service agent. The benchmark evaluates performance by deploying the agent against simulated users, revealing that current state‑of‑the‑art models achieve only 23.9% success while an expert‑written reference scores 82.2%.
By Quan Shi, Keshav Dhandhania, Karthik Narasimhan, Victor Barres
FINCOSMOS is an open‑world evaluation framework for financial language agents that tests their ability to acquire data from the Web, construct backtests within a data‑cutoff, and deliver verifiable results without any pre‑provided data or reference answers. In the benchmark, a simulated user hides a strategy and only reveals it upon request; agents must build their own backtests and pass 15 veto checks to be accepted. Across 150 sessions with four coding agents and a tool‑less model, no session was accepted, highlighting widespread failures and revealing that 88.33% of failures were attributed to the agents themselves.
By Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou