PAYWORTHY is a new benchmark that tests whether large language models can predict the success of an AI agent on freelance jobs before the agent attempts them. The dataset contains 1,494 real job postings sampled from 18.8 million postings on a freelancing platform, and 18 models are asked to forecast the likelihood of success. When the agent’s limitations are explicitly described, models achieve AUC scores between 0.66 and 0.86; when the agent is described generically, the models perform near random (AUC 0.45–0.56).
arXiv:2609.27051v1 Announce Type: new Abstract: Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retir...
Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent shoul...
The paper introduces World Reward Models (WRMs), a new approach for machine‑learning agents that selects the best candidate program directly from execution results instead of predicting intermediate outcomes. WRMs, exemplified by the ForeCritic model, are trained on thousands of executed program pairs and achieve a 79.4% success rate in choosing the better program, outperforming general‑purpose LLM judges. When integrated into systems like AIDE, ML‑Master, and ForeAgent, ForeCritic reduces mean task time by 4.4–7.5× while maintaining or improving overall performance.
arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
The paper "rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment" investigates whether advanced language models can act as research agents that propose and rank minimum viable experiments (MVEs) for recent NLP papers. Six frontier models were given seven papers and tasked with forecasting experiment outcomes and ranking them; a separate executor ran all MVEs and auditors evaluated the results. Of 415 MVEs, 194 were deemed sound, valuable, and new, but only 73 were judged beyond the seed paper and non‑obvious; while agents’ first choices matched auditors’ top picks more often than random, they did not outperform simple heuristics, and team models added cost without clear quality gains. "whyItMatters":"The study provides a dataset and evaluation harness for assessing AI‑driven research planning, highlighting both the potential and current limitations of using language models as research agents."
Vendster is an evaluation arena that studies how the bias and prediction‑interval width of time‑series forecasters influence the outcomes of buyer‑vendor discount negotiations conducted by large language model (LLM) agents. In its reference instance, Vendster‑M5, forecasts generated by LLMs increased sales target attainment by up to 17 percentage points, yet traditional forecast accuracy metrics did not predict negotiation success. Controlled experiments revealed that under‑prediction bias significantly improves outcomes for agents that react strongly to forecasts, while interval width has no notable effect when bias is zero.
arXiv:2609.37493v1 Announce Type: cross Abstract: Serving an answer from a large language model requires deciding when to abstain, yet a verifier's ranking accuracy alone does not determine the error...
The study examines how the choice of large language model (LLM) used to write job application cover letters influences hiring decisions, independent of the applicant’s qualifications. Using a multi‑model audit with eleven writer agents and nine evaluator agents across ten job descriptions, the authors find a consistent ranking of writer models—locally hosted models are disadvantaged and evaluators often favor their own outputs. The hierarchy has tangible effects, with up to 1.8 of 25 qualified candidates being rejected solely due to model mismatch, and the patterns are explained by shared stylistic cues that both writers and evaluators reward or penalise.
arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and...
The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.
The paper introduces $ au^ au$-Bench, a benchmark that turns the construction of AI agents into a measurable task. In this environment a developer agent receives real business records, client requirements, a production API, an existing codebase, and constraints on cost and models, and must deliver a complete customer‑service agent. The benchmark evaluates performance by deploying the agent against simulated users, revealing that current state‑of‑the‑art models achieve only 23.9% success while an expert‑written reference scores 82.2%.
arXiv:2610.18760v1 Announce Type: new Abstract: Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To...