Hugging Face Trending Papers
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
5d ago
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
PAYWORTHY is a new benchmark that tests whether large language models can predict the success of an AI agent on freelance jobs before the agent attempts them. The dataset contains 1,494 real job postings sampled from 18.8 million postings on a freelancing platform, and 18 models are asked to forecast the likelihood of success. When the agent’s limitations are explicitly described, models achieve AUC scores between 0.66 and 0.86; when the agent is described generically, the models perform near random (AUC 0.45–0.56).
Hugging Face Trending Papers
Oct 29
rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment
The paper "rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment" investigates whether advanced language models can act as research agents that propose and rank minimum viable experiments (MVEs) for recent NLP papers. Six frontier models were given seven papers and tasked with forecasting experiment outcomes and ranking them; a separate executor ran all MVEs and auditors evaluated the results. Of 415 MVEs, 194 were deemed sound, valuable, and new, but only 73 were judged beyond the seed paper and non‑obvious; while agents’ first choices matched auditors’ top picks more often than random, they did not outperform simple heuristics, and team models added cost without clear quality gains. "whyItMatters":"The study provides a dataset and evaluation harness for assessing AI‑driven research planning, highlighting both the potential and current limitations of using language models as research agents."
arXiv AI
Sep 24
Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors
arXiv:2609.27051v1 Announce Type: new Abstract: Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retir...
arXiv AI
Jul 21
FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches
arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
Hugging Face Trending Papers
Sep 22
Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors
Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent shoul...
arXiv AI
Oct 31
Predict the Reward, Not the World: World Reward Models for ML Agents
The paper introduces World Reward Models (WRMs), a new approach for machine‑learning agents that selects the best candidate program directly from execution results instead of predicting intermediate outcomes. WRMs, exemplified by the ForeCritic model, are trained on thousands of executed program pairs and achieve a 79.4% success rate in choosing the better program, outperforming general‑purpose LLM judges. When integrated into systems like AIDE, ML‑Master, and ForeAgent, ForeCritic reduces mean task time by 4.4–7.5× while maintaining or improving overall performance.