← Back to all news
arXiv AI By Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo, Andrew Rabinovich, Zhao Chen

PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?

Read the original on arXiv AI →

PAYWORTHY is a new benchmark that tests whether large language models can predict the success of an AI agent on freelance jobs before the agent attempts them. The dataset contains 1,494 real job postings sampled from 18.8 million postings on a freelancing platform, and 18 models are asked to forecast the likelihood of success. When the agent’s limitations are explicitly described, models achieve AUC scores between 0.66 and 0.86; when the agent is described generically, the models perform near random (AUC 0.45–0.56).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
6d ago

Predict the Reward, Not the World: World Reward Models for ML Agents

The paper introduces World Reward Models (WRMs), a new approach for machine‑learning agents that selects the best candidate program directly from execution results instead of predicting intermediate outcomes. WRMs, exemplified by the ForeCritic model, are trained on thousands of executed program pairs and achieve a 79.4% success rate in choosing the better program, outperforming general‑purpose LLM judges. When integrated into systems like AIDE, ML‑Master, and ForeAgent, ForeCritic reduces mean task time by 4.4–7.5× while maintaining or improving overall performance.

By Yuxiao Lee, Zherui Yang, Fan Liu, Hao Liu
llmsagents
More like this →
Hugging Face Trending Papers
Oct 29

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

The paper "rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment" investigates whether advanced language models can act as research agents that propose and rank minimum viable experiments (MVEs) for recent NLP papers. Six frontier models were given seven papers and tasked with forecasting experiment outcomes and ranking them; a separate executor ran all MVEs and auditors evaluated the results. Of 415 MVEs, 194 were deemed sound, valuable, and new, but only 73 were judged beyond the seed paper and non‑obvious; while agents’ first choices matched auditors’ top picks more often than random, they did not outperform simple heuristics, and team models added cost without clear quality gains. "whyItMatters":"The study provides a dataset and evaluation harness for assessing AI‑driven research planning, highlighting both the potential and current limitations of using language models as research agents."

agentsnlp
More like this →
arXiv AI
6d ago

Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Vendster is an evaluation arena that studies how the bias and prediction‑interval width of time‑series forecasters influence the outcomes of buyer‑vendor discount negotiations conducted by large language model (LLM) agents. In its reference instance, Vendster‑M5, forecasts generated by LLMs increased sales target attainment by up to 17 percentage points, yet traditional forecast accuracy metrics did not predict negotiation success. Controlled experiments revealed that under‑prediction bias significantly improves outcomes for agents that react strongly to forecasts, while interval width has no notable effect when bias is zero.

By Charin Polpanumas
llmsagentssafety
More like this →