← Back to all news
Hugging Face Trending Papers

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

The paper "rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment" investigates whether advanced language models can act as research agents that propose and rank minimum viable experiments (MVEs) for recent NLP papers. Six frontier models were given seven papers and tasked with forecasting experiment outcomes and ranking them; a separate executor ran all MVEs and auditors evaluated the results. Of 415 MVEs, 194 were deemed sound, valuable, and new, but only 73 were judged beyond the seed paper and non‑obvious; while agents’ first choices matched auditors’ top picks more often than random, they did not outperform simple heuristics, and team models added cost without clear quality gains. "whyItMatters":"The study provides a dataset and evaluation harness for assessing AI‑driven research planning, highlighting both the potential and current limitations of using language models as research agents."

Related stories

arXiv AI
Oct 16

Sci$\Pi$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Sciπvot is a 248‑case benchmark that evaluates how well large language model (LLM) agents perform scientific judgment during autonomous research. The benchmark is built from six months of real‑world experiments where senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly corrected the agents’ decisions. In each case, the agent must choose between its original plan and a human‑corrected alternative using only the project record available before the correction, allowing a direct comparison of LLM and human performance under varying reasoning effort and record access.

By Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye, Suying Liu, Jiaxuan Guo, Jianda Du, Weicheng Ye, Kecheng Liu, Hefan Zhang, Haixin Wang, Jie Xiao, Eric Yang, Haizhao Yang
llmsagentsbenchmarks
More like this →
arXiv Machine Learning
Sep 24

Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.

By Md Rezwanul Islam, Wael Mohammed
benchmarks
More like this →
arXiv Machine Learning
Oct 15

How Far Are We From True Auto-Research?

The paper "How Far Are We From True Auto-Research?" introduces ResearchArena, a lightweight scaffold that lets off‑the‑shelf agents (Claude Code, Codex, and Kimi Code) complete the full research cycle—ideation, experimentation, writing, and self‑refinement—across 13 computer science topics. In 39 trials, the agents produced 117 papers that were evaluated by three review methods: manuscript‑only review (SAR), artifact‑aware peer review (PR), and a human meta‑review. While SAR scores suggested competitive manuscript quality, PR revealed significant experimental shortcomings, with fabricated results, underpowered experiments, and plan/execution mismatches varying widely among agents; none of the papers met top‑tier venue acceptance thresholds, indicating a substantial gap to true auto‑research.

By Zhengxin Zhang, Ning Wang, Sainyam Galhotra, Claire Cardie
llmsagents
More like this →
arXiv AI
Sep 25

Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems

The paper introduces AgentX-Model, a dual‑agent framework that links proposal development with model experimentation in industrial recommender systems. The Research Agent drafts proposals from literature and prior findings, while the Model Agent runs multi‑round experiments, returning code, metrics, and open questions. The framework iteratively selects starting implementations and formulates new research questions, organizing work into Reproduce, Follow‑up, Composition, and Diagnose actions. Across production evaluations, most experiments exceeded business baselines, with recent A/B tests showing significant gains in acquisition efficiency, advertising spend, and watch time while reducing computational cost.

By Shuang Yang, Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Yusheng Huang, Han Gao, Guanchen Wang, Tianbao Ma, Linxun Chen, Peilin Song, Xuming Wang, Chen Li, Fan Wu, Tao Wang, Zibo Zhao, Xiangyu Wu, An Liu, Fei Pan, Peng Jiang, Chen Yang, Zhaojie Liu, Wenwu Ou
agentsbenchmarkssafety
More like this →
arXiv AI
Oct 23

CrashGym: Measuring the Decision Value of Experimentation for LLM Agents in Counterfactual Market Crashes

CrashGym is a deterministic, replayable limit‑order‑book simulator that lets language‑model agents decide between a deleveraging intervention and a liquidity backstop before a market crash. Agents can query intervention experiments in a laboratory version of the market, and their choices are scored by simulating every feasible intervention on an independently seeded target realization, allowing direct regret calculation. In a study of 24 new worlds, laboratory access reduced mean regret by 57.4% for one deployment, and exploratory results suggest lower regret across all evaluated deployments, while an evolving strategy mix offers no clear advantage over a fixed mix.

By Haocheng Lu, Haoxuan Cheng, Mingyang Chen, Minjun Zhu
llmsagentsbenchmarks
More like this →