arXiv AI By Jingjie Ning, Shanshan Zhong, Xiaochuan Li, Ji Zeng, Chenyan Xiong

One Run Is Not an Idea: The Implementation Lottery in Automated Research

Read the original on arXiv AI →

arXiv:2607. 26587v1 Announce Type: cross Abstract: Automated research systems use experimental scores both to deliver artifacts and to decide which ideas to retain, transfer, and pursue.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv AI
Aug 19

AutoResearch: Insight In, Hallucination Out

AutoResearch is a two‑stage autonomous research system that links Idea Generation with Idea Execution. In the generation phase it blends new research signals with existing domain knowledge, identifies transferable mechanistic insights, and produces grounded, testable research plans through multi‑model generation and cross‑review. The execution phase then decomposes these plans into experiments, iteratively implements and diagnoses them, and uses independent evidence‑based review to accept or revise conclusions, thereby turning ideas into measurable progress while minimizing hallucinations.

By Yiming Ren, Xiang Liu, Qumeng Sun, Xiao Zhang, Jiahao Li, Haoyang Zhang, Junjie Wang