Auto-RecSys is an autonomous research system designed to scale long-horizon experimentation for industry‑scale recommendation models. It tackles long feedback loops and system complexity by enabling distributed asynchronous execution, centralized cross‑server memory, and a cognitive‑procedural separation that combines natural‑language skill files with deterministic scripts. The system incorporates a dual‑loop self‑evolving architecture—Execution Evolution and Idea Evolution loops—to refine operational playbooks and guide future experiments, thereby reducing human effort per cycle and improving reliability as playbooks mature.
By Ming Li, Dai Li, Xuying Ning, Bo Sun, Rui Li, Yi Zhang, Silvia Gong, Xuan Cao, Rui Li, Cornelia Carapcea, Qunshu Zhang, Zhigang Wang, Yinglong Xia, Andy Wang
arXiv:2603. 05026v2 Announce Type: replace-cross Abstract: Language model (LM) agents have driven substantial progress in automated software engineering (SWE), yet building and testing software repositories at scale remains a largely manual and labor-intensive bottleneck.
By Kenan Li, Rongzhi Li, Linghao Zhang, Qirui Jin, Liao Zhu, Xiaosong Huang, Geng Zhang, Yikai Zhang, Shilin He, Chengxing Xie, Xin Zhang, Zijian Jin, Bowen Li, Chaoyun Zhang, Yu Kang, Yufan Huang, Elsie Nallipogu, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang
The paper explores whether natural‑language documentation aids coding agents in fixing software bugs and introduces a roundtrip benchmark that evaluates code descriptions by regenerating code and testing it. It finds that description completeness, not length, determines fidelity, and presents an optimizer that can produce fully faithful descriptions that generalize to new files. However, experiments across two model families and ten repositories show that such compact documentation does not improve an agent’s ability to resolve real repository issues compared to using the issue alone.
By Md Shohel Arman, Igor Molybog
RPCBench is a new benchmark designed to evaluate large language models’ ability to critique recommendation requests by detecting, diagnosing, and handling flawed premises. It includes evidence‑grounded test instances across five recommendation domains and ten types of premise failures, and introduces a fine‑grained evaluation framework covering detection, error localization, handling strategy, and evidence faithfulness. Experiments with 11 LLMs reveal that proactive detection is the main bottleneck, with models struggling most on underspecified‑premise errors and showing that optimal critique quality occurs at intermediate reasoning lengths.
By Zhongru Chen, Yuan Wu, Yi Chang
arXiv:2606. 10156v1 Announce Type: cross Abstract: As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace.
By Bharath Sivaram Narasimhan, Karthik R Narasimhan
arXiv:2601. 19072v3 Announce Type: replace-cross Abstract: Large Language models (LLMs) have shown strong capabilities in code review automation, such as review comment generation, yet they suffer from hallucinations -- where the generated review comments are ungrounded in the actual code -- poses a significant challenge to the adoption of LLMs in code review workflows.
By Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, Ming Wu
The paper demonstrates that large language models can generate executable procedural content generators, enabling direct search over generator programs rather than individual levels. Using Sokoban, Zelda, Dangerous Dave, and Lode Runner, the authors evolve complete Python generators via language‑model mutation and crossover, and introduce Continual Abstraction Discovery (CAD) to extract reusable primitives into a run‑specific helper module. Experiments show that CAD consistently improves mean final best fitness across all domain and API comparisons, with learned libraries being adopted by subsequent programs and repeatedly rediscovering useful utilities.
By Matthew Siper, Ahmed Khalifa, Julian Togelius
The paper presents EvalAgent, an AI assistant that automates agent evaluation by encoding domain expertise into evaluation skills such as procedural instructions, reusable code, and dynamic API retrieval. EvalAgent constructs a trace-based pipeline that outputs metrics, executable code, and reports, and is evaluated using a new meta-evaluation framework and AgentEvalBench. Results show that EvalAgent improves the Eval@1 metric from 17.5% to 65% and receives 79.5% human expert preference, while ablation studies confirm the importance of evaluation skills.
By Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Smeet Dhakecha, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong
arXiv:2501. 07892v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance in automated code generation, with few-shot prompting widely used for its simplicity and effectiveness.
By Shengsheng Zhou, Shuai Wang, Liang Ding, Yibing Zhan, Yong Luo, Zheng He, Fu Lin, Dapeng Tao
arXiv:2607. 18603v1 Announce Type: cross Abstract: We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system.
By Sam O'Nuallain, Nithya Rajkumar, Ramya Narayanasamy, Hanna Jiang, Shreyas Chaudhari, Andrew Drozdov
arXiv:2607. 02520v1 Announce Type: cross Abstract: Automated research agents increasingly generate code, retrieve literature, and draft scientific artifacts, but they often fail to verify whether generated experiments execute correctly or whether cited sources support generated claims.
By Rajesh Kumar, Waqar Ali, Junaid Ahmed, Abdullah Aman Khan, Shaoning Zeng
AutoLR is an autonomous harness designed to streamline the iterative research‑and‑engineering cycle for industrial recommender systems, exemplified by NetEase’s gaming‑community app DASHEN. It integrates a multi‑expert council for adversarial review, a deterministic evidence‑weighted selector to allocate trial budgets, and a layered knowledge system that fuses external research with domain‑specific insights and empirical evidence. Large language model agents handle semantic reasoning and code generation, while deterministic controllers maintain control over execution, metrics, guardrails, and state management.
By Qi Zhang, Yanlin Chen, Wenchao Xiao