The paper introduces Qwen-GuidePlay-2B, a 2B‑parameter language model fine‑tuned for dialogue‑game interaction. The training pipeline consists of three stages: supervised fine‑tuning on successful game trajectories, weighted turn‑level fine‑tuning, and teacher‑guided fine‑tuning that corrects formatting and evaluates examples. The resulting model achieves a clemscore of 57.12 and a statscore of 42.68 on Playpen’s public validation, ranking second in the official challenge and demonstrating that careful curation can outperform more aggressive procedural methods.
By Syed Mahbubul Huq, Pranava Madhyastha
CAST is a critique‑aware training framework that transforms sparse task outcomes into action‑level supervision for both critique learning and policy optimization. By analyzing agent trajectories, CAST synthesizes structured rationales that explain action validity under partial observability, enabling the creation of richer training data. Fine‑tuned Qwen3‑family models trained with CAST show significant reliability gains, outperforming GPT‑OSS‑120B by over 10% on Retail tasks and improving Telehealth performance by 9% in an out‑of‑domain setting.
By Amir Saeidi, Zehua Zhang, Rishitosh Singh, Naman Ahuja, Vivek Gupta, Ali Payani, Gaowen Liu, Jayanth Srinivasa, Chitta Baral
arXiv:2607. 17191v1 Announce Type: new Abstract: Human-like private chat requires more than fluent response generation: a system must preserve persona, relationship, memory, bounded knowledge, medium-specific timing, and a coherent multi-turn arc.
By Wentao Liu, Siyu Song, Xi Chen, Youjia Li, Xiaokun Wang, Min Ji, Ji Wang
The paper introduces Evaluation-as-Search (EaS), a feedback‑driven method that adaptively probes LLM‑powered meeting assistants by focusing on natural questions likely to reveal grounding failures. Using EaS, the authors build MeetingProbe, a benchmark of over 3,000 annotated question‑answer pairs from 20 transcripts across three meeting genres and three assistants. Ablation studies show that adaptive search uncovers 2.5× more failures than random probing, revealing a capability gradient and eight recurring failure categories dominated by discourse‑pragmatic challenges.
By Sami Khairy, Yasaman Hosseinkashi, Vishak Gopal, Ross Cutler
arXiv:2608.21833v1 Announce Type: new
Abstract: Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especial...
By Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno Zeng
The Agent Error Dataset (AED) presents 50,228 error–diagnosis pairs collected from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text‑based agent systems. A five‑stage Agentic Error‑to‑Training (AET) pipeline generates diagnoses and proposed corrections, verifies them against recorded evidence, and creates separate training views for diagnosis and actor recovery. Experiments show that first‑proposal corrections improve verifier pass rates from 18.4% to 51.1%, and fine‑tuning with full‑diagnosis data raises Qwen3‑8B’s exact‑step agreement from 47.2% to 63.6% on a holdout set.
By Kunlun Zhu, Xuyan Ye, Yibo Li, Cheng Qian, Beibin Li, Heng Ji
S3Gym is an interactive benchmark designed to evaluate large language models (LLMs) on their ability to self-improve through self-testing, self-judging, and self-improvement. It separates permissive exploration from strict held-out evaluation across seven text-based games with executable environment verifiers. Experiments show that self-improvement varies by task, with different experience incorporation pathways (direct history, summary memory, or parameter training) yielding mixed results and highlighting the need for agents to transform feedback into executable, transferable policies.
By Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang
arXiv:2609.25001v1 Announce Type: new
Abstract: Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, a...
By Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan
CivBench is an open‑source benchmark that evaluates language‑model agents in the long‑horizon, tool‑mediated game Civilization VI using the Model Context Protocol (MCP). Each episode lasts over 300 turns, generating thousands of tool calls across a 76‑tool action space, and includes a narration layer that translates visual game state into structured text. The study characterises agent behaviour across four model families, introducing Proactive Monitoring Rate (PMR) and RAG@10 as interface‑level metrics, and finds that agents often under‑monitor strategic state and fail to execute near‑term commitments despite tool access and explicit guidance.
By Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa
SIT‑Eval is a framework that evaluates conversational agents by sampling diverse situations at each turn and judging responses against situational rules. It uses an LLM‑interlocutor to generate uniformly sampled contexts and a context‑aware LLM‑judge to assess rule compliance. Benchmarking eight large language models on 1,292 turns uncovered hidden failure modes, showing that models with similar overall scores can differ greatly in rule adherence, especially in ambiguity clarification and Socratic questioning.
By Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko
arXiv:2610.17310v1 Announce Type: new
Abstract: Language-model agents are expected to adapt to others whose behavior changes through interaction. Yet adaptation may rely on learned behavioral pattern...
By Hanbo Xie, Yue Zhao, Robert C. Wilson
arXiv:2606. 24589v1 Announce Type: new Abstract: Scaling adversarial evaluation of large language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real.
By Khanak Khandelwal (Indian Institute of Technology Jodhpur)