arXiv AI

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

arXiv:2608. 09128v1 Announce Type: cross Abstract: LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents.

arXiv Computation and Language
Sep 1

S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

S3Gym is an interactive benchmark designed to evaluate large language models (LLMs) on their ability to self-improve through self-testing, self-judging, and self-improvement. It separates permissive exploration from strict held-out evaluation across seven text-based games with executable environment verifiers. Experiments show that self-improvement varies by task, with different experience incorporation pathways (direct history, summary memory, or parameter training) yielding mixed results and highlighting the need for agents to transform feedback into executable, transferable policies.

By Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang