arXiv Machine Learning

Reinforcement Learning of Communication in a Mesh of Small Language Models

The paper introduces TalkMesh, a decentralized network of small language model agents that learn to communicate effectively during inference. Each agent proposes an answer, scores it with a confidence head, and the most confident agent broadcasts a hint; lower‑confidence agents revise their proposals if a new suggestion scores higher. This gossip‑based consensus, trained via group relative policy optimization, enables a mesh of three agents to match the accuracy of majority voting over 32 samples, and scales to larger meshes to significantly boost performance on benchmarks like GSM8K and MATH-500.

arXiv Computation and Language
Nov 9

The Consensus Trap: Collaborative Interleaved Generation to Defend against a Corrupted Majority

The paper identifies a "consensus trap" where majority voting among large language model agents can reinforce incorrect answers when the majority follows misleading instructions. It proposes chunk‑interleaved Round‑Robin (RR) collaboration, allowing agents to take turns extending a shared reasoning trajectory so that minority agents can insert corrections before the final answer is produced. Experiments across model families and reasoning benchmarks show that RR improves accuracy under majority corruption while maintaining performance under minority corruption, and that shared reasoning during turns tends to increase support for the correct answer.

By Jiayuan Liu, Shiyi Du, Weihua Du, Xinhao Cheng, Mingyu Guo, Vincent Conitzer
arXiv AI
Oct 2

Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries

The paper introduces Bayesian Dialectical Argumentation (BDA), a method for aggregating answers from multiple large language models (LLMs) in a council setting. BDA treats each LLM’s typed moves—proposals, challenges, and concessions—as evidence in a classical annotator model, estimating per-agent reliability even when some agents are persistently unreliable. By weighting evidence according to these inferred reliabilities, BDA produces calibrated posterior probabilities for candidate answers and can invert unreliable agents instead of merely outvoting them, achieving superior calibration and robustness on both binary and multi-class benchmarks without extra LLM calls.

By Ionel Eduard Stan, Paolo Napoletano
arXiv AI
Oct 28

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

TeamGym is a new environment that evaluates and trains language‑model agents on team decision‑making tasks drawn from real‑world legal, finance, medical, and software engineering texts. It introduces the concept of pivotal information—a minimal set of facts necessary for a correct decision—and tracks how agents share and retrieve this information. Experiments show that distributing pivotal facts and requiring agents to extract them independently significantly reduces success rates, but reinforcement‑learning training can improve decision quality and foster coordination among team members.

By Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz, Yubin Kim, Taekyun Lee, Kaiqing Zhang, Asuman Ozdaglar
arXiv AI
Oct 31

Seq@K: Evaluating Language Models Under Sequential Feedback

Seq@K is a new evaluation protocol that measures a language model’s ability to solve a problem within K sequential attempts, receiving feedback after each failure. It isolates the core capability of iterative refinement from feedback, contrasting with existing metrics like Pass@K that do not capture this behavior. Experiments on seven benchmarks and sixteen frontier models show that Seq@K can change model rankings, reveal more efficient configurations, and provide insights into how models use feedback.

By Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
arXiv AI
Oct 16

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

The paper investigates why reinforcement learning with self‑generated labels—specifically majority‑vote rewards—can recover most of the performance gains of training with verifiable rewards. It argues that the key factor is batch‑level updates: a prompt’s own gradient contributes less than 1% to its output change, so only coherent label errors across prompts influence learning. Experiments on five models show that majority‑vote updates align almost perfectly with gold updates, with misdirection disappearing as batch size grows and incoherent errors behaving like random noise.

By Dylan P. Wilson, Chengsong Huang, Jiaxin Huang