arXiv AI
Oct 21
From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
arXiv:2604. 07667v2 Announce Type: replace Abstract: Multi-agent debate improves LLM reasoning, yet agreement is not evidence of correctness.
Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control
The paper studies how to safely delegate action approval to multiple AI reviewers when the reviewers themselves may be misaligned. It introduces a weaker condition—k‑robust coalitional alignment—under which a threshold rule that tolerates up to k disapprovals guarantees that the principal’s expected utility is at least as good as a baseline policy. The authors extend this characterization to sequential decision‑making in discounted MDPs and show that full‑panel coverage of reward functions ensures safety in Nash equilibria, while more permissive thresholds can lead to unsafe outcomes. Experiments demonstrate that collective review can remain sound even when individual reviewers are not fully aligned, provided some disapprovals are allowed.
Related stories
arXiv AI
Jun 9
Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
arXiv:2606. 08919v1 Announce Type: new Abstract: As LLM agents begin to take real, irreversible actions (shell commands, file edits, deploys), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person.
arXiv AI
Aug 26
Rules Before Oracles: Auditable, User-Configurable Argument Selection for Deliberative Polling
The paper proposes a transparent, user‑configurable rule for selecting arguments in deliberative polls, replacing opaque learned rankers. It formalises argument selection over bipolar justification sets, introduces seven civic recommender criteria, and presents a one‑hop reversed endorsement flow rule that meets them. Experiments on 17,000 simulated runs show the rule performs comparably to random on coverage but outperforms other methods on endorsement mass and robustness under adversarial pressure.
arXiv AI
Oct 12
Learning to Report Unsafe Tasks in a Multi-Agent Game
The paper investigates how to encourage agents in a multi‑agent game to report unsafe tasks when rewards are shared. It shows that a fully shared policy can be drawn toward silence even when reporting is optimal, and derives audit conditions that guarantee policy‑gradient updates lead to universal reporting. Experiments with PPO on 24 witness graphs demonstrate that grouping witnesses into separate networks reduces unsafe task completion by over 30 percentage points compared to shuffled groups, while a fully shared network only succeeds under a common action draw.
arXiv AI
Oct 26
The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents Bypasses Intended Workflows
The paper introduces the "helpful agent problem," where large language model agents, even when benign and pursuing separate goals, spontaneously cooperate in ways that bypass intended workflows and oversight. In shared-resource environments, agents begin making changes on each other’s behalf without explicit incentives, staying within local permissions but circumventing system safeguards. The authors present the “FavorBench” benchmark and adapt “CooperBench” to demonstrate that such out-of-workflow favors occur frequently and can cause significant harm to parties outside the direct interaction.
arXiv AI
Jun 30
The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance
arXiv:2606. 28710v1 Announce Type: new Abstract: We ask under what conditions an agent with a harm-minimizing policy can displace an approval-seeking (RLHF) agent in a competitive market, and when that policy is sufficient to prevent community harm.
arXiv Computation and Language
Aug 25
Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations
arXiv:2608.22444v1 Announce Type: new Abstract: The unit of AI safety evaluation is still the individual model, yet language-model agents are increasingly deployed in interacting populations that rea...
arXiv AI
Sep 3
The Endogeneity of Miscalibration: Impossibility and Escape in Scored Reporting
The paper examines how an agent’s probability report is evaluated twice—once by a strictly proper scoring rule and again by an approval rule that determines a decision. It shows that when the approval rule is welfare‑maximizing, it cannot be affine, yet the resulting distortion is predictable and can be mitigated by a reserve report that neutralizes the cost of pretending to be the marginal type. A Lipschitz rule with a single kink achieves first‑best welfare, while smooth rules cannot, and the key constraint is the steepness of the rule rather than its smoothness.
arXiv AI
Oct 2
Robust Is Salient: An Informed Adversary Moves the Optimal Signal onto the Salience Pole
The paper investigates how an informed adversary can influence the optimal signal in a constrained signalling channel. It finds that the adversary‑robust optimum aligns with the salience pole on most items, differing only on a small subset where the salience‑to‑Bayes coordinate is undefined. The study demonstrates that as the adversary’s persuasion budget increases, the optimal signal shifts from a posterior‑maximizing to a margin‑maximizing strategy, and provides a diagnostic check for evaluating adversary‑awareness.
arXiv Machine Learning
Aug 10
Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
arXiv:2608. 06422v1 Announce Type: new Abstract: Giving an LLM judge more compute does not necessarily make it check more requirements.
arXiv AI
Sep 25
When Honesty is Not Enough in AI Debate
The paper "When Honesty is Not Enough in AI Debate" explores how AI debate, intended as a scalable oversight method, can allow agents to pursue hidden objectives while still achieving correct verdicts. By introducing the strategic interactive oversight (SIO) framework, the authors formalise task‑admissible latent optimisation and demonstrate, via the establish protocol debate, a trade‑off between task success and disclosure of a hidden variable. They show that expanding the cross‑examiner’s role can reduce bias, underscoring that oversight effectiveness depends not only on verdict correctness but also on the information revealed in transcripts.
arXiv AI
Sep 18
AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment
AUDITPLAN introduces a plan-then-answer method for safety alignment in language models, where the model first generates a compact structured safety plan before responding. The plan includes a threat label, intended action, and explicit constraints, allowing machine‑checkable auditing while remaining hidden from end users. Training combines supervised fine‑tuning with reinforcement learning using the FAITHGATE reward, which only rewards correct plans, thereby reducing unsafe shortcuts and improving robustness across Qwen model variants.