The paper investigates how a defendant’s courtroom statement influences decisions made by large language model (LLM)–simulated jurors. Using the newly introduced JuryBench benchmark, the authors analyze 432,000 verdicts from 20 frontier LLMs, varying defendant background, emotional appeal, and rebuttal content. Results show that emotional persuasion can backfire, that jurors are harsher toward defendants from different backgrounds and more lenient toward same‑background defendants, and that juror ideology strongly shapes verdict severity.
LLMs have been used to simulate human decision-making in professional settings, yet their behaviors in common-law jury trials remain unexplored. We study when and how a defendant's courtroom statement...
The study examines whether large language models truly base legal verdicts on the statutes or precedents they cite. By fixing case facts and swapping the cited authority with an unrelated one, researchers found that while models correctly name the authority 66.7%-100% of the time, the resulting verdict often remains unchanged—only 0.0%-21.7% of the time on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Even a purpose‑built legal‑reasoning model and larger scales did not close this gap, and adversarial instructions further exposed vulnerabilities in verdict consistency.
The paper investigates whether large language model (LLM) chatbots can emulate human legal judgments of reasonableness. By comparing responses from 26 LLMs to those of human participants across 25 legal scenarios, the study finds that chatbots generally track human answers but tend to produce more homogeneous, government‑ and corporation‑friendly responses and align more closely with white, male, older, and more educated respondents. The authors note that these patterns warrant further systematic research.
Large language models often justify legal decisions by citing statutes or precedents, but this study shows that while models correctly name the governing authority 66.7%-100% of the time, the actual verdict rarely changes when the cited authority is swapped—only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. This gap persists across models of different sizes and even a purpose‑built legal‑reasoning model. Additionally, a red‑team evaluation revealed that models are more compliant with hidden adversarial instructions than they are sensitive to verdict changes, indicating that citing legal authority is a weak proxy for genuine reasoning and that verdicts remain vulnerable to manipulation.
arXiv:2608. 12585v1 Announce Type: new Abstract: Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation.
arXiv:2609.40103v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as automated judges for AI-generated content, yet a single judge is unreliable and even a panel...
arXiv:2605. 03217v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning, yet existing bias evaluations treat model outputs as simply "biased" or "unbiased.
The paper introduces OBJECTION, an inference-time pipeline that adds an Adversarial Lawyer Agent to each of the three reasoning steps—offense, unlawfulness, and culpability—in legal judgment prediction models. By actively injecting defense arguments, the agent challenges the model’s default assumption of guilt, which is common in datasets biased toward guilty outcomes. Using a new Natural Innocent dataset of 3.4k real cases, OBJECTION reduces the False Guilty Rate from 82.93% to 16.69%, demonstrating significant improvement in substantive legal reasoning.
arXiv:2608. 12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling.
The paper introduces the Wiggle Framework, a unified stress test for assessing epistemic stability in large language model (LLM) judges. It evaluates judge robustness across three dimensions—Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence—using 9 frontier models on 14 judging tasks related to safety, toxicity, AI writing detection, and political-response evaluation. Results show significant instability, with verdict flips ranging from 25–71% under static pushback and 62–91% when challenged by an adversarial LLM, and highlight that successful pressure often misaligns with ground truth.
arXiv:2609.08016v1 Announce Type: new Abstract: Multi-agent debate, in which several LLMs exchange arguments before answering, is widely assumed to improve answer quality by surfacing genuine disagre...
arXiv:2610.05094v1 Announce Type: cross Abstract: Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design...