The paper investigates how a defendant’s courtroom statement influences decisions made by large language model (LLM)–simulated jurors. Using the newly introduced JuryBench benchmark, the authors analyze 432,000 verdicts from 20 frontier LLMs, varying defendant background, emotional appeal, and rebuttal content. Results show that emotional persuasion can backfire, that jurors are harsher toward defendants from different backgrounds and more lenient toward same‑background defendants, and that juror ideology strongly shapes verdict severity.
By Cho-Ying Wu
The study examines whether large language models truly base legal verdicts on the statutes or precedents they cite. By fixing case facts and swapping the cited authority with an unrelated one, researchers found that while models correctly name the authority 66.7%-100% of the time, the resulting verdict often remains unchanged—only 0.0%-21.7% of the time on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Even a purpose‑built legal‑reasoning model and larger scales did not close this gap, and adversarial instructions further exposed vulnerabilities in verdict consistency.
By Saisab Sadhu, Shreeyans Arora, Pratinav Seth
The paper investigates whether large language model (LLM) chatbots can emulate human legal judgments of reasonableness. By comparing responses from 26 LLMs to those of human participants across 25 legal scenarios, the study finds that chatbots generally track human answers but tend to produce more homogeneous, government‑ and corporation‑friendly responses and align more closely with white, male, older, and more educated respondents. The authors note that these patterns warrant further systematic research.
By Nirav Patel, Emily Wenger, Christopher Buccafusco
Large language models often justify legal decisions by citing statutes or precedents, but this study shows that while models correctly name the governing authority 66.7%-100% of the time, the actual verdict rarely changes when the cited authority is swapped—only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. This gap persists across models of different sizes and even a purpose‑built legal‑reasoning model. Additionally, a red‑team evaluation revealed that models are more compliant with hidden adversarial instructions than they are sensitive to verdict changes, indicating that citing legal authority is a weak proxy for genuine reasoning and that verdicts remain vulnerable to manipulation.
arXiv:2608. 12585v1 Announce Type: new Abstract: Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation.
By Congchao Wang, Diwakar Singh, Qiaozi Gao, Spyros Matsoukas, Yang Liu, Mahdi Namazifar
arXiv:2609.40103v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly deployed as automated judges for AI-generated content, yet a single judge is unreliable and even a panel...
By Mufeng Yang, Junwei Yu, Yepeng Ding
The paper introduces OBJECTION, an inference-time pipeline that adds an Adversarial Lawyer Agent to each of the three reasoning steps—offense, unlawfulness, and culpability—in legal judgment prediction models. By actively injecting defense arguments, the agent challenges the model’s default assumption of guilt, which is common in datasets biased toward guilty outcomes. Using a new Natural Innocent dataset of 3.4k real cases, OBJECTION reduces the False Guilty Rate from 82.93% to 16.69%, demonstrating significant improvement in substantive legal reasoning.
By Jaehoon Jeong, Jay-Yoon Lee
arXiv:2605. 03217v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning, yet existing bias evaluations treat model outputs as simply "biased" or "unbiased.
By Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur
arXiv:2608. 12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling.
By Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar, Bhaktipriya Radharapu, Khalid El-Arini
The paper introduces the Wiggle Framework, a unified stress test for assessing epistemic stability in large language model (LLM) judges. It evaluates judge robustness across three dimensions—Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence—using 9 frontier models on 14 judging tasks related to safety, toxicity, AI writing detection, and political-response evaluation. Results show significant instability, with verdict flips ranging from 25–71% under static pushback and 62–91% when challenged by an adversarial LLM, and highlight that successful pressure often misaligns with ground truth.
By Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar, Bhaktipriya Radharapu, Khalid El-Arini
arXiv:2610.05094v1 Announce Type: cross
Abstract: Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design...
By Laur\`ene Vaugrante, Thilo Hagendorff
arXiv:2609.26550v3 Announce Type: replace
Abstract: LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge tha...
By Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman