The paper investigates whether large language model (LLM) chatbots can emulate human legal judgments of reasonableness. By comparing responses from 26 LLMs to those of human participants across 25 legal scenarios, the study finds that chatbots generally track human answers but tend to produce more homogeneous, government‑ and corporation‑friendly responses and align more closely with white, male, older, and more educated respondents. The authors note that these patterns warrant further systematic research.
arXiv:2603.19042v5 Announce Type: replace Abstract: The integration of artificial intelligence (AI) into judicial decision making -- particularly in pretrial, sentencing, and parole contexts -- has g...
arXiv:2608.28593v1 Announce Type: new Abstract: With the increasing development of AI regulatory frameworks, ensuring that artificial intelligence systems, particularly generative models, operate in...
arXiv:2607. 05680v1 Announce Type: cross Abstract: AI systems are increasingly used to provide legal advice, raising questions about whether laypeople accept guidance from algorithms--especially when that advice is legally correct but socially controversial.
arXiv:2606. 23716v1 Announce Type: cross Abstract: Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice, including for people who cannot access lawyers in order to understand and exercise their legal rights.
arXiv:2604. 26233v3 Announce Type: replace Abstract: As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions.
The paper examines whether AI‑driven debate can help humans and weaker language models (LLMs) correct their judgments. Experiments show that debate improves human accuracy over prior beliefs, especially with weighted voting, but offers little benefit—and sometimes harms—accuracy for weaker LLM supervisors. The study highlights debate as a promising tool for scalable human oversight while questioning its effectiveness for fully automated AI control.
The paper investigates how a defendant’s courtroom statement influences decisions made by large language model (LLM)–simulated jurors. Using the newly introduced JuryBench benchmark, the authors analyze 432,000 verdicts from 20 frontier LLMs, varying defendant background, emotional appeal, and rebuttal content. Results show that emotional persuasion can backfire, that jurors are harsher toward defendants from different backgrounds and more lenient toward same‑background defendants, and that juror ideology strongly shapes verdict severity.
LLMs have been used to simulate human decision-making in professional settings, yet their behaviors in common-law jury trials remain unexplored. We study when and how a defendant's courtroom statement...
Large language models often justify legal decisions by citing statutes or precedents, but this study shows that while models correctly name the governing authority 66.7%-100% of the time, the actual verdict rarely changes when the cited authority is swapped—only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. This gap persists across models of different sizes and even a purpose‑built legal‑reasoning model. Additionally, a red‑team evaluation revealed that models are more compliant with hidden adversarial instructions than they are sensitive to verdict changes, indicating that citing legal authority is a weak proxy for genuine reasoning and that verdicts remain vulnerable to manipulation.
The study examines whether large language models (LLMs) can perform legally meaningful reasoning by testing OpenAI GPT 5.4 on European Court of Human Rights case forecasting. Using various prompting strategies, the authors find that the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet only weakly aligned with human annotators. The expert-curated prompt yields more comprehensive reasoning but does not improve prediction accuracy, leading the authors to caution against relying solely on automated LLM evaluation or using task accuracy as a proxy for reasoning quality.
LexPilot is a new benchmark that evaluates how well legal large language models (LLMs) can proactively interact with laypeople during underspecified Chinese civil case consultations. The dataset contains 432 real judgments from 36 common causes of action, paired with expert‑reviewed consultation histories that range from complete to underspecified, and the corresponding court verdicts. Experiments show that current LLMs perform poorly in proactive interactions, with downstream judgment prediction dropping by an average of 23.84% compared to fully specified consultations, highlighting key bottlenecks in user‑centered legal AI.
arXiv:2608. 10268v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how.