arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
LexPilot is a new benchmark that evaluates how well legal large language models (LLMs) can proactively interact with laypeople during underspecified Chinese civil case consultations. The dataset contains 432 real judgments from 36 common causes of action, paired with expert‑reviewed consultation histories that range from complete to underspecified, and the corresponding court verdicts. Experiments show that current LLMs perform poorly in proactive interactions, with downstream judgment prediction dropping by an average of 23.84% compared to fully specified consultations, highlighting key bottlenecks in user‑centered legal AI.
The paper presents an expert‑validated benchmark of 1,042 items across ten areas of Colombian law to assess the reliability of large language models (LLMs) on national legal systems outside the U.S. It evaluates 15 contemporary proprietary and open‑weight models, finding that while closed‑choice accuracy can be high (up to 0.905), free‑text factual correctness never exceeds 0.45. The study reveals a strong dissociation between answer relevance and correctness, systematic variation by legal area, and suggests that expert supervision and grounding in authoritative sources are needed for reliable Colombian legal tasks.
arXiv:2609.22127v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to draft contractual language, yet conventional accuracy or preference-based evaluations are poorly...
CLASE is a hybrid evaluation method for Chinese legal text that combines linguistic feature-based scores with experience-guided LLM-as-a-judge scores. It learns from contrastive pairs of authentic legal documents and their LLM-generated counterparts, enabling transparent, reference-free assessment of stylistic quality. Experiments on 200 Chinese legal documents show that CLASE aligns better with human judgments than traditional metrics and offers interpretable score breakdowns and improvement suggestions.
Civil litigation is inherently a life-cycle process: what a lawyer drafts on day one constrains what unfolds at trial months later. Yet existing legal benchmarks evaluate isolated subtasks, and prior legal-agent simulators reinitialize each scenario from shared ground truth, leaving cross-stage causal dependencies unmodeled.
arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.
arXiv:2605. 21071v4 Announce Type: replace-cross Abstract: The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses.
arXiv:2606. 06679v1 Announce Type: cross Abstract: Court judgments are central to legal practice and jurisprudence, yet discourse analysis of Hong Kong judgments has received limited attention, owing largely to the absence of expert-annotated corpora.
arXiv:2608. 10268v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how.
arXiv:2509.00141v2 Announce Type: replace-cross Abstract: Statutory corpora and judicial decisions are growing faster than legal professionals can read them, while individual judgments often exceed t...