arXiv:2603. 22973v2 Announce Type: replace Abstract: Applying computational methods to law at scale requires separating genuine legal reasoning from surface similarity.
arXiv:2608. 20220v1 Announce Type: new Abstract: Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified.
arXiv:2605. 21071v4 Announce Type: replace-cross Abstract: The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses.
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
arXiv:2606. 23716v1 Announce Type: cross Abstract: Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice, including for people who cannot access lawyers in order to understand and exercise their legal rights.
arXiv:2606. 06679v1 Announce Type: cross Abstract: Court judgments are central to legal practice and jurisprudence, yet discourse analysis of Hong Kong judgments has received limited attention, owing largely to the absence of expert-annotated corpora.
The study examines whether large language models (LLMs) can perform legally meaningful reasoning by testing OpenAI GPT 5.4 on European Court of Human Rights case forecasting. Using various prompting strategies, the authors find that the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet only weakly aligned with human annotators. The expert-curated prompt yields more comprehensive reasoning but does not improve prediction accuracy, leading the authors to caution against relying solely on automated LLM evaluation or using task accuracy as a proxy for reasoning quality.
LEGO is a dual‑module framework that combines a Legal Expert GraphRAG system with an expert Chain‑of‑Thought approach to enhance complex legal reasoning. The GraphRAG component uses an expert‑annotated civil code graph and a greedy normative‑coverage retrieval algorithm to extract relevant provision subgraphs, while the Chain‑of‑Thought module structures retrieved provisions and case facts into a Provision‑Fact‑Conclusion reasoning flow. Using a Qwen3‑8B backbone, LEGO achieves 40.53% exact‑match accuracy on LawExamQA_Civil, surpassing baseline RAG and CoT models and matching larger models on multi‑hop and open‑ended benchmarks, with ablation studies confirming the complementary benefits of both modules.
arXiv:2610.19427v1 Announce Type: new Abstract: Legal reasoning is a fundamental capability for large language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal pra...
arXiv:2609.23726v1 Announce Type: new Abstract: Large language models have shown strong performance across a range of legal tasks, but existing benchmarks rarely evaluate the ability to take and defe...
arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.
arXiv:2605. 25240v2 Announce Type: replace-cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs.