arXiv:2605. 21071v4 Announce Type: replace-cross Abstract: The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses.
LexReward is a taxonomy-driven reward framework designed for legal language models, evaluating responses across three dimensions: Style (lexical and syntactic quality), Element (legal subjects, facts, statutes, and decisions), and Chain (order, completeness, correctness, and non-redundancy of reasoning). The framework introduces rubrics that define evaluation criteria and quality levels for each dimension, generating pairwise preference data used for Direct Preference Optimization (DPO) and reward-model training. Experiments demonstrate that rubric-based rewards effectively differentiate legal responses of varying quality, and that DPO training improves performance across all dimensions; the resulting reward models (LexRM) enable reinforcement learning to enhance policy performance in each specific dimension without needing reference answers.
Legal language models require reward signals that capture not only answer correctness but also the multidimensional quality of legal responses. Existing reward methods, however, often rely on coarse-g...
CanLegalRAGBench is a Canadian legal QA benchmark that uses realistic queries and expert‑annotated answers grounded in case law to evaluate retrieval‑augmented generation (RAG) systems. The study finds that retrieval performance varies with design choices and that open‑source embedding models can compete with closed‑source ones. It also highlights limitations of automatic evaluations, noting that RAG systems often produce hallucinations or overly detailed responses, with 8‑29% of claims unsupported by retrieved documents.
CoAL‑RAG is a complexity‑aware legal retrieval‑augmented generation method that adapts its retrieval strategy based on a multi‑dimensional evaluation of question essence and retrieval consistency. It quantifies reasoning demand from the logical structure of a question and uses the discrepancy between semantic and keyword retrieval to gauge problem complexity, thereby selecting the most suitable retrieval approach and filtering context dynamically. Experiments show that CoAL‑RAG outperforms baseline models on Chinese legal benchmarks (SocialLawQA, LawBench) with a 42.5% BLEU improvement and 3.6× ROUGE‑L, while also achieving strong cross‑jurisdictional performance on English datasets (LexGLUE, CaseHold).
arXiv:2605. 29738v2 Announce Type: replace-cross Abstract: Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible.
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
arXiv:2605.25920v2 Announce Type: replace Abstract: While large language models (LLMs) augmented with agentic search capabilities show promise for legal reasoning, they overlook a fundamental constra...
Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietnamese legal benchmarks provide limited coverage o...
arXiv:2604. 06173v2 Announce Type: replace-cross Abstract: Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning.
The paper introduces Legal Rule Induction (LRI), a task that seeks to extract concise, generalizable doctrinal rules from analogous judicial precedents. It presents a reproducible pipeline for constructing LRI datasets and, using Chinese law, releases the first benchmark comprising 5,121 case sets (38,088 court cases) for training and 216 expert‑annotated gold test sets. Experiments show that state‑of‑the‑art large language models struggle with over‑generalization and hallucination, but training on the new dataset significantly improves their ability to capture nuanced rule patterns across similar cases.
ViLegalExpert is a large-scale Vietnamese legal benchmark built from real citizen–lawyer consultations, comprising over 172,000 questions across 34 legal domains with professional answers and expert-verified evidence. It supports legal information retrieval, extractive QA, and abstractive QA. Experiments show that while pretrained language models perform well on QA, hybrid retrieval methods achieve the best evidence retrieval, highlighting significant challenges in grounding legal answers to authoritative sources.