arXiv AI
Building Legal Reward Models for Grounding and Abstention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 17
Fine-grained Claim-level RAG Benchmark for Law
arXiv:2605. 21071v4 Announce Type: replace-cross Abstract: The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses.
arXiv Computation and Language
Oct 1
LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models
LexReward is a taxonomy-driven reward framework designed for legal language models, evaluating responses across three dimensions: Style (lexical and syntactic quality), Element (legal subjects, facts, statutes, and decisions), and Chain (order, completeness, correctness, and non-redundancy of reasoning). The framework introduces rubrics that define evaluation criteria and quality levels for each dimension, generating pairwise preference data used for Direct Preference Optimization (DPO) and reward-model training. Experiments demonstrate that rubric-based rewards effectively differentiate legal responses of varying quality, and that DPO training improves performance across all dimensions; the resulting reward models (LexRM) enable reinforcement learning to enhance policy performance in each specific dimension without needing reference answers.
Hugging Face Trending Papers
Sep 30
LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models
Legal language models require reward signals that capture not only answer correctness but also the multidimensional quality of legal responses. Existing reward methods, however, often rely on coarse-g...
arXiv Computation and Language
Oct 22
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
CanLegalRAGBench is a Canadian legal QA benchmark that uses realistic queries and expert‑annotated answers grounded in case law to evaluate retrieval‑augmented generation (RAG) systems. The study finds that retrieval performance varies with design choices and that open‑source embedding models can compete with closed‑source ones. It also highlights limitations of automatic evaluations, noting that RAG systems often produce hallucinations or overly detailed responses, with 8‑29% of claims unsupported by retrieved documents.
arXiv AI
Aug 19
CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method
CoAL‑RAG is a complexity‑aware legal retrieval‑augmented generation method that adapts its retrieval strategy based on a multi‑dimensional evaluation of question essence and retrieval consistency. It quantifies reasoning demand from the logical structure of a question and uses the discrepancy between semantic and keyword retrieval to gauge problem complexity, thereby selecting the most suitable retrieval approach and filtering context dynamically. Experiments show that CoAL‑RAG outperforms baseline models on Chinese legal benchmarks (SocialLawQA, LawBench) with a 42.5% BLEU improvement and 3.6× ROUGE‑L, while also achieving strong cross‑jurisdictional performance on English datasets (LexGLUE, CaseHold).
arXiv AI
Aug 10
Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
arXiv:2605. 29738v2 Announce Type: replace-cross Abstract: Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible.