The paper investigates whether fine‑tuning improves how language models use supplied Bangladeshi legal text in bilingual question‑answering. Using a hierarchy‑preserving statutory corpus, 2,165 fine‑tuning examples, and a 150‑item control set, the authors evaluate six instruction‑tuned models with multiple LoRA seeds, separating scoring, retrieval, and model effects. Results show that while fine‑tuning can boost overall accuracy, it does not increase the models’ reliance on the governing provision, highlighting the need to disentangle scorer, retriever, and model contributions in legal adaptation studies.
arXiv:2607. 23446v1 Announce Type: cross Abstract: A small language model can receive the governing statutory provision and still answer incorrectly.
The paper presents a smartphone‑compatible, retrieval‑augmented language model tailored to Bangladeshi statutory law. By distilling a 9‑billion‑parameter Gemma‑2 teacher into a 2‑billion‑parameter student using supervised fine‑tuning and QLoRA, the authors achieve significant gains in ROUGE‑L and BERTScore on an English benchmark while keeping the model lightweight (1.6 GB) and operable offline on a Pixel 6. The system retrieves from 36,029 statutory passages using a hybrid dense/BM25 approach, and cross‑lingual evaluation shows effective Bangla query handling against an English‑only corpus, with a practicing lawyer rating the responses highly in a pilot study.
arXiv:2610.32144v1 Announce Type: new Abstract: When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, an...
The paper presents a retrieval‑augmented generation pipeline for answering regulatory compliance questions in finance. It builds a three‑stage retriever on LegalBERT and a compact 2B–12B generator served with 4‑bit quantization, achieving a Recall@10 of 0.774 on the ObliQA benchmark and improving answer quality via RAFT‑LoRA fine‑tuning. However, the adapted models fail to transfer to Australian case‑law questions, and a closed‑book model performs almost as well while lacking verifiable grounding.
The paper presents an expert‑validated benchmark of 1,042 items across ten areas of Colombian law to assess the reliability of large language models (LLMs) on national legal systems outside the U.S. It evaluates 15 contemporary proprietary and open‑weight models, finding that while closed‑choice accuracy can be high (up to 0.905), free‑text factual correctness never exceeds 0.45. The study reveals a strong dissociation between answer relevance and correctness, systematic variation by legal area, and suggests that expert supervision and grounding in authoritative sources are needed for reliable Colombian legal tasks.
arXiv:2604.19298v3 Announce Type: replace-cross Abstract: A benchmark leaderboard reports one accuracy number, but that number is a joint product of a model and a scoring rule. We ask what changes wh...
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
arXiv:2605. 25420v2 Announce Type: replace-cross Abstract: Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed globally.
arXiv:2610.18420v1 Announce Type: cross Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...
arXiv:2610.24764v2 Announce Type: replace Abstract: This paper presents BanglaBBQ, a question-answering bias benchmark designed to capture stereotypes grounded in the Bangladeshi context, in both Ban...
IndicQE-APE is a consolidated benchmark that unifies quality estimation (QE) and automatic post‑editing (APE) data for nine Indic language pairs, comprising 126,754 instances with multiple aligned labels such as direct assessment, human post‑edit, word‑level OK/BAD tags, and error explanations. The dataset includes a stratified test set across four difficulty axes and supports training and evaluation of six prompted large language models, three COMET metrics, and three APE systems. Experiments reveal that segments with conflicting holistic and token‑level quality signals are consistently ranked lower, while annotator disagreement shows no effect when controlled for score distribution. whyItMatters":"The benchmark provides a unified resource for training and evaluating QE and APE across Indic languages, enabling consistent comparison of models and metrics on a shared dataset."
arXiv:2607. 08731v2 Announce Type: replace-cross Abstract: National language models are becoming publicly funded epistemic infrastructure.