The paper investigates whether fine‑tuning improves how language models use supplied Bangladeshi legal text in bilingual question‑answering. Using a hierarchy‑preserving statutory corpus, 2,165 fine‑tuning examples, and a 150‑item control set, the authors evaluate six instruction‑tuned models with multiple LoRA seeds, separating scoring, retrieval, and model effects. Results show that while fine‑tuning can boost overall accuracy, it does not increase the models’ reliance on the governing provision, highlighting the need to disentangle scorer, retriever, and model contributions in legal adaptation studies.
arXiv:2607. 23446v1 Announce Type: cross Abstract: A small language model can receive the governing statutory provision and still answer incorrectly.
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
arXiv:2605. 25420v2 Announce Type: replace-cross Abstract: Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed globally.
IndicQE-APE is a consolidated benchmark that unifies quality estimation (QE) and automatic post‑editing (APE) data for nine Indic language pairs, comprising 126,754 instances with multiple aligned labels such as direct assessment, human post‑edit, word‑level OK/BAD tags, and error explanations. The dataset includes a stratified test set across four difficulty axes and supports training and evaluation of six prompted large language models, three COMET metrics, and three APE systems. Experiments reveal that segments with conflicting holistic and token‑level quality signals are consistently ranked lower, while annotator disagreement shows no effect when controlled for score distribution. whyItMatters":"The benchmark provides a unified resource for training and evaluating QE and APE across Indic languages, enabling consistent comparison of models and metrics on a shared dataset."
arXiv:2607. 08731v2 Announce Type: replace-cross Abstract: National language models are becoming publicly funded epistemic infrastructure.
arXiv:2608. 09028v1 Announce Type: new Abstract: Institutional policies stay in natural language while the systems that check compliance demand machine-readable constraints.
KhatianDoc is a new benchmark that tests multimodal large language models on Bengali legal land records, specifically the handwritten RS Khatians used in Bangladesh. The benchmark comprises four tasks—symbol recognition, base‑16 to decimal conversion, structured field extraction, and legal document question answering—drawn from 107 real records and 1,634 QA pairs. Six multimodal LLMs were evaluated under a zero‑shot protocol, revealing that many models fail to answer a significant portion of questions correctly and perform poorly on arithmetic conversion, highlighting a lack of capability rather than a performance gap.
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?
arXiv:2605. 29738v2 Announce Type: replace-cross Abstract: Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible.
Small language models can grade open‑ended exam answers as reliably as much larger models when they use an explicit rubric. In experiments with six cost‑efficient model configurations, the rubric decouples grading from judge intelligence, with answer identity explaining 95.6% of score variance and judge identity only 0.2%. Removing rubric criteria or the official answer collapses reliability and inflates scores, showing the rubric’s essential role.
The paper introduces THPT‑Ladder, a benchmark based on Vietnam’s 2025 National High School Graduation Examination’s convex grading scheme, which rewards partial credit non‑additively. It shows that standard accuracy metrics inflate model scores because they treat partial knowledge proportionally, whereas the official rubric penalizes incomplete correct sets. Using the benchmark, the authors demonstrate that this discrepancy can shift a model’s percentile ranking by up to 13 points among over 480,000 candidates.
arXiv:2607. 08731v1 Announce Type: cross Abstract: A national language model offers a linguistic community its own instrument for measuring what its citizens say and value.