The paper investigates whether fine‑tuning improves how language models use supplied Bangladeshi legal text in bilingual question‑answering. Using a hierarchy‑preserving statutory corpus, 2,165 fine‑tuning examples, and a 150‑item control set, the authors evaluate six instruction‑tuned models with multiple LoRA seeds, separating scoring, retrieval, and model effects. Results show that while fine‑tuning can boost overall accuracy, it does not increase the models’ reliance on the governing provision, highlighting the need to disentangle scorer, retriever, and model contributions in legal adaptation studies.
By Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen
arXiv:2608. 09393v1 Announce Type: cross Abstract: We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article when the applicable version is an earlier or future one.
By Rose Cymbler, Daniel Guez, Laurent Fabre
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
By Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
By Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu, Kuan Hsuan Yeh, Zih-Ching Chen, Kuan-Ming Chen, Patrick Chung-Chia Huang
The paper investigates how different scoring rules affect the ranking of large language models (LLMs) on the IndiaFinBench benchmark, which contains 406 expert‑annotated items covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. Using twelve LLMs evaluated under strict string‑matching, a cross‑model judge, and a composite regime, the authors find that model rankings vary dramatically across regimes, with no significant rank correlation (Spearman ρ = -0.273). Item‑level analysis reveals that a substantial portion of items are answered identically by all models, while a higher‑discrimination band isolates more challenging cases.
By Rajveer Singh Pall
The paper introduces THPT‑Ladder, a benchmark based on Vietnam’s 2025 National High School Graduation Examination’s convex grading scheme, which rewards partial credit non‑additively. It shows that standard accuracy metrics inflate model scores because they treat partial knowledge proportionally, whereas the official rubric penalizes incomplete correct sets. Using the benchmark, the authors demonstrate that this discrepancy can shift a model’s percentile ranking by up to 13 points among over 480,000 candidates.
By Nguyen Quoc Hung, Nguyen Dang Minh, Le Nhu Quynh, Tran Khanh Linh, Nguyen Kieu Linh
arXiv:2608. 08512v1 Announce Type: new Abstract: Evolving documents, such as laws, tax codes, and software documentation, are amended, replaced, and sometimes reverted over time, so a question has different correct answers at different dates.
By Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury, Md Adnan Arefeen, Farig Sadeque, Md. Faizul Bari, Swakkhar Shatabda
The paper presents a smartphone‑compatible, retrieval‑augmented language model tailored to Bangladeshi statutory law. By distilling a 9‑billion‑parameter Gemma‑2 teacher into a 2‑billion‑parameter student using supervised fine‑tuning and QLoRA, the authors achieve significant gains in ROUGE‑L and BERTScore on an English benchmark while keeping the model lightweight (1.6 GB) and operable offline on a Pixel 6. The system retrieves from 36,029 statutory passages using a hybrid dense/BM25 approach, and cross‑lingual evaluation shows effective Bangla query handling against an English‑only corpus, with a practicing lawyer rating the responses highly in a pilot study.
By MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan, Nadifa Zaman, Fariha Roushon Florin, Farig Yousuf Sadeque, Saadat Rafid Ahmed
arXiv:2607. 23446v1 Announce Type: cross Abstract: A small language model can receive the governing statutory provision and still answer incorrectly.
By Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen
arXiv:2610.19848v1 Announce Type: new
Abstract: We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench conta...
By Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang, Yu-Wei Liu, Che-Yu Wu, Po-Feng Lee
The paper presents an expert‑validated benchmark of 1,042 items across ten areas of Colombian law to assess the reliability of large language models (LLMs) on national legal systems outside the U.S. It evaluates 15 contemporary proprietary and open‑weight models, finding that while closed‑choice accuracy can be high (up to 0.905), free‑text factual correctness never exceeds 0.45. The study reveals a strong dissociation between answer relevance and correctness, systematic variation by legal area, and suggests that expert supervision and grounding in authoritative sources are needed for reliable Colombian legal tasks.
By Rub\'en Manrique, Michelle Castellanos, Jorge Morales, Juan David Guti\'errez, Antonio Barreto Rozo, Joaqu\'in V\'elez Navarro
KhatianDoc is a new benchmark that tests multimodal large language models on Bengali legal land records, specifically the handwritten RS Khatians used in Bangladesh. The benchmark comprises four tasks—symbol recognition, base‑16 to decimal conversion, structured field extraction, and legal document question answering—drawn from 107 real records and 1,634 QA pairs. Six multimodal LLMs were evaluated under a zero‑shot protocol, revealing that many models fail to answer a significant portion of questions correctly and perform poorly on arithmetic conversion, highlighting a lack of capability rather than a performance gap.
By Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim, S. M. Shah Nawaz Hossain, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi