LexFlip is a diagnostic tool that creates minimal perturbations of Quebec statutory French clauses to test whether legal meaning is preserved when lexical overlap remains high. The dataset contains 373 edits that reverse legal force while keeping 93% of tokens unchanged, and a harness evaluates various metrics, regressors, and judges on these edits. The study finds that most embedding and BERTScore metrics barely differentiate these edits, while bidirectional NLI performs better, and a simple length feature outperforms semantic metrics on the FrJudge human benchmark.
arXiv:2609.00378v1 Announce Type: new Abstract: Large language models pay a well-documented tax on non-English text: the same content costs several times more tokens, and because attention is quadrat...
arXiv:2605. 29738v2 Announce Type: replace-cross Abstract: Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible.
arXiv:2509.00141v2 Announce Type: replace-cross Abstract: Statutory corpora and judicial decisions are growing faster than legal professionals can read them, while individual judgments often exceed t...
arXiv:2606.09389v2 Announce Type: replace Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses...
arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its t...
arXiv:2609.39001v1 Announce Type: cross Abstract: LLM services are billed per token and context windows are measured in tokens, yet the number of tokens needed for the same content varies across lang...
arXiv:2609.08372v2 Announce Type: replace Abstract: A dense retriever encodes a question as one vector, but the question arrives one word at a time. We read 2,144 held-out headlines from Thu Vien Pha...
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
The paper critiques the common practice of stopword removal in legal text analysis, showing that standard stoplists actually degrade performance on binary classification tasks involving Supreme Court opinions. By exhaustively testing the removal of each of ~18,500 candidate words, the authors find that no stoplist—generic or optimized—outperforms a no‑removal baseline, and that models cannot predict which words are beneficial to remove. The study argues that inherited preprocessing defaults can distort the doctrinal and ideological signals that legal scholars aim to recover, calling into question the validity of such practices.
The paper introduces ComplianceAgent, a legal compliance sub‑agent within the COMPASS multi‑agent framework, which uses a large language model to evaluate texts on a fixed rubric across digital sovereignty, carbon impact, compliance, and ethics. The agent grounds its judgments in a reference legal corpus, and if a text scores at or below a critical threshold, it initiates a corrective chain that includes self‑healing rewrites, a compliance label, an explainability report, and escalation to a human reviewer. The authors evaluate the agent in a Canadian and Quebec context on a corpus of thirteen legal instruments and a hand‑annotated gold standard of fifteen texts, demonstrating perfect separation of conforming versus non‑conforming texts, stable scores under repeated queries, and relevant passage retrieval.
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.
Thunder‑DeID+ is a Korean de‑identification system that improves boundary precision, label granularity, and recall when applied to court judgments and other sensitive documents. It introduces a refined three‑tier PII taxonomy and a boundary‑aware tokenizer that augments a pretrained BPE vocabulary with N‑gram and boundary‑scoring components, enabling better alignment of token and PII span boundaries. Evaluations show the tokenizer raises the W. Strict Typed F1‑score from 0.76‑0.77 to 0.87‑0.88 across models, and fine‑tuning on 1,000 broader documents yields a 0.93 score, demonstrating effective transfer beyond court judgments.