LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics

LexFlip is a diagnostic tool that creates minimal perturbations of Quebec statutory French clauses to test whether legal meaning is preserved when lexical overlap remains high. The dataset contains 373 edits that reverse legal force while keeping 93% of tokens unchanged, and a harness evaluates various metrics, regressors, and judges on these edits. The study finds that most embedding and BERTScore metrics barely differentiate these edits, while bidirectional NLI performs better, and a simple length feature outperforms semantic metrics on the FrJudge human benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• rag