Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge

Evaluating RAG with LLM as a Judge

LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

arXiv:2606. 24585v1 Announce Type: new Abstract: While the validity of LLMs' use in the legal context remains subject to ethical and legal debate, legal professionals are already experimenting with personal LLMs, if only for translation and reformulation.

LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

While the validity of LLMs' use in the legal context remains subject to ethical and legal debate, legal professionals are already experimenting with personal LLMs, if only for translation and reformulation. However, even such a seemingly innocuous use can introduce biases through case processing speed if LLM assistants selectively refuse assistance on certain topics.

Judge Arena: Benchmarking LLMs as Evaluators

LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

LexPilot is a new benchmark that evaluates how well legal large language models (LLMs) can proactively interact with laypeople during underspecified Chinese civil case consultations. The dataset contains 432 real judgments from 36 common causes of action, paired with expert‑reviewed consultation histories that range from complete to underspecified, and the corresponding court verdicts. Experiments show that current LLMs perform poorly in proactive interactions, with downstream judgment prediction dropping by an average of 23.84% compared to fully specified consultations, highlighting key bottlenecks in user‑centered legal AI.

Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

arXiv:2606. 07874v1 Announce Type: new Abstract: LLMs-as-judges are the only way to evaluate safety at scale.