arXiv AI
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 1
One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
arXiv:2608.31017v1 Announce Type: cross Abstract: Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 1...
arXiv AI
Jun 10
Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
arXiv:2606. 10315v1 Announce Type: cross Abstract: LLM-as-judge is the default instrument for evaluating conversational agents, yet its reliability is almost always reported as agreement with human ratings, not recall of real defects.
arXiv AI
Jul 22
Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety
arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.
arXiv AI
Aug 26
More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight
The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.
arXiv AI
Sep 2
Commit-first LLM judging inherits the judge's own errors
The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.
arXiv AI
Sep 15
Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes
arXiv:2606.07834v2 Announce Type: replace-cross Abstract: Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately in...