← Back to all news
arXiv Computation and Language By Holly Baker, Michael Kuo, Kyana Pollas

Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

Read the original on arXiv Computation and Language →

Policy Central is a deployed system that assists police departments in identifying relationships between sections of their written policy, such as conflicts, redundancy, similarity, or difference. It offers two architectures: Retrieve‑Judge, a retrieve‑then‑classify pipeline that serves as a low‑cost library‑wide health check, and Screen‑Confirm, a selective audit that exhaustively compares reviewer‑chosen documents. On a benchmark of 302 pairs from one department’s 84 policy documents, Screen‑Confirm achieves 0.795 aggregate recall and 0.885 on conflict detection, while Retrieve‑Judge’s recall is 0.457, highlighting the challenge of efficiently finding relevant pairs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 28

When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

The paper investigates when a language‑model judge can truly ground its verdicts in code correctness. It shows that current multi‑agent verification methods rely on evidence that is both independent of the answer and distinct between candidates—conditions that fail in code judging. By analyzing two label‑free measurements from the judge’s logs, the authors demonstrate that gating on one measurement allows the system to decline uncertain comparisons, improving accuracy from 20.7% to 36.9% while still answering half of all cases.

By Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
llmsagentsbenchmarks
More like this →
arXiv AI
Aug 19

Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification

The paper investigates how personalized agents decide to use, ignore, update, or query retrieved user memory before acting on a task. An empirical audit protocol is developed to test structured intermediate outputs, revealing that while exposing state definitions improves accuracy, an explicit state-output field does not significantly enhance policy accuracy for large language models. The study also shows that example-level accuracy overstates consistency, with full four‑way family success being rare, and that providing benchmark‑associated state labels merely conditions predictions rather than proving internal fidelity.

By Yihang Chen, Pin Qian, Su Wang, Chong Peng, Huan Xu, Shuaiting Li, Yiqi Sun
llmsagentsbenchmarks
More like this →
arXiv AI
Sep 7

Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines

The paper investigates whether the object selected in a grounded language‑model pipeline actually reaches the reader, a failure that can break the handoff between stages. By auditing 600 HybridQA questions across three selector families, the authors find that exact key lookup and title matching recover the selected object in all 1,463 resolvable records, but body‑only BM25 omits it in 26.6% of cases at cutoff five, while hybrid retrieval with reranking omits it only 1.0%. The study also shows that misalignment between selected and retrieved objects can reduce exact match scores by up to 31 points, and introduces the Returned‑Object Profile (ROP) as a tool for reproducible auditing.

By Siddharth Vohra, Runmin Jiang, Xiaomo Li, Min Xu
ragbenchmarks
More like this →