arXiv AI

CrimeNER: Curated Dataset and Methods for Research and Investigation in Language Intelligence

arXiv AI
Jul 17

CrimeNER Demo: Named-Entity Recognition in the Crime Domain

arXiv:2607. 14800v1 Announce Type: new Abstract: We present CrimeNER Demo, an AI-powered platform that enables us to extract general crime-related information from documents and classify them into entity types with two levels of granularity.

By Miguel Lopez-Duran, Julian Fierrez, Aythami Morales, Daniel DeAlcala, Gonzalo Mancera, Javier Irigoyen, Ruben Tolosana, Oscar Delgado, Francisco Jurado, Alvaro Ortigosa
arXiv Computation and Language
3d ago

Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research

The paper presents an LLM-based pipeline for extracting structured data from criminal court decisions, using attribute exploration, text span extraction, and value normalization guided by expert-refinable annotation instructions. Evaluated on DUI and DDB traffic offenses, refining instructions improved extraction accuracy from 52% to 70% for DUI and increased annotator agreement. The study highlights that model consistency does not guarantee correctness and that disagreements can uncover ambiguous instructions.

By Tom\'a\v{s} Knap, Ivana Kvapil\'ikov\'a, Jan \v{C}ern\'y, Kl\'ara Bendov\'a, Jarom\'ir \v{S}avelka, Vojt\v{e}ch Pour, Jakub Dr\'apal
Hugging Face Trending Papers
Sep 17

Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

The paper introduces the Profiling, Investigation, and Judgment (PIJ) benchmark, which contains 2,500 real homicide cases from five countries to evaluate large language models (LLMs) on pre‑arrest criminal investigation tasks. It assesses LLMs across criminal profiling, crime process reconstruction, and sentence prediction, revealing that performance drops as tasks require more implicit reasoning about unknown suspect profiles. The study finds that LLMs lag behind human experts, especially in inferential categories like motivation and victim‑offender relationships, and exhibit biases in gender, age, and motive attribution.

arXiv Computation and Language
Sep 18

Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

The paper introduces the Profiling, Investigation, and Judgment (PIJ) benchmark, which contains 2,500 real homicide cases from five countries to evaluate large language models (LLMs) on pre‑arrest criminal investigation tasks. It assesses LLMs across criminal profiling, crime process reconstruction, and sentence prediction, revealing that performance drops as tasks require more implicit reasoning about unknown suspect profiles. The study finds that LLMs lag behind human experts, especially on inferential tasks like motivation and victim‑offender relationships, and exhibit biases in gender, age, and motive attribution.

By Yutong Yao, Yanjie Cao, Guanhua Chen, Xu Yang, Junchao Wu, Zeyu Wu, Lidia S. Chao, Derek F. Wong
arXiv Computation and Language
Sep 14

Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

The paper introduces a weakly supervised framework for extracting dataset mentions from forced displacement and Fragile, Conflict, and Violence (FCV) documents. It uses a lightweight model trained on general research literature to generate candidate mentions, which are then refined by a large language model that validates or rejects them and corrects boundaries. The refined annotations are augmented with synthetic and contrastive examples to fine‑tune the model, achieving 74.1% precision and 70.5% recall on a benchmark of 1,706 passages, with higher precision (89.5%) on passages that contain dataset references.

By Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock, Olivier Dupriez
arXiv AI
Jun 9

RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotations

arXiv:2606. 08376v1 Announce Type: cross Abstract: As artificial intelligence (AI) systems are increasingly deployed across socially consequential domains, reports of AI-related harms and failures have grown in frequency and diversity.

By Leihan Zhang, Wecheng Ye, Xianlong Ma, Haochuan Liu, Yang Li, Qianyu Zhang, Jinliang Chen, Qiang Yan
arXiv AI
Sep 17

A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages

The paper introduces a scalable, multi-step framework designed to improve the quality of Named Entity Recognition (NER) annotations, particularly in low-resource languages. It employs a frequency-based iterative approach that combines self‑training with a dual‑threshold mechanism to increase inference confidence. Experiments on various NER datasets show notable performance gains over the original data, and the study also investigates the use of generative Large Language Models for NER tasks.

By Toqeer Ehsan, Thamar Solorio