← Back to all news
arXiv Machine Learning

OpenSanctions Pairs: Large-Scale Entity Matching with LLMs

OpenSanctions Pairs is the first large‑scale public benchmark for entity matching on sanctions and OSINT data, comprising 755,540 expert‑labeled pairs drawn from over 1 million entities across 293 source datasets and 45 jurisdictions. The dataset spans multiple languages and writing systems, inconsistent structures, and time‑varying provenance, making it far more heterogeneous than prior benchmarks. Baseline experiments show a rule‑based matcher achieving 91.3 % F1, GPT‑4o reaching 99.0 % F1, and a locally deployable open‑source model scoring 98.2 % F1, with complementary failure modes that highlight the need to focus on downstream pipeline components.

Related stories

arXiv Computation and Language
Oct 16

Cost-Efficient RAG for Entity Matching with LLMs: A Blocking-based Batch Processing Framework

The paper introduces CE‑RAG4EM, a cost‑efficient framework that applies retrieval‑augmented generation (RAG) to entity matching (EM) by using blocking‑based batch retrieval and generation. It demonstrates that, in a zero‑shot setting, CE‑RAG4EM achieves comparable or better matching quality than naive RAG while significantly reducing end‑to‑end overhead. Experiments on nine real‑world EM benchmarks show that the approach closes the knowledge grounding gap for LLM‑based EM, especially on datasets with ambiguous or sparse attributes, and provides clear guidance on balancing accuracy and computational cost through blocking size and retrieval granularity.

By Chuangtao Ma, Zeyu Zhang, Arijit Khan, Sebastian Schelter, Paul Groth
llmsragbenchmarks
More like this →
arXiv Computation and Language
Aug 31

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

The paper introduces LongJudgeBench, a benchmark designed to evaluate large language models (LLMs) acting as judges for long-form text generation. It highlights that long-form evaluation requires complex, document-level assessments beyond simple length, such as organization, coverage, depth, consistency, and scenario-specific quality. Experiments show a significant reliability gap among current LLM judges, indicating instability across scenarios and limited effectiveness of rubrics or references.

By Junjie Chen, Yuxi Dong, Haitao Li, Weihang Su, Yujia Zhou, Min Zhang, Yiqun Liu, Qingyao Ai
llmsbenchmarks
More like this →
arXiv Computation and Language
Sep 10

MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

arXiv:2607.00890v2 Announce Type: replace Abstract: Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synth...

By Maximilian Idahl, J\"org Tiedemann, Sampo Pyysalo, David Salinas, Tomasz Galica, Shenbin Qian, Tudor Nicolae Mateiu, Zihao Li, Anna Lokrantz, Fedor Vitiugin, Andr\'e F. T. Martins, Jenna Kanerva, Filip Ginter, Matthias Lindemann, Tim Isbister, Birger Moell, Jonas Lindh, Jan Haji\v{c}, Jenia Jitsev, Andrey Kutuzov, Stephan Oepen, Gema Ram\'irez-S\'anchez
llmsbenchmarks
More like this →
arXiv AI
Sep 3

Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

The paper audits the reproducibility of knowledge‑graph extraction from threat reports by re‑implementing matching rules for only five of twelve systems and re‑scoring ten system outputs under eight protocols. The audit shows that different matching protocols can reverse most pairwise system rankings and that a fixed prediction set can vary from 0.16 to 0.70 F1. The authors also build CTIForge to isolate validation effects, finding that validation changes precision across backbones and increases entity‑type disputes, and they release the full pipeline, protocol suite, and audit records.

By Safayat Bin Hakim, Houbing Herbert Song
llmsrag
More like this →