arXiv AI
4d ago

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

The paper evaluates eight open‑source lightweight Vision‑Language Models (up to 7 B parameters) for converting document images into structured JSON, focusing on heritage collections with historical handwriting and non‑standard layouts. It compares zero‑shot, few‑shot, and fine‑tuned settings, measuring performance with CER, ANLS*, and mAP‑F1, and examines the effects of hyperparameter tuning, image preprocessing, and multi‑stage training. The study demonstrates that adapted VLMs can serve as a sustainable, private alternative to commercial OCR systems for institutional archives.

By Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen
arXiv Computer Vision
Sep 18

Automated Goldsmith's Mark Retrieval in Silverware

The paper introduces an AI-assisted pipeline for retrieving goldsmith marks from silverware images, combining mark localization with metric‑learning fine‑tuning on three backbone models (ResNet‑50, ViT‑S/16, and DINOv2 ViT‑S/14). Systematic tests of cropping strategies show that manual cropping and metric‑learning fine‑tuning yield the best performance, with DINOv2 ViT‑S/14 achieving 62.63% mAP and 73.74% Top‑1 accuracy. The authors release a manually annotated dataset, code, and a public web interface to support reproducibility and adoption in digital humanities.

By Atmik Tiwari, Vincent Christlein, Mark Fichtner, Freya Gohlke, Birgit Sch\"ubel, Theresa Witting, Heike Zech, Mathias Zinnen
arXiv AI
Sep 18

Fingerprinting Multimodal Large Language Models

The paper introduces AttnPrint, a white‑box fingerprinting method that extracts low‑frequency components of cross‑modal attention distributions to identify multimodal large language models (MLLMs). It also presents DistillTrace, a black‑box auditing tool that uses hypothesis testing of MLLM outputs to detect potential model infringement. Experiments on 154 model instances across 19 architectures show that AttnPrint effectively detects derivative models and remains robust to downstream modifications, while DistillTrace reveals distillation relationships under various techniques.

By Chao Huang, Meng Tong, Kejiang Chen
Hugging Face Trending Papers
Jul 21

RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency

Public institutions hold large volumes of sensitive documents and support tickets that cannot leave the premises, ruling out cloud-hosted language models entirely. We report on RAGAL, a retrieval-augmented assistant for the technical-support team of AFIR, the Romanian Agency for Financing Rural Investments, built and operated under three hard constraints: zero data egress (no external API calls, even for synthetic data), a read-only mandate (the assistant drafts, humans execute), and a single 8 GB consumer laptop as the only development and training machine.