SHELF is a Python system that creates synthetic, controlled benchmark data for evaluating large language models on bibliographic tasks such as classification, clustering, retrieval, pair classification, and instruction retrieval. It generates 62,899 model-written documents based on Library of Congress vocabularies and compares methods like TF, TF‑IDF, BM25, popular encoders, and zero‑shot decoders, reporting performance metrics such as 0.8887 for subject classification and 0.2605 for genre‑form classification. The tool also allows independent variation of bibliographic facets and can produce unseen documents beyond a model’s training cutoff, with results indicating that model rankings transfer more reliably than absolute scores when compared to other benchmarks.
By Michael J. Bommarito II
arXiv:2606. 04382v1 Announce Type: cross Abstract: Automated subject cataloging assigns controlledvocabulary headings to bibliographic records, but LCSH has no standard public benchmark.
By Kwok Leong Tang
arXiv:2506.17585v4 Announce Type: replace
Abstract: Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving docume...
By Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra
The Writerslogic team participated in the CLEF 2026 SimpleText shared task, tackling both text simplification (Task 1) and complexity spotting (Task 2). For simplification, they built a multi‑candidate pipeline with GPT‑4o‑mini, selecting the best candidate via a reference‑free heuristic, and their Claude Sonnet 4 submission achieved a SARI of 47.43 and BLEU of 14.21, ranking third overall on the Task 1 leaderboard. For complexity spotting, they fine‑tuned a DeBERTa‑v3‑large NLI model on 350 K labeled pairs, achieving a macro F1 of 0.8081 (0.8085 in an ensemble) on binary over‑generation identification and 0.804 accuracy on multi‑class error classification, placing them second among unique teams.
By David L. Condrey
arXiv:2610.02516v1 Announce Type: cross
Abstract: Zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expen...
By Haifeng Wu, Srinivasan Manoharan, Jian Wan, Fangbo Tu, Junhua Zhao, Xin Chen
arXiv:2610.16182v1 Announce Type: new
Abstract: Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit t...
By Daoping Wu, Dong Wang, Siyu Liu, Chen Liang, Jian Geng, Ziqi Li, Shuguang Xiong, Ying Zhang