arXiv Computation and Language

Improving Information Extraction with Learned Queries

The paper demonstrates that improving the design of queries used for information extraction can boost performance more than scaling up extraction models. By introducing List of Questions (LoQ) and a feedback-driven optimization method called FeedQ, the authors generate document-specific question sets that, when used to fine‑tune lightweight 4B‑parameter models, match or surpass expert‑derived baselines and outperform larger untuned models. They also release a dataset of 12,820 optimized questions to encourage treating question design as a primary research focus.

Hugging Face Trending Papers

Sep 24

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a scalable framework that automatically creates question–answer pairs from longitudinal electronic health record notes, validated by nineteen clinicians. It offers a continuously maintainable benchmark for evaluating large language models in clinical settings, addressing limitations of manual, costly, and quickly outdated existing benchmarks. Experiments across nine LLMs and five inference strategies reveal that even state‑of‑the‑art systems often miss clinically important information, especially for synthesis‑heavy queries.

llms benchmarks safety

arXiv AI

Sep 25

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a continuously maintainable benchmark for evaluating large language models (LLMs) in electronic health record (EHR) information retrieval. It presents a scalable framework that automatically generates question–answer pairs from longitudinal EHR notes, validated by nineteen clinicians. The benchmark allows assessment of multiple inference strategies and highlights that state‑of‑the‑art LLMs often miss clinically important information, especially when synthesis across documents is required.

By Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani, Philip Chung, Kevin R Keet, Kameron C. Black, Andrea T. Fisher, Sarita Khemani, Jerry Liu, Stephen Ma, Saloni K. Maharaj, Rita M. Pandya, Eduardo Perez-Guerrero, Priyanka Pillai, Lisa Shieh, David J. H. Wu, James Xie, James C. McAvoy, Teresa Nguyen, Jessica Tran, Lucy Yin, Bridget Lin, Alison Callahan, Jason A. Fries, Nigam H. Shah, Emily Alsentzer

llms benchmarks safety

arXiv Computation and Language

1d ago

Closing Ambient Clinical Documentation Gaps with Automated Provider Queries

arXiv:2610.07502v1 Announce Type: new Abstract: Provider queries are clarifying requests sent by clinical documentation specialists to physicians to close gaps in the clinical note and ensure accurat...

By Joseph Paul Cohen, Raj Shah, Han-Chin Shing, Fang Wang, Susan Nguyen, Chaitanya Shivade, Jack Moriarty

llms benchmarks

arXiv AI

Jun 16

Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning

arXiv:2606. 15598v1 Announce Type: new Abstract: Text-to-SQL aims to translate natural language questions into executable SQL queries over structured databases, enabling non-expert users to access data intuitively.

By Feng Lyu, Jinfeng Cen, Sijing Duan, Hao Wu, Shucheng Li, Weixu Zhang, Haolun Wu

llms fine-tuning benchmarks data-engineering

arXiv AI

Jul 8

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

arXiv:2607. 06452v1 Announce Type: cross Abstract: Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents.

By Taeyun Roh, Eunha Lee, Wonjune Jang, Sohyun Chung, Junha Jung, Jaewoo Kang

llms agents nlp

arXiv AI

1d ago

SAGE: Semantic Anchor-Guided Evolution for Grounded Medical QA Data Synthesis

arXiv:2610.08093v1 Announce Type: cross Abstract: Developing reliable models for clinical tasks, such as Medical Question Answering (QA), is severely constrained by the limited availability of high-q...

By Chuan Li, Chengyu Wang, Cen Chen, Ye Lyu, Mingyuan Fan, Ming Gao

llms nlp fine-tuning benchmarks safety

arXiv Computation and Language

Sep 22

Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions

arXiv:2411.10163v3 Announce Type: replace Abstract: Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmark...

By Yutao Hou, Yajing Luo, Zhiwen Ruan, Hongru Wang, Weifeng Ge, Yun Chen, Guanhua Chen

llms benchmarks

arXiv Machine Learning

Aug 6

RingSQL: Schema-Independent Synthetic Data Generation for Text-to-SQL Reinforcement Learning

arXiv:2601. 05451v2 Announce Type: replace Abstract: Recent advances in text-to-SQL have been driven by larger models, better datasets, and new training methods like RLVR.

By Marko Sterbentz, Kevin Cushing, Cameron Barrie, Kristian J. Hammond

llms reinforcement-learning benchmarks data-engineering

arXiv AI

Jul 21

Enhancing LLMs' Clinical Reasoning with Real-World Data from a Nationwide Sepsis Registry

arXiv:2505. 02722v2 Announce Type: replace Abstract: Although large language models (LLMs) have demonstrated impressive reasoning capabilities across general domains, their effectiveness in real-world clinical practice remains limited.

By Junu Kim, Chaeeun Shim, Sungjin Park, Su Yeon Lee, Gee Young Suh, Chae-Man Lim, Seong Jin Choi, Song Mi Moon, Kyoung-Ho Song, Eu Suk Kim, Hong Bin Kim, Sejoong Kim, Chami Im, Dong-Wan Kang, Yong Soo Kim, Hee-Joon Bae, Sung Yoon Lim, Han-Gil Jeong, Edward Choi

llms reinforcement-learning fine-tuning safety

arXiv AI

Aug 25

Future Querying: Can LLMs Serve as Implicit Medical World Models?

arXiv:2608.23248v1 Announce Type: cross Abstract: Traditional clinical prediction models rely on task-specific pipelines and curated, structured data, which scale poorly and underutilize unstructured...

By Siri Willems, James Butterworth, Lore Goetschalckx, Peter Vrancx, Philippe Modard, Elke Giets, Ludovic Denoyer

llms fine-tuning safety

arXiv AI

Sep 2

Multilingual Medical Reasoning for Question Answering with Large Language Models

The paper introduces a method for generating multilingual reasoning traces for medical question answering using large language models. It creates 500,000 reasoning traces in English, Italian, and Spanish by retrieving medical information from Wikipedia and applies them to MedQA and MedMCQA datasets extended into Italian and Spanish. The approach improves performance in both few‑shot in‑context learning and supervised fine‑tuning, achieving state‑of‑the‑art results for 8B‑parameter LLMs and releasing all resources for further research.

By Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri

llms rag nlp fine-tuning efficiency benchmarks

arXiv AI

Aug 11

Beyond Questions: Evaluating LLM's Knowledge Expression

arXiv:2605. 26937v2 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood.

By Luca Giordano, Simon Razniewski

llms benchmarks safety

Related stories