arXiv AI
AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 24
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
arXiv:2607. 20926v1 Announce Type: new Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources.
arXiv AI
Jul 2
GRACE-RAG: Governed Retrieval Architecture for Canonical Evidence Synthesis, Enabling Lightweight Deployment in Closed-Domain Institutional Settings
arXiv:2607. 00013v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems are widely used in institutional question answering settings where responses must be grounded in authoritative documentation (Gao et al.
arXiv AI
Aug 12
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
arXiv:2604. 13201v2 Announce Type: replace-cross Abstract: Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging.
arXiv AI
Aug 11
AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups
arXiv:2608. 08883v1 Announce Type: new Abstract: Recent advances in retrieval-augmented generation (RAG) and large language models (LLMs) enable researchers to integrate AI into scientific workflows.
arXiv AI
Sep 2
EGT-KG: Evidence-Grounded Typed KG Retrieval for Practical Scientific QA with Small Language Models
The paper introduces EGT-KG, an evidence‑grounded typed knowledge graph retrieval framework designed to enhance scientific question answering with small language models (SLMs). It compares three QA settings—standard Retrieval‑Augmented Generation (RAG) and two EGT‑KG variants (automatically generated and expert‑defined relation schemas)—using a six‑dimensional evaluation on a biopolymer‑bound soil composite literature benchmark. Results show that both EGT‑KG variants outperform vanilla RAG, with the llama3:8b model achieving a final score of 70.37 (+14.67%) and 68.82 (+12.14%) for the AS and ES variants, respectively.
arXiv AI
2d ago
Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
The paper investigates whether domain-specific fine‑tuning benefits open‑ended scientific reasoning in astronomy. Using a curated 300‑question QA benchmark from 2017–2026 Olympiad‑style materials, the authors compare open‑weight, API‑served general‑purpose, multimodal, and astronomy‑specialized language models. Results show that strong general‑purpose models set the highest correctness baseline, but variations in metric agreement, judge sensitivity, benchmark composition, and modality suggest that domain specialization is task‑ and deployment‑dependent and that domain‑specific evaluation is crucial for scientific workflows.