arXiv AI
Beyond Questions: Evaluating the Stated Knowledge of LLMs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 11
Beyond Questions: Evaluating LLM's Knowledge Expression
arXiv:2605. 26937v2 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood.
arXiv AI
Jul 1
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.
arXiv AI
Jun 26
NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
arXiv:2606. 27047v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge.
arXiv Computation and Language
1d ago
KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues
arXiv:2509.21856v2 Announce Type: replace Abstract: Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains. How...
arXiv AI
Aug 25
What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"
The paper introduces Knowledge-Weighted Fine‑Tuning, a method that estimates an instance‑level knowledge score through multi‑sampled inference and uses it to scale the learning signal. This approach encourages large language models to explicitly say "I don't know" on out‑of‑scope queries while preserving accuracy on known questions. The authors also propose new evaluation metrics for uncertainty, demonstrating that better discrimination between known and unknown instances improves overall performance.
arXiv AI
Jul 17
Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation
arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.