arXiv:2608. 20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers.
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
The paper introduces the concept of LLM‑specific utility, defining it as the performance gain a target large language model (LLM) achieves when provided with a passage compared to answering without evidence. A benchmark of utilitarian passages is built for four LLMs (Qwen3‑8B/14B/32B and Llama 3.1‑8B) across three QA datasets, revealing that each model benefits most from its own tailored evidence and that evidence optimized for other models is consistently suboptimal. The authors also create SpecUBench, a benchmark for LLM‑specific utility judgment, and show that current utility‑aware retrieval methods largely capture model‑agnostic usefulness, struggling to estimate LLM‑specific utility. "whyItMatters":"The study demonstrates that retrieval‑augmented generation must consider model‑specific evidence selection to truly improve LLM performance, highlighting a gap in existing utility‑aware methods."
The paper introduces LLM‑PeerReview, an unsupervised ensemble method that selects the best response from multiple LLM-generated candidates by scoring each answer with several LLMs, aggregating those scores via averaging or a graphical model, and choosing the highest-scoring response. The approach is peer‑review inspired, transparent, and interpretable, and it outperforms the Smoothie‑Global model by 6.9%–7.3% across factual recall QA, math reasoning, and instruction‑following tasks. The authors also provide a curated benchmark suite of 12 ensemble methods evaluated on four datasets and three task families to aid reproducibility.
arXiv:2606. 05901v1 Announce Type: cross Abstract: Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing.
arXiv:2607. 25182v1 Announce Type: cross Abstract: The ability to retrieve relevant tables for answering questions is a key task for structured information retrieval.
arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
arXiv:2610.10091v1 Announce Type: cross Abstract: Knowledge-intensive tasks require answering many questions by reasoning about a shared corpus of artifacts (e.g., court cases, or scientific literatu...
arXiv:2608.23265v1 Announce Type: new Abstract: In long-term collaboration spanning multiple meetings, factual states such as decisions and risks are continually revised, overturned, and replaced. Ex...
arXiv:2607. 16848v1 Announce Type: cross Abstract: Long-term memory is becoming a core component of LLM agents, but most memory benchmarks evaluate conversations or compact summaries, while research agents need to restore evidence from full scientific papers.
arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.