LAVE: Zero-shot VQA Evaluation on Docmatix with LLMs - Do We Still Need Fine-Tuning?

Judge Arena: Benchmarking LLMs as Evaluators

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

arXiv:2609.39013v1 Announce Type: new Abstract: EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, sel...

What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework

The paper introduces LAST-CQ, a five-agent, training‑free, execution‑grounded framework for Text‑to‑Cypher that evaluates which components of an agentic pipeline contribute most to performance. Experiments on 2,471 live‑database queries across six backbones show that removing correction reduces execution‑BLEU by 3.1–12.3%, while substituting schema‑grounded feedback with raw error strings has negligible impact. Parallel sampling degrades quality by 10–11%, whereas failure detection and retry routing recover 91.7% of initially failed queries, highlighting that simple failure handling is more effective than sophisticated feedback or increased sampling.

MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc

arXiv:2608. 03397v1 Announce Type: new Abstract: MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs.

Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...

LLM-Guided Planning for Multi-hop Reasoning over Multimodal Nuclear Regulatory Documents

arXiv:2606. 29399v1 Announce Type: new Abstract: Reviewing nuclear regulatory documents requires multi-hop reasoning across tens of thousands of pages, where judgments depend on evidence assembled across multiple chapters.

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

arXiv:2607. 08535v1 Announce Type: cross Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.

Retrieved-Span Training for Efficient Query-Focused Meeting Summarization on QMSum

The paper introduces a retrieved‑span training approach for query‑focused meeting summarization on the QMSum benchmark. By fine‑tuning a 406 M Fusion‑in‑Decoder model on 2,000‑word retrieved spans, the authors recover a 6.30 ROUGE‑1 loss incurred when moving from capped long input and achieve a test score of 36.33 ROUGE‑1, comparable to a larger 1.2 B system. The smaller model uses roughly one‑third the parameters and less than half the peak inference memory, while span‑regime fine‑tuning adds significant gains over the baseline. "whyItMatters":"The study demonstrates that efficient, smaller models can match or exceed larger systems on QMSum using span‑based fine‑tuning, offering a practical path for scalable query‑focused meeting summarization."

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

The paper introduces LongJudgeBench, a benchmark designed to evaluate large language models (LLMs) acting as judges for long-form text generation. It highlights that long-form evaluation requires complex, document-level assessments beyond simple length, such as organization, coverage, depth, consistency, and scenario-specific quality. Experiments show a significant reliability gap among current LLM judges, indicating instability across scenarios and limited effectiveness of rubrics or references.

VERITAS: Verifier-Guided Proof Search for Zero-Shot Formal Theorem Proving

arXiv:2606. 19399v1 Announce Type: cross Abstract: LLM-based formal provers often collapse rich verifier signals (syntax errors, type mismatches, partial goal progress) into a binary pass/fail bit.

Evaluation of LLMs for Mathematical Formalization in Lean

arXiv:2606. 05632v1 Announce Type: new Abstract: Within the past few years, the ability of Large Language Models (LLMs) to generate formal mathematical proofs has improved drastically.

Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents

arXiv:2606. 08151v1 Announce Type: new Abstract: Tool-using LLM agents often fail not because relevant text is absent, but because decisive evidence is not selected, compressed, or surfaced at action time.