arXiv:2609.23742v1 Announce Type: new Abstract: Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, functio...
arXiv:2608.31068v1 Announce Type: new Abstract: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of...
arXiv:2510. 06039v2 Announce Type: replace-cross Abstract: Reliable evaluation of knowledge-grounded Large Language Models (LLMs) in Chinese requires resources that explicitly align Chinese-language text with verifiable Knowledge Graph (KG) facts.
The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.
The paper introduces IndicReStruct, a benchmark dataset comprising two variants—GSM8K-Reordered and GSM8K-Voice—derived from GSM8K to test multilingual LLMs on Hindi and Malayalam. It evaluates six state‑of‑the‑art LLMs under constrained constituent reordering and active‑passive voice transformation, finding consistent and significant drops in mathematical reasoning performance. Qualitative error analysis and residual‑stream activation patching reveal that failures often stem from disrupted entity‑quantity alignment, with intermediate transformer layers playing a key role in restoring reasoning.
EviSI is an evidence‑based evaluation agent for low‑latency simultaneous speech‑to‑speech translation. It combines Multidimensional Quality Metrics with interpreter‑developed criteria, using shared source evidence to assess four dimensions—Anchor, Event, Logic, and Fluency—while deduplicating verified errors before scoring. On English‑to‑Chinese and Chinese‑to‑English data, EviSI’s rankings correlate strongly with human judgments, outperforming BLEU and COMET, and its multilingual extension maintains these correlations across five language directions.
The paper investigates why large language models often answer factual questions incorrectly in closed‑book settings. It shows that many errors are due to first‑token recall failures: the model commits to an incorrect first token but can complete the correct answer once the right token is provided. Using a First‑Token Cueing probe, the authors demonstrate that correcting just the first token recovers a significant portion of errors across 16 open models and improves accuracy on multilingual benchmarks.
Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors.
arXiv:2510.23160v2 Announce Type: replace Abstract: Data selection for supervised fine-tuning (SFT) often discards low-scoring instruction examples that may retain useful supervision. We propose a ru...
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.
arXiv:2607. 29378v1 Announce Type: cross Abstract: Large language models (LLMs) generate text by auto-regressively sampling the next token.
arXiv:2605.28190v2 Announce Type: replace Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...