Granuscore is a reference‑free metric that measures the granularity of text by exploiting the structure of a hierarchical embedding space. It successfully reproduces known hierarchical orderings on the Granola‑EQ dataset, distinguishes granularity across different discourse contexts, and explains sentence‑specificity variations beyond sentence length. The authors also apply Granuscore to four question‑answering benchmarks, revealing systematic differences in granularity among questions, gold answers, and model outputs, thereby offering a new lens for assessing QA dataset difficulty.
By Lukas Ellinger, Alexander Fichtl, Miriam Ansch\"utz, Georg Groh
arXiv:2602. 23971v4 Announce Type: replace-cross Abstract: Sycophancy, the tendency of large language models to favour user-affirming responses over critical engagement, has been identified as an alignment failure, particularly in high-stakes advisory and social contexts.
By Magda Dubois, Cozmin Ududec, Christopher Summerfield, Lennart Luettgau
arXiv:2505.22767v4 Announce Type: replace-cross
Abstract: Dialogue with a large language model can lead a person to insight: a sudden change in how they understand a problem. This perspective asks ho...
By Eleni Vasilaki
arXiv:2607. 05365v1 Announce Type: cross Abstract: Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech.
By Thomas Thebaud, Yuzhe Wang, Hao Zhang, Sathvik Manikantan Napa Ugandhar, Ashish Hallur, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez
Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality.
The paper compares two common ways of evaluating large language models (LLMs): prompting them to answer questions directly and scoring candidate answers using likelihood-based metrics. The authors introduce a new protocol that ranks declarative statements derived from question–answer pairs, and test it across 95 decoder-only models (0.1B–104B parameters) on 10 multiple-choice QA datasets. They find that while prompted answering accuracy improves sharply with model scale and instruction tuning, statement‑likelihood ranking accuracy stays relatively stable, indicating that the two evaluation methods probe different aspects of model behavior.
By Alessandro Bondielli, Lucia Passaro, Davide Bacciu, Alessandro Lenci
The paper introduces "question archaeology," an evaluation task that asks models to infer the single, authentic question that motivated a text. It presents a new dataset of commissioned texts paired with their original research questions and distractors, and evaluates both proprietary and open‑source LLMs. Results show newer models outperform older ones, with BERT-based models lagging, and current LLMs even surpassing human performance on this task.
By Claudiu Creanga, Liviu P. Dinu
arXiv:2609.24821v1 Announce Type: new
Abstract: The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-relate...
By Manjiang Yu, Hongji Li, Zihan Wang, Junwei Chen, Xue Li, Priyanka Singh, Yang Cao, Lijie Hu
CueMem is a cue‑guided framework for long‑term conversational memory that reconstructs query‑relevant dialogue context from compressed memory records. Instead of treating memory units as self‑contained evidence, it extracts fine‑grained cues linked to their source turns and, at query time, expands from these cues over a turn graph to rebuild a compact evidence context. Experiments on LoCoMo and LongMemEval show that CueMem outperforms baseline memory methods, reduces input tokens and latency, and improves long‑term conversational question answering.
By Changjian Wang, Rongzhen Li, Weili Guan, Shuming Shi, Quan Lu, Ning Jiang
Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability for such systems is effective question clarificat...
The paper presents a tri‑agent framework for evaluating large language models’ question‑clarification abilities. It involves a Question Clarifying Agent that identifies ambiguities and asks follow‑up questions, a Respondent Agent that simulates human replies, and an Evaluator Agent that judges the dialogue using metrics such as ambiguity handling, question quality, dialogue efficiency, language appropriateness, and intent alignment. The authors illustrate the approach with synthetic supply‑chain data and discuss validating the evaluator against human judgments.
By Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-related linear structures are organized within the mode...