The paper investigates whether replies generated by large language models (LLMs) stay semantically consistent when the underlying model changes. Using real collaborative conversation messages, the authors compared the semantic similarity of LLM replies across different models, both with and without preceding chat history. They found that both the choice of model and the conversational context influence response similarity and alignment with human replies, suggesting that prompting and context alone may not guarantee consistent responses as LLMs evolve.
arXiv:2606. 18258v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit a wide range of human-like behaviors, from expressing thoughts and emotions, to engaging in relationship-building with users, to refusing requests and maintaining boundaries.
arXiv:2607. 20734v1 Announce Type: new Abstract: As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction.
arXiv:2601.12208v2 Announce Type: replace Abstract: Evaluating conversational systems in multi-turn settings remains a fundamental challenge. Conventional pipelines typically rely on manually defined...
arXiv:2607. 12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.
PRAGMA is a benchmark designed to evaluate personalized guidance in long‑term conversations. It includes curated longitudinal conversation histories, evidence annotations, and guidance scenarios that reflect evolving user contexts and incorrect assumptions. Experiments show that current retrieval, memory, and long‑context models struggle to recover relevant conversational evidence and to use it effectively for personalized guidance.
The paper introduces CoCoEval, a framework for evaluating large language model (LLM)–simulated conversations by detecting 10 types of inconsistent and uncollaborative behaviors at the turn level. Using CoCoEval, the authors compare human conversations with those generated by GPT‑4.1, GPT‑5.1, and Claude Opus 4, finding that LLMs produce far fewer such behaviors under vanilla prompting and that prompt engineering or fine‑tuning often over‑produces specific behaviors. The study highlights gaps between human and LLM‑simulated interactions that conventional Likert‑scale evaluations miss, raising concerns about using LLMs as proxies for human social interaction.
arXiv:2601. 02813v3 Announce Type: replace Abstract: Aligning language models to qualitative behavioral traits, such as human-likeness, remains difficult because they are hard to define, measure, and optimize.
The paper introduces mutable transcripts, an interaction paradigm that lets users edit prior turns in a chat, turning the conversation history into an editable state rather than a fixed record. A prototype was built and tested with 17 participants, who preferred mutable transcripts over standard chat for clarity, confidence, and ease of use, and reported less need to restart conversations. Analysis of user study transcripts shows that mutable transcripts can shorten conversations and remove outdated context, suggesting that user-driven revisions improve interaction quality.
arXiv:2602. 24287v2 Announce Type: replace-cross Abstract: In multi-turn conversations, large language models typically condition on the full conversation history: both past user prompts and assistant responses.
The paper introduces UPHELD, a large benchmark of human-to-human dialogues written by professional script writers, featuring realistic turn densities and over 36,000 per-turn human annotations. It evaluates existing automatic metrics and LLM-as-a-judge methods, finding them unreliable against expert human judgment. Using UPHELD, the authors develop a Mixture-of-Judges framework that improves correlation with human assessments by about 30%.
arXiv:2606. 06614v1 Announce Type: cross Abstract: Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data.
arXiv:2609.36700v1 Announce Type: cross Abstract: When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up...