arXiv Computation and Language August 27, 2026 By Jiangang Hao

Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment

Read the original on arXiv Computation and Language →

The paper investigates whether replies generated by large language models (LLMs) stay semantically consistent when the underlying model changes. Using real collaborative conversation messages, the authors compared the semantic similarity of LLM replies across different models, both with and without preceding chat history. They found that both the choice of model and the conversational context influence response similarity and alignment with human replies, suggesting that prompting and context alone may not guarantee consistent responses as LLMs evolve.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 18

Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts

arXiv:2606. 18258v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit a wide range of human-like behaviors, from expressing thoughts and emotions, to engaging in relationship-building with users, to refusing requests and maintaining boundaries.

By Sunnie S. Y. Kim, Margit Bowler, Leon A Gatys

llms

arXiv Machine Learning

Jul 24

LLMs Get Lost in Evolving User Intent

arXiv:2607. 20734v1 Announce Type: new Abstract: As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction.

By Jihoon Tack, Philippe Laban, Jennifer Neville

llms agents benchmarks

arXiv Computation and Language

Sep 1

CoReflect: A Reflective Co-Evolution Framework for Improving Conversational Evaluation

arXiv:2601.12208v2 Announce Type: replace Abstract: Evaluating conversational systems in multi-turn settings remains a fundamental challenge. Conventional pipelines typically rely on manually defined...

By Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu

arXiv AI

Jul 15

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

arXiv:2607. 12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.

By Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran

llms agents benchmarks safety

arXiv AI

Sep 11

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

PRAGMA is a benchmark designed to evaluate personalized guidance in long‑term conversations. It includes curated longitudinal conversation histories, evidence annotations, and guidance scenarios that reflect evolving user contexts and incorrect assumptions. Experiments show that current retrieval, memory, and long‑context models struggle to recover relevant conversational evidence and to use it effectively for personalized guidance.

By Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung

llms benchmarks safety

arXiv Computation and Language

Sep 11

Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction

The paper introduces CoCoEval, a framework for evaluating large language model (LLM)–simulated conversations by detecting 10 types of inconsistent and uncollaborative behaviors at the turn level. Using CoCoEval, the authors compare human conversations with those generated by GPT‑4.1, GPT‑5.1, and Claude Opus 4, finding that LLMs produce far fewer such behaviors under vanilla prompting and that prompt engineering or fine‑tuning often over‑produces specific behaviors. The study highlights gaps between human and LLM‑simulated interactions that conventional Likert‑scale evaluations miss, raising concerns about using LLMs as proxies for human social interaction.

By Ryo Kamoi, Ameya Godbole, Binglin Zhou, Xiaoxin Lu, Longqi Yang, Rui Zhang, Mengting Wan, Pei Zhou

llms fine-tuning benchmarks

One email a morning, machine-written

Related stories

Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts

LLMs Get Lost in Evolving User Intent

CoReflect: A Reflective Co-Evolution Framework for Improving Conversational Evaluation

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction