LLM companions are deployed at scale in personally consequential settings, yet poorly evaluated. Existing benchmarks use hand-authored scenarios and prompted simulators, aggregate empathy into one score, and overlook judge biases such as same-family favoritism and scale drift.
arXiv:2608. 02046v2 Announce Type: replace-cross Abstract: LLM companions are deployed at scale in personally consequential settings, yet poorly evaluated.
The paper introduces a three-tier persona vector for user simulation in evaluating LLM agents, comprising 23 dimensions across demographics, behavioral traits, and emotional states, plus a query-complexity overlay. It demonstrates that these nuanced personas generate diverse, scenario-reactive conversations, leading to significant variations in agent goal achievement and compliance across different contexts. The model’s design allows for reproducible, auditable user behavior patterns without relying on learned covariance matrices.
The paper introduces a three-tier persona vector to generate diverse, realistic user inputs for evaluating tool-augmented LLM agents. The vector includes 23 dimensions: categorical demographics, continuous behavioral traits, and continuous emotional states, plus a query-complexity overlay. Experiments on 64,698 conversations show that these persona dimensions produce measurable differences in agent performance and realistic scenario-reactive behavior.
arXiv:2606. 30256v1 Announce Type: new Abstract: Safety benchmarks often buy scalability by fixing the prompt, the language, and the turn structure.
arXiv:2609.22255v1 Announce Type: new Abstract: Existing approaches to persona simulation with Large Language Models (LLMs) mostly rely on shallow character descriptions that fail to sustain coherent...
AgentWorld is a simulation framework that evaluates agentic information retrieval by incorporating diverse user personalities based on the Big Five (OCEAN) traits, stateful tool-use environments, and a pass$^k$ consistency metric with structured fault classification and partial-credit scoring. It includes a risk analyzer that uses Monte‑Carlo rollouts and advanced scoring methods to quantify trajectory brittleness and attack attribution. Experiments with conversational analytics, customer‑support agents, and adversarial stress‑testing demonstrate that personality variation reveals failure modes hidden by uniform testing, such as cross‑domain leakage, contextual drift, and significant quality gaps across personas.
arXiv:2607. 07916v1 Announce Type: new Abstract: Large language models exhibit recurring behavioural patterns -- personas -- that shape generalisation and safety, but we lack reliable tools for decomposing, measuring, and controlling them.
arXiv:2609.00250v1 Announce Type: cross Abstract: Many people now see AI systems as not just productivity tools but as social companions. Researchers are eager to study the consequences of AI compani...
The paper examines whether fine‑tuning large language models (LLMs) with personality‑labelled data improves their ability to act as socially interactive agents. Two small open‑weight LLMs were fine‑tuned on a corpus of personality‑labelled social media posts and dialogues, and the resulting models were evaluated in various social interaction scenarios by independent LLM judges. The findings show that the fine‑tuned models do not outperform their baseline counterparts in role‑playing personalities, though they offer comparable text quality and increased linguistic diversity for the Qwen models; low inter‑rater agreement limits confidence in the results, suggesting future work should focus on training data quality and domain alignment.
The study examined how four large language models (GPT‑5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Fable 5) scored 18 simulated Japanese‑language AI‑to‑AI counseling sessions compared to ratings from 15 human counseling experts. Each model evaluated every transcript three times on four motivational‑interviewing‑informed dimensions and overall quality, consistently giving higher scores for softening sustain talk and overall quality than the expert panel, though the magnitude varied by model. Run‑to‑run reliability (intraclass correlation coefficients ranging from .33 to .96) did not predict closer alignment with expert judgments, and the models’ ability to discriminate counselor conditions was distinct from both reliability and alignment.
The paper introduces FIGS, a dual‑axis evaluation framework for multi‑turn sycophancy that avoids penalizing empathy. It uses a 10‑turn conversational simulator with 500 diverse scenarios to test whether models stay truthful while keeping praise proportional, and whether they show calibrated validation of user feelings. The study finds that current models either drift toward sycophancy or become overly detached, highlighting an unresolved trade‑off in sustained dialogue.
arXiv:2509.08494v2 Announce Type: replace-cross Abstract: As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures....