InterviewPlayground is a simulation environment designed to evaluate AI interviewers by using simulated participants whose behaviors are grounded in social theory. The platform generates an InterviewReportCard that assesses AI interviewers across validated measures. Empirical validation shows that performance in InterviewPlayground predicts human study outcomes with a Pearson correlation of 0.86, supporting its use for assessing AI interviewer performance and failure modes.
By Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang, Madeline Mandell, Ziang Xiao, Anjalie Field
SIT‑Eval is a framework that evaluates conversational agents by sampling diverse situations at each turn and judging responses against situational rules. It uses an LLM‑interlocutor to generate uniformly sampled contexts and a context‑aware LLM‑judge to assess rule compliance. Benchmarking eight large language models on 1,292 turns uncovered hidden failure modes, showing that models with similar overall scores can differ greatly in rule adherence, especially in ambiguity clarification and Socratic questioning.
By Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko
arXiv:2610.23149v1 Announce Type: cross
Abstract: Large language model chatbots have been proposed as AI-interviewers for collecting qualitative interview data at scale, potentially bridging social s...
By Semra Guven, Tobias G{\aa}rdhus, Andreas Bjerre-Nielsen, Anna Rogers, Hjalmar Bang Carlsen
arXiv:2607. 20773v1 Announce Type: cross Abstract: Large language models (LLMs) have shifted human--computer interaction from `traditional'' interface journeys toward more conversational exchanges.
By Zeshu Zhu, Natalie Friedman, Kevin Weatherwax, Emily Eiben
arXiv:2607. 08285v1 Announce Type: new Abstract: Current AI evaluation frameworks focus primarily on technical performance, including accuracy, robustness, reasoning ability, and policy compliance.
By Marcos Economides, Paul M. Sacher, Samuel Salzer, Alexis Michelle Abellar, Fendi Tsim, Antoine Ferr\`ere
arXiv:2609.00250v1 Announce Type: cross
Abstract: Many people now see AI systems as not just productivity tools but as social companions. Researchers are eager to study the consequences of AI compani...
By Jacy Reese Anthis, Mark D\'iaz, Renee Shelby
arXiv:2603. 09995v2 Announce Type: replace-cross Abstract: Behavioral interview evaluation using large language models presents unique challenges that require structured assessment, realistic interviewer behavior simulation, and pedagogical value for candidate training.
By Kewen Zhu, Zixi Liu, Yanjing Li, Jing Chen
arXiv:2601.12208v2 Announce Type: replace
Abstract: Evaluating conversational systems in multi-turn settings remains a fundamental challenge. Conventional pipelines typically rely on manually defined...
By Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu
arXiv:2602.00685v2 Announce Type: replace
Abstract: Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable an...
By Xuan Liu, Haoyang Shang, Zizhang Liu, Xinyan Liu, Yunze Xiao, Yiwen Tu, Haojian Jin
The paper introduces CoCoEval, a framework for evaluating large language model (LLM)–simulated conversations by detecting 10 types of inconsistent and uncollaborative behaviors at the turn level. Using CoCoEval, the authors compare human conversations with those generated by GPT‑4.1, GPT‑5.1, and Claude Opus 4, finding that LLMs produce far fewer such behaviors under vanilla prompting and that prompt engineering or fine‑tuning often over‑produces specific behaviors. The study highlights gaps between human and LLM‑simulated interactions that conventional Likert‑scale evaluations miss, raising concerns about using LLMs as proxies for human social interaction.
By Ryo Kamoi, Ameya Godbole, Binglin Zhou, Xiaoxin Lu, Longqi Yang, Rui Zhang, Mengting Wan, Pei Zhou
arXiv:2608. 09939v1 Announce Type: cross Abstract: Production teams deploying LLM chat agents face a specific quality assurance gap: existing evaluation tools test individual responses or simulate social interactions, but none systematically verify whether real users can achieve their goals through multi-turn conversation.
By Alexandre Cristov\~ao Maiorano
arXiv:2411. 10109v3 Announce Type: replace Abstract: Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes.
By Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, Michael S. Bernstein