InterviewPlayground is a simulation environment designed to evaluate AI interviewers by using simulated participants whose behaviors are grounded in social theory. The platform generates an InterviewReportCard that assesses AI interviewers across validated measures. Empirical validation shows that performance in InterviewPlayground predicts human study outcomes with a Pearson correlation of 0.86, supporting its use for assessing AI interviewer performance and failure modes.
By Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang, Madeline Mandell, Ziang Xiao, Anjalie Field
SIT‑Eval is a framework that evaluates conversational agents by sampling diverse situations at each turn and judging responses against situational rules. It uses an LLM‑interlocutor to generate uniformly sampled contexts and a context‑aware LLM‑judge to assess rule compliance. Benchmarking eight large language models on 1,292 turns uncovered hidden failure modes, showing that models with similar overall scores can differ greatly in rule adherence, especially in ambiguity clarification and Socratic questioning.
By Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko
arXiv:2610.23149v1 Announce Type: cross
Abstract: Large language model chatbots have been proposed as AI-interviewers for collecting qualitative interview data at scale, potentially bridging social s...
By Semra Guven, Tobias G{\aa}rdhus, Andreas Bjerre-Nielsen, Anna Rogers, Hjalmar Bang Carlsen
arXiv:2607. 20773v1 Announce Type: cross Abstract: Large language models (LLMs) have shifted human--computer interaction from `traditional'' interface journeys toward more conversational exchanges.
By Zeshu Zhu, Natalie Friedman, Kevin Weatherwax, Emily Eiben
arXiv:2607. 08285v1 Announce Type: new Abstract: Current AI evaluation frameworks focus primarily on technical performance, including accuracy, robustness, reasoning ability, and policy compliance.
By Marcos Economides, Paul M. Sacher, Samuel Salzer, Alexis Michelle Abellar, Fendi Tsim, Antoine Ferr\`ere
arXiv:2609.00250v1 Announce Type: cross
Abstract: Many people now see AI systems as not just productivity tools but as social companions. Researchers are eager to study the consequences of AI compani...
By Jacy Reese Anthis, Mark D\'iaz, Renee Shelby