← Back to all news
arXiv Computation and Language By Weihan Peng, Chenxu Zhang, Qianao Wang, Yuling Shi, Heng Lian, Qihong Mao, Jiahao Pang, Chunliang Feng, Bowen Li, Xiaodong Gu

HEART-Bench: Do LLMs Make Persona-Consistent Decisions?

Read the original on arXiv Computation and Language →

HEART-Bench is a benchmark designed to assess whether large language models (LLMs) can make decisions that are consistent with a given persona and past experiences. It features 11 synthetic characters anchored in the Big Five personality traits, each with 1,000 autobiographical memories, and 64 decision scenarios guided by the DIAMONDS framework, resulting in 673 multiple‑choice questions. Models are evaluated with and without memory retrieval, showing that memory‑augmented configurations improve accuracy to 64.9%, while human annotators achieve 80.8% on a matched subset.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Sep 7

ICM-Bench: Person-Level Identity Reasoning in Multimodal Agents with Long-Term Memory

ICM-Bench is a new benchmark for evaluating identity-centric reasoning in multimodal agents with long-term memory. It consists of 839 synthetic video clips totaling 141 minutes and 1,217 open-ended questions about six recurring adults in a one-year life album. The benchmark isolates the ability to maintain recurring person identities and reason over their cross-time relations, and compares various baseline systems, showing that while Gemini 3.1 Pro performs well overall, its accuracy drops on questions requiring long-term identity profiles.

By Shidu Ren, Yunze Liu, Xing Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen
llmsagentsnlpmultimodalbenchmarks
More like this →
arXiv AI
Oct 5

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

DyadMem introduces a new benchmark for evaluating how long‑term agents remember user‑specific relational information, called User‑conditioned Relational Agent Memory (URAM). The dataset contains 3,065 episodes, 50,961 sessions, and 61,210 QA pairs, with detailed annotations for memory capture, update, and recall across multi‑session trajectories. Experiments on 20 models show strong performance in a gold‑memory setting but a significant drop in full‑pipeline QA, highlighting gaps in current LLM memory capabilities.

By Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang
llmsagentsbenchmarks
More like this →