Mistral AI

Le Chat dives deep.

arXiv Computation and Language
Sep 23

Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

Hy‑MultiTurn is a Chinese benchmark designed to evaluate deep multi‑turn dialogue understanding over long interactions. It introduces six controlled evaluation modes—constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution—across 209 tasks ranging from 12 to 76 turns, incorporating dialogue length, irrelevant distractions, and colloquial phrasing. Testing 22 state‑of‑the‑art models shows the benchmark is highly challenging, with even the best model meeting all criteria only 41.1% of the time and no model excelling in every mode.

By Eileen Ye, Jiawen Tao, Yaoming Li, Chenxu Liu, Wenhan Yu, Yaxin Fan, Xiaokun Yuan, Mengzhou Wu, Yanbing Jiang, Maxm Pan