arXiv Machine Learning
Jul 24
Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models
arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.
Task-Centric Multimodal Learning for Cognitive Impairment Detection
The paper introduces a contactless, multimodal screening framework that uses video, audio, and text data collected during structured voice interactions with Amazon Alexa to detect mild cognitive impairment (MCI). It evaluates three conversational tasks—spontaneous speech, comprehension, and recall—to determine which provides the most informative signals, compares unimodal models, and assesses multimodal fusion. Results show that while individual modalities perform best on specific tasks, the multimodal model consistently achieves higher accuracy (up to 0.76) across tasks, indicating that integrated behavioral cues can reliably distinguish MCI from healthy aging.
Related stories
arXiv Machine Learning
4d ago
CogDrift-4D: Hierarchical Multimodal Modeling of Temporal Behavioral Decay for MCI Detection
CogDrift-4D introduces a hierarchical 4D Transformer framework for detecting mild cognitive impairment (MCI) by modeling temporal behavioral decay across seven unscripted multimodal interactions over 18 months. The approach represents continuous, multi‑session data as a dense 4D tensor and employs dual‑level modeling: intra‑session least‑squares regression to capture acute fading of facial and acoustic features, and an inter‑session recursive cross‑attention encoder to accumulate cumulative fatigue. A localized Quality Gate filters low‑integrity features, and early fusion integrates semantic and raw dynamics, achieving an MCI classification accuracy of 0.733 ± 0.13 and an F1‑score of 0.716 ± 0.14, surpassing advanced baselines.
arXiv AI
Sep 1
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
arXiv:2608.30857v1 Announce Type: cross Abstract: Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal a...
arXiv Machine Learning
Sep 15
CCMAN: Cognitive Instability-Aware Cross-Modal Attention Network for Interpretable Temporal Biomarkers of Verbal Fluency Speech
arXiv:2609.14764v1 Announce Type: cross Abstract: Early detection of cognitive decline from speech offers a scalable and non-invasive alternative to conventional clinical assessment. Verbal fluency t...
arXiv AI
Jul 1
ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection
arXiv:2606. 30646v1 Announce Type: cross Abstract: Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment.
arXiv AI
Jun 30
LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features
arXiv:2606. 28445v1 Announce Type: cross Abstract: Early detection of dementia enables timely intervention, and reflecting cognitive impairment, spontaneous speech offers a non-invasive screening modality.
arXiv Computation and Language
Oct 9
When History Helps and Hurts: Selective History Use across Multimodal Turns
The paper introduces ReTurn, a benchmark of 7,000 multimodal tasks designed to evaluate how models selectively use conversational history. ReTurn includes pairs that test task-carrying history (Reconfirm/Reground) and evidence-carrying history (Retrieve/Rebind), each preserving the same target question, media, and answer. Experiments on 13 vision‑language and audio‑language models show a drop in open‑ended accuracy from 93.7% to 72.3% when history is involved, revealing that high question recall does not guarantee effective task application and that competing media can misdirect responses.
arXiv Computation and Language
Sep 21
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.
arXiv Machine Learning
Jun 18
Fair Cognitive Impairment Detection Through Unlearning
arXiv:2606. 18571v1 Announce Type: new Abstract: Mild Cognitive Impairment (MCI) is a medical condition characterized by a noticeable decline in memory, language, or thinking abilities.
arXiv AI
Jun 11
Frozen Multimodal Embeddings for Personality and Cognitive Ability Assessment in Asynchronous Video Interviews
arXiv:2606. 11930v1 Announce Type: cross Abstract: Predicting psychological traits from asynchronous video interviews (AVIs) is a challenging multimodal learning problem because labeled datasets are limited while each response contains high-dimensional visual, acoustic, and verbal signals.
arXiv AI
Jul 1
Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection
arXiv:2606. 30675v1 Announce Type: cross Abstract: Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging.
arXiv AI
Jun 2
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.