arXiv AI
Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 9
Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
arXiv:2606. 07541v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning.
arXiv AI
Oct 16
Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
The study investigates how a large language model (LLaMA2‑7B) aligns with human brain activity during naturalistic language comprehension by using SCRIBE, an interpretable semantic representation of 65 neurocognitively grounded properties. Variance partitioning on fMRI data from seven participants reveals that shared variance between LLaMA2‑7B and SCRIBE dominates across the brain, yet both representations also contribute unique variance. Further analysis shows that concrete semantic properties correlate more strongly with shared variance, while abstract properties do not show increased unique variance, and ablation experiments demonstrate that multiple semantic dimensions cumulatively influence both shared and unique brain-predictive variance.
arXiv AI
Jul 21
What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?
arXiv:2607. 16214v1 Announce Type: cross Abstract: Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivity remain unclear.
arXiv Computation and Language
Sep 11
Cross-lingual brain-language model alignment is robust but challenges hierarchical and computational accounts
The study examined whether brain-language model alignment reflects shared computational mechanisms or merely stable lexical‑semantic correspondences. Using whole‑brain encoding across Mandarin, English, and French, transformer representations predicted activity in a distributed network that overlapped across languages and remained stable across layers. Contextual embeddings and measures of prediction or compression did not outperform static lexical embeddings, suggesting that alignment is robust but not informative about shared computational processes.
Hugging Face Trending Papers
Aug 11
E$^3$mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
Understanding both expressed and evoked emotions is critical for multimodal large language models (MLLMs) to achieve comprehensive affect-aware interactions. However, existing benchmarks typically examine expressed and evoked emotions in isolation or are constrained to coarse-grained and incomplete affective characterizations.
arXiv Machine Learning
5d ago
What Shapes Brain Alignment of Large Language Models during Dialogue? Insights from Language Production and Comprehension
arXiv:2610.22961v1 Announce Type: cross Abstract: Brain alignment of large language models (LLMs) offers a way to investigate representational similarities between LLMs and the human brain during lan...