← Back to all news
arXiv Computation and Language By Ning E, Changde Du, Yizhuo Lu, Huiguang He

Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

Read the original on arXiv Computation and Language →

The paper introduces a framework that aligns multimodal large language models (MLLMs) with human cognitive structure by leveraging a behavior-derived semantic embedding space built from 4.70 million human Odd-One-Out judgments. It maps each triplet to the most salient decision dimension and generates dimension‑grounded linguistic rationales, enabling MLLMs to learn choices and justifications consistent with human criteria. Experiments show improved consistency with human similarity judgments, competitive benchmark performance, and increased representational alignment with human fMRI data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 19

Domain-specific language alignment of multimodal embeddings towards explainable predictions

The paper introduces a framework that aligns multimodal embeddings with language by using captioning functions on interpretable auxiliary data, a soft contrastive loss, and a rank-based concept retrieval task. Applied to Earth observation embeddings, the method improves species distribution modelling, achieving state‑of‑the‑art performance on most metrics and better concept retrieval than prior vision‑language models. The aligned concepts enable recovery of species‑habitat associations and allow domain experts to interpret model predictions in natural language.

By Gabriel\.e Tij\=unaityt\.e, Thijs L. van der Plas, Mohamed-Bachir Belaid, Rob Knapen, Maciej J. Soja, Ioannis N. Athanasiadis
llmsragmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 19

Alignment of visual brain encoding models with individual perceptual similarity judgments

The study explores how to fine‑tune visual brain encoding models so that their internal representations better match individual human perceptual similarity judgments. Using similarity data from 100 images collected from eight participants in the Natural Scenes Dataset, the authors adapted a generic similarity model (DreamSim) to create personalized perceptual similarity models for each subject. They then fine‑tuned each participant’s baseline encoder with either generic or personalized similarity information, finding that models trained with personalized judgments were most aligned with human perceptual similarity and preserved inter‑subject correlations of those judgments.

By Daniel Chong, Haomiao Chen, Keith W Jamison, Raja Marjieh, Ilia Sucholutsky, Nori Jacoby, Amy Kuceyeski
fine-tuningsafety
More like this →
arXiv AI
Jun 26

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

arXiv:2606. 26196v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have recently made remarkable progress in unifying vision-language understanding and reasoning, especially following the introduction of models such as OpenAI's O-series and DeepSeek's R-series, which have driven a paradigm shift toward perception-centric intelligence.

By Haoxiang Sun, Tao Wang, Li Yuan, Jian Zhao, Jiancheng Lv
llmsmultimodal
More like this →