arXiv Machine Learning June 5, 2026 By Jaehyun Jeon, Min Soo Kim, Jang Han Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu

Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding

Read the original on arXiv Machine Learning →

arXiv:2505. 05026v5 Announce Type: replace-cross Abstract: User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 12

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

arXiv:2606. 13192v1 Announce Type: new Abstract: User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI).

By Ruichao Mao, Zhou Fang, Teng Guo, Hao Yang, Yaping Li, Shaohua Peng, Maji Huang, Xiaoyu Lin, Shuoyang Liu, Xuepeng Li, Yuyu Zhang, Hai Rao

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 2

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

arXiv:2604. 06995v2 Announce Type: replace Abstract: Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding.

By Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 1

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

arXiv:2608.29374v1 Announce Type: new Abstract: Tool-augmented multimodal reasoning integrates external tools (e.g., object detection, depth estimation) into multimodal large language models (MLLMs)...

By Yu Cheng, Arushi Goel, Hakan Bilen

llms computer-vision reinforcement-learning multimodal benchmarks

arXiv AI

Jun 10

Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use

arXiv:2606. 10803v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) excel at utilizing digital APIs and increasingly serve as the "brain" of embodied AI, instructing robots to interact with the physical world.

By Zhixin Ma, Yutong Zhou, Yongqi Li, Chong-Wah Ngo, Wenjie Li

llms agents robotics multimodal benchmarks

arXiv AI

Aug 21

A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems

arXiv:2608. 11907v2 Announce Type: replace-cross Abstract: As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge.

By Hao Zhang, Jiaxin Qi, Zhijiang Tang, Jianqiang Huang

llms multimodal benchmarks

arXiv AI

Aug 25

Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models

arXiv:2608.22232v1 Announce Type: new Abstract: Real-world situation appearances can deviate from their underlying physical states, challenging the reliability of multimodal large language models (ML...

By Zhiming Yang, Zhuoxi Xiong, Donglin Zhou, Wenjun Wei, Shiyao Cui, Jinqiao Shi

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use

A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems

Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models