arXiv AI

Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

arXiv:2606. 11854v1 Announce Type: cross Abstract: There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs).

arXiv Machine Learning

Aug 10

Symbolic Graphics Programming with Large Language Models

arXiv:2509. 05208v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel at program synthesis, yet their ability to produce symbolic graphics programs (SGPs) that render into precise visual content remains underexplored.

By Yamei Chen, Haoquan Zhang, Yangyi Huang, Zeju Qiu, Kaipeng Zhang, Yandong Wen, Weiyang Liu

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 10

PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting

arXiv:2605. 28066v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability.

By Yu-Che Tsai, Kuan-Yu Chen, Yuan-Hao Chen, Yu-Han Chang, Ching-Yu Tsai, Yu-Hsiang Chuang, Shou-De Lin

llms rag fine-tuning benchmarks safety

arXiv AI

Jun 16

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

arXiv:2606. 15231v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios.

By Zhengbo Zhang, Changtao Miao, Jinbo Su, Zhaowen Zhou, Chunxia Zhang, Xukai Wang, Ruiqi Liu, Kaiyuan Zheng, Jiansheng Cai, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan

llms agents multimodal benchmarks data-engineering

arXiv Machine Learning

Jun 3

Visual Instruction Tuning Aligns Modalities through Abstraction

arXiv:2606. 03871v1 Announce Type: cross Abstract: Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text.

By Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Jun 11

MLaGA: Multimodal Large Language and Graph Assistant

arXiv:2506. 02568v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis.

By Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, Qiaoyu Tan

llms fine-tuning multimodal

arXiv Machine Learning

Jul 27

Layer-wise LoRA fine-tuning: a similarity metric approach

arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.

By Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 12

M*: A Modular, Extensible, Serving System for Multimodal Models

arXiv:2606. 12688v1 Announce Type: cross Abstract: We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, action generators, and world-model predictors.

By Atindra Jha, Naomi Sagan, Keisuke Kamahori, Irmak Sivgin, Rohan Sanda, Steven Gao, Mark Horowitz, Luke Zettlemoyer, Olivia Hsu, Jure Leskovec, Baris Kasikci, Stephanie Wang

llms diffusion robotics efficiency multimodal

arXiv AI

Jul 31

Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.

By Mingkuan Feng, Zhengqi Wen, Jianhua Tao

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 4

Can Large Language Models Generalize Procedures Across Representations?

arXiv:2602. 03542v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are trained and tested extensively on symbolic representations such as code and graphs, yet real-world user tasks are often specified in natural language.

By Fangru Lin, Valentin Hofmann, Xingchen Wan, Weixing Wang, Zifeng Ding, Anthony G. Cohn, Janet B. Pierrehumbert

llms reinforcement-learning

arXiv Machine Learning

Jun 11

IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents

arXiv:2606. 11652v1 Announce Type: new Abstract: This paper investigates reinforcement learning (RL) methods for improving tool-calling capabilities in multimodal small language model (SLM) agents.

By Yifan Yang, Zhen Zhang, Jiayi Tian, Liyan Tan, Zheng Zhang

llms agents nlp reinforcement-learning multimodal

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv Machine Learning

Aug 10

An AI4AI Framework for Visual Token Pruning

arXiv:2608. 07193v1 Announce Type: new Abstract: Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error.

By Zhen Liu, Wenli Huang, Wei Song, Yuhan Liu, Zhiqin Yang, Jingwen Fu

llms efficiency multimodal benchmarks

Related stories