arXiv Computation and Language

SPyCE: Skill-Policy Co-evolution for Multimodal Agents

arXiv:2607. 13854v2 Announce Type: replace Abstract: Multimodal agents that think with images iteratively manipulate visual evidence and invoke tools across many steps.

arXiv AI
Jun 2

MMSkills: Towards Multimodal Skills for General Visual Agents

arXiv:2605. 13527v3 Announce Type: replace Abstract: Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines.

By Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu
Hugging Face Trending Papers
Aug 11

SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable.

arXiv Computer Vision
Sep 15

V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments

arXiv:2609.15683v1 Announce Type: new Abstract: While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particu...

By Ziqian Fan, Shibo Xu, Junjie Li, Xiangyu Zhao, Shengyuan Ding, Yifan Yang, Zhenjie Yang, Haodong Duan, Yue Zhou, Zhihang Zhong, Xue Yang
arXiv Machine Learning
2d ago

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

SkillRL is a framework that enhances large language model agents by automatically discovering and evolving skills from raw experience. It builds a hierarchical skill library called SkillBank, uses an adaptive retrieval strategy for heuristics, and allows the skill library to co‑evolve with the agent’s policy during reinforcement learning. These techniques reduce token usage and improve reasoning, achieving state‑of‑the‑art results on ALFWorld, WebShop, and seven search‑augmented tasks, outperforming baselines by 15.3% and remaining robust as task complexity grows.

By Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, Huaxiu Yao
arXiv AI
Aug 12

SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

arXiv:2608. 10775v1 Announce Type: new Abstract: Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress.

By Zhou Liu, Ligang Huang, Zeli Su, Zewei Pan, Zhaoyang Han, Xing Chen, Yuanfeng Song, Wentao Zhang
arXiv AI
Aug 7

SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

arXiv:2608. 05970v1 Announce Type: cross Abstract: Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks.

By Changyuan Wang, Chubin Zhang, Zhenyu Wu, Runhao Li, Angyuan Ma, Ke Chao, Yinan Liang, Xiuwei Xu, Ziwei Wang, Yansong Tang, Jiwen Lu