← Back to all news
arXiv Computer Vision By Deepak Sridhar, Yi Li, Kartikeya Bhardwaj, Shuangjun Liu, Taotao Jing, Yuan Li, Shuai Zhang, Jiancheng Lyu, Dashan Gao, Nuno Vasconcelos

Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Read the original on arXiv Computer Vision →

The paper introduces Diffusion Meta-Prompt (DMP), a framework that learns a distribution of prompts via diffusion models, enabling the synthesis of new prompts from natural language task descriptions without needing original task data or losses. DMP incorporates a test-time steering strategy that uses a best training prompt as a latent anchor to stabilize sampling, improving generalization across classification, retrieval, and text-to-image generation tasks. The method reduces storage and inference costs by over 90% compared to prompt retrieval, achieves up to 2.0% average gains over prior meta-learning methods on composite classification, and provides a theoretical bound on expected task loss.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Sep 28

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.

By Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
llmsmultimodalbenchmarks
More like this →
arXiv AI
Oct 1

Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Prompt2Skill is an unsupervised framework that constructs skills for Large Language Models directly from natural‑language task descriptions. It automatically derives task specifications, discovers or synthesizes datasets, and refines the skill through a reflective editing loop. In experiments across question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill outperforms direct prompting, improving performance by an average of 10.8 points on both open‑source and frontier models.

By Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr
llmsnlprobotics
More like this →