Hugging Face Blog

Zero-shot image-to-text generation with BLIP-2

arXiv Computer Vision
Sep 7

How far can we go with ImageNet for Text-to-Image generation?

The paper argues that large-scale text‑to‑image models can be trained effectively on ImageNet, provided the dataset is enriched with carefully crafted text and image augmentations. Using this approach, the authors match the performance of state‑of‑the‑art models such as FLUX, surpassing SD3 on GenEval by +5 points and SDXL on DPGBench by +12, while employing only 1/1000th of the training images and significantly fewer parameters. The method requires just 500 hours of H100 GPU time, making it a more reproducible and accessible alternative to massive web‑scraped datasets.

By L. Degeorge, A. Ghosh, N. Dufour, D. Picard, V. Kalogeiton
arXiv Computer Vision
Sep 25

AdaPilot: Towards Scene-Adaptive Policy Learning for Cross-Generator Text-to-Image Quality Optimization

AdaPilot introduces a scene-adaptive, cross-generator policy for optimizing text-to-image generation quality. By framing multi-turn image generation as a Markov Decision Process and using reinforcement learning, it decouples the policy from specific generator internals, incorporates scene-aware and process-level rewards, and achieves superior quality and generalization compared to baselines. Experiments demonstrate that a single AdaPilot policy can transfer zero‑shot to unseen generators while consistently improving performance across all evaluated models.

By Wenjin Liu, Fayuan Ke, Yue Lu, Zhe Cui, Anh Tuan Luu, Haoran Luo