arXiv Computer Vision By Niange Yu, Ye Tian, Biaolong Chen, Miao Lu, Aixi Zhang, Hao Jiang, Yunhai Tong, Pipei Huang

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

Read the original on arXiv Computer Vision →

The paper tackles two main issues in multi‑subject video generation—uncontrollable fidelity strength and semantic drift—by exploiting intrinsic attention patterns in Diffusion Transformers. It introduces an Intrinsic Spatial Grounding Map (ISGM) that accurately locates reference subjects and a Dual‑phase Intrinsic Attention Leveraging (DIAL) framework that uses ISGM during both training and inference. DIAL guides attention in low‑noise stages for precise fidelity control and builds preference pairs in high‑noise stages for reinforcement learning, resulting in superior identity consistency and controllable fidelity on the OpenS2V‑Eval benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Hugging Face Trending Papers
Sep 10

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

The paper tackles two main issues in multi-subject video generation—uncontrollable fidelity strength and semantic drift—by studying Diffusion Transformers (DiTs). It discovers that certain attention blocks naturally create an Intrinsic Spatial Grounding Map (ISGM) that accurately locates reference subjects. Leveraging this insight, the authors introduce Dual-phase Intrinsic Attention Leveraging (DIAL), which uses ISGM during low-noise stages to control fidelity strength without retraining and during high-noise stages to generate preference pairs for reinforcement learning, thereby anchoring attention and reducing semantic drift. Experiments on the OpenS2V-Eval benchmark show that DIAL outperforms baseline models, improving identity consistency and enabling controllable fidelity strength.

arXiv Computer Vision
Sep 7

Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation

The paper introduces Reflection-Aware GRPO (RA‑GRPO), a reinforcement‑learning framework that aligns diffusion generative models with human preferences. It uses Diffusion Reflection to correct intermediate sampling paths by reversing the diffusion process, and Counterfactual Path Synthesis to embed these corrected trajectories into the policy, avoiding extra inference cost. Experiments on text‑to‑image and text‑to‑video models show RA‑GRPO outperforms existing methods, reducing reward hacking and improving generalization while remaining architecture‑agnostic.

By Junlong Wu, Jiuzhou Lin, Jia Sun, Boheng Zhang, Huaiqing Wang, Dewen Fan, Houde Liu, Qianqian Gan, Fan Yang, Tingting Gao
arXiv Computation and Language
2d ago

ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint Learning

ViSSFlow is a unified flow‑matching framework that jointly tackles Video‑to‑Sound (V2S) and Visual Text‑to‑Speech (VisualTTS) generation. It incorporates a Diffusion Transformer with a disentangled condition aggregation mechanism, using cross‑attention for semantic cues and self‑attention for dense visual information. The authors also introduce a feature‑level data synthesis strategy to compensate for limited joint training data, and demonstrate that a single model can be trained end‑to‑end without sacrificing performance on either task.

By Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, Meng Cao, Ruihua Song
arXiv AI
Jun 12

ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

arXiv:2506. 01274v2 Announce Type: replace-cross Abstract: Recent progress in Large Multi-modal Models (LMMs) has enabled effective vision-language reasoning, yet the ability to video understanding remains constrained by suboptimal frame selection strategies, albeit with the rapid development of video-specialized LMMs.

By Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro