← Back to all news
arXiv Computer Vision

Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

The paper introduces “Spend Where It Matters”, a two‑stage, training‑free framework that combines particle search with Metropolis‑Hastings refinement for budget‑constrained audio‑visual alignment. In the first stage, particles are adaptively allocated across denoising checkpoints based on online variance estimates, reducing estimation variance while focusing compute where it matters most. The second stage uses MH refinement with local noise‑and‑denoise proposals to further improve reward, starting from the strong candidate produced by the particle search. Experiments on “JavisBench‑Mini” show that this combined approach yields higher rewards than either particle search or MH alone, achieving significant relative gains over baseline methods. The method also demonstrates benefits for image generation on SD1.5, improving aesthetic and image reward scores compared to uniform particle allocation.

Related stories

arXiv Machine Learning
Oct 16

Noise-Space Reward Alignment for Few-Step Generative Models

The paper introduces Noise Policy Optimization (NPO), a method for aligning few-step generative models with black-box reward signals. NPO learns an amortized flow policy that transforms initial Gaussian noise into a reward-tilted distribution, allowing the pretrained generator to produce higher-reward outputs without fine-tuning. The approach achieves superior reward-quality trade-offs across image, language, and protein generation tasks while drastically reducing inference cost compared to test-time search methods.

By Woocheol Shin, Taeyoung Yun, Jiwon Sung, Hyeongyu Kang, Jeongmin Park, Jaewoo Lee, Kyuil Sim, Dohyun Kim, Yutong He, Jinkyoo Park
diffusionreinforcement-learningfine-tuningsafety
More like this →
arXiv Machine Learning
Aug 28

GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

The paper introduces GRAS, a method that improves training‑free reward alignment for discrete diffusion models by reducing variance in guided proposals and adapting the resampling temperature during search. It achieves this without adding denoiser cost, using Rao‑Blackwellized estimates for differentiable rewards and a leave‑one‑out baseline for non‑differentiable ones. Experiments on regulatory DNA and protein design show GRAS outperforms existing training‑free techniques and rivals reward‑fine‑tuned models.

By Kwanyoung Kim
diffusionfine-tuningsafety
More like this →
arXiv AI
Oct 12

Unsupervised Multimodal Reward Modeling

The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.

By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang
llmsreinforcement-learningfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv AI
Oct 27

VaLoP: Value Gradient Lookahead and Pullback for Reward-Guided Diffusion

VaLoP introduces a new method for reward-guided diffusion called Value Gradient Lookahead and Pullback, which evaluates the value function at a less noisy state and pulls its gradient back to the current state to reduce posterior covariance errors. The approach unifies two common gradient estimators and adds a correction term computed with a single vector-Jacobian product, while VaLoP‑Fast amortizes this correction with a lightweight head trained on a single reward. Experiments across multiple models, rewards, and datasets show that both variants consistently outperform existing value‑gradient guidance, and VaLoP‑Fast achieves the highest success rates on robot policy benchmarks such as LIBERO and RoboCasa.

By Kwanyoung Kim
diffusionreinforcement-learningroboticsbenchmarks
More like this →
Hugging Face Trending Papers
Aug 19

Falcon Perception-HD: High Density Perception via Reinforcement Learning

Falcon Perception-HD applies reinforcement learning (GRPO) to autoregressive perception models, aligning them directly with precision and recall metrics rather than relying on maximum‑likelihood fine‑tuning. The RL framework introduces reward design for set‑structured outputs and multi‑head sampling control, enabling state‑of‑the‑art performance in very dense scenes (up to 500 objects) and eliminating common issues such as mask repetitions, NMS, and coordinate deduplication. Hybrid self‑annotation pipelines tailored for difficult referring expressions and dense scenes further boost RL training, with improvements observed across all difficulty levels on PBench and SACO‑Gold, and the model preserves object existence knowledge without negative samples.

computer-visionreinforcement-learningfine-tuningbenchmarks
More like this →