arXiv AI

Physically Based Rendering in the Latent Space

The paper introduces a method that applies physically based rendering (PBR) within the latent space of variational autoencoders used in image diffusion models. By modifying the rendering equation and using a differentiable renderer, the authors can generate latent maps that guide content creation with physically accurate lighting. The approach is trained on a single rendered image and then shown to generalize to changes in scene geometry, lighting, and camera viewpoint.

arXiv AI
Oct 19

RaPD: Resolution-Agnostic Pixel Diffusion via Semantics-Enriched Implicit Representations

RaPD (Resolution‑agnostic Pixel Diffusion) introduces a continuous Neural Image Field (NIF) latent space for diffusion, enabling resolution‑free generation. It incorporates Semantic Representation Guidance to make latent learning generation‑aware and a Coordinate‑Queried Attention Renderer for scale‑aware, coordinate‑conditioned rendering. A single denoised latent can be rendered at any resolution by adjusting query coordinates, keeping diffusion cost constant while improving generation quality and scalability.

By Yanhao Ge, Shanyan Guan, Weihao Wang, Ying Tai, Mingyu You
arXiv Computer Vision
Sep 22

GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

The paper introduces the Geometry‑Native Autoencoder (GAE), a compact latent space that can be decoded into appearance, depth, camera parameters, and point maps, enabling 3D‑consistent world generation. By reparameterizing a geometry foundation model’s features, GAE replaces traditional appearance‑centric latents and improves visual quality and 3D coherence, achieving significant reductions in FVD and camera‑trajectory error on benchmark datasets. The work demonstrates that a geometry‑native latent space can serve as a shared interface between perception and generation models.

By Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu