arXiv:2610. 22221v1 Announce Type: cross Abstract: Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave.
arXiv:2609.13012v1 Announce Type: new Abstract: Vision-language models retain a substantial amount of pixel-decodable visual content in their visual key-value cache. We show, in our setting, that thi...
The paper investigates how joint audio–video generation models learn to associate sound with visual appearance rather than the underlying causal event, a problem termed the visual shortcut. By constructing a structural causal model where audio is independent of video appearance, the authors demonstrate that cross‑attention and shared latent approaches fail when appearance‑event correlations are broken, and that common‑cause routing does not solve the issue. They propose blocking the shortcut via interventions on nuisance variables, proving that counterfactual invariance is necessary and sufficient for identifying the causal predictor, and validate this approach on synthetic and real datasets, including a pretrained video‑to‑audio generator.
arXiv:2610.07355v1 Announce Type: cross Abstract: Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and c...
The paper investigates why joint audio–video generators often learn to predict sound from visual appearance rather than from the underlying event, a problem termed the visual shortcut. By constructing a controlled causal model where audio is independent of video appearance, the authors show that common remedies such as shared latent spaces fail to prevent this shortcut. They propose that intervening on the nuisance appearance is necessary and sufficient for counterfactual invariance, and validate this approach across synthetic and real datasets, highlighting the remaining challenge of unknown nuisances.
CALIPER is a new benchmark that tests whether pretrained visual encoders can infer physical properties such as mass and friction from images. The test involves striking an object twice at known speeds, showing a third strike only up to contact, and asking a linear readout on frozen features to predict how far the object slides. Results show that in clean, fixed‑camera scenes all representations perform similarly, but when camera, lighting, and clutter are varied, only encoders that truly infer physics—like V‑JEPA 2—maintain performance, while random or raw pixel representations fail.
arXiv:2606. 09646v1 Announce Type: cross Abstract: We study whether pretrained video foundation models encode intuitive-physics information in their frozen representations, and how this information varies across model families, layers, and probe types.
arXiv:2512.23994v5 Announce Type: replace-cross Abstract: Text-to-audio-video (T2AV) generation is central to applications such as filmmaking and world modeling. However, current models often fail to...
arXiv:2610.10343v1 Announce Type: new Abstract: Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregr...
The paper revisits the use of contrastive loss for post‑training CLIP models, showing that the reported catastrophic forgetting is mainly due to an overly large temperature parameter in the InfoNCE objective. By setting a smaller temperature, contrastive post‑training can actually improve CLIP. The authors introduce ComCLIP, a lightweight single‑epoch recipe that freezes the text encoder, trains the vision encoder with a properly tempered contrastive loss, an MSE anchoring loss, and a relational distillation loss, achieving comparable zero‑shot classification to CLIP‑Refine while significantly enhancing visual feature transferability and maintaining downstream compatibility with LLaVA.
The paper introduces modality‑gated deep adapters, a parameter‑efficient method for adding new modalities to a frozen multimodal embedding language model without altering its existing outputs. These adapters are bottleneck modules attached to each decoder layer, grouped into modality‑specific packs that activate only during encoding of their own modality, ensuring exact preservation of the base model’s computation graph. Experiments on a 2B base model show significant gains in audio‑to‑text and thermal‑to‑text retrieval metrics, and the authors release the audio and thermal packs along with training and evaluation code.
arXiv:2606.09646v2 Announce Type: replace-cross Abstract: We study whether pretrained video foundation models encode intuitive-physics information in their frozen representations, and how this inform...
The paper introduces a multi‑party backchannel prediction benchmark built from the AMI meeting corpus, featuring 682 masked‑listener views, 190 speakers, and 18,697 backchannel events. A state‑of‑the‑art dyadic model performs at chance when applied zero‑shot to meetings, but its frozen acoustic features are still informative, and retraining improves performance to an AUROC of 0.751. The study reveals that listener conditioning helps only for listeners seen during training, that speaker identity is entangled with useful cues, and that backchannel rates vary significantly across individuals, prompting the authors to report both AUROC and event‑F1 metrics. whyItMatters":"The benchmark and evaluation tools provide a standardized, person‑disjoint testbed for advancing multi‑party backchannel prediction research."