The paper introduces FailSAE, a method that uses Sparse Autoencoders to predict failures in vision‑language models (VLMs) such as CLIP. By treating failure prediction as a classification over sparse SAE latent activations and employing a three‑stage training pipeline, the approach yields higher prediction accuracy than existing confidence‑score or auxiliary‑classifier baselines. Analysis shows that the SAE captures class‑specific concepts and reveals a shift toward ambiguous or style‑related concepts during failures, offering insights for runtime failure recovery.
arXiv:2602.06652v2 Announce Type: replace Abstract: The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions r...
arXiv:2606. 24716v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence.
arXiv:2602. 00462v4 Announce Type: replace-cross Abstract: Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM.
arXiv:2609.09909v1 Announce Type: new Abstract: Although text-to-image diffusion models generally exhibit strong prompt-following ability, we identify a persistent and previously underexplored failur...
arXiv:2609.17790v1 Announce Type: new Abstract: Pre-trained vision-language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustnes...
arXiv:2607. 04593v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model.
arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.
arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.
arXiv:2606. 16193v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret.
arXiv:2605.27136v2 Announce Type: replace Abstract: Uncertainty quantification (UQ) remains a critical challenge in Large Vision Language Models (LVLMs) for reliable predictions and real-world deploy...
arXiv:2608. 10289v1 Announce Type: cross Abstract: Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios.
arXiv:2608. 09011v1 Announce Type: new Abstract: Uncertainty Quantification (UQ) aims to measure the reliability of model predictions, serving as a critical safeguard for deploying Vision-Language Models (VLMs) in safety-critical scenarios.