FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoders

The paper introduces FailSAE, a method that uses Sparse Autoencoders to predict failures in vision‑language models (VLMs) such as CLIP. By treating failure prediction as a classification over sparse SAE latent activations and employing a three‑stage training pipeline, the approach yields higher prediction accuracy than existing confidence‑score or auxiliary‑classifier baselines. Analysis shows that the SAE captures class‑specific concepts and reveals a shift toward ambiguous or style‑related concepts during failures, offering insights for runtime failure recovery.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• llms
• rag
• multimodal
• safety