Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

The paper introduces ECCBench, a new benchmark for evaluating the memory of vision‑language models beyond simple accuracy. ECCBench measures memory along three axes—efficiency (FLOPs needed to retrieve information), compression (how well compressible inputs are remembered), and calibration (whether the model appropriately signals uncertainty). The study finds that pretrained VLMs compress text but not video, are poorly calibrated, and that some non‑Transformer backbones outperform RoPE Transformers in compression‑calibration trade‑offs, indicating potential benefits for long‑horizon agents.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• multimodal
• benchmarks