You Cannot Photograph the Same Street Twice: Reliability Limits in Vision-Language Measurement of Urban Change

Vision‑language models used to gauge urban change from repeated street‑level images exhibit limited reliability at single locations. In a study of 4,648 image pairs from 435 Google Street View points across five U.S. cities, re‑photographing the same street altered perception scores by an average of 0.80 points—about two‑thirds of the difference between distinct streets—while repeated model calls added negligible variation. Although image re‑encoding, prompt order, and various image statistics contributed modestly, a small systematic drift (~0.1 points) persisted and grew with time between captures, suggesting minor unrecorded physical changes. Controlled experiments revealed that varying camera and image properties can shift scores, and that camera geometry alone caused a model to falsely report change in 45% of identical scenes; normalising to a common virtual camera reduced this to 7.5%. Despite these individual‑point unreliabilities, aggregating many paired observations recovers a clear redevelopment signal, indicating that such models are dependable at large scales but not for single‑location assessments.