arXiv:2610.01670v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model...
NumBench is a large benchmark of 640,000 text‑to‑image prompts that tests how well models count objects, covering 1,600 categories and counts from 1 to 100. It uses a factorial design to vary composition, spatial guidance, and appearance while balancing counts, and introduces a process model that predicts a near‑quadratic collision deficit at low occupancy. The authors also propose the Confidence‑Weighted Numeric Precision Score for scalable evaluation and find that all tested systems perform poorly above 50 objects, with count range, layout, and composition having the largest effects.
arXiv:2606. 00188v1 Announce Type: cross Abstract: While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle.
arXiv:2606. 01213v1 Announce Type: cross Abstract: Despite tremendous recent progress, current text-guided image editing methods still struggle with many aspects of editing involving instruction following, minimally editing the source image, and ensuring high visual quality.
The paper introduces a counterfactual benchmark to evaluate whether vision models understand physical constraints or merely learn rendering shortcuts. Using Mitsuba 3, it generates 5,500 synthetic images across 500 scene families, each with a clean image and ten violations (shadows, reflections, support, surface response, occlusion). Diagnostic baselines (LLaVA‑1.5‑7B, Qwen2.5‑VL‑7B, InternVL3.5‑8B) achieve 64.0–67.8 % macro‑F1 on a held‑out synthetic test set, but performance drops to 40.8 % for LLaVA‑1.5‑7B under an intervention shift, indicating reliance on rendering cues rather than true physical reasoning.
ABACUS is a 3‑billion‑parameter vision‑language model that simultaneously tackles object counting, crowd counting, referring‑expression counting, and count‑faithful image generation. It introduces density‑aware adaptive zooming with an objectness map, a boundary‑aware count policy trained via GRPO to avoid over‑ or under‑counting at crop edges, and a cycle‑consistent GRPO strategy that scores generated images for count accuracy and aesthetic quality without external critics. The model sets new state‑of‑the‑art performance on seven benchmarks, outperforming both specialized and larger generalist models.
The paper introduces WeGenEditBench, a benchmark with 2,740 evaluation instances and 148 task types for image editing, focusing on local failure detection and prioritization for model improvement. It also presents WeGenEvoStudio, a multi‑agent harness that supports recursive self‑improvement by diagnosing failures, constructing retrieval‑grounded data, and adapting models. Experiments show that models such as GPT‑Image‑2 and Seedream‑5.0‑Pro excel in single‑image and text‑intensive editing, and evolution experiments demonstrate significant targeted gains on selected benchmarks.
CompoSET is a new vision‑language benchmark that tests compositionality by presenting each image‑caption pair with a foil that differs by only a single controlled edit—such as color, count, spatial relation, or object state—while keeping the overall scene largely unchanged. The dataset contains 1,776 instances across 80 scenes, 16 edit types in four semantic categories, and captions at three verbosity levels, with an average of only 8% pixel change between base and foil images. Evaluation of 22 dual‑encoder VLMs shows that all models perform at or below chance on CompoSET, contrasting with their strong performance on prior benchmarks that conflate global image differences with compositional reasoning.
arXiv:2607. 22705v1 Announce Type: cross Abstract: Object-centric learning aims to represent scenes as objects whose properties can be reused in new combinations.
MatReplace is a new reference‑free benchmark for evaluating material replacement in interior scenes, assessing edits on local material correctness, global lighting harmony, outside preservation, and inside structure. It offers three tracks that vary the conditioning signal—instruction only, instruction plus region mask, and material reference image—allowing systematic comparison of different editing approaches. Results show that while closed‑source editors excel at named‑material rendering, grounding materials from pixel references remains a significant challenge.
arXiv:2610.28356v1 Announce Type: cross Abstract: Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many exis...
arXiv:2609.01409v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown strong performance in generating scientific figures from text or images. However, producing publication-ready...