The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.
By Jhen-Ke Lin, Hong-Yun Lin
arXiv:2608.23921v1 Announce Type: new
Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...
By Yuanhao Sun, Huawei Ji, Yuan Jin, Cheng Deng, Luoyi Fu, Xinbing Wang
arXiv:2609.08188v1 Announce Type: new
Abstract: Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retriever...
By Zhan-Lun Chang, Dong-Jun Han, Seyyedali Hosseinalipour, Mung Chiang, Christopher G. Brinton
arXiv:2608. 09111v1 Announce Type: new Abstract: AI video generation has advanced rapidly and entered widespread commercial use.
By Ziheng Jia, Jiaying Qian, Zicheng Zhang, Xiaorong Zhu, Lancheng Gao, Xiongkuo Min
arXiv:2609.00667v1 Announce Type: cross
Abstract: Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning es...
By Siyi Liu, Hanjun Yang, Chenchen Zhang, Xiaorong Zhu, Xinyu Zuo, Lisheng Duan, Haijin Liang, Jin Ma, Junfu Pu, Yongqi Zhang
AI video generation has advanced rapidly and entered widespread commercial use. As a result, quality differences among videos produced by state-of-the-art AI video generation models~(AIVGMs) have become increasingly difficult to discern using conventional evaluation criteria, such as visual fidelity and semantic instruction following.
The paper introduces the Learned Perceptual Image Fusion Measure (LPIFM), a model trained on dense human pairwise comparisons to assess infrared-visible image fusion. LPIFM jointly processes both source images and fused candidates using a shared hierarchical encoder, triadic interaction, and a tie-aware objective, achieving high agreement with human judgments and outperforming 19 conventional metrics. The authors release a large comparison corpus, model weights, and code, demonstrating LPIFM’s rapid adaptability to new fusion-evaluation protocols.
By Haoran Liu, Mingzhe Liu, Peng Li, Guibin Zan
arXiv:2608. 01301v2 Announce Type: replace-cross Abstract: Infrared-visible image fusion (IVIF) has no ideal fused reference, so fusion algorithms are routinely ranked by scalar objective metrics that formalize different proxies for information transfer, structure, or source similarity.
By Haoran Liu, Mingzhe Liu, Peng Li, Guibin Zan
The paper introduces prediction‑powered evaluation, a framework that blends limited human judgments with large‑scale automatic scores to produce unbiased, data‑efficient system comparisons. It offers both parametric and non‑parametric methods, examines the trade‑off between paired and unpaired designs, and validates the approach on six WMT datasets. Additionally, the authors propose the Prediction‑Powered Saving Ratio (PPSR), a meta‑metric that quantifies how much human annotation can be saved by using an automatic metric within this framework, providing more discriminative and stable metric rankings than existing system‑level meta‑metrics.
By Mingqi Gao, Anthony Sicilia, Weiyan Shi
arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.
By Lei Shi, Anlan Zhang, Rita Lyu, Zhengmian Hu, Tong Yu, David Arbour, Avi Feller, Saayan Mitra, Ritwik Sinha
ET‑Prune is a training‑free framework that dynamically allocates visual token budgets in multimodal large language models based on question‑conditioned evidence. It protects text‑like spatial regions, converts evidence uncertainty into a token floor, and progressively prunes concentrated evidence while retaining diffuse or text‑dense tokens. In six backbone‑benchmark comparisons, ET‑Prune matches or outperforms other pruned methods while keeping roughly half the visual tokens, achieving notable gains on OCRBench‑v2 and MMBench v1.1.
By Zizhong Ding, Junxian Li, Kai Liu, Shaoqiu Zhang, Xiao Xiao, Linghe Kong, Yulun Zhang
arXiv:2609.37515v1 Announce Type: cross
Abstract: Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities...
By Hyunjong Ok, Seunggu Kang, Jaeho Lee