arXiv:2609.09184v1 Announce Type: new Abstract: Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within indivi...
arXiv:2605.00326v2 Announce Type: replace Abstract: Safety pipelines often block content by thresholding a model's first-token unsafe score. We show that a single prompt's score is not a reliable pro...
arXiv:2609. 31450v1 Announce Type: cross Abstract: Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy.
arXiv:2609.14284v1 Announce Type: new Abstract: Criterion-level grading connects examination performance to learning outcomes, but manual marking introduces workload and variation between markers. Th...
VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.
The paper introduces Behavior Pack Optimization (BPO), a post‑training method for video multimodal large language models that replaces single‑response rewards with a set of outputs across counterfactual views. BPO enforces stability when interventions are irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains, using an anchor‑relative advantage to keep the objective stable with small pack sizes. Experiments on datasets such as TempCompass, MVBench, and NExT‑QA show that BPO improves macro accuracy and abstention metrics for models like Qwen2.5‑VL‑7B‑Instruct, with gains that transfer to other benchmarks and models.
Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video descript...
arXiv:2608. 09111v1 Announce Type: new Abstract: AI video generation has advanced rapidly and entered widespread commercial use.
arXiv:2608. 10406v1 Announce Type: cross Abstract: Web search, product search, and question-answering retrieval systems often assign a relevance label and confidence score to each query-candidate pair.
The study compares two ways of obtaining predictions from language models fine‑tuned on customer behavior: scoring answer tokens directly versus generating a written rationale and then scoring the resulting answer. Across 13 model‑domain cells covering four retail tasks, scored readouts consistently rank outcomes more accurately than generated readouts, with an AUC improvement ranging from 1.5 to 14.5 points. The authors also find that a third readout—eliciting a probability before any verdict—improves calibration but only when outcome rates are represented in training, and they recommend using generated rationales for interpretability while relying on scored heads for ranking.
arXiv:2608.21839v1 Announce Type: new Abstract: Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference effic...
Multimodal large language models (MLLMs) have demonstrated strong capabilities in vision-language understanding and natural-language response generation. However, these systems can still produce overconfident predictions and hallucination-like outputs, particularly when the visual evidence is weak, ambiguous, or semantically inconsistent.