arXiv:2606. 20364v1 Announce Type: new Abstract: A companion study established a de-biased, cross-model VLM-as-3D-judge that reliably ranks single-image-to-3D mesh quality where cheap geometry and CLIP proxies fall short.
The paper investigates whether panels of vision‑language models (VLMs) can reliably judge image aesthetics. It shows that a panel of holistic judges rarely outperforms its best member, but when each model scores images on five rubric‑defined dimensions and these dimension scores are fused across model families, the panel consistently beats the best single VLM on two datasets (EVA and PARA). The study demonstrates that the value of a panel depends on the type of input it receives, and that dimension‑based fusion yields measurable gains at the cost of additional labeling and API usage.
arXiv:2609.27110v2 Announce Type: replace-cross Abstract: Image quality assessment predicts a mean opinion score (MOS), but for image aesthetics the mean hides the individual, so personalized assessm...
The study analyzes 373,019 judgments from LLM‑scored benchmarks, decomposing variance into system, item, judge, and interaction components via generalizability theory. It finds that with a single judge, generalizability converges to a ceiling determined by the system‑by‑judge variance, which is substantially lower in pairwise preference settings, allowing one judge to suffice. The research also reveals significant biases in presentation order and highlights that many published win‑rate claims fall below the measured floor of the benchmarks.
The study investigates how multi‑harness reinforcement learning (RL) affects coding agents by comparing two grouping strategies—Within (one group per task‑harness pair) and Cross (harnesses pooled within a task)—using a Qwen3‑8B policy trained on frozen task‑harness records from Aider, OpenHands, Qwen Code, and SWE‑agent. Across 24,000 sealed evaluations, the choice of evaluation harness dramatically increases solve rates (from 2.14 % to 9.27 %), while the grouping rule has a negligible effect. Both grouping rules yield similar gains on the same source harness, and Cross‑harness credit does not improve portability beyond Within‑harness credit, suggesting that multi‑harness RL reports should specify grouping boundaries and test on unseen harnesses.
arXiv:2610.14707v2 Announce Type: new Abstract: Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad bench...
The paper introduces CoC-CFG, a method that applies classifier‑free guidance to the reasoning trace of vision‑language‑action (VLA) policies, weighting each Euler step to modulate the influence of reasoning. By treating the weight selection as a Markov decision process with measured action values, the authors develop Q‑adaptive guidance that conservatively improves the controller. Experiments on the Alpasim driving simulator and the Waymo end‑to‑end benchmark show that Q‑adaptive guidance outperforms fixed weights and online RL baselines, achieving a new state‑of‑the‑art score on the hidden test set.
Agentic systems have widened the gap between producing candidate outputs and reviewing them. This paper asks a practical architectural question: should domain specialization be built into an evaluator's weights, or into the rule that decides when its judgment can be trusted?
The paper investigates how the scoring rules used in multimodal large‑language‑model (LLM) benchmarks can dramatically alter reported accuracy. By reproducing official scorers for eight benchmarks and comparing them to a blinded human‑plus‑LLM adjudication, the authors show that the same model outputs can yield accuracy differences of up to 22 points depending on the rule. They argue that no single scoring rule is reliable and recommend that benchmarks report non‑compliance rates and use at least two distinct scoring rules.
arXiv:2606. 19162v1 Announce Type: new Abstract: Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself.
arXiv:2607. 02104v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.
arXiv:2608.29765v1 Announce Type: new Abstract: Structured pruning uses surrogate objectives because direct task evaluation over every feasible mask is too expensive. Most evaluations report average...