Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv:2509.01107v2 Announce Type: replace Abstract: Layout-to-image (L2I) generation has exhibited promising results in natural domains, but suffers from limited generative fidelity and weak alignmen...
arXiv:2509.02807v2 Announce Type: replace Abstract: Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension...
arXiv:2602.20569v2 Announce Type: replace Abstract: We present AIForge-Doc, the first dedicated benchmark targeting exclusively diffusion-model-based inpainting in financial and form documents with p...
arXiv:2603.16063v2 Announce Type: replace Abstract: Vision Transformers (ViTs) based vision foundation models (VFMs) have achieved remarkable performance across diverse vision tasks, but suffer from...
arXiv:2603.23794v2 Announce Type: replace Abstract: Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent represe...
arXiv:2604.04554v3 Announce Type: replace Abstract: We demonstrate Graph Representation Learning to address challenges that arise in visual odometry. In this context, the change in pose of a camera i...
arXiv:2605.01165v2 Announce Type: replace Abstract: This paper proposes a novel Zero-Shot Action Recognition~(ZSAR) method based on contrastive learning. In ZSAR, we aim to classify examples from cla...
arXiv:2607.28571v2 Announce Type: replace Abstract: Operational Earth observation increasingly calls for answering queries such as ``find the image pairs where a new building appeared.'' This means s...
arXiv:2610.14403v2 Announce Type: replace Abstract: Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and...
arXiv:2306.08757v2 Announce Type: replace-cross Abstract: While diffusion models excel at generating high-quality samples, their latent variables typically lack semantic meaning and are not suitable...
arXiv:2610.24016v2 Announce Type: replace-cross Abstract: Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-wor...
arXiv:2610.35428v1 Announce Type: new Abstract: Structured sparse learning is a leading paradigm for high-dimensional data modeling, as it enables rigorous integration of domain-validated structural...
The paper introduces BoardSafety, a benchmark that transforms manufacturer datasheets and hardware guides into a board‑specific safety rule base for evaluating large language models (LLMs) on hardware safety judgments. Each data item pairs a development board and a proposed operation with a safety rule and a ground‑truth label (safe, unsafe, or insufficient‑info). Experiments show that providing safety rules improves model accuracy, while structured‑rationale prompting yields mixed results, indicating that current models cannot reliably assess operation safety.
InfiniteWorld is a new benchmark that pairs physical‑reasoning questions with simulator videos whose physical parameters are systematically altered, allowing researchers to test whether models rely on visual evidence or on familiar physics. The benchmark’s generation pipeline connects simulation, scene screening, paired video translation, question construction, and input checks, and it includes five tasks—law identification, parameter estimation, temporal reasoning, analytic counterfactual comparison, and law induction. Evaluation across five matched input settings (text‑only, ordered, shuffled observations) reveals that video‑language models often default to Earth‑normal physics, showing strong accuracy on normal questions but poor recognition of altered dynamics. "whyItMatters":"The benchmark provides a measurable way to separate visual evidence from familiar physics, exposing persistent biases in current video‑language models toward Earth‑normal rules."
The paper "Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners" introduces a benchmark of 324 scenarios across 23 collaboration categories to test how well large language models translate partner evidence into calibrated trust decisions. Each scenario requires three sequential decisions with defined transition dynamics, feedback, and cost schedules, allowing precise analytical evaluation of choices against expected outcomes. The study evaluates seven models and 36 controlled-history probes, finding that while the best frontier model approaches near‑optimal decisions, other models display pathologies such as excessive safeguards, zero‑value verification purchases, prioritizing immediate savings over cooperative learning, and misallocating assistance.
MultiEvoSkill is a skill‑optimization framework that transforms skill documents into typed graphs, enabling explicit procedural topology and parallel branches. It employs a heterogeneous agent system to generate revision suggestions based on diverse experiences, and a structure‑aware debate mechanism to resolve edits while respecting dependencies and importance constraints. Experiments show that this approach outperforms state‑of‑the‑art methods and improves cross‑model generalization of skill artifacts.
The paper investigates how users should set the reasoning‑effort level in large language models, a parameter that trades off inference cost for potential performance gains. Experiments on proprietary and open‑weight models across four benchmarks show that higher effort does not consistently improve accuracy, can be up to 42× more expensive on easy tasks, and alters agent behavior by increasing verification and search steps. The authors propose a simple escalation strategy—starting at the lowest effort and only increasing after a verified failure—that matches or outperforms fixed high‑effort settings while reducing cost.
The paper presents the first dataset that captures how humans learn and use mechanisms in unfamiliar environments, using ARC‑AGI‑3 as a testbed. It includes 144 trajectories from 64 participants across 25 public games, each paired with time‑synchronized actions and think‑aloud audio, and 250 trajectories from large language model agents with and without harnesses. The dataset is annotated for mechanism triggers, action purposes, errors, and recovery, enabling a comparison of human and LLM cognition, highlighting differences in hypothesis testing, exploration, and use of qualitative representations.
InferPriv is a new benchmark that evaluates inferential privacy leakage in large language model (LLM) assistants. It measures whether an honest-but-curious recipient can deduce a user’s sensitive attributes from an LLM’s reply during routine tasks, a type of leakage not captured by existing direct‑leakage benchmarks. Testing five frontier models, the study finds that inferential leakage rates (16.2–25.2%) far exceed direct leakage, even when direct leakage is near zero under privacy‑aware prompts.
The paper introduces CEPO, a framework that separates coordination and execution in multi‑agent systems by using a shared coordinator prompt and agent‑specific executor prompts. CEPO guides the optimization of each prompt type according to its role, enabling consistent refinement of coordination rules while allowing execution instructions to remain adjustable. Experiments on 11 benchmarks across four domains show that CEPO consistently improves the performance of four existing prompt optimizers, and additional studies demonstrate better optimization budget efficiency and scaling with more agents.