VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.
By Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
The paper investigates whether code large language models (CodeLLMs) inadvertently reproduce proprietary or sensitive code by evaluating seven state‑of‑the‑art training data detection (TDD) methods on eight CodeLLMs. It introduces CodeSnitch, a benchmark of 9,000 function‑level code samples across three languages, each labeled as included or excluded from training data, and applies mutation strategies based on the Type‑1 to Type‑4 code clone taxonomy to test TDD robustness. The study offers a systematic assessment of current TDD techniques for code and suggests directions for developing more effective detection methods.
By Tianlin Li, Yunxiang Wei, Zhiming Li, Aishan Liu, Qing Guo, Xianglong Liu, Dongning Sun, Yang Liu
MEGA is a new framework that extracts object-level, watertight meshes from 3D Gaussian Splatting (3DGS) scenes. It uses a segment-then-mesh approach, leveraging Spatial Visual Distillation (SVD) to sample diverse camera views of each segmented object and train a mesh reconstruction model with photometric supervision. Experiments on popular benchmarks show that MEGA outperforms existing methods in accurately recovering object-level 3D occupancy and supports complex physical interactions by combining high-quality meshes with photorealistic 3DGS rendering.
By Liwei Liao, Yingkui Zhang, Qianqian Tong, Ronggang Wang
HiPhy introduces a hierarchical reinforcement learning framework for video generation that enforces physical laws at both local and global levels. It addresses the challenge of multi-principle interactions—such as buoyancy and fluid dynamics occurring simultaneously—by ensuring each principle’s temporal dynamics and the overall scene’s coherence. The authors also provide a 50K-prompt dataset and the MultiPhyBench benchmark, demonstrating that HiPhy outperforms existing methods, especially in scenes with multiple concurrent physical principles.
By Tahira Kazimi, Shubhankar Borse, Munawar Hayat, Fatih Porikli, Pinar Yanardag
The paper introduces FaV-A, a multimodal agent that generates positive and negative space compositions in a staged manner. It first creates a base object, analyzes its shape to identify negative‑space semantics, and then produces compositional instructions for the final image. Experiments show that this approach yields more visually coherent and semantically aligned compositions than direct zero‑shot multimodal large language model baselines.
By Shiwen Wang, Jian Yang, Xu Wang, Xincan Wang, Weiming Dong
The paper investigates how changes in evaluation settings—such as camera angles and caption wording—affect the rankings of text-to-3D generators. Using 300 fixed scenes and varying eight render and caption factors, the authors find that configuration variance often exceeds generator variance, leading to frequent shifts in the top-scoring model across 19 alignment evaluators. They conclude that observed winner changes are descriptive rather than definitive, and recommend detailed reporting of generator, score, and protocol specifics to account for uncertainty.
By Anson Y. Lam, Shuqing Li, Michael R. Lyu
Incident‑Arena is a new benchmark for AI coding agents focused on production incident response, featuring 20 tasks derived from real‑world open‑source software. Each task deploys a production application on an ephemerally created Kubernetes cluster, injects faults at various layers, and applies a sustained load profile. The benchmark introduces functional verifiers that maintain system‑level metrics while ensuring safe repairs, and shows that current frontier models achieve below 64.3% across the tasks, highlighting challenges in diagnosis, repair, and regression safety.
By Andre Fu, Malik Drabla, Leon Liu, Meji Abidoye, Marek Suppa, Lata Mishra, Adnan El Assadi, Yiyuan Li
MemFit is a long‑term memory system designed for conversational agents that stores each dialogue turn verbatim in an append‑only store, enabling near‑instantaneous, LLM‑free insertion. It indexes turns using segment summaries and employs an LLM‑free, multi‑path retrieval strategy that blends lexical and semantic signals with cross‑encoder reranking over caption‑augmented episodes. Experiments on LoCoMo, MemGallery, and LongMemEval‑S demonstrate state‑of‑the‑art performance while drastically reducing memory construction time and cost.
By Mitchell Piehl, Muchao Ye
The paper highlights that as Large Language Models grow in capability and prevalence, their environmental footprint is increasing, yet the machine learning community lacks standardized carbon accounting practices. An automated review of 5,285 NeurIPS 2025 papers shows almost no reporting of environmental impact. To address this, the authors propose standardized sustainability metrics for training efficiency, heuristics for estimating inference carbon costs, a software tool called carbonbenchmark for tracking emissions, and the SMAJ framework to encourage prioritizing computational efficiency and environmental accountability over marginal accuracy gains.
By Lachlan McGinness, Dan Pagendam, Robert Offner
The paper introduces a formal framework and benchmark for time‑series world models (TSWMs) that separates state, actions, and exogenous inputs, and defines a new metric called mechanism consistency to evaluate whether model predictions move in the expected direction when actions change. Experiments on eight public datasets show that using a frozen latent prediction space and gated output fusion improves prediction accuracy, while prediction error and mechanism consistency often diverge, with the best‑performing models sometimes failing to exhibit consistent directional responses. Adding a directional supervision loss significantly boosts mechanism consistency without affecting mean‑absolute error, providing a practical recipe for building more reliable TSWMs.
By Haochen Zhang, Jiaheng Guo, Zhen Xu, Zachary Plotkin, Nicholas Konz, Zhen Tan, Tianlong Chen
The paper examines dataset contamination in public brain‑tumor MRI classification benchmarks, introducing a three‑layer framework—duplicate, patient, and source‑label leakage—to evaluate integrity. Audits of the three most used corpora reveal severe leakage: up to 28.8% of test images duplicate training images, 22.3% are byte‑identical, and 95.5% share patients with training data. Even after removing all leaked images, model accuracy remains unchanged, indicating that benchmark integrity cannot be inferred from stable leaderboard scores alone.
By Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala
The paper investigates how well large language models specify the software environments needed to run AI-generated code. Using a new agent protocol and a three‑layer dependency framework, the authors evaluate three coding agents across four languages and fifty tasks, finding that dependency specifications are often inconsistent, redundant, or incomplete. Agreement on dependency sets is as low as 7% for identical tasks, and newer agents show no improvement, indicating that environment specification remains a significant, unaddressed challenge in code generation.
By Bhanu Prakash Vangala, Tanu Malik
PACT is an end‑to‑end model that jointly learns human pose, contacts, and contact forces from monocular video. It augments a human reconstruction foundation model with learnable contact‑force tokens and a temporal transformer, and uses physics‑based supervision to enforce consistency between motion and forces. The authors also create a data annotation pipeline and a real‑world climbing benchmark, ForceWall, to train and evaluate the system, achieving state‑of‑the‑art performance and better generalization than staged approaches.
By Rikhat Akizhanov (MBZUAI), Yangsong Zhang (MBZUAI), Nikolai Kaliazin (MBZUAI), Peter Wolf (ETH Z\"urich), Yoshihiko Nakamura (MBZUAI), Pascal Fua (EPFL), Fabio Pizzati (MBZUAI), Ivan Laptev (MBZUAI)
EurekaBench is a cross‑domain benchmark designed to assess AI agents’ capacity to discover new scientific insights. It comprises 26 long‑horizon tasks spanning fields such as neuroscience, chemistry, and astrophysics, with 306 expected insights that mechanisms should support. The evaluation measures adherence to scientific constraints, predictive accuracy, and the ability to generate insights or guide future research, revealing that current AI agents excel at prediction but lag in producing meaningful scientific insights.
By Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig
The paper investigates collusive behavior in a two‑player Almgren‑Chriss liquidation game using independent proximal policy optimisation agents. By training agents that learn a punitive mechanism, the study shows that when one agent deviates from the mean liquidation schedule, the other responds by accelerating liquidation, thereby negating the deviator’s gains while keeping its own payoff stable. The authors formalize checks confirming that punishment outweighs deviation gains and that behavioral changes are sufficient to offset losses, providing both behavioral and economic evidence of collusion.
By Christos Spyridon Koulouris, Carlo Campajola
The paper introduces RuleEvolve, a self‑evolving framework that automatically refines coding rules for AI coding agents. It maintains a pool of candidate rules, uses an LLM‑powered mutator to generate variants, and a judge module to evaluate and update the pool with the best performers. Experiments across multiple frameworks, LLMs, and benchmarks show that RuleEvolve surpasses manual engineering and prompt‑optimization baselines in functional correctness, code length, and generation cost.
By Zhengyuan Jiang, Reachal Wang, Yuepeng Hu, Yupu Wang, Yuqi Jia, Neil Zhenqiang Gong
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
By Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin, Inessa Fedorova, Dmitry Bocharov, Yuliana Shakhvalieva, Maria Tikhonova, Valerii Ternovskii
The paper introduces a unified benchmark for personalized image generation that uses users' historical data—such as reviews, posts, images, captions, and metadata—to create images aligned with their lifestyle and aesthetic preferences. It defines two tasks: Personalized Scene Generation, which places objects in scenes reflecting user preferences for product presentation, and Personalized Creative Generation, which produces novel images faithful to a user's aesthetic for social media content. The authors also propose PEARL, a method that interleaves multimodal reasoning with a frozen image generator, achieving a 15% average improvement over baselines on personalization metrics.
By Bo Ni, Ngoc N. Tran, Qinwen Ge, Franck Dernoncourt, Seunghyun Yoon, Samyadeep Basu, Sungchul Kim, Puneet Mathur, Nedim Lipka, Tong Yu, Yu Wang, Ryan A. Rossi, Tyler Derr
TabJoinBench is a benchmark designed to evaluate join discovery methods across semantic, relational, and hybrid data lake scenarios. It constructs query‑candidate pairs using source‑specific validation strategies and introduces systematic perturbations—structural, representation, and semantic—while preserving reliable ground truth. The benchmark evaluates set‑based, feature‑based, learned, and language‑model embedding approaches, and publicly releases processed datasets, annotations, and a generation pipeline for reproducible research.
By Sandipan De, Jin Wang, Vivek Gupta
FORALL-LEAN-AGENT is a frontend‑agnostic framework that enhances auditable reasoning in formal mathematics and software verification. It integrates isolated workspaces, Lean tools, and fresh review processes—including statement comparison, axiom audits, and independent proof checking—to bind verification evidence and reviewer decisions to the same candidate artifact, ensuring traceable acceptance. Evaluations on VeriSoftBench, PutnamBench, and the Lean Eval track show that the framework improves benchmark success rates and reduces costs while providing evidence beyond aggregate solve counts.
By Naing Oo Lwin