Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 15

Sanity Checking Causal Representation Learning on a Simple Real-World System

The paper evaluates causal representation learning (CRL) methods on a controlled optical experiment where the true causal factors are known. All selected methods fail to reliably recover these factors, even when the data-generating process is simplified to a synthetic version. The study highlights reproducibility issues and the importance of mixing function assumptions, underscoring the gap between theoretical promise and practical performance of CRL techniques.

By Juan L. Gamella, Simon Bing, Jakob Runge
arXiv Machine Learning
Sep 15

Ensemble-Conditioned Molecular Design

The paper proposes a new framework called ensemble-conditioned guidance that reframes molecular design as an optimisation over both the modes and properties of a molecule’s conformational ensemble. It allows 3D generative models to be conditioned simultaneously on multiple axes—such as shapes, pharmacophore profiles, or protein pockets—by adaptively combining vector fields from each condition. The authors introduce adaptive symmetry learning for composable conditions across reference frames, extend the framework to support flexible-size generation, and demonstrate its effectiveness on new benchmarks and practical drug‑discovery tasks, showing improved outcomes when conditioning on additional states compared to single‑state approaches.

By Ross Irwin, Alessandro Tibo, Jon Paul Janet, Simon Olsson
arXiv Computer Vision
Sep 15

MorphoStyle: Motion Style Transfer with Morphology Control

MorphoStyle is a new framework for shape‑aware motion style transfer that uses a shape‑conditioned FSQ‑VAE. It disentangles style from content through a contrastive style encoder, a text‑guided style‑routing mechanism, and a manifold‑preserving style modulator. Experiments on benchmark datasets show that MorphoStyle outperforms existing baselines in both shape control and motion style transfer.

By Xin Feng, Eleonora D'Arnese, Mohan Sridharan
arXiv Machine Learning
Sep 15

Neuron Activation-based Computation of Logical Explanations for Deep Neural Networks

The paper introduces a flexible symbolic framework that efficiently computes logical explanations for deep neural networks by parameterizing explanations with internal neuron activations and leveraging general-purpose logical engines like SMT solvers. Unlike previous methods that rely on specialized verifiers or are limited to individual input features, this approach is not restricted in shape and can scale to deep architectures. Experiments on image recognition and medical benchmarks demonstrate improved computational efficiency and the ability to explain networks that were previously intractable for logic-based methods.

By Tom\'a\v{s} Kol\'arik, Faezeh Labbaf, Fabrizio Leopardi, Grigory Fedyukovich, Michael Wand, Natasha Sharygina
arXiv Machine Learning
Sep 15

Parameter isolation with domain-specific experts for incremental audio classification

The paper introduces a domain‑specific parameter‑isolation architecture for domain‑incremental learning (DIL) in audio classification, aiming to preserve knowledge from earlier domains without accessing their data. By employing data‑free generative replay and cross‑domain feature generation, the method constructs new experts conditioned on all previously frozen models, thereby mitigating catastrophic forgetting. Applied to the DCASE 2026 Challenge Task 7, the approach achieves micro and macro accuracies of 78.4 % and 78.9 %, outperforming the baseline by 33 and 25 percentage points, respectively, with ablation studies confirming the contribution of each component.

By Jongyeon Park, Do-Hyeon Lim, Sang-won Park, Hong Kook Kim, Kyungdeuk Ko, Hyeongcheol Geum, Jeong Eun Lim
arXiv Machine Learning
Sep 15

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

TwinICL is a procedurally generated benchmark that pairs matched text and image versions of tasks to enable controlled comparison of in‑context learning (ICL) across modalities. Experiments on six open‑weight models and 38 tasks show that multimodal ICL consistently underperforms text‑only ICL, with varying gaps by task family. Interventions targeting visual access, task framing, and reasoning can recover strong multimodal performance on a diagnostic subset, yet a modality gap remains even when explicit task instructions are provided, highlighting the dual role of demonstrations as context and evidence.

By Zihan Xue, Po-Yi Lu, Serhii Honcharenko, Zih-Ching Chen, Hsuan-Tien Lin, Nanyun Peng, I-Hung Hsu, Kuan-Hao Huang
arXiv Machine Learning
Sep 15

Eigenvalue-Decomposition Cost Denoising as an Alternative to Predict-then-Optimize for Shortest-Path Problems

The paper proposes using eigenvalue decomposition (or PCA) to denoise noisy cost observations for shortest‑path problems, instead of the traditional predict‑then‑optimize approach. By projecting new cost vectors onto the top‑k eigenvectors of the training covariance matrix before running Dijkstra’s algorithm, the method can recover the true underlying costs. Experiments on a 5×5 grid benchmark show that choosing k equal to the true latent feature dimension (k=5) yields the best performance, outperforming the SPO+ method especially under high model misspecification.

By Henry Aldridge-Krawciw, Irene Aldridge
arXiv AI
Sep 15

SAILOR: Solver-Assisted Interactive LLM-based Optimization Recovery

SAILOR is a proof‑of‑concept system that helps language models translate natural‑language optimization problem descriptions into executable code by detecting missing numerical values. It asks users targeted follow‑up questions, prioritizing them based on uncertainty and solver estimates of impact, and updates the model before returning a solution. In tests on 1,723 benchmark instances, SAILOR achieved exact objective‑value agreement between 27.0% and 87.6% while asking an average of 1.4–5.7 questions per instance.

By Shaghayegh Sadeghi, Stephen L. Smith, David C. Del Rey Fern'andez
arXiv Machine Learning
Sep 15

Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarks

The survey reviews frequency‑domain techniques for time‑series analysis, covering classical Fourier methods to modern neural operators. It identifies three main research challenges: preserving causal structure during spectral transformations, quantifying uncertainty in learned frequency representations, and performing topology‑aware analysis for non‑Euclidean data. By reviewing over 100 studies, the authors propose a unified taxonomy, establish standardized benchmarks, and highlight gaps in geometric deep learning and quantum‑enhanced spectral analysis.

By Qianru Zhang, Yuting Sun, Honggang Wen, Peng Yang, Xinzhu Li, Ming Li, Kwok-Yan Lam, Siu-Ming Yiu, Hongzhi Yin
arXiv AI
Sep 15

To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation

The paper introduces PriCoder, a method for teaching large language models (LLMs) to effectively use private library APIs for code generation. PriCoder synthesizes training data by constructing a graph and applying two operators—Progressive Graph Evolution to increase diversity and Multidimensional Graph Pruning to enhance quality. Experiments on three mainstream LLMs demonstrate that PriCoder boosts private‑library code generation by over 20% in pass@1, while leaving general code generation largely unchanged.

By Yitong Zhang, Chengze Li, Ruize Chen, Guowei Yang, Xiaoran Jia, Yijie Ren, Jia Li
arXiv Computer Vision
Sep 15

Context-Aware Mutual Learning for Blind Image Inpainting and Beyond

The paper introduces Context-Aware Mutual Learning (CAML) for blind image inpainting, a two-stage approach that jointly learns mask estimation and image inpainting. CAML employs two complementary learners—Inpainting-Guided Context-Mutual (IGCM) and Estimation-Guided Context-Mutual (EGCM)—to share contextual details and semantics between the two tasks. Experiments demonstrate that CAML outperforms existing methods on blind inpainting and related vision tasks such as snow, shadow, and watermark removal.

By Haoru Zhao, Yufeng Wang, Zhaorui Gu, Bing Zheng, Haiyong Zheng
arXiv Machine Learning
Sep 15

Graph Neural Networks for Influence Maximization in Social Networks: An Unsupervised Minimum Dominating Set Approach

The paper introduces an unsupervised graph neural network framework for solving the Minimum Dominating Set problem in social networks. By training on 12,000 synthetic graphs, the method achieves up to 55× faster inference than metaheuristic baselines and 14× faster than supervised approaches while producing optimal or near‑optimal dominating sets on real‑world benchmarks. The learned heuristic generalizes well to unseen graph distributions, indicating strong practical applicability for large‑scale social network analysis.

By Erfan Ahmadi, Mina Shirazi, Behnam Bahrak
arXiv Computer Vision
Sep 15

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation

MoVT is a new framework for text‑to‑motion generation that uses a cross‑modal augmented motion tokenizer to project 3D motion tokens into 2D, enriching the motion codebook with real‑world video patterns. The enriched tokens are mapped back to 3D, creating aligned 3D and 2D codebooks that better capture intricate motions. These codebooks feed a generative masked transformer, which predicts masked motion tokens in a modality‑agnostic way, allowing text‑index pairs from the 2D codebook and annotated videos to further improve generation quality. Empirical tests show MoVT outperforms previous state‑of‑the‑art methods on several key metrics.

By Beibei Jing, Tianle Guo, Youjia Zhang, Zikai Song, Yawei Luo, Junqing Yu, Tao Guan, Wei Yang
arXiv Computer Vision
Sep 15

G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity

The paper introduces G-ray, a ray-level relative position encoding for multi-view vision Transformers that remains consistent across different camera projections. By parameterizing rotary phases with camera-local ray angles, G-ray achieves projection-invariant positional consistency and can be integrated with existing encodings without extra learned parameters. Experiments on 3D reconstruction and novel-view synthesis benchmarks show that G-ray improves performance, notably reducing mean pointmap relative error by 45.8% over MapAnything.

By Shuo Zhang, Xin Su, Wei Wang, Jun Liu, Xinrui Zeng, Yongsen Chen, Chenjie Wang, Guibo Zhu, Jinqiao Wang, Bin Luo, Liangpei Zhang
arXiv AI
Sep 15

Atria Dawn: The Dawn of Agentic Superintelligence

The paper introduces Atria Dawn Preview, a foundation agentic language model aimed at scientific research and engineering workflows. Trained through a Verifiable Experience Pipeline, it performs competitively across 16 real‑world benchmarks, achieving the highest scores on five. The authors also present a detailed case study of human–AI collaboration, showing that while AI proposes methods and revisions, humans retain final decision‑making and guide the research direction.

By Honglin Guo, Tao Gui, Yicheng Chen, Guanting Dong, Qiming Ge, Yuyang Hu, Zixian Huang, Jiajie Jin, Alexander Lam, Yining Li, Jiahang Lin, Yanjiang Liu, Xinyu Lu, Haijun Lv, Junlin Shang, Qisheng Su, Guoqiang Wang, Rui Wang, Zhecan Wang, Hao Xiang, Xinchen Xie, Shuhao Xing, Xiaoyu Xing, Wanghan Xu, Xinyu Yang, Yajie Yang, Chengfeng Zhao, Haoran Zhao, Ruojun Zhou, Yunhua Zhou, Yicheng Zou, Kun Cai, Qiye Cai, Xinmeng Che, Haodong Chen, Jiabei Chen, Jiahao Chen, Jiayi Chen, Yujia Chen, Lizhi Cui, Youheng Dai, Xin Deng, Yi Dong, Shihan Dou, Chenya Gu, Xu Guo, Ding Han, Feiyang Hao, Haotan He, Jie Hou, Binze Hu, Zijian Hu, Junhao Huang, Huicheng Jiang, Jiazhen Jiang, Shufan Jiang, Jiahao Kuang, Bowen Lai, Bo Li, Jiaqiang Li, Peng Li, Qilong Li, Zhuoqun Li, Jiaxiang Liu, Shuainan Liu, Tong Liu, Yi Liu, Zhonghang Lu, Jianwen Luo, Yanyi Luo, Huijie Lv, Ningsheng Ma, Zerun Ma, Houcheng Min, Chengjun Pan, Qiyuan Peng, Xiaoxuan Peng, Jianmin Qian, Jiantao Qiu, Wanying Ren, Huayu Sha, Jifei Shan, Zixin Shang, Bing Shao, Zhuohui Sheng, Jiayang Shi, Yang Shu, Aierpanjiang Simayi, Sirui Song, Yuxiao Song, Zhe Sun, Zhichao Sun, Wenzhe Tan, Wenhui Tian, Zhongbo Tian, Hanchen Wang, Pengbo Wang, Rui Wang, Yiding Wang, Yuhui Wang, Zhiheng Xi, Caijun Xu, Chao Xu, Yongfeng Xu, Xiaolei Yang, Zhixiong Yang, Qian Yao, Shihong Yi, Yuankai Ying, Jia Yu, Dingbo Yuan, Hao Yuan, Junjie Yuan, Bo Zhang, Caixian Zhang, Qiuyinzhe Zhang, Jiyuan Zhao, Penghao Zhao, Ying Zhao, Pujun Zheng, Xiaoxue Zhong, Xiaohao Zhou, Xinyu Zhou, Dongsheng Zhu, Guanru Zhu, Yulun Zhu, Yaojie Lu, Tao Ji, Hongyu Lin, Yutao Zhu, Pengfei Cao, Guoxiu He, Xianpei Han, Ben He, Zhicheng Dou, Kang Liu, Qi Zhang, Le Sun, Jun Zhao, Ji-Rong Wen, Xuanjing Huang, Yu-Gang Jiang, Bowen Zhou
arXiv AI
Sep 15

Transfer Learning for Socioeconomic Estimation in Forced-Displacement Settings

The paper presents a transfer‑learning approach that adapts a multimodal spatiotemporal vision transformer, originally trained on Demographic and Health Survey data, to estimate socioeconomic conditions in forced‑displacement settings. Using satellite‑derived geospatial covariates, the adapted model explains up to 66% of variation in socioeconomic outcomes in camp‑intersecting grids and 41% in non‑camp areas, achieving mean absolute errors of 4.37 and 5.41 index points respectively. This framework supplements periodic household surveys by providing regularly updated, spatially granular socioeconomic estimates that bridge data gaps between survey rounds.

By Steven Ndung'u, Adel Daoud, Ismael Yacoubou Djima, Hai-Anh H. Dang, Patrick Michael Brock
arXiv AI
Sep 15

EvoOntology: A Self-Evolving Ontology Layer for Data Agents

EvoOntology introduces a self‑evolving ontology layer for data agents, encapsulating the ontology as an MCP server with schema, content, and tool layers. It enables agents to query and interact with the ontology at runtime, using a builder agent for autonomous construction and a self‑evolution loop that refines the ontology through attribution‑guided edits validated by backbone‑conditional evaluation. Experiments on three data‑agent benchmarks with four LLM backbones show that EvoOntology consistently outperforms strong baselines and existing semantic‑layer approaches, effectively bridging the agent‑data gap for heterogeneous data.

By Meiduo Chong, Shaolei Zhang, Ju Fan, Xiaoyong Du