Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning
Sep 14

Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference

Dynamic Expert Quantization (DynaExq) is a runtime-aware mixed-precision serving system designed for single‑GPU Mixture‑of‑Experts (MoE) inference under a hard high‑bandwidth memory (HBM) envelope. It treats the problem as an online, budget‑constrained precision allocation task, keeping the most frequently used experts at higher precision while relegating the rest to low‑precision fallbacks. By estimating expert hotness from router traces and asynchronously promoting or demoting experts, DynaExq maintains a fully materialized expert set during the forward pass, improving accuracy and throughput compared to static post‑training quantization and offloading/prefetch baselines. whyItMatters":"DynaExq enables efficient deployment of large MoE models on memory‑limited GPUs by dynamically allocating precision based on runtime expert usage, thereby reducing memory footprint and latency while boosting accuracy and throughput."

By Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, Wei Zhang
arXiv Machine Learning
Sep 14

DenseTRF: Texture-Aware Unsupervised Representation Adaptation for Surgical Scene Dense Prediction

DenseTRF is a self‑supervised framework that adapts texture‑aware representations for dense prediction in surgical computer vision. It uses slot attention to learn invariant visual structures and then conditions dense prediction on these representations, merging models to adapt to target distributions without supervision. Experiments on multiple surgical procedures show that DenseTRF improves cross‑distribution generalization compared to state‑of‑the‑art segmentation models and test‑distribution adaptation methods.

By Guiqiu Liao, Matja\v{z} Jogan, Daniel A. Hashimoto
arXiv Computation and Language
Sep 14

Automated Detection and Structuring of Social Tipping Point Evidence in Climate related Documents: A Modular AI Framework

The paper introduces an open, modular AI framework that automatically detects and structures evidence of social tipping points in climate literature at the passage level. It integrates a DistilBERT boundary splitter, an iteratively augmented RoBERTa classifier, a Mistral 7B rewrite model, a LLaMA 3.2 3B rating model, and a Milvus vector store, all accessible via a Streamlit interface. Evaluation on a GPT‑4.1‑labelled benchmark and expert‑reviewed set shows the splitter outperforms competitors and the RoBERTa detector achieves high accuracy and agreement.

By Kavindu Perera, Mohammad Abaeiani, Ekaterina Gilman, Lauri Loven, Mourad Oussalah, Tassos Kanellos, Beatrice Gobbo, Dante Adami, Nicol\`o Ferriani, Maximiliano Romero, Pierre Rossel, Marc Bonazountas, Christina Deligianni, Nikos Xyderis, Artur Bogucki, Lampros Argyriou, Prasasthy Balasubramanian
arXiv Computation and Language
Sep 14

Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models

The paper introduces Tasks over Application Manuals (TAM), a benchmark designed to test long‑horizon procedural reasoning in large language models. TAM uses real‑world tasks from ICD‑10‑CM clinical coding and U.S. federal sentencing, requiring models to follow extensive, rule‑based manuals and perform interdependent steps to produce exact answers. Experiments with GPT‑5 and various prompting strategies show very low exact‑match accuracy—1% for coding and 15.5% for sentencing—highlighting a gap between current benchmarks and the ability to reliably follow complex procedures.

By Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie, Sachin Chandrasekhar, Eugene Wen
arXiv Computation and Language
Sep 14

Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?

The study investigates how deictic ambiguity—specifically the shifting reference of expressions like "previous"—affects Draft‑Verify‑Revise pipelines that use multiple large language models (LLMs). Using a synthetic dataset of 10 base examples and 21 reasoning‑effort configurations, six LLMs were evaluated for their ability to correctly resolve the ambiguous expression across the draft, verify, and revise stages. Results show wide variance in balanced accuracy, with GPT‑5.2 improving from 0.156 to 0.942 with increased reasoning effort, while Gemini 3 Pro consistently achieved high accuracy above 0.94 even at low reasoning effort, and meta‑evaluators often relied on surface cues when making errors.

By Obinna I. Ekekezie
arXiv Computation and Language
Sep 14

UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking

The paper introduces UrduFactBench and UrduFactQA, two hand‑annotated benchmarks for claim verification and factual consistency evaluation in Urdu, created through a multi‑stage annotation process with native speakers. It also presents UrduFactCheck, a modular fact‑checking framework that uses both monolingual and translation‑based evidence retrieval to address the scarcity of high‑quality Urdu evidence. Experiments on twelve LLMs show that translation‑augmented pipelines outperform monolingual ones, highlighting ongoing challenges for open‑source models in Urdu.

By Sarfraz Ahmad, Hasan Iqbal, Momina Ahsan, Numaan Naeem, Muhammad Ahsan Riaz Khan, Arham Riaz, Muhammad Arslan Manzoor, Yuxia Wang, Preslav Nakov
arXiv Computer Vision
Sep 14

RoES: Rotational Equivariant Selective-frequency Fusion for Multimodal Images

RoES is a Rotational Equivariant Selective-frequency fusion network that dynamically separates low- and high-frequency components of infrared-visible images. It uses a trainable rotation-enhanced updater to decouple frequencies, then fuses them with a dual-branch module: a rotation-equivariant Mamba for low-frequency structural dependencies and a polar spectral attention Dual-Fourier block for high-frequency detail refinement. Experiments show RoES outperforms existing methods in fusion quality and downstream object detection, offering a robust multimodal fusion solution.

By Jiabao Wang, Wenjian Liu, Yaoming Cai, Gengyu Zhang, Boyan Zhao, Zijia Zhang, Yao Ding, Xiaobo Liu
arXiv Computer Vision
Sep 14

Partition-Invariant Tuning for 3D Scene Understanding

The paper introduces PointPiT, a partition‑invariant tuning framework designed for scene‑level point cloud understanding. It combines a Scene‑aware Structural Adapter (SSA) that fuses local geometry with global context, and Gradient Subspace Optimization (GSO) that selects stable update directions to reduce partition‑induced representation shifts. Experiments on multiple benchmarks show that PointPiT matches or surpasses full fine‑tuning while using less than 1% of the backbone’s parameters, achieving state‑of‑the‑art performance among parameter‑efficient fine‑tuning methods.

By Hongqiang Lin, Tianle Wang, Shuiwang Li, Dongxu Zhang, Yiding Sun, Zihao Guo, Dongfu Yin
arXiv Computer Vision
Sep 14

PhysioAI: Clinical Knowledge-Guided Semantic Supervision for Skeleton-Based Physiotherapy Action Recognition

PhysioAI introduces a clinical knowledge‑guided framework that injects structured physiotherapy knowledge into skeleton‑based action recognition models. By combining graph‑based spatiotemporal modeling with semantic anchors derived from a Clinical Knowledge Dictionary encoded via a frozen CLIP model, PhysioAI improves training of skeleton representations while requiring only skeleton inputs at inference. In subject‑disjoint evaluations, it outperforms existing methods on KiMoRe, Hard‑67, and UI‑PRMD benchmarks, achieving up to 99.03% accuracy on KiMoRe overall.

By Jie Cao, Euijoon Ahn, Anwar Hassan, Jinman Kim
arXiv Computer Vision
Sep 14

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

ChitraMiti introduces a synthetic benchmark of 12,874 Bengali planar geometry problems with structured 15‑attribute descriptions, alongside a complementary set of 500 textbook diagrams. Using a three‑phase protocol that tests diagram‑only, diagram‑plus‑description, and description‑only inputs, the study finds that description‑only performance matches diagram‑plus‑description performance across several VLMs, yet models still struggle with cross‑modal verification. Fine‑tuning on ChitraMiti improves results on both datasets, though a gap remains compared to the best zero‑shot model.

By Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi, Md Adnan Arefeen
arXiv Computer Vision
Sep 14

KAD-Net: Kinematics-Aware Decoupled Learning for Robust 3D Hand Pose Estimation from a Single Depth Image

KAD-Net introduces a Kinematics-Aware Decoupled Learning Network for 3D hand pose estimation from a single depth image. It employs a Finger Topology Constraint module that uses local kinematic representations of three consecutive finger joints to better model distal joint relationships and handle occlusion. The architecture also decouples 2D joint localization from depth estimation in a hierarchical multitask framework, reducing feature interference and improving accuracy on benchmark datasets such as ICVL, NYU, and MSRA.

By Jun Lu, Zhenming Chen, Lin Chen, Kanlun Tan, Xiaoling Li, Qiao Liu
arXiv Computer Vision
Sep 14

RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

RA‑SOD is a new RGB‑Thermal salient object detection framework that explicitly models the reliability of each modality. It introduces a reliability‑conditioned representation, an uncertainty‑guided dual‑stream refinement, and a pixel‑wise modality competition mechanism to adaptively compensate degraded features and suppress unreliable evidence. Experiments on four benchmarks show that RA‑SOD achieves state‑of‑the‑art performance and remains robust under severe modality degradation.

By Hongbo Gao, Zhengyu Li, Xueru Nie, Dihao Zhu, Lijun Zhao, Yunke Wang, Chang Xu
arXiv Computer Vision
Sep 14

DementiaCare-Bench: A Modality-Validated Video Benchmark

DementiaCare-Bench is a new video benchmark designed to evaluate video-language models (VLMs) on understanding behavioral and psychological symptoms of dementia (BPSD). It contains 56 caregiver training videos, 94 clips across nine BPSD categories, and 2023 questions that are grounded in transcript spans and labeled by their visual demand. The benchmark reveals that most VLMs perform well on questions answerable by language alone but struggle with those requiring ordered frames, and a lightweight LoRA fine‑tune (DemCare‑VLM) can improve video dependence.

By Afrouz Sheikholeslami, Yuankai Qi, Xuyun Zhang, Luping Zhou, Amin Beheshti, Quan Z. Sheng, Ming-Hsuan Yang
arXiv Computer Vision
Sep 14

Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimization

PhysPlan is a training‑free guidance framework that enhances video diffusion models by incorporating physical awareness through agentic physics simulation. It uses a vision‑language model to generate a Chain‑of‑Visual‑Thought representation of kinematic trajectories and 3D depth, which then drives an object‑centric test‑time optimization that isolates kinematic changes and locks the passive environment. The framework also employs Kinetic Intensity Profiling to adapt hyperparameters to varying physical deformations, and demonstrates superior performance on PhyGenBench and Physics‑IQ benchmarks compared to existing VDM baselines.

By Minh-Loi Nguyen, Xuan-Vu Le, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
arXiv Computer Vision
Sep 14

AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation

AnchorVLN is an open‑vocabulary vision‑language navigation system that separates semantic proposals from geometric metrics. It uses a VLM to generate semantics while a geometry module supplies reliable metric quantities such as range and bearing, all within a Model Context Protocol server. The system achieves 64.4% on instruction following and improves object‑reference accuracy, reducing median center error from 3.37 m to 2.48 m.

By Long Giang Vu, Chengkai Yao, Yuxin Liu, FNU Aryan, Rajath Chandrashekar Aralikatti
arXiv Computer Vision
Sep 14

TestDG: Test-time Domain Generalization for Continual Test-time Adaptation

This paper introduces TestDG, an online test-time domain generalization framework for continual test-time adaptation (CTTA). TestDG learns features invariant to both current and past test domains during testing, using a new model architecture, adaptation strategy, and prototype selection/update mechanisms. It achieves state‑of‑the‑art results on four CTTA benchmarks and demonstrates superior generalization to unseen test domains.

By Sohyun Lee, Nayeong Kim, Juwon Kang, Seong Joon Oh, Suha Kwak
arXiv Computer Vision
Sep 14

UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation

UniLayDiff is a Unified Diffusion Transformer that tackles content‑aware layout generation across a wide range of tasks—unconditional, element‑type, size, and relationship‑conditioned generation—using a single end‑to‑end trainable model. It treats layout constraints as a distinct modality within a Multi‑Modal Diffusion Transformer framework, capturing interactions among background images, layout elements, and constraints. The model is further refined with LoRA fine‑tuning to incorporate relation constraints, achieving state‑of‑the‑art performance and, according to the authors, the first unified solution for all content‑aware layout generation tasks.

By Zeyang Liu, Le Wang, Sanping Zhou, Yuxuan Wu, Xiaolong Sun, Gang Hua, Haoxiang Li
arXiv Computer Vision
Sep 14

EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

EgoFun3D introduces a coordinated task, dataset, and benchmark for creating simulation-ready interactive 3D objects from egocentric videos. The approach captures general cross-part functional mappings via function templates, enabling precise evaluation and direct compilation into executable code. A 4‑stage pipeline—2D part segmentation, reconstruction, articulation estimation, and function template inference—is proposed, and a dataset of 517 videos with detailed annotations is released for comprehensive benchmarking.

By Weikun Peng, Denys Iliash, Manolis Savva
arXiv Computer Vision
Sep 14

GNM Head: A Generative aNthropometric Model of the human head

The paper introduces GNM Head, a generative anthropometric model that extends beyond traditional head models by including detailed ocular and intra‑oral structures such as eyeballs, teeth, and tongue. Built from a large collection of high‑resolution 3D scans and artist‑crafted samples, GNM offers improved geometric fidelity and state‑of‑the‑art fitting performance for target 3D face scans. The authors release the full framework publicly to support further research and development.

By Stylianos Ploumpis, Jan Bednarik, Gaspard Zoss, Ruslan Guseinov, Luca Prasso, Prashanth Chandran, Oliver Boyne, Vasileios Choutas, Timo Bolkart, Daoye Wang, Menglei Chai, Di Qiu, Sebastian Winberg, Gilles Rainer, Lewis Bridgeman, Leonhard Helminger, Edo Collins, Delio Vicini, J\'er\'emy Riviere, Yannick Boetzel, Alexander Koumis, Stylianos Moschoglou, Jay Busch, Cynthia Herrera, Jacob Still, Scott Ysebert, Peter Lincoln, Sergio Orts Escolano, Christoph Rhemann, Erroll Wood, Thabo Beeler, Stefanos Zafeiriou
arXiv Statistics ML
Sep 14

An ab initio foundation model of wavefunctions that accurately describes chemical bond breaking

The paper introduces Orbformer, a transferable wavefunction model that uses deep neural networks to pretrain on 22,000 equilibrium and dissociating molecular structures. Fine‑tuning Orbformer on unseen molecules achieves an accuracy‑cost ratio comparable to classical multireference methods, consistently reaching chemical accuracy (1 kcal/mol) on standard benchmarks, challenging bond dissociations, and Diels‑Alder reactions. This demonstrates that amortizing the cost of solving the Schrödinger equation across many molecules is feasible in quantum chemistry.

By Adam Foster, Zeno Sch\"atzle, P. Bern\'at Szab\'o, Lixue Cheng, Jonas K\"ohler, Gino Cassella, Nicholas Gao, Jiawei Li, Frank No\'e, Jan Hermann