Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Computer Vision
Sep 14

UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction

arXiv:2609.12898v1 Announce Type: new Abstract: Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized b...

By Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang
arXiv Computer Vision
Sep 14

MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment

arXiv:2609.12081v1 Announce Type: cross Abstract: Mobile manipulation requires perceptual evidence at different spatial scales for base motion and arm control, while the two action modalities remain...

By Guangyu Chen, Qiwei Liang, Shaolong Zhu, Tianxing Chen, Zikuan Xiao, Yifan Xie, Lingfeng Zhang, Ping Luo, Renjing Xu, Wenbo Ding
arXiv Machine Learning
Sep 14

Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization

The study benchmarks tokenization choices for generative medical event models, evaluating quantization granularity, reference-range anchoring, code–value fusion, numeric and temporal encodings, and native versus harmonized event representations. Using Llama and Qwen architectures, 156 models were trained and assessed on early hospitalization data, showing that fusing codes with value deciles and using event-order or admission-relative RoPE embeddings improved predictive performance. The Common Longitudinal Intensive Care Unit Data Format (CLIF) reduced token count by 30.8% while enhancing outcomes in most families.

By Inhyeok Lee, Luke Solo, Michael C. Burkhart, Bashar Ramadan, Sahil Sethi, Sarah Jabbour, William F. Parker, Brett K. Beaulieu-Jones
arXiv Computation and Language
Sep 14

HypoKG: Evidence-Disciplined Biomedical Hypothesis Generation Beyond Endpoint Knowledge

The paper introduces HypoKG, a unified biochemical knowledge graph built from KEGG, Rhea, and UniProt, and uses it to benchmark 13,200 biomedical hypotheses generated by six large language models (LLMs). By varying the biological information provided—source enzyme only, full biological path, or source and disease endpoint—the study finds that LLMs produce higher-scoring hypotheses when given minimal information, but these are less evidence‑grounded. When supplied with the full biological path, the models generate hypotheses that align more closely with known mechanistic relationships, a phenomenon the authors term evidence‑disciplined reasoning, which is confirmed by shuffling intermediate path steps. "whyItMatters":"The study demonstrates that knowledge graphs can both uncover novel disease–enzyme pairs and guide LLMs to reason more accurately from evidence, improving the reliability of AI‑generated biomedical hypotheses."

By Dominic Okonkwo, Adetayo Okunoye, Ismailcem Budak Arpinar
arXiv Computation and Language
Sep 14

GraphProfiler: Source-Linked Sensitive Attribute Inference via Personal Knowledge Graphs

GraphProfiler is an auditable LLM-based profiler that infers sensitive attributes such as age, income, and occupation from user-generated content by aggregating indirect cues across many posts. It represents each user's post history as a source-linked personal knowledge graph, allowing predictions to be traced back to specific posts, concepts, and relationships. The system achieves an 86.7% success rate on the SynthPAI benchmark and 84.6% on PANDORA, citing supporting evidence for over 98% of predictions and demonstrating that cited posts significantly contribute to attack success.

By Ahmed Sohair Khan, Estrid He, Chenglong Ma, Monica Wachowicz, Elham Naghizade
arXiv Computer Vision
Sep 14

DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

DRS‑VPT is a feed‑forward transformer that performs image‑to‑scan registration by predicting the scan pose, point maps, and a coarse‑to‑fine feature pyramid for direct reprojective alignment. It unifies tasks like camera‑LiDAR calibration and indoor camera‑to‑map relocalization, achieving state‑of‑the‑art results in autonomous driving and competitive performance indoors without map‑specific training. The model also learns complex scan‑to‑image projection behaviors, such as occlusion of back‑facing points.

By Lanke Frank Tarimo Fu, Maurice Fallon
arXiv Computer Vision
Sep 14

GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models

GRACE is a new framework for concept erasure in text-to-image diffusion models that uses a semantically weighted sensitive subspace to guide localized interventions and lightweight subspace-constrained adapters to avoid global semantic disruption. It replaces manual counterfactual prompts with an automatically decoupled safe-anchor mechanism and controls intervention strength through an energy-driven dynamic gating system. Experiments show GRACE improves NSFW reduction by 17.86% over five state-of-the-art methods while also reducing target CLIP Score and FID, indicating stronger concept suppression with better preservation of generative quality.

By Qinghui Gong, Yihuai Liang, Yuanlun Xie, Deepak Kumar Jain, Vitomir \v{S}truc, Zhengchun Zhou
arXiv Computer Vision
Sep 14

VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation

VideoTok4D introduces a 4D‑aware video tokenizer that transforms videos into compact tokens representing a dynamic 3D world. It employs spatiotemporal disentanglement to separate static and dynamic content, a track‑aware dynamic attention mechanism for motion consistency across views, and a diffusion prior (Co4DGen) for efficient 4D scene generation. Experiments show state‑of‑the‑art results with up to four orders of magnitude less storage than dense 4D representations, and shorter diffusion sequences for faster generation.

By Xinyi Chen, Hanxin Zhu, Xijun Wang, Xingrui Wang, Sen Liang, Xin Li, Zhibo Chen
arXiv Computer Vision
Sep 14

Adaptive AI: Energy Efficient Multi-exit TinyML on Intelligent Vision Systems at the Edge

The paper presents a novel multi‑exit computational scheme for TinyML on an ultra‑low‑power GAP9 SoC, adding confidence‑based gating points to a MobileNetV2 CNN for ImageNet‑100. By allowing inference to stop early, the approach cuts average MAC operations by 41 % (from 313 MMAC to 185 MMAC), reduces inference time by 29 % (49 ms to 35 ms), and saves 24 % in energy (2.1 mJ to 1.6 mJ per frame) with only a ~1 % drop in accuracy. Compared to a state‑of‑the‑art adaptive CNN on the same hardware, the method more than doubles computational efficiency, raising MAC/cycle from 8.1 to 17.2.

By Luca Crupi, Lorenzo Lamberti, Alessandro Giusti, Daniele Palossi
arXiv Machine Learning
Sep 14

CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

CanvasAnneal is a curriculum‑guided reinforcement learning framework designed to improve Diffusion Language Models (DLMs) on complex reasoning and tool‑use tasks. It starts training by injecting reasoning traces from a stronger teacher model into the diffusion canvas, then gradually reduces this guidance so the model learns to generate reasoning independently. Experiments on mathematical reasoning and tool‑use benchmarks show that CanvasAnneal outperforms standard diffusion RL methods such as diffu‑GRPO on tasks like MATH500, Countdown, and Tau2, and accelerates reward improvement, though the gains vary by task.

By Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan
arXiv Machine Learning
Sep 14

MAxBench: A Multinomial Concept Recovery Benchmark

MAxBench is a geometry‑agnostic benchmark for evaluating how well language models recover multinomial concept representations. The study compares ten localization methods across five geometry types, six concepts, and four models, finding that affine subspaces generally steer more reliably and recall more instances than rank‑one or linear subspaces. The results also show that manifold steering can match the best methods when applicable, and that no method consistently outperforms prompting for these complex concepts.

By Divya Appapogu, Freya Behrens, Yonatan Belinkov, Aaron Mueller
arXiv Machine Learning
Sep 14

AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems

The paper introduces AIM, a privacy‑aware memory framework that lets multi‑agent, multi‑user large language models manage both private and shared memory. AIM classifies data as private (user‑specific) or public (shared) and enforces index‑level access controls to protect sensitive information while enabling shared knowledge to improve coordination. The authors also present MUMBench, a new dataset for evaluating memory operations in multi‑user settings, and report high accuracy metrics for AIM on this benchmark.

By Zachary Johnson, Nigel Boachie Kumankumah, Somya Chatterjee, Tejas Sathyamurthi, Min Chen, Xinyi Alice Li, Xiao Wang, Emily Morgan Gelchie, Jessica Lin, Sadid A. Hasan, Sulaiman Vesal
arXiv Machine Learning
Sep 14

Optimizing for the decision not the prediction: an exploration of Smooth Net Benefit as a training objective

The study introduces Smooth Net Benefit (σNB), a differentiable approximation of Net Benefit, as a training objective aimed at aligning predictive models with threshold‑specific clinical decisions. Experiments on the Framingham cardiovascular risk dataset and 44 TabZilla datasets show that σNB training yields modest improvements for logistic regression but little to no benefit for more flexible models such as GAMs and XGBoost. The authors conclude that σNB is not a universal replacement for negative log‑likelihood training, though it may be worth exploring in contexts where model flexibility is limited.

By Koen M. F. Gorgels, Lasai Barre\~nada, Maarten van Smeden, Ben Van Calster, Ewout W. Steyerberg, Wouter A. C. van Amsterdam