Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 15

Fraglingo: Molecular Design via Attachment-Aware Autoregressive Fragment Generation

Fraglingo is an autoregressive fragment-based molecular generator that jointly models fragment identity and attachment in a continuous latent space. It predicts attachment-aware fragment embeddings using a wildcard-anchored readout that captures the growing molecule’s active attachment site, then retrieves the next fragment via latent-space nearest-neighbor search. This approach allows new fragments to be added at inference time without retraining and achieves stronger joint property control on benchmarks while maintaining high validity, uniqueness, and novelty.

By Thao Nguyen, Jeonghwan Kim, Zhenhailong Wang, Heng Ji
arXiv AI
Sep 15

A primer on evaluation methods for large language models in healthcare

arXiv:2609.14819v1 Announce Type: cross Abstract: Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and...

By Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan, Timothy J Keyes, Bernardo C Bizzo, Keith J Dreyer, Sarah F Mercaldo, James M Hillis
arXiv AI
Sep 15

CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback

CHAI for LLMs is a framework that improves large language models’ performance on code‑mixed translation tasks by using LLMs as annotators to create preference data, applying reinforcement learning from AI feedback, incorporating LLM‑generated domain knowledge for iterative refinement, and evaluating on real‑world datasets. The approach yields a 68.45% average win rate over state‑of‑the‑art open‑source models in human‑adjudicated tests. It demonstrates a scalable method to enhance code‑mixed language understanding in open‑source LLMs.

By Wenbo Zhang, Aditya Majumdar, Asif Ekbal, Amulya Yadav
arXiv Computer Vision
Sep 15

Robust Multi-Model Fitting through Learning Neighbor Regions

The paper introduces Learning Neighbor Regions (LNR), a coarse‑to‑fine framework for robust multi‑model fitting in noisy environments. LNR’s coarse module uses a neural network to analyze geometric features of minimum sets, selecting a small number of promising hypotheses to improve efficiency. The fine module encodes neighbor region features for each hypothesis, allowing simultaneous refinement and scoring while avoiding differentiation of the sampling process and solvers. Experiments on four classic tasks show state‑of‑the‑art performance and suggest easy adaptation to other robust fitting problems.

By Chang Nie, Guangming Wang, Zhe Liu, Hesheng Wang
arXiv Computer Vision
Sep 15

ViCo-SAM3: Vision-Conditioned Alignment for Open-Vocabulary Camouflaged Object Segmentation

ViCo-SAM3 introduces a Vision-Conditioned alignment framework for open-vocabulary camouflaged object segmentation. The approach adds a vision-conditioned (ViCo) module that dynamically adjusts text embeddings based on global visual context, and a vision-conditioned cross-modal binding (ViCoBind) module to improve interaction between visual and textual representations. These innovations close the semantic gap between text and pixel-level cues, enabling state‑of‑the‑art performance on the OVCamo benchmark without heavy parameter overhead.

By Qiangqiang Zhou, Wenjun Tang, Yong Chen, Dandan Zhu, Jiawei Xu
arXiv Computer Vision
Sep 15

SAR-FAH: A Frequency-Adaptive Hybrid Network based on Neural ODEs for Structural-Preserving SAR Despeckling

SAR-FAH is a Frequency‑Adaptive Hybrid network that uses Neural Ordinary Differential Equations (NODEs) to despeckle Synthetic Aperture Radar (SAR) images. It separates homogeneous and heterogeneous regions in the frequency domain via wavelet transform, then applies a NODE‑based module to low‑frequency sub‑bands for smooth denoising and an enhanced U‑Net with deformable convolutions to high‑frequency sub‑bands for edge and texture preservation. Experiments on synthetic and real SAR data show that SAR‑FAH outperforms current state‑of‑the‑art despeckling methods both quantitatively and qualitatively.

By Ziqing Ma, Chang Yang, Zhichang Guo, Yao Li
arXiv AI
Sep 15

Freeze, Share, Shrink: Rethinking the Action Backbone in Diffusion Policies

The paper argues that diffusion-based action policies can use a frozen, observation‑free backbone as a reusable trajectory prior, with task adaptation handled entirely by the conditioning pathway. By pretraining a general action head on forward‑kinematics data and then freezing it, the authors show that a single backbone can match or outperform normally trained models on MimicGen and LIBERO. Their experiments reveal that a small 5 M‑parameter MLP backbone can rival large U‑Net and transformer backbones, indicating that action backbones are often over‑parameterized and that image‑style architectures may not be the best fit for low‑dimensional action generation.

By Jian Zhou, Sihao Lin, Shuai Fu, Zerui Li, Gengze Zhou, Qi WU
arXiv AI
Sep 15

DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generation

DuoTok is a source‑aware dual‑track music tokenizer designed for vocal‑accompaniment generation. It first learns a semantic audio representation via self‑supervised pretraining, then refines source‑aware structure with feature‑replacement noise and multi‑task supervision (spectral reconstruction, source separation regularization, and an ASR head for lyric alignment). The encoder is frozen and hard‑routed codebooks for vocals and accompaniment are learned, while a diffusion decoder restores fine acoustic detail from the discrete tokens, achieving a favorable predictability‑fidelity trade‑off at ultra‑low bitrate across public benchmarks.

By Rui Lin, Zhiyue Wu, Jiahe Lei, Kangdi Wang, Weixiong Chen, Junyu Dai, Tao Jiang