Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computer Vision

Oct 7

DistScene: Object-to-Scene Distillation for 3D Scene Generation

DistScene is a framework for generating 3D scenes from a single image by jointly modeling the environment and individual objects. It introduces Scene-Frame Generation to produce separate environment and object components in a shared coordinate frame, and Object-Centric Refinement to fine‑tune each object with scene context. The method also employs Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation using automatically composed synthetic scenes, achieving improved spatial coherence on indoor and outdoor benchmarks.

By Kunming Luo, Hongyu Yan, Ken Deng, Chengcheng Zhou, Tianyu Liu, Haipeng Li, Haibin Huang, Xuelong Li, Ping Tan

efficiency benchmarks

arXiv Computer Vision

Oct 7

Towards benchmarking Western Bluebird detection in the wild

The paper introduces a new benchmark dataset of over 6,000 high‑resolution images for detecting and segmenting Western bluebirds in natural, cluttered scenes. Experiments show that supervised detectors such as Faster R‑CNN and RT‑DETR outperform open‑vocabulary models in zero‑shot settings, though fine‑tuned YOLO‑World can compete. Segmentation results favor Mask R‑CNN for mask quality, while YOLOv8‑Seg offers the best precision and speed. The study also identifies multiple factors—scale, brightness, contrast, clutter, blur, crowding, and session variation—as contributors to detection failures.

By Estela Monserrat Arriaga Santana, Julian Rosas Scull, Ibeth P. Alarc\'on, Bibiana Montoya, Aylin Sosa Mej\'ia, Hugo Jair Escalante

computer-vision fine-tuning benchmarks

arXiv Computer Vision

Oct 7

UltraDiff: Differentiable Ray Tracing in Ultrasound for Shape Optimization

UltraDiff is a modular framework that extends differentiable rendering to medical ultrasound by modeling image formation as a path‑space integral gated by travel time. It derives a Monte Carlo estimator for both the forward ultrasound model and its gradients, enabling gradient‑based optimization of scene parameters. The authors demonstrate the method on inverse geometry estimation, recovering vertebral surfaces from simulated and real B‑mode sweeps without requiring pre‑segmented images, achieving competitive geometric accuracy.

By Felix Duelmer, Magdalena Wysocki, Nassir Navab, Mohammad Farid Azampour

robotics benchmarks

arXiv Computer Vision

Oct 7

FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs

The paper introduces FindIt, the first comprehensive benchmark for evaluating the promptable localization abilities of generalist multimodal large language models (MLLMs). It covers four core task categories—object detection, referring expression detection, instance-level detection, and video-based detection—and provides a unified framework that standardizes inputs, enforces parsable bounding box outputs, and defines transparent evaluation protocols. Using this benchmark, the authors assess a range of open-source and proprietary MLLMs, revealing both their strengths and limitations, particularly their sensitivity to formatting constraints and difficulty generalizing to minor variations.

By Eshika Khandelwal, Jingjing Pan, Mingfang Zhang, Quan Kong, Lorenzo Garattoni, Hilde Kuehne

llms agents computer-vision nlp multimodal benchmarks

arXiv Computer Vision

Oct 7

Multitask Conditional Generative Adversarial Network Enables Automatic Whole Knee Cartilage and Menisci Segmentation and Reliable $T_{1\rho}$ and $T_2$ Quantification Without High-Resolution Morphological Images

The study introduces a multitask conditional generative adversarial network (MT‑cGAN) that simultaneously synthesizes high‑resolution DESS‑like images and segments knee cartilage and menisci directly from quantitative MRI echo images. Evaluated on 508 knee MRIs from 361 subjects, MT‑cGAN achieved a mean Dice score of 0.84 for segmentation and the lowest coefficient of variation for T1ρ (1.84%) and T2 (1.81%) quantification, outperforming existing conditional GAN approaches. By eliminating the need for separate high‑resolution morphological scans, the method shortens scan times and supports clinical adoption of quantitative MRI.

By Ahmed Tahseen Minhaz, Richard Lartey, Zhiyuan Zhang, Jeehun Kim, Kunio Nakamura, Mingrui Yang, Jiasen Zhang, Weihong Guo, Naveen Subhas, Carl S. Winalski, Xiaojuan Li

diffusion computer-vision fine-tuning benchmarks safety

arXiv AI

Oct 7

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

PERSIST is a persistent memory system designed for multi-session, multi-speaker spoken dialogue, explicitly modeling Who, What, and When to enable voice assistants to recall past conversations and adapt to individual users. It structures cross-session histories into readable event records and retrieves them using a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. By reusing intermediate representations from the dialogue backbone, PERSIST reduces retrieval latency from 578.42 ms to 7.03 ms and achieves 85.08% end‑to‑end task accuracy on the SpokenTrace benchmark, improving EM@3 from 49.01% to 82.10%.

By Achira Lin, Siyuan Hou, Wenyi Yu, Xinnian Zhao, Haoyu Niu, Wang Geng, Longshuai Xiao, Shihai Xiao, Mangsuo Zhao, Chao Zhang

benchmarks

arXiv Machine Learning

Oct 7

Towards Explainable Benchmarking for Data-driven Post-Wildfire Debris Flow Prediction

arXiv:2610.07358v1 Announce Type: new Abstract: Post-wildfire debris flows (PFDFs) are destructive sediment-laden hazards triggered when intense rainfall strikes recently burned terrain, destabilizin...

By Zhisheng Qi, Li Zhu, Utkarsh Sahu, Douglas Tommey, Josh Roering, Yu Wang

reinforcement-learning benchmarks safety

arXiv Machine Learning

Oct 7

Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models

arXiv:2610.07247v1 Announce Type: new Abstract: Large language models have shown strong reasoning capabilities, but their high inference costs make knowledge distillation an important approach for tr...

By Heng Liang, Xinwen Zhang, Hongchang Gao

llms efficiency benchmarks

arXiv AI

Oct 7

Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axes

arXiv:2610.06889v1 Announce Type: cross Abstract: We study the application of large language models (LLMs) to the visual exploration of textual corpora. We introduce zero-shot visualization (ZSV), a...

By Arnau Bueno Tricas, Jose A. Rodr\'iguez-Serrano

llms rag benchmarks safety

arXiv AI

Oct 7

CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

arXiv:2610.07132v1 Announce Type: cross Abstract: Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful rea...

By Berke Arda, Ahmetcan Yavuz, Paul Gerry, Sebastian Lobentanzer, Nobin Sarwar, Joan Giner-Miguelez, Kongtao Chen, Luyao Zhang, Mrinmaya Sachan, Mubashara Akhtar

llms agents benchmarks

arXiv AI

Oct 7

Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models

arXiv:2610.07848v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing...

By Dayan Pan, Jingyuan Wang, Xie Yu

llms rag fine-tuning benchmarks

arXiv AI

Oct 7

When Agent Context Goes Stale: Incoherence in Volatile Agent Context

arXiv:2610.05281v2 Announce Type: replace Abstract: Modern agents increasingly ground their reasoning in observations returned by tools, such as file contents read from a workspace. However, the data...

By Yingying Liu, Junzhou Fang, Chenxiong Qian

llms agents benchmarks

arXiv Machine Learning

Oct 7

Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks

arXiv:2610.06942v1 Announce Type: new Abstract: Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time serie...

By Nilushika Udayangania, Kishor Nandakishora, Marimuthu Palaniswami

computer-vision efficiency benchmarks