Building an omni-modal foundation model means evaluating it across text, image, video, and audio. Excellent evaluation toolkits exist for each modality, but their inference engines, prompt conventions...
arXiv:2603.13966v3 Announce Type: replace
Abstract: Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pi...
By Suhwan Choi, Yunsung Lee, Yubeen Park, Chris Dongjoo Kim, Ranjay Krishna, Dieter Fox, Youngjae Yu
Dooly is a profiling tool for large language model (LLM) inference that eliminates the need to re‑profile every operation for each configuration. By labeling input dimensions with taint propagation and only profiling operations missing from its latency database, Dooly builds a configuration‑agnostic, redundancy‑aware latency database that can be used as a drop‑in backend for simulators. In experiments across two GPU platforms, three attention backends, and multiple model architectures, Dooly achieves simulation accuracy within 5% MAPE for time‑to‑first‑token and 8% for total‑processing‑time while cutting profiling GPU‑hours by 56.4% compared to traditional methods.
By Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda, Daehyeok Kim
HoliBench is a modular benchmarking and deployment toolkit that jointly measures accuracy, latency, and energy for foundation models across a wide range of devices, from single-board computers to GPU servers. It provides a platform abstraction layer that calibrates cross-device measurements and supports multiple model modalities, inference engines, and quantization levels. Using HoliBench, the authors evaluated 20 models on 7 device types, revealing tradeoffs such as limited latency gains from quantization on low‑precision hardware and diminishing accuracy returns relative to energy consumption, while also showing that single-model profiles can predict multi-model pipeline performance within a few percent.
By Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
arXiv:2607. 24762v1 Announce Type: new Abstract: Machine learning models are increasingly embedded in everyday software, and most of their runtime is spent in a small set of compute kernels such as matrix multiplication, convolution, and normalization.
By Joshua Brodsky, Dhravid Kumar, Savini Kashmira, Jayanaka Danatanarayana, Jason Mars, Krisztian Flautner, Lingjia Tang
arXiv:2606. 02963v1 Announce Type: new Abstract: Production inference increasingly targets a heterogeneous mix of accelerators.
By Taras Sereda, Burak Bartan, Ankita Nayak, Tom St. John, Natalie Serrino, Zain Asgar