The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper introduces OmniEvaluator, a composable evaluation system designed to streamline reproducible testing of omni‑modal foundation models across text, image, video, and audio. It unifies disparate inference engines, prompt conventions, and metric implementations by providing a single interface that supports four inference backends, four evaluation frameworks, and over a thousand benchmarks. Each evaluation run is logged as an artifact for exact reproducibility, with results displayed on a shared dashboard; a federated mode allows GPU inference servers to be shared, and a lightweight verifier ensures stable scoring across engines and prompts without incurring API costs.
arXiv:2603.13966v3 Announce Type: replace Abstract: Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pi...
HoliBench is a modular benchmarking and deployment toolkit that jointly measures accuracy, latency, and energy for foundation models across a wide range of devices, from single-board computers to GPU servers. It provides a platform abstraction layer that calibrates cross-device measurements and supports multiple model modalities, inference engines, and quantization levels. Using HoliBench, the authors evaluated 20 models on 7 device types, revealing tradeoffs such as limited latency gains from quantization on low‑precision hardware and diminishing accuracy returns relative to energy consumption, while also showing that single-model profiles can predict multi-model pipeline performance within a few percent.
arXiv:2512. 22219v2 Announce Type: replace-cross Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel.
arXiv:2606. 02963v1 Announce Type: new Abstract: Production inference increasingly targets a heterogeneous mix of accelerators.
KernelFoundry is a hardware‑aware evolutionary framework that optimizes GPU kernels by combining MAP‑Elites quality‑diversity search, meta‑prompt evolution, and template‑based parameter tuning. It generates SYCL and CUDA kernels, outperforming baseline methods with an average 2.3× speedup on KernelBench. The system is distributed, supports remote hardware access, and offers a flexible user interface for diverse real‑world kernel generation tasks.