Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,014 stories · RSS feed

arXiv Machine Learning
Aug 11

OD-Gear: Online Decomposition and Group Sampling for Expert-Guided Adversarial Routing in Scalable Capacitated Vehicle Routing

arXiv:2602. 00488v3 Announce Type: replace Abstract: Solving large-scale capacitated vehicle routing problems (CVRP) is hindered by the high complexity of classical heuristics and the limited generalization of neural solvers.

By Dongbin Jiao, Zisheng Chen, Xianyi Wang, Jintao Shi, Shengcai Liu, Shi Yan
arXiv Machine Learning
Aug 11

Test-time Generalization for Physics through Neural Operator Splitting

arXiv:2602. 00884v2 Announce Type: replace Abstract: Neural operators have shown promise in learning solution maps of partial differential equations (PDEs), but they often struggle to generalize when test inputs lie outside the training distribution, such as novel initial conditions, unseen PDE coefficients or unseen physics.

By Louis Serrano, Jiequn Han, Edouard Oyallon, Shirley Ho, Rudy Morel
arXiv Machine Learning
Aug 11

Are Latent Reasoning Models Easily Interpretable?

arXiv:2604. 04902v2 Announce Type: replace Abstract: Latent reasoning models (LRMs) have attracted significant research interest due to their low inference cost (relative to explicit reasoning models) and theoretical ability to explore multiple reasoning paths in parallel.

By Connor Dilgren, Sarah Wiegreffe
arXiv Machine Learning
Aug 11

Hybrid Quantum-Classical PINNs for Scientific Computing: A Multi-GPU Open-Source Framework

arXiv:2604. 15645v2 Announce Type: replace Abstract: We present QPINNACLE, an open-source computational framework for physics-informed neural networks (PINNs) that integrates modern training strategies, multi-GPU acceleration, and hybrid quantum-classical architectures within a unified modular workflow.

By Ziv Chen, Hemanth Chandravamsi, Shimon Pisnoy, Aaron Goldgewert, Gal Shaviner, Boris Shragner, Steven H. Frankel
arXiv Machine Learning
Aug 11

Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamics

arXiv:2604. 26836v3 Announce Type: replace Abstract: Predictive safety filters (PSFs) leverage model predictive control to enforce constraint satisfaction during deep reinforcement learning (RL) exploration, yet their reliance on first-principles models or Gaussian processes limits scalability and broader applicability.

By Bernd Frauenknecht, Lukas Kesper, Daniel Mayfrank, Henrik Hose, Sebastian Trimpe
arXiv Machine Learning
Aug 11

Statistically-Lossless Quantization of Large Language Models

arXiv:2605. 02404v2 Announce Type: replace Abstract: Model quantization has become essential for efficient large language model deployment, yet existing approaches present clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but lack inference acceleration.

By Michael Helcig, Eldar Kurtic, Dan Alistarh
arXiv AI
Aug 11

Entropy-based Code Adversarial Translation for Real-world Repository Migration

arXiv:2608. 09273v1 Announce Type: new Abstract: LLMs have demonstrated strong capabilities in code generation and automated program repair, but migrating an entire repository rarely produces a runnable application because long-horizon translation challenges LLM-based agents' ability to maintain repository-level migration objectives.

By Yushun Tang, Yisen Cao, Zhicheng Chen, Lin Peng, Junkang Mao, Fengyi Song, Yantao Jia
arXiv AI
Aug 11

TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability

arXiv:2608. 09538v1 Announce Type: cross Abstract: We introduce TCS-Bench, a benchmark for evaluating Large Language Models (LLMs) on research-level Theoretical Computer Science (TCS) proof generation.

By Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland, Max Springer, Julien Canitrot-Paradis, Honghao Lin, David Woodruff, Adarsh Kumarappan, Rajesh Jayaram, Rudrajit Das, Lalit Jain, Ola Svensson, Silvio Lattanzi, Mislav Balunovic, Theophane Weber, Vahab Mirrokni
arXiv AI
Aug 11

Towards Expert-level Medical AI for Real-time Video Consultations

arXiv:2608. 09861v1 Announce Type: new Abstract: Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues.

By Mahvish Nagda, Jihyeon Lee, Matthew Thompson, Chunjong Park, Tim Strother, Valentin Li\'evin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fukuzawa, Sam Schmidgall, Craig Schiff, Joseph Xu, Aliya Rysbek, Yana Lunts, Jan Freyberg, Rebecca Hemengway, Sunny Virmani, David Racz, Carey Radebaugh, Jo\"elle Barral, Kavi Goel, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Gregory Wayne, Tao Tu, Yun Liu, Ethan Goh, Christina Chen, Ryutaro Tanno, Po-Hsuan Cameron Chen, Mike Schaekermann, Anil Palepu