Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,048 stories · RSS feed

arXiv Machine Learning

Aug 6

RiboSphere: Learning Unified and Efficient Representations of RNA Structures

arXiv:2603. 19636v2 Announce Type: replace Abstract: Accurate RNA structure modeling remains difficult because RNA backbones are highly flexible, non-canonical interactions are prevalent, and experimentally determined 3D structures are comparatively scarce.

By Zhou Zhang, Hanqun Cao, Cheng Tan, Fang Wu, Pheng Ann Heng, Tianfan Fu

llms diffusion efficiency benchmarks

arXiv Machine Learning

Aug 6

GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs

arXiv:2606. 23299v2 Announce Type: replace Abstract: Configuring the hyperparameters of Mixed-integer programming (MIP) solvers is a high-dimensional, instance-dependent optimization problem where suboptimal settings can degrade solving time by orders of magnitude.

By Yidong Luo, Xuemin Chen, Chenguang Wang, Fangzhou Zhu, Tao Zhong, Tianshu Yu

llms benchmarks

arXiv Machine Learning

Aug 6

BIM-Native Tokenization for Constraint-Aware Room Layout Synthesis

arXiv:2512. 04832v3 Announce Type: replace-cross Abstract: We present a BIM-native tokenization for room-level layout synthesis in Building Information Modeling (BIM) scenes.

By Manuel Ladron de Guevara, Jinmo Rhee, Ardavan Bidgoli, Vaidas Razgaitis, Michael Bergin

llms rag nlp benchmarks

arXiv AI

Aug 6

NeuMoSync: End-to-End Neuromodulatory Control for Plasticity and Adaptability in Continual Learning

arXiv:2608. 04358v1 Announce Type: new Abstract: Continual learning (CL) requires models to learn tasks sequentially, yet deep neural networks often suffer from plasticity loss and poor knowledge transfer, which can impede their long-term adaptability.

By Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mandana Samiei, Mo Chen

benchmarks

arXiv AI

Aug 6

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

arXiv:2608. 04726v1 Announce Type: new Abstract: Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels.

By Yongxin Wang, Ruizhe Zhou, Yueling Tang, Yingying Zhu, Xuemin Zhao, Xiaojun Chang, Xiaodan Liang

llms computer-vision multimodal benchmarks

arXiv AI

Aug 6

From Score Matrices to Football-Aware Match-State Simulation: An Auditable LLM Harness for Exact-Score Reranking

arXiv:2608. 05030v1 Announce Type: new Abstract: Football score forecasting combines a strong statistical core with a difficult contextual edge.

By Shaopeng Liang

llms rag benchmarks

arXiv AI

Aug 6

When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit

arXiv:2608. 04896v1 Announce Type: new Abstract: Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not.

By Ziang Wei, Minjun Yu, Zheyuan Lai, Mingjie Pang, Wei Li

agents benchmarks

arXiv AI

Aug 6

EDATracer: An Agentic Framework for Large-Scale EDA Artifact Analysis

arXiv:2608. 04032v1 Announce Type: cross Abstract: Modern chip design relies on electronic design automation (EDA) tools that generate large, heterogeneous artifacts, including source files, scripts, logs, netlists, and reports.

By Phat Tieu, Sayanti Jana, Matthew DeLorenzo, Jiawen Wu, Narendran Srinivasan, Srinivas Shakkottai, Jiang Hu, Jeyavijayan Rajendran

llms agents benchmarks

arXiv AI

Aug 6

MIDAS: Multi-LLM Iterative Data-Adaptive Summarization

arXiv:2608. 04307v1 Announce Type: cross Abstract: Text summarization is deceptively difficult.

By Karen Lee, Dhanashree Balaram, Seojun Shon, Umair Rasheed

llms nlp benchmarks

arXiv AI

Aug 6

ATLAS: Adaptive Topological Learning with Abstract Successors for Continual Learning

arXiv:2608. 04334v1 Announce Type: cross Abstract: Contemporary model-free reinforcement learning algorithms can achieve very high performance, but have low sample efficiency and are not robust to changes in the environment.

By R. Blake Lawlor, Daniel S. Brown

reinforcement-learning benchmarks

arXiv AI

Aug 6

Towards Trustworthy Hypergraph Neural Networks under Label Noise

arXiv:2608. 04377v1 Announce Type: cross Abstract: Hypergraph neural networks (HGNNs) have demonstrated remarkable capabilities in processing complex higher-order relationships.

By Mengyao Zhou, Zhiheng Zhou, Xiao Han, Guiying Yan

benchmarks

arXiv Machine Learning

Aug 6

EvtGraph: Event-Adaptive Compression for Sparse Temporal Graph Learning in Multimodal Time Series

arXiv:2608. 04368v1 Announce Type: new Abstract: Multimodal temporal data are inherently irregular and uneven in information density, yet most models rely on uniform discretization, leading to inefficient representations.

By Ziqian Wang, Tingxiong Xiao, Yuxiao Cheng, Jinli Suo

llms multimodal benchmarks

arXiv Machine Learning

Aug 6

Benchmarking Deep Learning Models for Dense Event Classification of Offshore Wind Infrastructure in Sentinel-1 Time Series

arXiv:2608. 04706v1 Announce Type: new Abstract: Monitoring of offshore wind energy infrastructure life cycles, especially during the deployment phase, is an important contribution for stakeholders to make informed decisions in a phase of increasing deployment activities.

By Thorsten Hoeser, Felix Bachofer, Claudia Kuenzer

llms benchmarks

arXiv Machine Learning

Aug 6

Continual-Learning Physics-Informed Neural Networks for Parameterized Partial Differential Equations

arXiv:2608. 04778v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) incorporate governing equations into neural-network training and can approximate PDE solutions without requiring large observational datasets.

By Xujia Chen, Xinyue Hu, Letian Chen, Yi Liu, Wenhui Fan

benchmarks

arXiv Machine Learning

Aug 6

Above-ground Biomass Estimation with Geospatial Foundation Models

arXiv:2608. 04792v1 Announce Type: new Abstract: Accurate estimation of Above-Ground Biomass (AGB) from satellite imagery is essential for the large-scale monitoring of carbon stocks, yet it remains a challenging regression task at global scale.

By Ghjulia Sialellia, Linus Scheibenreif, Jan Dirk Wegner, Konrad Schindler

rag computer-vision benchmarks

arXiv Machine Learning

Aug 6

BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

arXiv:2608. 05104v1 Announce Type: new Abstract: Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges.

By Sajib Hossain, Md Kamrus Samad, Anan Ghosh, Labib Imam Chowdhury, Nabeel Mohammed

llms nlp efficiency benchmarks

arXiv Machine Learning

Aug 6

The Cost of Binarizing Survival Outcomes in Clinical Prognostic Modeling

arXiv:2608. 04046v1 Announce Type: cross Abstract: Survival analysis is an established framework for analyzing time-to-event data, yet many clinical machine learning studies still binarize the outcome before model training.

By Shashank Yadav, David M. Routman, Andrew Y. K. Foong

benchmarks

arXiv Machine Learning

Aug 6

From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models

arXiv:2608. 04200v1 Announce Type: cross Abstract: Financial sentiment classifiers are commonly evaluated against human labels, but strong linguistic performance does not necessarily imply economically useful return predictability.

By Fusheng Luo

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 6

PriDyG: Privacy-preserving Dynamic Graph Inference with LLM-GNN Collaboration

arXiv:2608. 04255v1 Announce Type: cross Abstract: Graph inference over relational data can expose sensitive edge information, and this risk becomes more severe in dynamic graphs, where repeated model updates cause privacy loss to accumulate.

By Yuyang Xia, Ruixuan Liu, Li Xiong

llms benchmarks safety

arXiv AI

Aug 6

D$^2$F-ReAG: Dynamic Decomposition and Filtering for Multi-Hop Reasoning-Augmented Generation

arXiv:2608. 04444v1 Announce Type: cross Abstract: Large language models (LLMs) often generate inaccurate answers due to their reliance on static internal knowledge.

By Jiaoyang Li, Junhao Ruan, Shengwei Tang, Kaiyan Chang, Zhengtao Yu, Tong Xiao, Jingbo Zhu

llms rag benchmarks

Get the daily briefing