Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning
Sep 10

Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

Squeeze10-LLM is a staged mixed‑precision post‑training quantization framework that reduces 16‑bit LLM weights to an average of 1.6 bits per weight by assigning 80% of weights to 1 bit and 20% to 4 bits. It introduces Post‑Binarization Activation Robustness (PBAR), a weight significance metric that considers activation impact, and Full Information Activation Supervision (FIAS), a strategy that preserves activation information to limit error propagation. Experiments on LLaMA and LLaMA2 demonstrate that Squeeze10‑LLM achieves state‑of‑the‑art performance for sub‑2‑bit weight‑only quantization, raising average accuracy from 43% to 56% on six zero‑shot classification tasks.

By Qingcheng Zhu, Yangyang Ren, Linlin Yang, Yanjing Li, Sheng Xu, Haodong Zhu, Juan Zhang, Runqi Wang, Baochang Zhang
arXiv AI
Sep 10

Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?

The paper introduces Open Tabular Insight Extraction (OpenTI), a unified framework aimed at democratizing access to insights from large table corpora. It highlights how current research is fragmented across domains like table QA, text‑to‑SQL, and data analysis agents, and shows that existing systems and benchmarks fall short of covering the full end‑to‑end scope of OpenTI. The authors propose a consolidated terminology, conduct a systematic review, and outline a research agenda for developing comprehensive OpenTI systems, evaluation methods, and interaction paradigms.

By Daniel Gomm, Maarten de Rijke, Madelon Hulsebos
arXiv AI
Sep 10

Latent Preference Modeling for Multi-Session Personalized Tool Calling

The paper introduces the Multi-Session Personalized Tool Calling (MPT) benchmark, containing 4,695 instances across 459 multi‑session histories that test Preference Recall, Induction, and Transfer. It proposes PRefine, a test‑time memory method that refines a user’s latent preference via a generate‑verify‑refine loop. Experiments with five LLMs show that PRefine outperforms existing memory systems and even full‑history prompting on Preference Transfer, suggesting that personalized agents should encode behavior as preferences rather than merely storing past interactions.

By Yejin Yoon, Minseo Kim, Taeuk Kim
arXiv AI
Sep 10

Optimal Experiments for Partial Causal Effect Identification

The paper tackles selecting a cost‑constrained set of experiments that most effectively tighten bounds on a partially identifiable causal query. It formalizes this as the NP‑hard max‑potency problem, introduces efficient graphical pruning rules to reduce the search space, and demonstrates the approach on synthetic graphs and real NHANES data to estimate the effect of physical activity on diabetes.

By Tobias Maringgele, Jalal Etesami
arXiv AI
Sep 10

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

The paper introduces Suan, a new preference optimization algorithm designed to improve safety alignment in large language models. Suan operates directly at the gradient level, avoiding traditional variational derivations, which yields more interpretable and robust training dynamics. Experiments show that Suan outperforms existing methods, achieving superior safety alignment while maintaining response utility.

By Oleksandr Cherednichenko, Roman Klypa
arXiv Machine Learning
Sep 10

Conditional Validity for Adaptive Modality Acquisition: When the Policy Chooses Its Own Calibration Group

The paper introduces RouteCert, a method for ensuring risk control in multimodal systems that acquire inputs adaptively. It shows that conditional calibration can remain valid even when the acquisition policy determines the calibration group, and provides two finite‑sample constructions: threshold‑free routing with terminal‑pattern calibration and simultaneous validation of policy‑pattern pairs. Experiments on a clinical ECG task and masked multimodal benchmarks demonstrate that RouteCert achieves low disagreement rates and competitive answered fractions while validating each acquisition stage separately.

By Melika Baghi
arXiv AI
Sep 10

HyCO: A Hybrid Neural Solver for Combinatorial Optimization

HyCO is a hybrid neural solver that combines a sequential reinforcement learning (RL) solver with a global diffusion model (DM) to tackle combinatorial optimization problems. The RL component builds an initial solution prefix, after which HyCO switches to a conditional DM to finish the remaining decisions. The authors provide a theoretical framework showing that this hybrid approach yields lower expected regret than either method alone, identify an optimal trigger step for the switch, and implement a lightweight adaptive trigger based on policy entropy and RL‑DM disagreement, achieving consistent performance gains across benchmarks.

By Yuheng Li, Di Yang, Haipeng Chen, Yanhai Xiong