Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 26

HarnessMM: Recursive Self-Improvement for Harness Token Efficiency

HarnessMM (Harness Minus Minus) is a method that improves harness token efficiency by using guarded decision reuse and recursive self‑improvement (RSI). It refines harness configurations from training feedback, reuses resolved decisions to replace later reasoning callbacks, and stores actions in Executable Decision Graphs. Experiments show significant token savings while maintaining or improving success rates on benchmarks such as HumanEval and DeepSeek Harness.

By Yuhua Jiang, Chang Xu, Feifei Gao, Li Zhao
arXiv AI
Oct 26

Distributional Alignment of LLMs and Its Generalization Properties

The paper investigates how large language models (LLMs) can be distributionally aligned with human output distributions, particularly for applications like survey simulation. It examines both the alignment process and the models’ ability to generalize to unseen questions within the same domain and to related but different topics. The study finds that while LLMs can achieve alignment and generalize within a domain, cross‑topic generalization is variable and aligned models may unexpectedly lose creative capabilities, indicating unpredictable and unstable generalization properties.

By Nelson Jo