Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 7

Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking

The paper introduces the Offline AI Modules workstream, which provides a voice‑first offline architecture, a low‑cost hardware reference bill of materials, and a reproducible quantization and benchmarking pipeline for instruction‑tuned language models in the 2‑5B parameter range. It evaluates three models across four quantization formats on two hardware tiers—NVIDIA Jetson Orin NX and Raspberry Pi5—measuring deployment metrics and multilingual quality. The key result is that Q4_K_M quantization offers the best size‑to‑quality trade‑off, enabling high decode throughput and strong topic classification accuracy while staying within memory limits on both tiers.

By Sunday Afariogun, Odunolaoluwa Jenrola, Zeinab Nezami
arXiv AI
Oct 7

Inference-Time Projection for Physically Valid Biomolecular Diffusion Models

The paper introduces two closed‑form projection operators that enforce physical validity in biomolecular diffusion models during inference. One operator resolves inter‑chain van der Waals clashes, while the other restores ligand geometry, including bond lengths, angles, planarity, and chirality. These lightweight, model‑agnostic corrections preserve structural accuracy and ligand placement across multiple benchmarks without adding runtime or memory overhead.

By Qurat-ul-ain, Yee Whye Teh, Charlotte M. Deane, Matteo Cagiada
arXiv AI
Oct 7

Internalizing Agent Experience into Diffusion Model Weights via On-Policy Context Distillation

The paper introduces Diffusion On-Policy Context Distillation (D-OPCD), a method that internalizes the benefits of an agentic harness—such as memory, skills, and iterative prompt refinement—into the weights of a diffusion model. By treating the agent‑improved prompt as privileged context, D-OPCD distills harness knowledge into the model, raising direct‑generation scores from 60.52 to 65.09 across four benchmarks. This internalization allows the harness to shed saturated skills and continue evolving, enabling a second round of skill evolution that further improves performance by 1.83 points.

By Wenxuan Wang, Zekai Liu, Weinan Zhang, Yu Cheng, Yang Yang
arXiv AI
Oct 7

Verifying Coordination in Parallel Coding Agents: NP-Bench and a Scheduling Planner

The paper introduces NP‑Bench, a benchmark and a proactive scheduling planner that coordinates parallel large‑language‑model coding agents. By partitioning work scopes and ordering merges ahead of time, the planner improves clean‑integration rates from 1/9 to 9/9 and eliminates merge conflicts, outperforming both no‑coordination and reactive‑detection baselines. It also demonstrates that cross‑session memory can eliminate repeated mistakes and that routing facts to agents does not improve long‑context accuracy at scale.

By Sumanyu Muku
arXiv AI
Oct 7

The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory

The paper presents a retrieval‑admissibility verification framework for long‑term memory agents, classifying each memory‑query pair as admissible, inadmissible, or unresolved. It evaluates the framework on public benchmarks (RHELM and MemOps), showing improved anchor recall and reduced exact similarity errors, while also revealing that existing verifiers miss certain inadmissible exposures. The study highlights the need for separate checks on candidate support, admissibility, prompt exposure, and answer disclosure to ensure safe memory retrieval.

By Zi Wang, Xingqiao Wang, Emmanuel Addai, Devika Ambekar, Xiaowei Xu
arXiv AI
Oct 7

MemCo: Memory-Centric Collaboration for Generalizing LLM Agents to Unseen Environments

MemCo introduces a memory‑centric collaboration framework that equips large language model agents with both local and global memory spaces. Local memory preserves environment‑specific details, while global memory captures transferable workflows from local trajectories. During online interaction, MemCo selectively routes relevant memories based on the agent’s state and decision phase, enabling reuse of other agents’ experiences without over‑generalizing environment‑specific information.

By Xinting Liao, Siyan Liu, Rabab K. Ward, Holger R. Roth, Xiaoxiao Li
arXiv AI
Oct 7

COMPASS: Finding Where Reasoning Lives in Language Models

The paper introduces COMPASS, an inference-time steering technique that enhances large language model performance on mathematical reasoning tasks. By using the correctness of a model’s direct answer attempts as a latent signal, COMPASS identifies a small subset of attention heads whose activations can be steered to elicit reasoning. Experiments across three model families and multiple math benchmarks show that COMPASS improves GSM8K accuracy by an average of 16 percentage points and achieves near Chain‑of‑Thought performance while generating 20–70% fewer tokens.

By Pratyay Dutta, Kowshik Thopalli, Vivek Narayanaswamy
arXiv AI
Oct 7

Jailbreaking Open-Weight LLMs via Random Embedding Perturbations

arXiv:2610.07125v1 Announce Type: cross Abstract: While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important co...

By Abhinav Sudhakar Dubey (University of California Santa Cruz), Scott Sirri (University of California Santa Cruz), Vaggos Chatziafratis (University of California Santa Cruz), C. Seshadhri (University of California Santa Cruz)