Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 22

MolSC: Leveraging Substituent Contributions to Enhance Fine-grained Molecular Understanding in LLMs

MolSC is a new dataset of 181,000 substituent-level examples that captures how attaching specific substituents to molecular scaffolds changes properties such as bioactivity and physicochemical descriptors. The authors also provide MolSC-Bench, a held‑out benchmark of 1,541 examples that are disjoint from MolSC at scaffold, substituent, and molecule levels. Experiments show that training molecular large language models on MolSC markedly improves their ability to predict substituent contributions, outperforming existing models on a range of downstream chemistry tasks.

By Hyuntae Park, Sooyeon Kim, Jiwon Park, SangKeun Lee
arXiv Machine Learning
Sep 22

ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs

ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.

By Junyoung Park, Jungwook Choi, Mingu Lee
arXiv Computation and Language
Sep 22

RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents

RPMem introduces a two‑stage architecture that compiles each session into a model‑independent latent memory and then consolidates it with retained memory via a task‑trained recurrent gate. The consolidated memory is mapped to backbone‑specific low‑rank adaptation (LoRA) parameters, enabling the memory to transfer when the backbone is replaced. Across three long‑term memory benchmarks and five diverse backbones, RPMem achieves broad generalization with near‑constant update cost and memory footprint, outperforming existing parametric and text‑based baselines on the PERMA benchmark.

By Fanyu Zhao, Ruike Cao, Liang Dong, Fugen Yao, Jian Xu, Guanjun Jiang, Han Zhang, Yifei Zhao, Yinsheng Li
arXiv Machine Learning
Sep 22

VISTA: An Attention-Based Multi-Agent Reinforcement Learning Architecture for Space Situational Awareness Sensor Tasking

arXiv:2609.23875v1 Announce Type: new Abstract: The rapid growth of resident space objects is increasing the complexity of space situational awareness sensor tasking, challenging classical optimizati...

By Miguel Leiva-V\'elez, Adalberto Claudio Quiros, Nicolas Gaston Rozado, Hodei Urrutxua, V\'ictor Rodr\'iguez-Fern\'andez
arXiv Machine Learning
Sep 22

MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes

arXiv:2609.23980v1 Announce Type: cross Abstract: AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application...

By Andy K. Zhang, Ava Huang, Joey Ji, Wai Han, Thomas Qin, Nardos Demilew, Michael Tian-Yue Liu, Brian Song, Riya Dulepet, Brian Wang, Kyleen Liao, Cuiyuanxiu Chen, Nishka Kacheria, Andrew Wu, Pratham Rangwala, Xinjie Wang, Laura Gomezjurado Gonzalez, Anita Ding, Benjamin Yi, Daniel E. Ho, Dan Boneh, Dawn Song, Ion Stoica, Percy Liang
arXiv Computer Vision
Sep 22

Action-Slot: Structured Action-Centric Representation Learning for Multi-Agent Atomic Activity Understanding

The paper introduces Action‑Slot, a structured action‑centric representation learning framework for multi‑agent atomic activity understanding. It reformulates slot attention into activity‑aligned slots, parallel spatio‑temporal updates, and background regularization to disentangle concurrent, asynchronous activities directly from raw video. Additionally, an attention‑difference pseudo‑mask method enables weakly supervised localization, and a new synthetic dataset, TACO, provides balanced atomic activity coverage with pixel‑level annotations.

By Yu-Ho Chang, Chi-Hsi Kung, Yi-Hsuan Tsai, Yi-Ting Chen
arXiv Machine Learning
Sep 22

Data Agents: Agentic Data Systems

arXiv:2609.24137v1 Announce Type: cross Abstract: Traditional data systems face profound limitations in the AI era, relying on human-crafted pipelines, lacking semantic understanding of heterogeneous...

By Guoliang Li, Peiyao Zhou, Xuanhe Zhou, Ji Sun, Yuyu Luo, Ju Fan