Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI
Sep 11

Generative AI for Analysts

The paper examines how generative AI (GenAI) changes the way financial analysts produce information, using the 2023 integration of GenAI into FACTSET as a natural experiment. It finds that FACTSET reports become richer—adding 26% more distinct sources, 24% broader topics, and 21% more analytical methods—while also being more timely. However, the increased information load leads to lower forecast accuracy for analysts, suggesting a human attention bottleneck, whereas a machine-learning benchmark does not suffer the same decline.

By Jian Xue, Qian Zhang, Wu Zhu
arXiv AI
Sep 11

Meta-RL with Bayesian Linear Task Models

The paper introduces GLiBRL, a deep Bayesian reinforcement learning framework that uses generalized linear task models with learnable nonlinear basis functions. GLiBRL performs exact, sequential Bayesian inference without variational approximations, providing closed‑form posterior updates and a marginal likelihood. The method is permutation‑invariant, compatible with both off‑ and on‑policy algorithms, and achieves the best zero‑shot performance among eight meta‑RL baselines on MuJoCo locomotion and MetaWorld manipulation tasks.

By Jingyang You, Hanna Kurniawati
arXiv Machine Learning
Sep 11

Adaptive Margin Ordinal Loss: Penalizing Center-Class Hedging in Ordinal Classification

The paper introduces the Adaptive Margin Ordinal Loss (AMOL), a new loss function designed to reduce the tendency of neural networks to predict center classes in ordinal classification tasks—a problem called center‑class hedging. AMOL applies a multiplicative weight to per‑class loss terms that is large only when a candidate class is near the center while the true label is far from it, thereby discouraging hedging. The authors also propose the Center‑Hedging Rate (CHR) metric to quantify this failure mode and demonstrate that AMOL achieves state‑of‑the‑art Quadratic Weighted Kappa scores on four benchmarks, with an asymmetric variant eliminating hedging on the Abalone dataset.

By Manisha Kandel
arXiv Machine Learning
Sep 11

EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

The paper introduces EMMI, a framework that enables communication‑efficient inference of multimodal large language models (MLLMs) on edge devices. EMMI encodes each sensor modality separately, fuses the representations, and compresses them into a compact latent vector that is transmitted to a server for high‑capacity reasoning. Experiments on a multimodal benchmark show that EMMI can cut the communication payload by 32× while keeping accuracy comparable, achieving up to a 3.4× reduction in end‑to‑end inference latency under bandwidth‑constrained conditions.

By Motahare Mounesan, Irfan Khan
arXiv Machine Learning
Sep 11

From Cycle Space to Cycle Manifold: Limits and Achievability of Blind False Data Injection Attacks

The paper investigates blind false data injection attacks (FDIAs) on power grids, showing that for a connected DC branch‑flow model the residual‑sensitive subspace equals the weighted cycle space. This space is both necessary and sufficient for constructing complete stealthy attacks, revealing that only cycle‑space knowledge is required. The authors propose a benchmark, a measurement‑only reconstruction method, and extend the analysis to AC systems via a cycle manifold, demonstrating practical attack generation on GPUs.

By Xin Li, Chenhan Xiao, Jonathan Cohen, Aviad Elyashar, Yang Weng, Rami Puzis
arXiv AI
Sep 11

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

PRAGMA is a benchmark designed to evaluate personalized guidance in long‑term conversations. It includes curated longitudinal conversation histories, evidence annotations, and guidance scenarios that reflect evolving user contexts and incorrect assumptions. Experiments show that current retrieval, memory, and long‑context models struggle to recover relevant conversational evidence and to use it effectively for personalized guidance.

By Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung
arXiv Machine Learning
Sep 11

SynCo: Synthetic Community-Aware Attributed Graph Generator for Graph Neural Network Benchmarking

SynCo is a synthetic graph generator that lets users control node degree distributions and sub‑community structures, addressing limitations of existing generators that rely on power‑law distributions and lack flexibility. It is evaluated on graph mimicking, hyperparameter tuning, and node clustering, outperforming state‑of‑the‑art methods while preserving original data distributions. SynCo can generate large graphs with up to 2.1 million nodes.

By Guilherme Henrique Messias, Mariana Caravanti de Souza, Sylvia Iasulaitis, Alan Dem\'etrius Baria Valejo
arXiv Machine Learning
Sep 11

When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents

The paper investigates how synthetic data used for fine‑tuning large language model (LLM) skill routers can lead to catastrophic forgetting of real and out‑of‑distribution (OOD) skills. By evaluating a production router with 34,396 skills, the authors show that synthetic‑data fine‑tuning improves in‑distribution retrieval but degrades performance on real and OOD data. They test several continual‑learning inspired mitigation methods—embedding‑anchor regularization, Learning without Forgetting, Elastic Weight Consolidation, and L2‑initialization—and find that these approaches both preserve OOD retrieval performance and boost synthetic in‑distribution retrieval by up to 13.98% for a 0.6B Qwen retriever and reranker.

By Syed Shariyar Murtaza, Yifan Nie, Utkarsh Soni, Eugene Wen, Arvid Frydenlund
arXiv Machine Learning
Sep 11

DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents

DriftNet is a dual‑head trajectory Transformer designed to detect and localize prompt injection attacks in large language model agents. It processes logged tool‑call trajectories, classifying each as compromised or not while labeling every step as benign, injection point, hijacked, or failed injection. On the AgentDrift benchmark, DriftNet achieves high accuracy, with an F1 score of 0.983, 98.7% exact injection‑point recovery, and low false‑alarm rates.

By Asif Pinjari, Mithun Paul Saint-Germain
arXiv Machine Learning
Sep 11

Symmetry-aware super-resolution of crystal orientation maps via invariant latent-space learning

arXiv:2609. 10898v1 Announce Type: cross Abstract: Crystal-orientation maps are physical fields defined only up to crystal symmetry; electron backscatter diffraction (EBSD) resolves them experimentally, but acquisition-time constraints limit spatial resolution.

By Umang Garg, Warren Zamudio, McLean P. Echlin, Samantha H. Daly, Tresa M. Pollock, B. S. Manjunath
arXiv Machine Learning
Sep 11

Empirical Evaluation of Membership Inference Attacks on NLP Text Classifiers: A Baseline Study on SST-2

The paper evaluates membership inference attacks (MIAs) on NLP text classifiers using the GLUE SST‑2 sentiment dataset. It compares a TF‑IDF + Logistic Regression pipeline with a fine‑tuned DistilBERT model under a loss‑threshold MIA, finding that both models leak membership signals despite high accuracy. The study also tests mitigations, showing that stronger regularization reduces leakage for Logistic Regression at a utility cost, while fine‑tuning DistilBERT for fewer epochs lowers leakage with minimal accuracy loss.

By William Novak (Minot State University), Muhammad Abusaqer (Minot State University)
arXiv Machine Learning
Sep 11

Hologram Representation via Quadratic Phase Gaussian Splatting

The paper introduces Complex-Valued Quadratic Phase Gaussian (CVQPG), a new hologram representation that replaces standard 2D Gaussian splatting with 2D quadratic phase functions. CVQPG adds learnable parameters to control the curvature of these bases, and the authors demonstrate that it improves visual quality by +0.19 dB (RGB) and +0.33 dB (grayscale) over state‑of‑the‑art methods while keeping the same parameter count. Frequency‑domain analysis shows that CVQPG preserves the mid‑to‑high frequency band of natural images.

By Haolong Wang, Yicheng Zhan, Kaan Ak\c{s}it, Simeng Qiu
arXiv Machine Learning
Sep 11

Breaking the Central Bias: Spatially Partitioned Experts for Coordinate-Based Neuroevolution

The paper investigates a spatial-concentration bias in Evolvable-Substrate HyperNEAT (ES‑HyperNEAT) when applied to MNIST, where evolved networks focus on a central cluster of input pixels. By partitioning the input image into 13 non‑overlapping spatial segments and evolving a separate expert network for each, the authors achieve a 43% mean accuracy—an 106% relative improvement over the baseline—without relying on data‑driven weighting. The study also introduces a receptive‑field diagnostic to detect silent input‑coverage collapse and a spatial‑partitioning remedy to restore full image coverage.

By Romain Claret, Arthur Gygax, Michael O'Neill, Paul Cotofrei, Michael Palma Mendes, Pascal Felber
arXiv Machine Learning
Sep 11

CertDW: Towards Certified Dataset Ownership Verification via Conformal Calibration

The paper introduces CertDW, a certified dataset watermark and ownership verification method that remains reliable even under malicious perturbations. By leveraging conformal prediction, it defines two statistical measures—principal probability (PP) and watermark robustness (WR)—to evaluate model stability on benign versus watermarked samples. The authors derive certification conditions linking WR to a PP-based threshold and provide a high‑probability bound on false positives, enabling robust ownership verification when a suspicious model’s WR exceeds the PP values of benign models.

By Ting Qiao, Yiming Li, Jianbin Li, Yingjia Wang, Leyi Qi, Junfeng Guo, Ruili Feng, Dacheng Tao
arXiv Machine Learning
Sep 11

When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification

The paper introduces Murmur2Vec, a lightweight, alignment‑free embedding that uses k‑mer counts hashed with MurmurHash to create a compact representation for biological sequences. It provides a full theoretical analysis, including bias/variance formulas, a Johnson–Lindenstrauss‑style concentration bound, and an excess‑risk bound that clarifies the trade‑off between hash‑table size and classifier performance. Empirically, Murmur2Vec matches or surpasses a fine‑tuned 650M‑parameter ESM‑2 protein language model across several classification tasks, including SARS‑CoV‑2 spike lineage and HIV‑1 Env subtype identification.

By Sarwan Ali, Taslim Murad, Imdadullah Khan, Safi Faizullah
arXiv Machine Learning
Sep 11

Time-Varying Graph Learning with Constraints on Graph Temporal Variation

The paper introduces a new framework for learning time‑varying graphs from spatiotemporal data, leveraging a prior on signal temporal behavior to estimate graphs from few measurements. It adds three convex regularization terms that enforce sparsity in the temporal changes of the network, and presents a scalable algorithm to solve the resulting optimization problem. Experiments on synthetic data and real datasets—including point clouds, temperature readings, and EEG signals—show that the method outperforms existing state‑of‑the‑art approaches.

By Haruki Yokota, Koki Yamada, Yuichi Tanaka, Antonio Ortega
arXiv Machine Learning
Sep 11

PitchFlower: A flow-based neural audio codec with pitch controllability

PitchFlower is a flow‑based neural audio codec that offers explicit pitch controllability by flattening and randomly shifting F0 contours during training while conditioning on the true F0 to reconstruct the original audio. A vector‑quantization bottleneck blocks pitch recovery, and a flow‑based decoder produces high‑quality audio. Experiments demonstrate that PitchFlower matches DSP baselines in pitch accuracy, surpasses state‑of‑the‑art neural codecs in audio quality, and remains robust even when trained on WORLD‑transformed audio, effectively removing vocoder artifacts.

By Diego Torres, Axel Roebel, Nicolas Obin
arXiv Computation and Language
Sep 11

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

The paper introduces a new perspective on entity rarity in multimodal entity linking by using knowledge‑graph structural metrics instead of popularity metrics, revealing many rare entities previously overlooked. Experiments show that state‑of‑the‑art models suffer a 15.4–39.9% accuracy drop on these rare‑entity slices. The authors propose a training‑free framework that combines reasoning and retrieval with a vision‑language model, achieving a 6.9% overall accuracy gain and up to 23.3% improvement on rare entities, and release a new benchmark MERLIN‑Rare for focused evaluation.

By Parinthapat Pengpun, Simran Khanuja, Graham Neubig
arXiv Computation and Language
Sep 11

ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

ProMediConv is a new benchmarking framework for evaluating proactive conversational agents in legal dispute mediation. It models mediation as a multi-stage, party-aware dialogue that incorporates 11 mediation strategies and four party behavior pattern states, and it is built on 972 real-world cases with utterance-level annotations. The framework introduces a fine-grained metric, MAD (Mean Attribute Difference), to capture shifts in party behavior throughout the dialogue, and provides a comprehensive benchmark with diverse models and a tailored baseline, ProMediAgent.

By Zesheng Wei, Mengfan Li, Wenhao Liu, Yixin Zhang, Zilei Wang, Yang Deng