Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv:2609. 27930v1 Announce Type: cross Abstract: We propose a Bayesian nonparametric mixture of regression trees with a Dirichlet process prior over tree-parameter pairs, enabling data-driven selection of ensemble size and unifying CART, BART, random forests, and boosting.
EvEMTBench is an open, executable, and versioned benchmark designed to standardize the evaluation of machine‑learning methods for power system protection. It defines 12 protection and event‑analysis functions across four grids (20–345 kV) as 24 scored tasks, enabling structured assessment under varied observability, distribution shifts, and cross‑grid transfer scenarios. The benchmark includes committed data partitions, leakage controls, and reproducible reporting, and demonstrates that wider observability does not always help, that shifted conditions expose hidden failures, and that fault detection transfers better than fault localization.
The paper introduces a quantum score‑matching framework that extends classical score matching to quantum states, addressing challenges posed by noncommuting density operators. It demonstrates that this method can learn thermal (Gibbs) states without extra state preparation, achieving optimal sample complexity in high‑temperature regimes for local Hamiltonians. Numerical tests and experiments on IBM quantum hardware confirm the approach’s effectiveness and NISQ‑friendly performance, reducing Hamiltonian‑parameter error from 64% to about 10%.
The paper introduces PIECE, a Parameter Importance-Driven Continual Learning method that selectively updates only 0.1% of core parameters to preserve general abilities while learning new domain knowledge. PIECE employs two importance estimators—PIECE‑F using Fisher Information and PIECE‑S combining gradient and curvature information—to guide updates. Experiments on three language models and two multimodal models demonstrate that PIECE maintains general capabilities and achieves state‑of‑the‑art continual learning performance without accessing prior training data or adding parameter overhead.
The paper introduces localized diffusion models, which exploit locality structure—sparse conditional dependencies among target variables—to reduce the dimensionality of the score function. By training a localized neural network with a localized score matching loss, the authors demonstrate that diffusion models can achieve dimension‑independent error bounds, balancing statistical and localization errors with a moderate radius. This approach also enables parallel training, potentially improving efficiency for large‑scale applications.
The paper introduces the Ensemble-conditioned Inverse Problem (EIP), a multivariate statistical framework for inferring an ensemble that follows the pushforward of a prior through a forward process. It applies to fields such as high‑energy physics, full waveform inversion, and inverse imaging, and proposes non‑iterative inference‑time methods using ensemble inverse generative models that avoid explicit forward model use during inference. The authors demonstrate the approach on synthetic and real datasets and provide code for replication.
The paper introduces iAmTime, a time‑series foundation model that uses instruction‑conditioned in‑context learning to adapt to tasks at inference time. iAmTime represents each episode as a structured prompt with semantic tokens that focus on specific time‑series regions, enabling the model to infer task structure from input‑output demonstrations. Trained on large real and synthetic corpora across forecasting, imputation, reconstruction, classification, anomaly detection, and source de‑mixing, iAmTime outperforms strong baselines on zero‑shot probabilistic and point forecasting while matching or exceeding performance on several non‑forecasting tasks.
The paper introduces Hierarchical Reinforcement and Collective Learning (HRCL), a framework that combines multi‑agent reinforcement learning (MARL) with decentralized coordination. HRCL uses MARL at a high level to generate strategic guidance that limits the decision space for low‑level agents, enabling efficient short‑term coordination while considering long‑term effects. Experiments on synthetic, energy‑management, and drone‑swarm scenarios demonstrate faster convergence and significant reductions in system‑wide and individual costs compared to standalone MARL.
The paper introduces an SMT-based active learning algorithm for nondeterministic weighted finite automata (WFAs), offering a practical alternative to traditional Hankel/L*-style methods. The algorithm is parametric over a chosen semiring and, upon termination, guarantees the production of minimal WFAs. Experiments demonstrate that it learns numerous minimal WFAs over both finite and infinite semirings, outperforming a naive baseline and competing with state‑of‑the‑art algorithms while yielding smaller automata and requiring less teacher interaction.
The paper introduces a sentence‑level benchmark for judging large language models’ ability to classify interpretive canons used by the German Federal Constitutional Court, based on Larenz’s framework. It operationalizes these canons as classification criteria, provides a dataset of court decisions annotated at the sentence level, and evaluates four LLMs with both expert hand‑written prompts and prompts optimized via Genetic‑Pareto. The results show mean F1 scores between 70.4 and 79.2, with grammatical interpretation being the easiest and systematic interpretation the hardest, and indicate that expert prompts already offer a strong baseline.
LeakScale is an interventional framework that estimates the causal effect of benchmark exposure on model performance. By creating new executable tasks that require private, family‑specific information and controlling access to that information, LeakScale measures the control‑adjusted change in accuracy. Across 2,048 families, two model families, two domains, and 262,144 generations, exposure consistently improves accuracy by 7.17 to 27.31 percentage points, separating the question of whether contact occurred from how much a score depends on it.
The paper introduces WebMRE, an offline benchmark comprising 541 tasks and 5,293 steps extracted from WebArena trajectories, designed to provide deterministic scoring for web agents without live environments. It enables the first systematic study of how guide sentences and grounded actions reinforce each other, showing that jointly decoding a guide improves element selection accuracy and that the guide acts as a causal instruction channel. The authors fine‑tune models that outperform leading zero‑shot baselines on all offline metrics.
The paper introduces RoPA Manager, a system that automates the extraction of Records of Processing Activities (RoPA) required by Vietnam’s new Personal Data Protection Law. It combines hybrid retrieval techniques—lexical ranking, dense‑vector search, and Reciprocal Rank Fusion—with locally deployed large language models to avoid data‑sovereignty issues. A Vietnamese RoPA benchmark of 32 organizations and 77 processing activities was created, and the system achieved robust scorer performance (F1 ≈ 0.95) and moderate end‑to‑end token coverage (≈ 50‑55%).
The paper introduces MORSE, a method that orders contexts in a way that preserves evidence during compression. By applying a reverse query‑evidence principle to both individual contexts and compressed outputs, MORSE constructs an evidence‑first anchor and guides permutation selection to avoid information preemption. Experiments on multi‑hop QA benchmarks show that MORSE consistently outperforms static reverse ordering and random search, improving evidence retention and downstream QA performance.
AraGenre 2026 is a shared task that challenges participants to perform hierarchical, definition-guided Arabic genre classification. Systems must assign each Arabic text segment both a broad communicative genre and a fine-grained specific genre, using natural-language definitions for 74 unseen specific genres in a zero-shot setting. The released datasets include limited synthetic examples for training and development, while the hidden final benchmark contains noisier, naturally occurring text across Modern Standard Arabic, Classical Arabic, and multiple dialects.
The paper compares Jev with nine language models on the ContractNLI task, assessing inference cost, response time, average correctness, and correctness under repeated requests. Controlled experiments vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev achieves the lowest cost and median response time, whereas hosted language models show higher baseline accuracy, but rankings differ when evaluating correctness across all conditions and repeats.
The paper introduces Self‑Conditioning, an unsupervised, information‑theoretic estimator that measures the amount of external information in peer reviews. It compares the likelihood of a review under its original production context with the likelihood when that context is augmented by hints extracted from the review itself. On the IntelLabs benchmark, Self‑Conditioning can perfectly distinguish fully‑delegated reviews from machine‑polished ones, remains largely insensitive to surface rewriting, and shows that increased external input drives scores toward human‑like values, unlike standard ATD baselines.
The paper introduces Causal Shortcut Learning (CSL), a framework that identifies token chains—called causal shortcuts—that guide Diffusion Language Models (DLMs) toward correct reasoning paths. By extracting these shortcuts and applying parallel prioritized masking during training, CSL improves both convergence speed and generation accuracy. Experiments on several reasoning benchmarks and two base models show CSL outperforms existing SFT-variant baselines, achieving an average 1.92% improvement over SFT-only models and up to 4.20% on MATH-500.
RapidUn is a parameter reweighting framework that uses influence estimates to guide LoRA-only updates for efficient unlearning of targeted behaviors in large language models. It operates in a practical PEFT setting with a small forget set and limited retain buffer, converting cross-sample influence into fixed sample-specific weights for weighted LoRA unlearning. Experiments on Llama‑3‑8B with Dolly‑15k and Alpaca‑57k datasets show RapidUn achieves lower trigger ASR than Fisher, GA, and LoReUn while preserving clean utility, and delivers a 77× wall‑clock speedup over clean‑corpus LoRA retraining, with additional evaluations supporting its effectiveness.
SafeTutors is a benchmark designed to evaluate both safety and pedagogical effectiveness of AI tutoring systems across mathematics, physics, and chemistry. It introduces a risk taxonomy of 11 harm dimensions and 48 sub‑risks based on learning‑science literature, focusing on issues such as answer over‑disclosure, misconception reinforcement, and loss of scaffolding. The study finds that all tested models exhibit broad harms, that larger scale does not mitigate these issues, and that multi‑turn interactions significantly increase pedagogical failures from 17.7% to 77.8%.