Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 22

Extending FAIR Principles with Observability for Trustworthy Federated Learning

The paper proposes extending the FAIR principles with an observability layer—FAIR-O—to enhance trust in federated learning. It argues that merely keeping data local is insufficient for trustworthiness, and introduces a framework where each participant generates signed, minimal trust evidence for every FL round, while raw data and logs stay local. A prototype demonstrates evidence generation and a three‑level trust model, and the authors outline how this evidence could inform participant selection and aggregation in future work.

By Hangli Ge, Noboru Koshizuka
arXiv Machine Learning
Oct 22

How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?

The paper investigates how well foundation models can be distilled for efficient jet tagging at the Large Hadron Collider. Six teacher models, varying in architecture, symmetry, and pretraining—including two foundation models—are distilled into five small student models. The strongest distilled students, Transformer and L‑GATr‑slim, achieve state‑of‑the‑art performance with only 20,000 parameters, and even a 2,000‑parameter version matches a much larger model, ParT, while offering significantly lower inference cost.

By Jay Agarwal, Siddharth Khare, Dhruv Kumar
arXiv Machine Learning
Oct 22

Asymmetry Tunneling Beyond Adaptive Equivariance Relaxation

The paper investigates whether adaptive control of equivariance relaxation is necessary for neural networks to capture symmetry-breaking structure in data. By adding a residual path to a strictly equivariant model and limiting their interaction, the authors observe that the residual can either collapse or survive, a phenomenon they term asymmetry tunneling. They analyze this behavior across five benchmarks using four diagnostics and conclude that adaptive control of equivariance relaxation is not required to model symmetry-breaking patterns.

By Eunho Jeong, Joon Jang, Hyeonjin Kim
arXiv Computation and Language
Oct 22

Towards English - Mizo Legal Machine Translation: Corpus Creation, Analysis, and Benchmarks

The paper introduces the first English‑Mizo parallel dataset of Indian court judgments, comprising 13,168 sentence pairs from the Gauhati High Court and the Supreme Court of India. Professional native Mizo translators produced complete documents, after which sentence pairs were manually aligned and independently reviewed. Analysis reveals selective borrowing of English words—official and legal terms remain in English while everyday terms are rendered in Mizo—and inconsistent use of Mizo diacritics. Fine‑tuning the NLLB‑200 1.3B model on this corpus yields a 27.47 BLEU score on a balanced legal test set, improving by 7.16 BLEU over the base model, yet the model still over‑uses English words compared to human translation.

By Telem Joyson Singh, Sanabam Boyshana, Sanasam Ranbir Singh, Priyankoo Sarmah
arXiv Computation and Language
Oct 22

CulturalRiddles: A Multicultural Benchmark of Riddles

CulturalRiddles is a new benchmark consisting of original riddles created by a community effort, covering 62 cultures across 45 countries and 51 languages. The riddles are deeply rooted in their respective cultures and require abductive, compositional reasoning, factual and social knowledge, language skills, and an understanding of culturally specific associations. The authors evaluated 28 large language model configurations, finding that even state‑of‑the‑art models achieve at most 22% accuracy in some communities, and they are releasing all artifacts and an open contribution process to expand the benchmark.

By My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, Jos\'{e} Mat\'{i}as Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Neboj\v{s}a Damnjanovi\'{c}, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dhol\'{e}, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier Garc\'{i}a Gilabert, \'{A}lvaro Gonz\'{a}lez-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Joki\'{c}, Ulugbek Khalilov, Dominik Krzemi\'{n}ski, Roy Ka-Wei Lee, Haichang Li, Chen Cecilia Liu, Zhengyuan Liu, Cedric Lothritz, Houman Mehrafarin, Antonio Miranda-Escalada, Fengran Mo, Mehrnaz Mofakhami, Wafaa Mohammed, Felipe Moreno-Vera, Prahitha Movva, Jannatul Nayem, Van Ngo, Joy Olusanya, Marc P\`{a}mies, Avinash Kuney Pandey, Yurii Paniv, Fred Philippy, Vladislav Poritski, Dikshya Poudel, Joanna Rado{\l}a, Sara Rajaee, Swati Rajwal, Prerana Rane, Muhammad Ravi Shulthan Habibi, Lucas Resck, Shubhashis Roy Dipta, Ananya Sahu, Eduardo S\'{a}nchez, Borja S\'{a}nchez Clemente, Aleix Sant, Nobin Sarwar, Istiak Shihab, Santhoshini Pranathi Singaraju, Jane Smith, David Stap, Emily Z Su, Ananth K Suresh, Sajag Swami, Ruiyang Tan, Nazia Tasnim, Khalid Tounoussi, Patteera Triamamornwooth, Muhammad Saad Umar, Shankar Venkitachalam, Oksana Volchek, Lingbo Wang, Debela Desalegn Yadeta, Rushikesh Zawar, Jinghan Zhang, Mike Zhang, Weihua Zheng, Marzieh Fadaee, Akhil Arora, K\'{a}roly Boczka, Julia Kreutzer
arXiv Machine Learning
Oct 22

The Data Differentiator Framework (DDF): Quantifying Marginal Data Value with Information-Theoretic and Game-Theoretic Measures in an Agentic Evaluation Pipeline (SDSAE)

The Data Differentiator Framework (DDF) is a composite scoring method that evaluates candidate data assets by combining a mutual‑information redundancy penalty, a cost‑normalized marginal utility score from an ablation study, and explicit governance inputs such as trust and strategic weights. It is executed through the Strategic Data Sourcing Agent Ecosystem (SDSAE), which bypasses costly model retraining for assets that fail governance or redundancy checks. In a credit‑card fraud benchmark, only the genuinely differentiated asset passed the significance gate and received an acquisition recommendation, while a fully redundant asset scored lowest.

By Sukesh Kumar, Dilip Singh
arXiv Machine Learning
Oct 22

Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift

The paper investigates how changes in inference-time settings—such as quantization and temperature—affect conformal prediction (CP) for language models. It introduces the concept of inference-time model shift, showing that such shifts break split-conformal exchangeability even when the underlying model and data distribution remain unchanged. Experiments on eight language models, four benchmarks, and twelve inference configurations reveal that matched evaluation protocols can significantly overestimate deployment-time coverage, with effects varying across models and datasets.

By Achref Doula, Max M\"uhlh\"auser
arXiv Machine Learning
Oct 22

DISRL: Dual-Scale Incremental Semi-Supervised Reinforcement Learning for Dynamic Community Tracking

DISRL is a Dual-Scale Incremental Semi-Supervised Reinforcement Learning framework designed for dynamic local community detection. It tackles temporal supervision uncertainty by generating seed-specific guidance from sparse known communities and mitigates unbounded error cascades through a dual-scale community exploration module that separates short-term adaptation from long-term parameter consolidation. Experiments on a newly constructed dynamic graph benchmark show DISRL consistently outperforms state‑of‑the‑art baselines across various dynamic settings.

By Tianming Zhang, Jinxu Wang, Yixuan Zhang, Zhengyi Yang, Chenyu Hou, Bin Cao, Jing Fan
arXiv Machine Learning
Oct 22

Data Schedules and Learning Rate Schedules Should Be Designed Together

The paper investigates how the timing of introducing mid‑training (MT) data and the decay of the learning rate (LR) interact in large‑language‑model training. It shows that when the LR decays, mixing MT data earlier with pretraining (PT) data reduces final MT loss, whereas with a constant LR the opposite holds. The authors conduct ablation studies and propose a theoretical model explaining why the optimal data schedule depends on the LR schedule.

By Haodong Wen, Kairong Luo, Tingkai Yan, Wenguang Chen, Kaifeng Lyu
arXiv Machine Learning
Oct 22

WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

The paper introduces Weighted Refusal Direction Optimization (WRDO), a method that constructs a composite refusal direction by learning aggregation weights over per‑prompt refusal directions. WRDO aims to strengthen refusal interventions in aligned language models by allowing heterogeneous harmful prompts to contribute unequally. Experiments on three open‑weight models, using guard judges and HarmBench, demonstrate that weighted aggregation outperforms single‑direction methods and can also be used in reverse to restore refusal behavior in a damaged checkpoint.

By Grigoriy Evseev, Vladimir Solodkin, Dmitry Bylinkin, Daniil Medyakov, Vladimir Aletov, Aleksandr Beznosikov
arXiv Machine Learning
Oct 22

DriveMLP: When Training Strategy Outperforms Model Complexity in Autonomous Driving

The paper introduces DriveMLP, a lightweight MLP policy for autonomous driving that is trained using an Expert‑Guided RL (EGRL) framework. EGRL leverages reinforcement learning solely as an optimization tool, guided by a trajectory‑distance reward derived from logged expert data, and the study shows that the training strategy itself yields significant performance gains over model complexity. DriveMLP achieves an 86.51 score on the nuPlan Val14 Reactive benchmark, surpassing the previous best IL planner by 3.2 points while offering more than 15× faster inference.

By Alessandro Canevaro, Julian Schmidt, Hang Yu, Ye Jin, Sajad Marvi, Georg Martius, Julian Jordan
arXiv Machine Learning
Oct 22

GPU-Accelerated Construction of the Gabriel Graph: Parallelization Strategies and Applications to Large-Margin Classification and Graph-Based Clustering

The paper introduces GPU-based parallelization techniques for constructing the Gabriel graph, a proximity graph used in computational geometry and machine learning. By mapping thread indices to vertex pairs, employing a bitmask-based edge-list construction, and using thread coarsening, the authors achieve up to 8484× speedups over a CPU baseline without sacrificing accuracy. The accelerated graph enables efficient large‑margin classification and a new density‑aware clustering method that competes with DBSCAN while eliminating the need for a density radius parameter.

By Diego Sanches Nere dos Santos, Luiz Carlos Bambirra Torres
arXiv Machine Learning
Oct 22

Recursive Self-Improvement on a Budget: Scoring Code Edits Before They Run

The paper introduces a coding agent that selects and expands code edits before evaluation by scoring proposed changes using a linear readout of the parent’s performance and the edit’s features. It compares several selection strategies—including Huxley‑Gödel Machine selection, parent‑rate ablation, and calibration‑best edits—across HumanEval+ and Polyglot benchmarks, reporting held‑out performance for each. Experiments with GPT‑4o and Claude Sonnet 4.5 show that the proposed methods achieve high accuracy, with the best nodes reaching 87.0% on HumanEval+.

By Erfan Darzi, Sree Bhargavi Balija, Sheng He
arXiv Machine Learning
Oct 22

A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

The paper introduces a unified evaluation framework for assessing the physical adversarial robustness of Vision‑Language Models (VLMs) deployed at the edge. It defines a conditional attack success rate (cASR) metric that counts only images where the model still localizes objects under a gray occluder, and categorizes failures into omission, substitution, and displacement. Benchmarking shows that compact grounding VLMs can be as vulnerable as traditional object detectors, with omission being the only failure visible to downstream systems, while displacement and substitution pose potentially more dangerous risks, especially under distribution shift.

By Eric Liang, Alan Luo, Kaiwen Yuan
arXiv Machine Learning
Oct 22

Cache Locally, Recompute Selectively for Streaming Video Understanding

The paper introduces FrameBlend, a plug‑and‑play technique for streaming video understanding that selectively recomputes key‑value (KV) caches at query time. By re‑prefilling only query‑relevant video tokens, FrameBlend allows shorter attention windows during idle prefill, cutting prefill cost while boosting accuracy. Experiments on three SVU frameworks and five benchmarks show up to 3.9‑point accuracy gains and significant reductions in FLOPs and processing time.

By Junhyeok Lee, Sangyoon Lee, Jaeho Lee
arXiv Machine Learning
Oct 22

Decodability Is Not Controllability: Local and Anisotropic Speaker Control in Continuous-Latent TTS

The paper demonstrates that high linear decodability in continuous‑latent TTS does not guarantee causal controllability of speaker identity. In VoxCPM2, probes at the final residual acoustic language model layer show strong accuracy, yet aligned edits fail to produce target‑dominant dyads, whereas early‑position edits succeed. Cross‑architecture experiments reveal that backbone edits shift the source voice without attracting the target, while x‑vector ablation eliminates source similarity, underscoring the need for geometric specificity and a computational path that embeds the intended speaker condition into synthesis.

By Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang
arXiv Machine Learning
Oct 22

Think Harder Where It Matters: Item-Level Adaptive Recursion for Sequential Recommendation

The paper introduces LoopRec, an item‑level adaptive recursion mechanism for sequential recommendation that replaces the conventional fixed‑depth computation with a learned, per‑item depth assignment. LoopRec recursively applies a shared block and uses a lightweight gate to decide whether each item should exit or continue processing, allowing early exit for items that converge quickly while allocating deeper computation to critical items. Experiments on multiple backbone families show that LoopRec consistently outperforms fixed‑depth counterparts, with learned depth patterns aligning with lower‑frequency and more recent items.

By Zhanshuo Cao, Jiakai Tang, Yuqi Zhou, Sunhao Dai, Jun Xu
arXiv Machine Learning
Oct 22

ChemBART: Pre-training a Reaction Language Model for Comprehensive Synthesis Planning

ChemBART is a BART‑based language model trained on holistic reaction expressions (Reactant > Reagent > Product) using a mask‑filling objective. It can solve diverse chemical tasks such as precursor/reagent generation, temperature‑yield regression, and property classification, and it is integrated with Monte Carlo tree search to design multi‑step synthesis routes. The model’s predictions were experimentally validated, yielding shorter synthetic pathways with over 20 % higher yield than literature benchmarks.

By Kenan Li, Yijian Zhang, Jin Wang, Haipeng Gan, Zeying Sun, Xiaoguang Lei, Hao Dong
arXiv Machine Learning
Oct 22

Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification

The paper introduces Evidential Uncertainty Quantification (EUQ), a fine‑grained method for detecting misbehaviors in large vision‑language models (LVLMs). EUQ interprets model output features as supporting or opposing evidence and aggregates them using Evidence Theory to quantify internal conflict and ignorance in a single forward pass. Experiments on hallucinations, jailbreaks, adversarial attacks, and out‑of‑distribution failures show that EUQ outperforms strong baselines, linking hallucinations to high conflict and OOD failures to high ignorance, and provides layer‑wise dynamics for interpreting internal representations.

By Tao Huang, Rui Wang, Xiaofei Liu, Yi Qin, Li Duan, Liping Jing
arXiv Computation and Language
Oct 22

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

The paper introduces DiagSafe, a diagnostic evaluation framework that replaces binary safe/unsafe labels with prompt-level rubrics grounded in AI safety policies. DiagSafe uses a bilingual benchmark with an 84‑label safety taxonomy, comprising DiagBase and DiagAttack subsets, to assess safety and usefulness independently. Experiments show DiagSafe achieves high agreement with human annotations (95.34%) and strong inter‑judge consistency (Cohen’s κ = 0.864), revealing significant differences in unsafe rates and vulnerabilities across leading LLMs, especially in Chinese and multi‑turn contexts.

By Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wang, Rui Chen, Blaze Chen, Fengzong Lian, Renfen Hu