Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI

Sep 10

Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation

arXiv:2609.08275v1 Announce Type: new Abstract: Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute e...

By Tianyi Zeng, Junchao Liao, Yujie Wei, Ziying Zhang, Litao Li, Tianyi Wang, Zhichao Wei, Shuyao Xu, Wenwen Qiang, Siyu Zhu, Zhenghao Zhang, Long Qin

agents diffusion nlp multimodal benchmarks safety

arXiv Machine Learning

Sep 10

Content-Based Addressing for Long Context

The paper proposes a content‑based addressing scheme for long‑context models that replaces the growing token counter in Rotary Position Embedding (RoPE) with unit‑level addresses derived from the content of each unit. By dividing the token stream into units, the method preserves local RoPE behavior while allowing new units to be addressed via learned content maps, avoiding positional mismatches when extending context length. Experiments on character‑level Tiny Shakespeare show that a model trained on 256‑character contexts achieves lower perplexity at 4096 characters using this scheme, and a second diagnostic demonstrates retrieval of multiple serialized facts.

By Mahesh Godavarti

rag benchmarks safety

arXiv AI

Sep 10

Deposon: An Auditable, Conservation-Guaranteed, Game-Theoretically Tested Scattering Layer over LLM Reasoning Paths

arXiv:2609.09001v1 Announce Type: new Abstract: Multi-step LLM reasoning lacks a machine-recheckable ledger: discarded reasoning paths leave no auditable record. We propose the Deposon scattering lay...

By Qihao Yuan

llms benchmarks

arXiv AI

Sep 10

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv:2609.08966v1 Announce Type: new Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the...

By Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges

fine-tuning benchmarks

arXiv AI

Sep 10

XAI-SDN: An Explainable Entropy-Guided Machine Learning Framework for Real-Time DDoS Detection in Software Defined Networks

arXiv:2609.05701v1 Announce Type: cross Abstract: One of the biggest risks faced by Software Defined Networks (SDN) is the Distributed Denial of Service (DDoS) attack in which a compromised controlle...

By Adeel Ahmad, Ali Akarma, Ahmad Ali, Hammad Muneer, Toqeer Ali Syed

benchmarks

arXiv Machine Learning

Sep 10

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

arXiv:2609.08657v1 Announce Type: cross Abstract: Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This str...

By Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui

benchmarks

arXiv AI

Sep 10

SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation

arXiv:2609.05742v1 Announce Type: cross Abstract: American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion represen...

By Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu

llms nlp efficiency benchmarks safety

arXiv AI

Sep 10

Do Quantum AIs Dream in Paths? Path-Integral Slow Thinking through Grover Interference

arXiv:2609.05842v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards enables large language models to think slowly, but the same training can induce policy collapse: proba...

By Xiansheng Cai, Xiu-Hao Deng, Kun Chen

llms reinforcement-learning benchmarks

arXiv Computer Vision

Sep 10

Shape-guided Gaussian Splatting for Sparse-View X-ray 3D Reconstruction

arXiv:2609.10376v1 Announce Type: new Abstract: Sparse-view X-ray 3D reconstruction is essential for reducing radiation exposure, but recovering a density field from a handful of X-ray projections is...

By Pranav Poudel, Florence Dell'Aniello Picard, Nairouz Shehata, Fr\'ed\'eric Lavoie, Herve Lombaert

benchmarks

arXiv Computer Vision

Sep 10

CHIMERA Challenge Task 2 and 3: Response Subtypes Classification and Progression Survival Prediction in Bladder Cancer Patients using Multimodal Datasets

arXiv:2609.09510v1 Announce Type: cross Abstract: High-risk non-muscle-invasive bladder cancer (HR-NMIBC) carries substantial risks of recurrence and progression, while current clinical risk stratifi...

By Catherine Chia, Tongjie Wang, Robert Spaans, Maryam Mohammadlou, Farbod Khoraminia, J. Alberto Nakauma-Gonz\'alez, Adam Kowalewski, Parandzem Khachatryan, Domingos Oliveira, Khrystyna Faryna, CHIMERA Challenge Consortium, Marlies Wakkee, Sita Vermeulen, Tahlita Zuiverloon, Nadieh Khalili

multimodal benchmarks

arXiv AI

Sep 10

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

arXiv:2609.05899v1 Announce Type: cross Abstract: Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective...

By Peng Lai, He Zhu, Zhiwen Ruan, Dongdong Zhang, Yun Chen, Peng Li, Furu Wei, Yang Liu, Guanhua Chen

llms benchmarks safety data-engineering

arXiv AI

Sep 10

A visual large language foundational model for medical image recognition using clinician-oriented social media

arXiv:2609.06914v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their a...

By Lingxuan Hou, Yuhua Xie, Yue Hu, Yan Zhuang, Junqi Li, Chengzhi Xia, Binh Phu Nguyen, Abubakar Siddique, Minh Nguyen, Yao Hou, Yanju Bao, Kexin Liu, Ke Chen, Jianjun Sun, Zeqi Li, Trung Nguyen, Jiangli Lin

llms computer-vision nlp multimodal benchmarks safety

arXiv AI

Sep 10

iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes

arXiv:2609.06960v1 Announce Type: new Abstract: Invasive neural recordings provide high-fidelity measurements of brain activity, with signals such as intracranial EEG (iEEG) and intracortical spiking...

By Ying Chen, Tiou Wang, Zhifeng Yue

llms benchmarks safety

arXiv AI

Sep 10

Intent Drift at SME Scale: Deployment Practice, Not Model Capability, Determines Agentic Compliance

arXiv:2609.05975v1 Announce Type: cross Abstract: We introduce Chain of Intent, a governance framework for agentic AI at small regulated firms, and validate it against a failure it was built to addre...

By Ilia Voroshilov

agents benchmarks safety

arXiv AI

Sep 10

Beyond Sparse Rewards: A New Benchmark and Structure-Aware Graph Alignment for Micro-Drama Understanding

arXiv:2609.07107v1 Announce Type: new Abstract: Micro-dramas, characterized by ultra-short durations and hyper-dense storylines, pose unique challenges for video understanding that conventional bench...

By Yixin Qin, Shi-Zhe Chen, Zhiqi Yu, Siyuan Cheng, Tao Cheng, Jinwen Luo, Zheng Wei

reinforcement-learning benchmarks safety

arXiv Machine Learning

Sep 10

Interpretable and Fair Generalized Additive Neural Networks via Multi-objective Learning

arXiv:2609.05946v1 Announce Type: new Abstract: Interpretability and fairness are two of the most emphasized dimensions in trustworthy artificial intelligence (AI). Various explainable AI methods hav...

By Ziming Wang, Changwu Huang, Ke Tang, Yew-Soon Ong, Xin Yao

benchmarks safety

arXiv AI

Sep 10

Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

arXiv:2609.06011v1 Announce Type: cross Abstract: Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexp...

By Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen

llms multimodal benchmarks safety

arXiv AI

Sep 10

From Event Logs to Governed Action: A BlueSky Agenda for Agentic Process Mining

arXiv:2609.07984v1 Announce Type: new Abstract: Process mining has long turned event logs into process knowledge: discovered models, conformance evidence, bottleneck diagnoses, and runtime prediction...

By Yiyuan Yang, Zheshun Wu, Yong Chu, Zhenghua Chen, Zenglin Xu, Qingsong Wen

llms agents benchmarks safety

arXiv AI

Sep 10

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

arXiv:2609.08025v1 Announce Type: new Abstract: Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms su...

By Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray

llms agents nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 10

SeaCausal-FL: Federated Fuzzy Causal Learning for Maritime IoT Fault Diagnosis and Counterfactual Reasoning

arXiv:2609.06257v1 Announce Type: new Abstract: Reliable marine-engine fault diagnosis in maritime IoT is challenged by distributed data ownership, heterogeneous fault distributions, and continuously...

By Yuhang Qiu, Haihan Zhu, Koteeswaran Seerangan, Longsheng Zhu, Xiong Wang, Yijun Lu, Zheng Lin, Fangmin Ren, Jialiang Xie

benchmarks

Get the daily briefing