Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 21

Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking

The paper proposes a multi‑signal value model for Instagram Feed ranking that separates baseline predictions from incremental gains, arguing that retaining raw predictions and adding incremental terms improves ranking quality. It describes how three independent debiasing families—viewer habit, item popularity, and request context—are combined into a single parameterized operator, with a design procedure selecting optimal points. Controlled ablation studies show that the two‑term model outperforms single‑term variants in most metrics, and the approach has been adopted across Meta’s products, boosting key engagement metrics.

By Hsin-Chiang Chen, Hao Wang, Gedi Zhou, Mingze Gao, Abhishek Kumar, Yiben Yang, Haoyu Wang, Hongjing Xia, Bo Chen, Zhengyu Su, Xian Sun, Chang Xu, Kevin Han, Weiwei Li, Zhongchang Sun, Yao Zhang, Qifei Wang, Peggy Yang, Yihan Wu, Haoran Liu, Siyang Sun, Xingyu Pan, Shannon Phu, Haoming Chen, Kevin Zhao, Chen Liang
arXiv Machine Learning
Oct 21

From Tokens to Files: Multi-Granularity Detection and Localization of AI-Involved Code

The paper introduces a multi-granularity framework for detecting AI involvement in code, distinguishing human-written from AI-generated or modified files and localizing the specific lines or blocks. It uses a frozen code language model with low-rank adapters trained on machine- and human-written data, producing per-token log-likelihood ratios that feed lightweight heads for line, block, and file-level predictions. Evaluated on thousands of mixed-authorship files, the method achieves high macro F1 scores and maintains low false-positive rates on real human code.

By Yue Xu, Xiyang Wang, Hui Liu, Bolei He
arXiv Computation and Language
Oct 21

Decomposed Relation Extraction from Aviation Accident Narratives: Evaluating Transformer-Based Safety Knowledge Discovery under Relational Sparsity

The paper investigates relation extraction from aviation accident narratives, noting that true safety relations are extremely sparse among concept pairs. It evaluates a decomposed framework that separates relation classification from concept identification, using a benchmark of 300 NTSB narratives expanded to 21,038 candidate pairs, of which only 248 represent key safety relations. Experiments with a TF‑IDF SVM and a weighted RoBERTa classifier show high false‑positive rates and a significant drop in performance from development to exhaustive evaluation, highlighting challenges in controlling false positives and managing candidate‑space sparsity.

By Aziida Nanyonga, Mustafa Tahsin Yilmaz, Saad Ezzini, Ugur Turhan, Graham Wild
arXiv Computation and Language
Oct 21

Do LLMs Really Lack Difference Awareness? Uncovering Heuristic Substitution in Group-Differentiating Fairness

The paper examines whether large language models (LLMs) truly lack awareness of meaningful differences between social groups in fairness decisions. It proposes that instead of lacking difference awareness, LLMs often substitute a simpler equal‑treatment heuristic, a phenomenon called Fairness Heuristic Substitution. The authors introduce a Two‑stage Distinguish‑Decide (TDD) framework that separates the identification of relevant group differences from the fairness decision, and show that TDD outperforms one‑stage baselines across multiple benchmarks and models, while also revealing mechanistic evidence for the heuristic substitution hypothesis.

By Jingshen Zhang, Bo Wang, Yanlin Fu, Yuewen Dang, Dongming Zhao, Zifei Yu, Yuexian Hou
arXiv Computer Vision
Oct 21

HiQGS: Hierarchical Spatiotemporal Querying for Feed-Forward Gaussian Reconstruction

HiQGS is a query-based hierarchical framework that performs feed-forward Gaussian reconstruction from sparse monocular frames. It introduces a Hierarchical Spatiotemporal Encoder that organizes multi-resolution visual features by image location, observation time, and camera pose, preserving complementary appearance and structural cues. A Time-Adaptive Decoder then uses target timestamp queries to retrieve relevant spatiotemporal evidence and predict the corresponding Gaussian scene state, achieving strong rendering quality and geometric accuracy on static and dynamic scene benchmarks.

By Qiaowei Miao, Qifeng Cheng, Yijing Guo, Yuru Xiao, Ziqian Ni, Sheng Yang
arXiv Computer Vision
Oct 21

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

The paper introduces Pangaea-Latents, an extension of the Pangaea benchmark that provides frozen embeddings from five geospatial foundation models (GFMs) across eleven datasets, showing that these embeddings are neither redundant nor disjoint. It then proposes GeoMixture, a framework that aligns a Vision Transformer (ViT) to multiple GFMs per dataset using a multi-representation alignment (mREPA) objective, resulting in an encoder that achieves the best average rank across nine Pangaea datasets while maintaining cross-task consistency.

By Arthur Ouaknine, David Rolnick
arXiv Computer Vision
Oct 21

Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting

arXiv:2610.27877v1 Announce Type: new Abstract: Anticipating how users will interact with their surroundings is essential for proactive systems that provide timely assistance in AR, VR, and embodied...

By Chuanyang Ma, Chenye Wang, Yixuan Wang, Huiling Zhou, Xingqun Qi, Muyi Sun, Man Zhang, Caifeng Shan, Zhenan Sun
arXiv Computer Vision
Oct 21

Decoupling Action from Egocentric Observation for World Simulation

arXiv:2610.27946v1 Announce Type: new Abstract: Egocentric action transfer aims to decouple the semantic action from egocentric observations and reproduce them in new visual contexts, enabling scalab...

By Yue Ma, Pengjie Song, Xinyu Wang, Yi He, Zeqian Long, Fangneng Zhan, Kaichen Zhou, Hongyu Liu, Hongfa Wang, Peihao Li, Haoyang Huang, Nan Duan, Qifeng Chen
arXiv Computer Vision
Oct 21

Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

arXiv:2610.27974v1 Announce Type: new Abstract: The rapid progress of generative artificial intelligence has led to a significant increase in manipulated multimedia content, including synthetic faces...

By David Fischinger, Marcel Hasenbalg, Silvia Poletti, Davide Antonutti, Mina Sch\"utz, Lam Pham, Justin Ilyes, P\'eter Rjabcsenk\'o, Simon Freitter, Martin Boyer
arXiv Computer Vision
Oct 21

RACE: Learning Multimodal Search through Referential Abstraction and Compositional Evidence

arXiv:2610.28382v1 Announce Type: new Abstract: Training multimodal search agents requires tasks that make visually grounded search necessary for determining the answer. Textual identity clues and un...

By Eric Tao, Minhao Sun, Liyan Liu, Zhijing Wu, Xiaohan Yu, Lang Mei, Li Huang, Chao Feng, Jinchao Ma, Sichen Kang, Siyu Mo, Chong Chen