Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI

Oct 31

PAPERBENCHX: Can AI Reproduce Science Across Disciplines?

PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.

By Ruoyu Wu, Zengji Tu, Aolong Sun, Tianyi Ma, Junren Li, Xinbo Xu, Li Fu, Ziyan Zhang, Pinrui Huang, Xiaochen Wang, Chengda Wu, Yutao Zhou, Jianjun Wu, Lin Chang, Baobao Chang, Liang Chen, Kuan Li

agents robotics benchmarks

arXiv AI

Oct 31

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

Nemotron 3.5 Super and Lightning are open, efficient Mixture‑of‑Experts hybrid Mamba‑Transformer models aimed at reasoning, long‑context understanding, and agentic workloads. Super is a 120 B‑parameter, 12 B‑active multimodal model, while Lightning is a 30 B‑parameter, 3 B‑active text model, both optimized for efficient deployment. The models combine large‑scale distillation, long‑context training, native speculative decoding, and a comprehensive post‑training pipeline that includes supervised fine‑tuning, reinforcement learning, and multi‑teacher on‑policy distillation, enabling Super to function as a multimodal agent capable of reasoning over text, images, video, and interactive tools.

By NVIDIA, :, Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahamed, Asma Kuriparambil Thekkumpate, Aswin Visva, Atefeh Sohrabizadeh, Avinash Vem, Ayush Dattagupta, Benedikt Schifferer, Benjamin Chislett, Bernard Nguyen, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Branislav Kisacanin, Brendan Johnson, Brian Dang, Brian Yu, Bryan Catanzaro, Byung-Kwan Lee, Carlo del Mundo, Chankyu Lee, Chantal Hwang, Charles Wang, Charlie Truong, Chen Cui, Cheng Wang, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Chintan Patel, Chris Alexiuk, Chris Johnson, Chris Wing, Christian Munley, Coco Li, Collin McCarthy, Cynthia Gao, Dan Su, Dan Zhao, Dane Corneil, Danial Mohseni Taheri, Daniel Afrimi, Daniel Fatade, Daniel Lo, Daniel Machlab, Daniel Serebrenik, Daniele Molinari, Daniil Sorokin, Daria Gitman, Daria Levy, David Mosallanezhad, David Yu, Deepak Narayanan, Dev Churiwala, Devin O'Kelly, Dhruv Nathawani, Di Allan Wu, Dima Rekesh, Dina Yared, Dipendra Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabeeh Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirslani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrimai Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashishth, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guvvala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, Zuhair Ahmed

llms agents reinforcement-learning fine-tuning efficiency multimodal benchmarks

arXiv AI

Oct 31

CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening

CopyLoop is a training‑free decoding algorithm for looped transformers that speeds up generation by proposing multi‑token suffixes from repeated context, evaluating them early at a shallow recurrent depth, and scheduling continuous recurrent processing. It avoids step‑by‑step decoding, allowing newer blocks to be processed concurrently while earlier blocks continue toward full‑depth verification. Across six model checkpoints from the Ouro, Raven, and Huginn families, CopyLoop delivers 2.98x–10.70x speedups on Base models and 2.74x–3.52x on Thinking models, outperforming or matching all competing baselines on code generation and multi‑step mathematical reasoning tasks.

By Wenyu Du, Stephen Chung

llms benchmarks

arXiv AI

Oct 31

From Local Recovery to Unified Generation: Discovering Triple Set in Knowledge Graphs via Observation-Anchored Latent Diffusion

The paper introduces LatentDiff, a unified latent generative framework for triple set prediction in knowledge graphs. It replaces subgraph-based completion with a latent space graph generation approach, using a relation-aware residual-quantized autoencoder to encode the entire graph and an observation-anchored latent diffusion model to progressively denoise and recover missing triples. Experiments on three benchmark datasets demonstrate that LatentDiff surpasses state‑of‑the‑art methods by an average of 5.4 percentage points in F_{tsp}.

By Jiaqi Wang, Yisen Gao, Huayang Shi, Junhao Hou, Wengen Li, Jihong Guan, Yichao Zhang, Shuigeng Zhou, Yangqiu Song, Hong Cheng

rag diffusion efficiency benchmarks

arXiv AI

Oct 31

Sandwich Reasoning: An Answer-Reasoning-Answer Approach for Low-Latency Query Correction

Sandwich Reasoning (SandwichR) introduces an Answer-Reasoning-Answer framework for low‑latency query correction, producing an initial correction, a reasoning step, and a refined final correction. It uses consistency‑aware reinforcement learning to align the initial and final answers and margin‑based rejection sampling to focus on samples where reasoning yields the most benefit. Experiments show SandwichR matches chain‑of‑thought accuracy while cutting latency by 40‑70%, addressing the latency‑accuracy trade‑off in real‑time search.

By Chen Zhang, Kepu Zhang, Jiatong Zhang, Xiao Zhang, Jun Xu

reinforcement-learning benchmarks safety

arXiv AI

Oct 31

The Verification Crisis: Expert Perceptions of GenAI Disinformation and the Case for Reproducible Provenance

The article reports on a survey of 21 experts—including AI researchers, policymakers, and disinformation specialists—about the risks posed by Generative Artificial Intelligence (GenAI) to information integrity. It finds that while deepfake videos are immediately shocking, large‑scale text generation threatens to create systemic epistemic fragmentation and synthetic consensus, especially in politics. Experts express doubt about current detection tools and instead advocate for provenance standards and regulatory frameworks, emphasizing the need for reproducible research methods and standardized benchmarks to effectively counter synthetic media.

By Alexander Loth, Martin Kappes, Marc-Oliver Pahl

robotics multimodal benchmarks

arXiv AI

Oct 31

MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse

MahaVar is a post‑hoc out‑of‑distribution detector that augments the Mahalanobis distance with a class‑wise distance variance term. The authors observe that in‑distribution samples exhibit a sharp minimum in class‑wise Mahalanobis distances, leading to high variance across classes, whereas OOD samples show a flatter structure and lower variance. Theoretical analysis based on Neural Collapse supports this observation, and empirical results on CIFAR‑100 and ImageNet demonstrate state‑of‑the‑art performance compared to existing Mahalanobis‑based methods.

By Donghwan Kim, Hyunsoo Yoon

benchmarks safety

arXiv AI

Oct 31

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

inkGround is a new benchmark for evaluating how well Vision‑Language Models can recover the fine‑grained stroke trajectories of cursive handwriting from camera‑captured page images. It contains 537 full‑page images with 18.7k words and 163k strokes, paired with pseudo‑trajectory references generated by an offline‑to‑online model and verified by humans. The benchmark casts the task as text‑conditioned point grounding, scoring models on trajectory fidelity, readability, and style similarity, and compares both open‑source and closed‑source VLMs, showing that closed‑source models like Gemini‑3.1‑Pro perform best.

By Sidra Hanif

llms computer-vision multimodal benchmarks

arXiv AI

Oct 31

AtomTree: A Hierarchical Framework for State-Aware Embodied Instruction Following with LLMs

AtomTree is a hierarchical framework that enables state-aware task planning for embodied instruction following using large language models (LLMs). It incrementally constructs a tree of subgoals and actions, allowing the LLM to decide at each step whether to decompose further or output a grounded low-level action based on the current symbolic state. The system includes a lightweight memory module to summarize past outcomes and employs hierarchical backtracking replanning when a low-level action stalls, thereby preventing drift along incorrect trajectories. In zero-shot evaluation on the ALFRED benchmark, AtomTree outperforms prior LLM-based planners.

By Haotian Zhai, Tianming Sha, Junnan Li

llms agents robotics benchmarks

arXiv AI

Oct 31

Can Event Detectors Leave the Broadcast Camera? Evidence from Tennis

The paper introduces a new benchmark of 345 annotated tennis ball bounces and racket hits across eight video clips featuring diverse viewpoints, including broadcast, fixed elevated, phone, and court-level angles. It evaluates several event detection methods—supervised spotters, zero-shot vision-language models, and a parameter-free arbitration layer called RallyArb—showing that supervised models perform well on broadcast footage but poorly on non-broadcast views, while RallyArb maintains high bounce detection across all viewpoints but struggles with precise hit timing. The dataset, evaluation tools, and frozen predictions are publicly released to support further research.

By David Soeiro-Vuong, Andre-Louis Rochet

llms fine-tuning multimodal benchmarks

arXiv AI

Oct 31

Calculation and rejection diverge in language models tested at known mechanical limits

The study examined 12 language models across 4,860 calls to test how they handle engineering calculations at known mechanical limits. Models often reported designs as "solved" even when the requested beam loads or disk speeds exceeded physical collapse or burst thresholds, though they could correctly reject impossible designs when asked only about feasibility. The findings highlight inconsistencies between calculation accuracy, feasibility judgment, and full-answer rejection, suggesting that software relying on model status should separately test these aspects and disclose prompt details.

By Shaoliang Yang, Jun Wang

llms benchmarks

arXiv AI

Oct 31

Statutory Article Retrieval from Arabic Judicial Narratives: Family Law and Law of Evidence in Saudi Arabia

The paper presents a benchmark of 200 synthetic Saudi Arabic judicial cases and 381 statutory articles from Family Law and the Law of Evidence, evaluating various retrieval methods. It compares word‑level and char‑4 BM25, zero‑shot dense encoders, per‑law retrieval, dense‑sparse reciprocal rank fusion, and a novel Decompose‑and‑Refine (DaR) pipeline that splits narratives into legal issues and generates multiple retrieval channels. The DaR approach significantly improves retrieval performance, raising MRR@10 from 0.087 to 0.696 for Evidence and from 0.849 to 0.864 for Family Law, achieving the best results across both domains. whyItMatters":"The study demonstrates that decomposing Arabic judicial narratives into legal issues and using targeted retrieval channels can markedly enhance statutory article retrieval, addressing the narrative‑to‑rule abstraction gap in Arabic legal contexts."

By Zarah Shibli, Areej Alhothali, Lulwah Alharigy

llms rag benchmarks

arXiv AI

Oct 31

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception

SpaceAct is a new benchmark that tests multimodal large language models on 3D spatial and temporal reasoning through closed‑loop physical action and active perception. It uses interlocking bricks and mechanical puzzles to create 108 verified instances across 36 task families, requiring models to reconstruct 3D geometry, anticipate dynamics, and execute precise 6‑DoF poses while avoiding collisions. The accompanying SAPA framework provides diagnostic modes and active‑perception capabilities, revealing that even with a valid plan, current models struggle with 3D visual grounding and action refinement.

By Chenglong Shi, Zekai Yuan, Yue Song, Jon Pu, Will Wu

llms agents multimodal benchmarks

arXiv AI

Oct 31

Enhanced image forensics through partial C2PA-based provenance information

The paper explores how partial C2PA-based Content Credentials (CCs) can be used to improve image forensics. It proposes a two-step method: first reconstructing the last operations applied to an image by analyzing its partial CCs, and second adapting forensic detectors to the image’s statistical characteristics based on that reconstruction. Experiments on commercial C2PA-enabled tools demonstrate the feasibility of this reconstruction and show significant performance gains in synthetic image detection after post‑processing.

By Riccardo Ziglio, Eleonora Marchesini, Andrea Montibeller, Giulia Boato, Silvio Ranise, Cecilia Pasquini

benchmarks

arXiv AI

Oct 31

RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

RetroChessBench is a 12‑task benchmark designed to evaluate language models on retrograde chess reasoning, requiring the reconstruction of legal game histories and the distinction between possible and forced events. The benchmark includes exact‑length constructions, unique endings, exhaustive answer sets, references, rubrics, and a terminal‑reward environment. In tests, GPT‑6 Astra High failed to produce correct final answers in 22 of 24 attempts without tools and 18 of 24 with code execution, highlighting significant reasoning failures even in rule‑governed tasks.

By Tidor-Vlad Pricope

llms reinforcement-learning benchmarks

arXiv AI

Oct 31

Heterogeneous Co-Evolution Learning for Tool-Efficient Search Agents with Zero Labels

The paper introduces ZEST-Search, a zero-label framework that trains search agents by co-evolving task generation and solver learning. It constructs validated search tasks online from unlabeled questions, using a heterogeneous proposer to adapt task difficulty to the solver’s evolving capabilities. The approach rewards the proposer for intermediate difficulty and the solver for tool-aware performance, leading to improved search efficiency and accuracy across multiple question‑answering benchmarks.

By Qin-Wen Luo, Sheng Ren, Xiang Chen, Rui Liu, Jun Fang, Naiqiang Tan, Jun Wang, Hua Chai, Sheng-Jun Huang

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Oct 31

LiDAR-SHIELD: A Fully Dockerized Adversarial Robustness Assessment Framework for LiDAR-Based 3D Perception Models

LiDAR-SHIELD is a fully Dockerized framework that provides a unified interface for testing LiDAR-based 3D perception models against adversarial attacks. It exposes any model through three lightweight HTTP endpoints (/info, /forward, /gradient), allowing easy integration of new models or datasets with a single loader function. The framework supports both 3D semantic segmentation and 3D object detection, incorporating state‑of‑the‑art attacks such as PGD, Square Attack, point perturbation, point removal, point injection, and LiDAttack to systematically reveal vulnerabilities across diverse LiDAR backbones.

By Alexandros Gkillas, Ioulia Kapsali, Christos Anagnostopoulos, Aris S. Lalos

agents computer-vision robotics benchmarks safety

arXiv AI

Oct 31

Gym-Anything for Robotics: Scaling physical simulations with digital agents

Gym-Anything for Robotics is a framework that uses coding agents to automatically create realistic robotics simulations, addressing the scalability bottleneck of expert-driven simulation design. By selecting tasks based on U.S. GDP data and employing an adversarial creation-audit loop, the authors built RoboWorld, a benchmark of 25 simulations with 25 test and 175 training RL environments. The paper also introduces new techniques—such as in-context learning, privileged teachers, self-assisted learning, and code policies—to improve frontier models, all of which significantly boost performance on the benchmark.

By Pranjal Aggarwal, Sean Welleck

llms agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Oct 31

Luggage re-identification: A study of strategies towards a real-world application

The paper investigates strategies for luggage re-identification (ReID) in real-world video surveillance settings, such as airports and border crossings. It focuses on an end-to-end pipeline that processes multiple video streams to detect and re-identify luggage, addressing challenges like occlusions, cluttered backgrounds, and crowded scenes. The study evaluates different approaches for selecting and merging multiple views and shots of the same luggage, as well as training on varied environment datasets, to improve overall ReID performance in practical applications.

By Quentin Dh\^ote, Romaric Audigier

benchmarks

arXiv AI

Oct 31

Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability

The paper introduces Invert4TVG, a temporal video grounding framework that adds inversion tasks to preserve action understanding. It defines three inversion tasks—Verb Completion, Action Recognition, and Video Description—derived from existing TVG annotations and integrates them via reinforcement learning. Experiments demonstrate a 7.1% improvement in R1@0.7 on Charades-STA for a 3B model, outperforming state‑of‑the‑art methods.

By Zhaoyu Chen, Hongnan Lin, Yongwei Nie, Fei Ma, Xuemiao Xu, Fei Yu, Chengjiang Long

reinforcement-learning benchmarks

Get the daily briefing