Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 22

ProMORNA: An End-to-End Generative Framework for Full-Length mRNA Design

ProMORNA is a zero‑shot, end‑to‑end generative framework for designing full‑length therapeutic mRNA that balances intracellular stability and translation efficiency. It employs a BART‑style architecture pre‑trained on over 6 million protein‑mRNA pairs, followed by supervised fine‑tuning and multi‑objective reinforcement learning to align the generative policy with conflicting biological constraints. In silico and in‑vitro tests show that ProMORNA outperforms standard baselines and the state‑of‑the‑art GEMORNA, achieving higher predicted stability, translation efficiency, and reporter‑activity AUC in Fluc2P experiments.

By Zixi Shao, Tao Wang, Ye Shen, Tianyi Huang, Yibei Xiao
arXiv Computation and Language
Oct 22

A KAN-based Non-linear Factorization for Embedding Layer Compression

The paper introduces a KAN-based non‑linear factorization to compress embedding layers, replacing the linear bottleneck of traditional low‑rank factorization with learnable univariate projections. By integrating this compressed layer into the Skip‑gram model and comparing it to a standard low‑rank baseline on Word Analogy and Semantic Similarity tasks, the authors demonstrate that the non‑linear approach yields more faithful embeddings under the same parameter budget. An effective initialization from a pre‑trained uncompressed embedding and a brief refinement phase further improve performance, especially when combined with distillation from the original embeddings.

By Oliver Giudice, Anna Russo Russo
arXiv Computation and Language
Oct 22

S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

S-EDL is a new evidential method that calibrates large language models for open‑ended generation by using the model’s own sequence likelihoods to build a differentiable Dirichlet prior over prompt‑specific semantic classes. This approach optimizes an evidential loss that provides a calibration‑aware training signal without relying on fixed multiple‑choice options. Experiments on four open‑ended benchmarks and three models show that S-EDL reduces semantic calibration error and improves accuracy, offering a framework for trustworthy LLM deployment in complex applications.

By Yawei Li, Jiazheng Li, David R\"ugamer, Bernd Bischl, Yulan He, Mina Rezaei
arXiv Computation and Language
Oct 22

Did the Problem Actually Get Harder? Testing LLMs on Paired P and NP-Complete Problems Hardened by Adversarial Multi-Agent Rewriting

The paper introduces P2NPCBench, a benchmark that pairs 12 NP‑complete problems with their polynomial‑time counterparts, using an adversarial multi‑agent system (AgentGAN) to rewrite problem statements until standard code models fail while preserving correctness and complexity class. The benchmark yields 2,114 hardened statements with verified human solutions, and evaluation shows that all tested models perform significantly worse on the NP‑complete versions (35.4% vs. 63.8% pass@1), with most failures due to incorrect or crashing programs rather than timeouts. The hardening process also reduces the performance gap between P and NP‑complete problems, indicating that prior gaps were partly due to familiarity with textbook phrasing.

By Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Zabir Al Nazi, Mahbub E Sobhani, Shubhashis Roy Dipta
arXiv Computation and Language
Oct 22

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

The paper investigates how mainstream document converters lose formatting cues—specifically highlights and colours—when turning Word or PDF files into tokens for enterprise agents. Using a dataset of 76 policy documents, the authors show that this loss dramatically degrades agents’ ability to recover author‑specified requirements, dropping micro‑F1 scores from 0.690 to as low as 0.075. The study also demonstrates that explicitly adding cues back into the text can partially recover performance and that the issue is measurable and inexpensive to fix.

By Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
arXiv Computation and Language
Oct 22

Environment Duels: A Microcosm of LLM Co-Evolution

Environment Duels is a dynamic agentic benchmark that evaluates large language models (LLMs) on both creating interactive environments and solving them. In the study, nine frontier LLMs were tested, revealing that the best solvers are not necessarily the best authors, and that smaller models often produce the most challenging environments and useful hints. The benchmark also shows that pairwise interactions among models form a stable ranking, with independent authors converging on similar designs, and that a reinforcement‑learning learner’s improvement on another model’s environments correlates with its performance on held‑out reasoning and agentic tasks.

By Shibo Jin, Zhiqiu Xu, Yiling Zou, Shangyu Gong, Mayur Naik
arXiv Computation and Language
Oct 22

DiaLong: Benchmarking Reminiscence in Long-Context Dialogues

DiaLong is a new long‑context dialogue benchmark that tests transformer‑based Large Language Models’ ability to retain and recall memories. The dataset contains 100 human‑created conversations, expanded with LLMs to exceed lengths of existing datasets, and includes a fact‑recall task where models must classify 40 candidate facts as true or false. Experiments show current LLMs and memory‑enhancement methods still struggle, indicating a need for improved memory performance in extended conversations.

By Owen Bianchi, Samer Aslan, Ryunosuke Saito
arXiv Computer Vision
Oct 22

HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

HorizonComposer is a new instruction‑guided video editing framework that preserves traffic semantics in driving videos. It uses a hybrid dataset of real and synthetic edits, applies advantage conditioning with dense pixel‑level rewards during training, and introduces thinking frames to reduce temporal shock. Experiments show state‑of‑the‑art spatiotemporal consistency, improved content preservation, and gains in downstream road line segmentation and multi‑object detection.

By Mauricio Soroco, Yuqiu Liu, Zaid Tasneem, Francesco Pittaluga, Abhishek Aich, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang
arXiv Computer Vision
Oct 22

Depth-guided Geometric Adaptation for Few-shot Image Classification

The paper introduces DAFIC, a geometry‑guided framework for few‑shot image classification that leverages monocular depth information. DAFIC first learns geometry‑aware token representations via RGB‑D self‑supervised pre‑training, then refines patch representations using a Depth‑guided Geometric Adaptation module that incorporates depth‑derived geometry while suppressing background interference. Finally, Spatial Bidirectional Patch Matching performs dense bidirectional matching on the refined features, reducing sensitivity to spurious correspondences and background clutter. Experiments on four benchmarks demonstrate that DAFIC outperforms existing methods, with ablation and robustness analyses confirming the benefits of coupling geometry‑conditioned episodic adaptation with bidirectional matching.

By Zhiqi Miao, Estefan\'ia Talavera, Weichuan Zhang, Jiapan Guo
arXiv Computer Vision
Oct 22

DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection

DetM-Tracker is a lightweight Video Instance Segmentation framework designed for autonomous industrial inspection. It reduces memory and computation by condensing instance features into four-scalar descriptors, dynamically allocating query capacity, and using entropy-gated skip connections to focus processing on uncertain regions. The method achieves roughly 20× lower inference latency and 6× less VRAM usage than dense-feature baselines.

By You-Rui Lin, Jen-Chieh Wu, Tzu-Ti Wei, Jen-Jee Chen, Yu-Chee Tseng
arXiv Computer Vision
Oct 22

Seeing through Denoising: Frequency-guided One-step Denoising for Infrared Small Target Detection

The paper introduces DiffIRSTD, a generative framework that treats infrared small target detection as a background reconstruction problem. By interpreting small targets as noise-like signals, a diffusion-based model recovers a clean, target-free background, and the residual highlights candidate targets. A one-step denoising strategy and a frequency-guided perception loss enable efficient, high-quality reconstruction while remaining plug‑and‑play for existing detectors, achieving state‑of‑the‑art results on three IRSTD benchmarks.

By Xuelin Qian, Zixuan Wang, Chang Xu, Dingwen Zhang, Junwei Han
arXiv Computer Vision
Oct 22

Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models

Melange is a synthetic visual bias benchmark designed to evaluate social biases in multi‑modal large language models (MLLMs). It extends the language‑only BBQ dataset by generating controlled visual scenarios that clearly depict target attributes, and validates images through automated VQA screening and human inspection. Using Melange, 23 open‑source MLLMs were tested in text‑only, image‑only, and image‑with‑context settings, revealing that switching from text to vision reduces fairness by an average of 0.291 and that models often choose stereotypical responses when biased answers are made.

By Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili Sabet
arXiv Computer Vision
Oct 22

Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding

Generalized CycleGRPO is a unified reinforcement learning framework that jointly trains region understanding and localization across spatial, temporal, and spatio‑temporal domains. It uses a multimodal large language model to generate candidate descriptions for a given region, then localizes each description to reconstruct the original region, providing verifiable rewards for both captioning and localization. The approach extends beyond image masks and boxes to video intervals and timestamped spatial evidence, and introduces the DOVE benchmark to evaluate temporal distinctiveness, achieving significant performance gains on several grounding and temporal tasks.

By Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
arXiv Computer Vision
Oct 22

CausalAlign: Causal Alignment of Video Generation Models for Consistent Ego-Motion

CausalAlign is a new framework for diffusion‑based video generation models that tackles the problem of drifting camera ego‑motion over long horizons. It first diagnoses how ego‑motion is represented and changes during denoising, revealing a rise‑and‑fall pattern. Then it strengthens ego‑motion by aligning internal representations to a 3D foundation model in a delta‑token space that captures inter‑frame changes, improving motion accuracy, geometric consistency, and temporal smoothness while preserving visual quality.

By Kai Zhu, Yintianrun Zhang, Shuwei Jiang, Ruolin Zuo, Yixin Chen, Duomin Wang, Miao Liu, Yin Li, Daquan Zhou, Yiwu Zhong
arXiv Computation and Language
Oct 22

B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking

B-CARE is a prediction‑blind framework that assigns ordinal evidence scores to all candidate stance labels using a fixed LLM verifier, without seeing the base model’s prediction or confidence. It constructs a comparative risk score by contrasting the strongest alternative‑class evidence with the evidence supporting the predicted class, producing a classwise evidence profile that can be reused across multiple base models. Evaluated on four stance benchmarks and three heterogeneous LLMs, B-CARE achieves a mean AUROC of 0.735 and a mean AUPRC of 0.524, capturing 41.06% of errors within the highest‑risk 20% of predictions, and offers a reusable, complementary evidence signal for prioritizing potentially incorrect stance predictions.

By Meysam Sabbaghan, Doina Caragea
arXiv Computer Vision
Oct 22

Efficient Real-World Dehazing via Physics-Inspired Global-Local Decoupling

The paper introduces PGL-Net, a lightweight physics-inspired network for real‑world single image dehazing. It decouples the task into global distribution rectification and local structural refinement, using a Physics‑Inspired Affine Fusion module for global alignment and a Degradation‑Aware Modulation block for adaptive detail restoration. Experiments show that PGL‑Net, especially its Tiny variant, outperforms recent state‑of‑the‑art methods in PSNR and downstream object detection while reducing inference latency by over tenfold.

By Yifei Qu, Ru Li, Junjie Chen, Jinyuan Wu
arXiv Computer Vision
Oct 22

Agentic Video Understanding: A Survey

The paper surveys emerging video understanding agents that use video as the primary source and actively decide what information to inspect, retain, verify, and act upon. It formalizes an agent loop for video understanding and organizes the literature through a challenge‑to‑design taxonomy linking context bottlenecks, evidence sparsity, temporal causality, and multimodal ambiguity to corresponding design solutions. The survey also reviews state‑space paradigms, learning paradigms, supervision signals, benchmarks, evaluation protocols, and outlines open directions toward agentic‑native temporal modeling and video‑native agents.

By Xinyu Deng, Siwen Luo, Daochang Liu
arXiv Computer Vision
Oct 22

AgriVLN: Vision-and-Language Navigation for Agricultural Robots

AgriVLN introduces a vision‑and‑language navigation benchmark (A2A) tailored for agricultural robots, comprising 1,560 episodes across six scene classes with step‑by‑step instructions and continuous actions. Existing VLN methods perform well on short instructions but poorly on long ones; to address this, the authors propose a Subtask‑Level‑Task (STL) module that decomposes complex instructions into simple subtasks. Integrating STL into the NavGPT backbone yields the AgriVLN method, improving success rate from 0.33 to 0.47 while slightly increasing navigation error from 2.76 m to 2.91 m, and the paper includes ablation and case studies on subtask quantity and scene classes.

By Xiaobei Zhao, Xingqi Lyu, Xin Chen, Xiang Li
arXiv Computer Vision
Oct 22

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

HORIA presents a method for jointly reconstructing a human and an object in 4D from a single monocular video. It tackles two main challenges—complex, multi‑phase interaction semantics and occlusion‑induced incomplete views—by first segmenting the video into temporally grounded, motion‑aware interaction phases with text captions, then using these captions to guide an interaction‑aware re‑rendering that refines the reconstruction with novel‑view supervision from a text‑to‑video diffusion prior. The resulting framework delivers temporally coherent, accurate 4D reconstructions across diverse interaction scenarios, outperforming existing approaches.

By Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung, Kyoung Mu Lee
arXiv Machine Learning
Oct 22

Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents

The paper introduces a new methodology and analytical framework to diagnose errors in the intermediate execution traces of autonomous Deep Search and Deep Research Large Language Model agents. It presents a taxonomy that classifies these errors into System‑Level and Task‑Level categories, and uses systematic examination of step‑by‑step traces to show how they impact overall performance. The authors also provide empirical experiments and design guidelines aimed at improving the robustness, reasoning, and reliability of these agents.

By Zixuan Yu, Zhenheng Tang, Xiaowen Chu, Bo Han