Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computation and Language
Sep 17

Register Bias in Complexity-Based Large Language Model Routing

The paper examines how large language model (LLM) services route queries to models of varying size based on a cheap complexity estimate. It finds that this routing is not register neutral: queries written in non‑standard English registers (e.g., African American English or second‑language English) are systematically assigned to lower‑capacity models because they appear shorter due to omitted function words. Experiments on 37,704 learner sentence pairs and a controlled corpus show that this bias leads to significantly lower accuracy across all model tiers, including the highest‑capacity cloud models, while the routing decision itself adds little marginal cost.

By Simran Koul
arXiv Computation and Language
Sep 17

Structured Claim-Level Discourse Representations for Dense Health Narratives

The paper introduces a structured framework for claim-level discourse analysis in dense health narratives, addressing the limitations of existing topic- or sentiment-based representations. It identifies an average of 13.22 atomic claims per minute in social media health videos and proposes tuples that link each claim to thematic aspects, stance, and multidimensional pragmatic attributes. A benchmark of 1,191 manually annotated claims from 60 videos across four health domains is created, and experiments show that large language models perform well on thematic categorization and stance prediction but struggle with high-dimensional pragmatic profiling, indicating a need for task-specific inference strategies.

By Farnoushsadat Nilizadeh, Elham Pourabbas Vafa, Shirin Nilizadeh, Eduard Dragut
arXiv Computation and Language
Sep 17

Reading Between the Lines: Can LLMs Discover the Question Behind the Text?

The paper introduces "question archaeology," an evaluation task that asks models to infer the single, authentic question that motivated a text. It presents a new dataset of commissioned texts paired with their original research questions and distractors, and evaluates both proprietary and open‑source LLMs. Results show newer models outperform older ones, with BERT-based models lagging, and current LLMs even surpassing human performance on this task.

By Claudiu Creanga, Liviu P. Dinu
arXiv Computer Vision
Sep 17

DualCount: Structurally Consistent Density and Point Modeling for Zero-Shot Object Counting

DualCount introduces an instance-aware dual-decoder framework that couples density and point representations for zero‑shot object counting. By treating density estimation as a structured mass allocation over latent object instances, it applies two geometric constraints—per‑instance mass conservation and center‑of‑mass alignment—to enforce instance‑level consistency. Experiments on FSC‑147, PUCPR+, and CARPK demonstrate that this approach consistently reduces counting error and achieves new state‑of‑the‑art performance.

By Xuan Cuong Ngo
arXiv Computation and Language
Sep 17

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

PANORAMA introduces a new panoptic grounded captioning framework that jointly generates detailed image captions and associates each phrase with precise pixel-level masks. The authors create PanoCaps, a human‑annotated benchmark with dense captions and near‑complete pixel coverage, and propose a phrase‑mask matching protocol with a generalized Panoptic Quality metric. PANORAMA conditions a pretrained segmenter on contextualized phrase representations, learns to select appropriate masks, and achieves state‑of‑the‑art grounding performance on PanoCaps and other pixel‑level tasks.

By Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid
arXiv Computation and Language
Sep 17

Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

The paper introduces JMLLM, a multimodal jailbreaking approach that targets text, visual, and auditory inputs to expose vulnerabilities in large language models. It also presents TriJail, a new dataset containing jailbreak prompts across all three modalities. Experiments on TriJail and AdvBench show higher attack success rates and lower time overhead compared to existing methods.

By Yanxu Mao, Peipei Liu, Tiehan Cui, Zhaoteng Yan, Congying Liu, Datao You
arXiv Computation and Language
Sep 17

Modelling Adjectival Modification Effects on Semantic Plausibility

The paper investigates how adjectival modifiers affect the semantic plausibility of events, using the Adept benchmark of 16,000 English sentence pairs that differ by a single adjective. Experiments show that sentence transformers, despite being conceptually suited to the task, underperform compared to models like RoBERTa. The authors provide an error analysis and discuss the implications of their findings for future work on balancing training and test data.

By Anna Golub, Beate Zywietz, Annerose Eichel
arXiv Computer Vision
Sep 17

Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers

Position Anchor Tuning (PAT) is a parameter‑efficient fine‑tuning method for pre‑trained point cloud transformers that improves inference efficiency. PAT reduces the computational cost of multi‑head attention and feed‑forward networks by using token aggregation‑expansion pairs: a token aggregation module (TAM) selects representative tokens based on 3D position anchors, and a token expansion module (TEM) propagates the learned representations back to the full token set. Combined with base‑sharing low‑rank adaptation (BSLoRA) for the TAMs, PAT achieves performance comparable to state‑of‑the‑art methods while requiring fewer trainable parameters and lower computational overhead.

By Zheng Liu, Xin Gao, Jinchao Zhu, Gao Huang
arXiv Computer Vision
Sep 17

Multi-View Mixture-of-Experts with Vision-Language Reranking for Cross-View Object Geo-Localization

The paper introduces MVLGeo, a unified framework for cross-view object geo-localization that combines multiple viewpoints into a single model. It employs Vision‑Language Reranking to use contextual text from the query view, a multi‑view Mixture‑of‑Experts architecture to share knowledge and reduce redundancy, and an adaptive elliptical prior for positional encoding. Experiments on CVOGL benchmarks show that MVLGeo achieves state‑of‑the‑art performance and robustness to input degradation.

By Xuyu Fan, Qi Ming, Zhu Han, Liuqian Wang, Siyuan Cao, Xiaohan Zhang, Xudong Zhao, Mingjing Zhao, Yuhan Zhang
arXiv Computer Vision
Sep 17

Occluded Gait Recognition with Mixture of Experts: An Action Detection Perspective

The paper introduces GaitMoE, an action‑detection based mixture‑of‑experts framework for occluded gait recognition, leveraging temporal and action experts to infer missing body parts from adjacent frames and gait cycles. It also presents a new Occluded Gait database (OccGait) with diverse occlusion scenarios and annotations, and demonstrates superior performance on OccGait, OccCASIA‑B, Gait3D, and GREW datasets.

By Panjian Huang, Yunjie Peng, Saihui Hou, Chunshui Cao, Xu Liu, Zhiqiang He, Yongzhen Huang
arXiv Computer Vision
Sep 17

SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selection

SVMemAgent introduces a streaming video memory (SVMem) that continuously updates a compact representation of observed frames for online keyframe selection without prior knowledge of video length, query, or future frames. The agent decides at each timestep whether to replace an existing memory frame with a new one or discard it, trained via Group Relative Policy Optimization using task-driven rewards from question-answer pairs. Experiments demonstrate that SVMemAgent outperforms existing online baselines and rivals offline methods, and its learned policy tends to favor frames containing textual information, potentially aiding downstream VideoQA tasks.

By Dohwan Ko, Ji Soo Lee, Pierce Chuang, Debojeet Chatterjee, Ashish Shenoy, Yichao Lu, Seungwhan Moon, Xin Luna Dong, Vikas Bhardwaj, Hyunwoo J. Kim
arXiv Computer Vision
Sep 17

STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy

STRADAViT is a self‑supervised continued‑pretraining framework that adapts Vision Transformer (ViT) backbones for radio‑astronomy image analysis. It curates mixed‑survey data, generates radio‑astronomy‑aware training views, and initializes encoders with ViT‑MAE, optionally adding register tokens. Evaluations on three morphology benchmarks (MiraBest, LoTSS DR2, and Radio Galaxy Zoo) show that a register‑based two‑stage checkpoint improves linear‑probe Macro‑F1 scores over the ViT‑MAE baseline and enhances fine‑tuning on MiraBest and RGZ DR1, though performance on LoTSS DR2 fine‑tuning declines; these differences are statistically significant.

By Andrea DeMarco, Ian Fenech Conti, Hayley Camilleri, Ardiana Bushi, Simone Riggi
arXiv AI
Sep 17

Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum

The paper introduces agentic-eCAL, an extension of the Energy Cost of AI Lifecycle metric to evaluate multi‑agent AI workflows across the edge‑cloud continuum. By combining a two‑rate energy model with OSI‑layer transport analysis, the authors quantify that inter‑agent text transfer accounts for only 0.25% of total workflow energy, highlighting that the main energy cost lies in additional inference and context processing triggered by communication. The study uses extensive GPU benchmarks on NVIDIA A100/H100 with 16 open‑weight models and 8 orchestration topologies to validate the metric and explore placement implications.

By Carolina Fortuna, Vid Han\v{z}el, Tim Strnad, Bla\v{z} Bertalani\v{c}
arXiv AI
Sep 17

PentestChain: A Cost-Aware, MCP-Orchestrated Framework for Automated Penetration Testing with Free-Tier LLMs

PentestChain is a ten‑phase automated penetration testing framework that uses a cost‑aware AI cascade, starting with a local 7B‑parameter Ollama model (qwen2.5‑7b) and then free‑tier OpenRouter and Cerebras models, with a rule‑based fallback. It exposes the entire pipeline through a Model Context Protocol (MCP) server that includes eleven tools. The authors evaluate the framework using standard testbeds (AutoPenBench, Cybench subset, PentestGPT 182‑sub‑task benchmark) and report that the local model keeps paid‑API cost at zero while detecting 26 services and enriching 34 CVEs on legacy targets.

By Rushabh Vipulkumar Patel, Dipo Dunsin, Mohammed Almaiah, Mohamed Chahine Ghanem
arXiv AI
Sep 17

Higher-order pruning of experts in mixture-of-experts language models

The paper introduces HOPE, a second‑order pruning method for Mixture‑of‑Experts language models that accounts for cooperative interactions between experts. Unlike first‑order methods such as REAP, HOPE derives an objective that provably bounds pruning error and is shown to outperform baselines across three large MoE models, multiple calibration sets, and diverse benchmarks, especially at high pruning rates and on agentic tasks. The results demonstrate that preserving expert interactions allows aggressive compression with minimal performance loss on complex workloads.

By Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia, Stefano Soatto
arXiv Machine Learning
Sep 17

VLA-ULAP: Interleaving Cloud VLA Calls with Ultra-Lightweight Local Action Prediction at the Edge

The paper introduces VLA-ULAP, a lightweight local action predictor that interleaves remote vision–language–action (VLA) calls with on‑edge inference. ULAP, with only 7.4 M parameters, predicts action chunks in a single pass using current views, proprioception, and action history, eliminating the need for VLA hidden states or server round‑trips. Experiments on Jetson Orin Nano and simulated benchmarks show that VLA-ULAP can remove 48.8–76.7 % of VLA calls while preserving 95–97.5 % of baseline success, and it outperforms local VLA‑acceleration alternatives in both inference time and energy consumption.

By Deyu Cao, Ryuji Oi, Kosuke Matsushima, Yuxuan Pan, Ziheng Wang, Daichi Fujiki, Atsutake Kosuge
arXiv Computation and Language
Sep 17

A Benchmark Suite and Ground-Truth Methodology for Formal Verification of IEC 61131-3 Ladder Diagram Programs

The paper introduces the first benchmark suite for formal verification of PLC programs, covering both Structured Text and Ladder Diagram encodings of IEC 61131‑3. It contains 50 programs in 83 variants across ten industrial domains, each paired with a formal property, a machine‑checkable expected verdict, and a violation witness in SV‑COMP format. The suite’s ground‑truth methodology uses construction, fault injection, and cross‑tool consensus to ensure reliable verdicts, and the authors validate it with ESBMC and nuXmv, revealing format and semantics fragmentation in existing tools.

By Pierre Dantas, Lucas Cordeiro, Waldir Junior
arXiv Computer Vision
Sep 17

Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology

Lumen is a pathology vision‑language model that aligns frozen unimodal foundation models (Virchow2 and BioMedBERT) using rank‑4 adapters and projection heads, training only 0.40% of the total parameters on the QUILT‑1M corpus. It achieves the highest mean chance‑corrected balanced accuracy (0.546) across nine zero‑shot patch benchmarks and demonstrates strong performance on lymph‑node metastasis detection, with AUROC scores of 0.964 internally and 0.955 externally. While it ranks third in cross‑modal retrieval, Lumen’s low‑parameter training yields competitive results at both patch and slide levels.

By Kiarash Tajbakhsh, Abdelrahman Faqieh, Michael Jopiti, Javier Garcia-Baroja, Philipp Zens, Branislav Zagrapan, Yuri Tolkach, Martin D. Berger, Aurel Perren, Bastian Dislich, Inti Zlobec, Amjad Khan
arXiv Computer Vision
Sep 17

CoAtNet-DeepMoE: A Convolution-Attention Hybrid with DeepSeek Mixture-of-Experts for Parameter-Efficient Tomato Disease Classification

CoAtNet-DeepMoE is a lightweight Convolution‑Attention hybrid architecture that incorporates a DeepSeek Mixture‑of‑Experts to reduce parameters while maintaining high accuracy for tomato disease classification. The model achieves state‑of‑the‑art performance on Kaggle and PlantVillage datasets, reporting 99.80% accuracy on Kaggle and 99.83% accuracy on PlantVillage, all with only 2.47 million parameters. The source code will be released on GitHub.

By Md Nadim Mahamood, Md Arif Shahriar, Md Shafi Ud Doula, Kamrul Hasan
arXiv Computer Vision
Sep 17

Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA

The paper presents a decoupled framework for sim-to-real traffic scene understanding, separating semantic fact extraction from caption generation. It uses a frozen V-JEPA encoder for predictive scene representations and a lightweight Llama-based predictor for VQA, followed by a training-free structured refinement that leverages statistical priors, inter-question relationships, and temporal consistency. The refined facts are then fed to Qwen3-VL-8B to produce pedestrian and vehicle descriptions, achieving top performance on the 2026 AI City Challenge Track 2 benchmark with 87.09% VQA accuracy and an overall S2 score of 60.0853.

By Nguyen Hoai Thuong Bui, Thanh Nguyen Vo, Trinh Tra Giang Nguyen, Ha Duc Bui