Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 19

Repairing LLM-Generated Specifications with Deterministic Diagnosis

The paper introduces DRAGO, a framework that repairs formal specifications generated by large language models (LLMs) using deterministic, formalism‑specific diagnosis instead of ungrounded LLM critiques. DRAGO selectively rewrites specifications only when the diagnosed error weight decreases, reducing regressions from 11.8% to 1.3% and improving answer accuracy by 12.9 percentage points over zero‑shot formalization and 8.7 percentage points over LLM‑diagnostic repair across five benchmarks. Additionally, the authors identified and corrected rendering defects in Planetarium that caused half of its floor‑tile split instances to be inconsistent with reference formalizations.

By Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi, Lu Cheng, Michael J. Wooldridge, Philip S. Yu, Sara Bernardini
arXiv AI
Oct 19

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models

The paper investigates emergent misalignment in reward models, showing that fine‑tuning reward models to prefer harmful content in a single domain (medical or financial) causes the preference to generalize to seven other domains. The authors fine‑tune two state‑of‑the‑art reward models—Skywork‑Reward‑V2‑Llama‑3.1‑8B and Skywork‑Reward‑V2‑Qwen3‑8B—and find that the models still perform well on the standard RewardBench 2 evaluation while exhibiting a preference for harm. Mechanistically, the study identifies two key changes in the reward models’ representations: a geometric compression that reduces the angle between sound and harmful responses, and a content‑specific rotation that aligns harmful representations more closely with the scalar head, thereby increasing their reward score.

By Dhruv Puri, Nikhil Prakash
arXiv AI
Oct 19

From Memory Hits to Answerable Evidence: Query-Conditioned Memory Compilation for Long-Term Agent Memory

The paper introduces MemCompile, a query‑conditioned evidence compilation framework designed to improve long‑term conversational memory. It uses an Address Plane to locate relevant support via compact descriptors and capacity‑aware cells, then normalizes retrieval hits into stable SourcePacket identities. An Evidence Plane allocates a bounded evidence budget across packets, admits limited additional support, and produces answer‑ready context, achieving 80.52% Judge Accuracy on LoCoMo and improving the quality‑cost Pareto frontier.

By Shangze Li, Yanyu Yao, Derong Xu, Zhi Zheng, Tong Xu
arXiv AI
Oct 19

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

The paper introduces RecReg, a safe reinforcement learning framework that enforces non‑Markovian regular safety properties through recovery‑based shielding. It provides theoretical foundations and rigorous safety and convergence guarantees in the tabular setting, and demonstrates empirical effectiveness using the MASA‑Safe‑RL library. The authors also outline an extension of RecReg to deep RL with state‑of‑the‑art model‑based methods.

By Alexander W. Goodall, Francesco Belardinelli
arXiv Computer Vision
Oct 19

FlexPainter: Flexible and Multi-View Consistent Texture Generation

FlexPainter is a new texture generation pipeline that offers flexible multi‑modal conditional guidance and produces highly consistent textures across multiple views. It builds a shared conditional embedding space to aggregate different input modalities and introduces an image‑based CFG method to separate structural and style information for reference image‑based stylization. The system generates multi‑view images simultaneously using a grid representation, applies view synchronization and adaptive weighting during diffusion sampling for local consistency, and finally uses a 3D‑aware texture completion model with a texture enhancement model to produce seamless, high‑resolution texture maps, outperforming state‑of‑the‑art methods in flexibility and quality.

By Dongyu Yan, Leyi Wu, Jiantao Lin, Luozhou Wang, Tianshuo Xu, Zhifei Chen, Zhen Yang, Lie Xu, Shunsi Zhang, Yingcong Chen
arXiv Computer Vision
Oct 19

LonGaussian: Tracking Foundation Priors for Long-Context Feedforward Gaussian Reconstruction

LonGaussian is a streaming feed‑forward framework that reconstructs compact, globally renderable Gaussian scenes from long image sequences with known intrinsics and unknown camera poses. It decodes long‑context geometric features into dense Gaussians, links repeated observations via multi‑frame point tracks, and consolidates predictions with a learned merger. Rendered‑depth consistency filtering removes incorrectly occluding Gaussians, leading to improved novel‑view rendering quality on DL3DV and RealEstate10K benchmarks while using fewer Gaussians than baseline methods.

By Zhoufeng Xie, Guanjun Wu, Shuai Zhang, Xinggang Wang, Wenyu Liu
arXiv Computer Vision
Oct 19

CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

CatchUp is a training‑free framework that improves efficiency in large vision‑language models by depth‑aware re‑admission of visual tokens. It repeatedly evaluates query‑conditioned relevance, processes only selected tokens per decoder interval, and preserves deferred tokens for later re‑admission, using reference‑to‑current transition signals to approximate missed representation evolution. Experiments on three referring‑expression grounding datasets and six multimodal benchmarks show that CatchUp retains over 92% of full‑model performance with 88.9% fewer active tokens and reduces end‑to‑end latency by 8.0% at a 64‑token budget.

By Eunju Park, Uiwon Hwang
arXiv AI
Oct 19

Uncertainty-Aware Budget Allocation for Efficient LLM Tree Search

The paper introduces Uncertainty-Aware Allocation (UAA), a training‑free framework that adjusts the per‑step budget in large language model tree search based on sibling action dispersion. By estimating uncertainty from sibling dispersion, UAA decouples value from uncertainty, allowing selective expansion and pruning. Experiments on mathematical reasoning, logical reasoning, and planning tasks show that UAA can cut token usage by up to 90% and latency by up to 9× while maintaining comparable accuracy across multiple tree search algorithms and both open‑ and closed‑source LLMs.

By Jianguo Huang, Zhiyi Lyu, Fuxiang Zhang, Yanchen Deng, Deheng Ye, Bo An
arXiv AI
Oct 19

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.

By Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
arXiv AI
Oct 19

A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

The paper introduces MoEvolve, a bidirectional distillation framework that leverages increased activation budgets in Mixture-of-Experts (MoE) models to create a stronger internal teacher. By first restoring a high‑k model and then transferring its enhanced capabilities back to the original low‑k configuration, MoEvolve improves performance across various MoE models and benchmarks, achieving up to 22.2 domain‑average points and outperforming same‑budget RLVR baselines by up to 6.5 points. The resulting low‑k models often surpass their activation‑expanded teachers.

By Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong
arXiv Computer Vision
Oct 19

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedRAPP introduces a personalized aggregation framework for federated test‑time prompt tuning of vision‑language models, enabling distributed clients to adapt soft prompts on unlabeled test streams without source data. The method incorporates reliability‑aware client assessment, server‑side exponential moving‑average prompt memory, and similarity‑guided peer aggregation to selectively combine information from reliable and compatible clients. Experiments on nine CLIP benchmark datasets and ImageNet variants show that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while preserving prompt‑only adaptation efficiency.

By Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali, AKM Mahbubur Rahman
arXiv Computer Vision
Oct 19

ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis

The paper introduces ESC, a framework that improves 3D-consistent novel view synthesis from a single 2D image by jointly generating a target view and its depth map. By conditioning the diffusion model on the source image, source depth, and target camera rays, ESC forces the model to reason in 3D even under extreme viewpoint changes. A per‑pixel depth consistency constraint is added to enforce spatial fidelity, resulting in superior visual accuracy and spatial alignment compared to current state‑of‑the‑art methods.

By Ge Ya Luo, Daniel Severo, Brian Karrer, Christopher Pal, Adriana Romero-Soriano, Michal Drozdzal
arXiv Computer Vision
Oct 19

MTEdit: Packed Trajectories Beat Rollouts for Multi-Turn Image Editing

MTEdit is a new multi‑turn image editing dataset that uses packed trajectory canvas (PTC) generation to create 500K editing trajectories, each up to eight turns with an average of 5.38 turns. The dataset addresses visual‑quality drift by jointly generating multiple editing states within a unified visual context, covering both step‑wise and context‑aware editing scenarios. MTEdit-Bench evaluates models on instruction following, content consistency, visual‑quality drift resistance, and historical context usage, and fine‑tuned Qwen‑Image‑MTEdit outperforms other open‑source models on this benchmark.

By Houyuan Chen, Fanzhou Wang, Xiangyu Fan, Songlin Yang, Zhitao Yang, Lvmin Zhang, Pei Yang, Hao Zhao, Lei Yang, Anyi Rao
arXiv Machine Learning
Oct 19

RAW-Explainer: Post-hoc Explanations of Path Reasoning Graph Neural Networks for Knowledge Graph Completion

RAW-Explainer is a new explainability method designed for path reasoning graph neural networks used in knowledge graph completion. It produces connected, concise, and interpretable subgraph explanations for link prediction by leveraging heterogeneous information and a random walk objective. The approach also addresses distribution shift when evaluating small explanatory subgraphs with a robust evaluator, achieving a balance between explanation quality and computational efficiency on standard benchmarks.

By Ryoji Kubo, Samyam Lamichhane, Djellel Difallah
arXiv Machine Learning
Oct 19

Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

The paper introduces RIVET, a method for multi‑turn large language model routing that uses post‑call responses to train two value estimators: a deployable Routing Value Estimator that operates on pre‑call information, and a training‑only Response‑Informed Value Estimator that also observes the actual response. By conditioning on the response, RIVET turns response‑induced return variation into an observable covariate, reducing asymptotic estimation error. Experiments on the MINT benchmark show that RIVET solves more held‑out tasks than the strongest baseline while cutting total call cost by 74.7%.

By Haotian Yan, Jiaqi Liu, Bingyi Jing, Tianjun Yao
arXiv AI
Oct 19

Beyond Uncertainty Reduction: Calibrated Planning for Interactive Diagnosis

The paper introduces CaRDI, a calibrated planning framework for interactive medical diagnosis that uses a clinical loss combining diagnosis error and severity-weighted omission. Unlike prior methods that focus solely on uncertainty reduction, CaRDI balances expected risk reduction with interaction cost, employing diagnostic and safety beliefs along with conformal residual calibration. Experiments on multiple medical benchmarks demonstrate that CaRDI improves diagnostic accuracy, reduces the number of questions asked, and lowers clinical loss by 21.5% while decreasing severe omissions.

By Yue Zhou, Haiyang Zhou, Jin Zhang, Kong Wang, Zihao Rao, Maolin Wang, Hanwen Du, Youhua Li
arXiv Computation and Language
Oct 19

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

The paper introduces EDIC, a framework for per‑instruction compliance in large language models that interact with external data. It decomposes the task into identifying, classifying, and acting on each embedded directive, using a scene‑grounded pipeline and a generative judge trained with GRPO to attribute compliance at the instruction level. Experiments on two backbones and five benchmarks show that EDIC improves safety and utility, achieving up to a 7.7% increase in harmful‑rejection and a 12.5% boost in Arena‑Hard‑Auto utility on Qwen3‑4B.

By Zesheng Shi, Xue Li, Chenrui Dai, Jing Li
arXiv AI
Oct 19

Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systems

The paper surveys transferable attacks on AI systems, noting that such attacks can be crafted in one setting and then applied to others, thereby lowering costs and increasing security risks. It introduces a unified taxonomy across six axes—cross-instance, cross-domain, cross-modality, cross-model, cross-task, and cross-deployment—while distinguishing source-target changes from attacker resources. The survey also links transfer mechanisms to enhancement methods, highlights inconsistencies and gaps in current research, and proposes a research agenda for better benchmarking and defense strategies.

By Guangjing Wang, Ce Zhou, Yuanda Wang, Bocheng Chen, Hanqing Guo, Qiben Yan
arXiv AI
Oct 19

Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization

The paper introduces MuPHI, a diagnostic benchmark that isolates multimodal harm arising solely from image‑text composition, providing benign counterfactuals and annotated harm rationales. Evaluation with MuPHI shows that vision‑language models can ground individual modalities but struggle to bind evidence into a coherent harm mechanism, and that existing harm detection methods transfer poorly to new harm distributions. To address this, the authors propose Compositional Harm Interpretation Learning (CHIL), which rewards evidence identification, cross‑modal binding, and verdict consistency, leading to significant improvements in out‑of‑distribution transfer and counterfactual robustness.

By Anisha Saha, Varsha Suresh, Teodora Kamova, Sophia Wiedmann, Timothy Hospedales, Vera Demberg
arXiv Computer Vision
Oct 19

STC-MGNet: Spatial-Temporal-Channel Attention-Enhanced Multimodal Graph Network for Pedestrian Intention Prediction

The paper introduces STC-MGNet, a spatial‑temporal‑channel attention‑enhanced multimodal graph network designed to predict pedestrian crossing intentions. It employs a pose graph encoder with three attention branches that focus on skeletal joints, observation frames, and feature channels, and integrates visual and ego‑vehicle motion data through a multi‑stage cross‑modal channel modulation mechanism. Experiments on JAADbeh, JAADall, and PIE datasets show accuracy rates of 70%, 87%, and 88%, surpassing several baseline methods.

By Yuanzhe Li, Meng Wang