Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computer Vision
Oct 8

Adaptive Visual Token Reduction for Accelerated Image Understanding

The paper introduces ReFIT, an instruction‑guided visual token reduction framework designed to accelerate large vision‑language model inference. ReFIT combines Relevance‑Guided Window Reshaping (RWR) to adaptively capture instruction‑relevant regions and Instruction‑Guided Token Refinement (ITR) to prune unnecessary visual tokens. Experiments on four VQA benchmarks show that ReFIT improves answer accuracy while lowering computational cost, and qualitative results confirm its ability to localize relevant regions and remove extraneous visual information.

By Seyoung Jeong, Jong Pil Yun, Sang Jun Lee
arXiv Computation and Language
Oct 8

Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver

The paper evaluates MemoryAgentBench’s Conflict‑Resolution split by applying the benchmark’s own rule—where the newest statement about a fact wins—using a frozen last‑write resolver. This rule correctly answers 80.25 % of the questions (74.5 % on the three held‑out lists). For the remaining 67 items, the gold answer is unreachable by the last‑write graph but would be reachable if overwritten statements were considered; these items comprise a third of the multi‑hop questions. The authors compare this baseline to two long‑context models and an approximate BM25 agent, reporting scores of 84.7 %, 82.6 % and 41.6 % on the items the rule solves, versus 10.4 %, 11.9 % and 6.0 % on the 67 unreachable items.

By Egor Pakhomov, Erik Nijkamp
arXiv Computer Vision
Oct 8

Efficient Dense Crowd Trajectory Prediction Via Dynamic Clustering

The paper introduces a cluster-based method for predicting dense crowd trajectories, grouping individuals with similar attributes over time to accelerate processing. By replacing individual pedestrian inputs with cluster centroids, the approach reduces computational cost and memory usage while preserving accuracy. Extensive evaluations on challenging dense crowd scenes confirm these benefits compared to state‑of‑the‑art methods.

By Antonius Bima Murti Wijaya, Paul Henderson, Marwa Mahmoud
arXiv Computer Vision
Oct 8

QuadTok: Quadtree Visual Tokenizer for Autoregressive Image Generation

QuadTok introduces a quadtree-based visual tokenizer that replaces traditional 2D grids or 1D token sequences with a hierarchical structure, allowing dynamic allocation of tokens to visually complex regions while using coarser resolution for homogeneous areas. The tokenizer, trained on ImageNet, reduces token usage by about 10% compared to a fixed 256-token grid and retains comparable reconstruction fidelity. Leveraging the tree’s natural causality, a 947 M GPT-style model conditioned on a quadtree topology achieves a 2.08 gFID on ImageNet 256×256 and supports zero‑shot spatially controlled image generation.

By Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, Divyansh Srivastava, Bingnan Li, Zhuowen Tu
arXiv Computer Vision
Oct 8

Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery

The paper introduces JoHan, a generative framework that directly recovers 2D and 3D hand motion from video sequences without intermediate per‑frame pose predictions. By jointly learning temporal dynamics and 2D‑3D correspondence, JoHan generates aligned pose sequences that improve temporal consistency and enable accurate estimation of the hand’s global position and orientation relative to the camera. Experiments on challenging benchmarks show that JoHan achieves higher accuracy and faster performance, producing smoother hand‑motion dynamics while maintaining high per‑frame pose accuracy.

By Chen Xu, Yunqi Li, Binbin Huang, Brent Yi, Shenghua Gao, Yi Ma
arXiv Machine Learning
Oct 8

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

The paper examines the vulnerability of open‑weight large language model safeguards that aim to block harmful usage by assuming new harmful behavior arises only through fine‑tuning. It demonstrates that simple, low‑cost attacks—abliteration and prefilling—can significantly increase attack success rates across multiple benchmarks, and proposes an abliteration‑resistant tuning (ART) method that mitigates these attacks. The study shows that safeguarding defenses must be evaluated against a broader range of attack strategies beyond fine‑tuning.

By Kevin Kuo, Virginia Smith, Chhavi Yadav
arXiv AI
Oct 8

One-Shot Localisation of the Global Minimum of a Noisy One-Dimensional Function: An Iterative Neural Minimizer Compared with Set Transformers and Classical Estimators

The paper introduces the Neural Function Minimizer (NFM), an iterative neural model that predicts the global minimum of a noisy one‑dimensional function from only twenty samples. It compares the NFM to two Set Transformers and classical zero‑query estimators, finding that all learned models outperform classical methods on held‑out data, with the NFM achieving the lowest error relative to the Gaussian‑process posterior argmin. The study also examines how different estimator outputs affect tie‑breaking between equally deep valleys and plans to release a benchmark and self‑checking harness.

By Qusay Muzaffar, David Levin, Michael Werman
arXiv Machine Learning
Oct 8

Directional Evidence Guided Search-Space Reduction for Exact DAG Learning

The paper introduces DECO, a non‑parametric hybrid framework that uses directional evidence from observational data to prune the search space of parent‑set configurations before performing exact DAG optimization. By eliminating empirically unsupported parent configurations, DECO achieves an exponential reduction in the admissible configuration space while maintaining flexibility for plausible edge orientations. Experiments on benchmark Bayesian networks and synthetic DAGs show substantial search‑space reduction and competitive structure‑recovery performance, with favorable structural Hamming distances across many settings.

By Upala Junaida Islam, Abdelmonem Elrefaey, Rong Pan
arXiv Machine Learning
Oct 8

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

The paper introduces Conditional Accuracy Profiling (CAP), a diagnostic framework that breaks down LLM‑judge accuracy into eight deployment conditions covering content sensitivity, robustness, and rationale quality. CAP is benchmark‑agnostic and can be applied directly when annotations are available, via task‑subset proxies, or through controlled augmentation. Applied to seven judges across six benchmarks, CAP reveals that aggregate accuracy masks important differences—for example, judges highly sensitive to omitted qualifications rank poorly on a Hard‑Constructed subset, and Position Robustness is generally stable but can degrade sharply under adversarial stress.

By Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li
arXiv Computation and Language
Oct 8

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

The paper introduces ARCS, a new benchmark for text-to-SQL that focuses on naturally occurring ambiguities in user queries over real-world databases. It proposes a structured disambiguation approach, where ambiguity is resolved through explicit, constrained interactions instead of free-form dialogue. Experiments show that even advanced models struggle, with the best open-source model achieving only 27% execution accuracy.

By Yihao Hu, Yanlin Feng, Naoki Otani, Nikita Bhutani
arXiv Computation and Language
Oct 8

Which Language Should a Skeleton Speak? Language Choices in Multilingual Reasoning

The paper introduces the Language-Aware Skeleton Exploration Framework (LASEF) to investigate how the language of skeleton prompts affects multilingual mathematical reasoning in large language models. Experiments across various math benchmarks, model sizes, and languages show that English skeletons provide a modest advantage on average, especially for smaller models and low-resource languages, but this advantage is not universal and often not statistically significant after correction. The study identifies three distinct patterns of skeleton-language effects—directionally consistent, evaluation- and benchmark-dependent, and asymmetric negative—highlighting that skeleton language is a context-dependent design choice that requires comprehensive exploration.

By HyeonSeok Lim, SeungWoo Song, Inho Won, Hoyun Song, Jihyo Kim, KyungTae Lim
arXiv Computation and Language
Oct 8

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

Alice is a new large-scale German benchmark for Automatic Short Answer Scoring that focuses on rubric-based evaluation. It contains three subtasks—learning performance (Alice‑LP), knowledge elements (Alice‑KE), and skills (Alice‑SK)—and is designed to assess how well students master underlying concepts and skills rather than just answer questions directly. Experiments show that large language models struggle with zero‑shot scoring of knowledge elements and skills, and that rubric text is especially helpful for Alice‑KE and Alice‑SK.

By Zhifan Sun, Sebastian Gombert, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Marcus Kubsch, Knut Neumann, Hendrik Drachsler