DeclineCraft is a dynamic abstention benchmark for retrieval‑augmented generation (RAG) that tests whether models can recognize when neither retrieved documents nor parametric knowledge can answer a question. The benchmark uses unanswerable Minecraft feature‑addition queries that appear after a corpus cut‑off, allowing evaluation of future models. Across 27 large language models, the study measures Correct Abstention Rate (CAR), Answerable Attempt Rate, and Answer Accuracy, finding that single‑shot retrieval boosts CAR, iterative RAG reduces CAR with minimal impact on accuracy, and larger models improve accuracy but not CAR.
By Atharva Thakur, Abulhair Saparov, Bruno Ribeiro
The paper introduces a new evaluation setting called penalty‑framed no‑valid‑option MCQA, where multiple‑choice questions may contain no correct answer. By removing the correct option from the MMLU‑Pro mathematics subset and allowing models to either pick an option or abstain, the authors penalize forced‑choice responses that are invalid. Experiments reveal that even models with high standard MCQA accuracy can still produce invalid forced‑choice answers, indicating that traditional accuracy metrics miss an important aspect of model reliability.
By Jinhyeok Kim, Hye-Young Jung
The paper investigates how compact an encoder can be for closed‑set decision tasks in language‑model pipelines, comparing joint versus per‑option scoring. In a 32‑million‑parameter study, per‑option scoring outperforms joint scoring by 5.8 chance‑corrected skill points on a 37‑benchmark suite, and two external benchmarks corroborate this advantage. Across six released joint encoders ranging from 17 million to 1 billion parameters, the 68‑ and 150‑million‑parameter models surpass a 435‑million‑parameter DeBERTa‑v3 zero‑shot classifier on 209 overlapping tasks and match it on 145 unseen tasks, while the smallest model processes a median request in 14 ms versus 728 ms for the classifier.
By Tardelli Ronan Coelho Stekel
EDU-Jev is a family of single‑pass decision models (0.8B, 2B, and 4B parameters) designed for educational tasks that require selecting from a fixed set of options. The models output a probability distribution over predefined choices without generating text, using a pointer head to score each option. In twelve education benchmarks, EDU‑Jev 4B achieved 67.6% accuracy, outperforming five commercial decision APIs by 9.4 percentage points and the best open 4B model by 18.7 points, while the 2B version can process about 150 decisions per second on a single H100 GPU.
By Jiazheng Li, Yulan He
LEV is a decision model that performs reasoning by looping its layers instead of generating text, allowing it to deepen its computation on a per-question basis. Trained jointly across all loop steps, a single LEV checkpoint can answer at any depth, improving accuracy—especially on hard questions—while keeping inference fast. On benchmark tests, LEV outperforms dense and cascade models by 5.4 to 8.8 accuracy points with sub‑50 ms latency, and on‑demand depth selection saves 15–24% of computation for minimal accuracy loss.
By Dingjie Song, Zhiling Yan, Yuxuan Zhang, Lichao Sun
The paper introduces a penalty‑aware evaluation framework for Retrieval‑Augmented Generation (RAG) systems that uses asymmetric scoring, knowledge‑gap canaries, and a failure‑attribution pipeline. Applying this framework to three commercial RAG products and a baseline on SimpleQA‑Verified, the authors find that while overall accuracy is similar across systems, canary violation rates vary dramatically, showing that systems differ more in when they answer than in what they answer. The study demonstrates that penalty‑aware scoring can reorder system rankings and is robust across different penalty settings.
By Alden Do Rosario, Hussein Younes, Felipe Pires
arXiv:2606.16011v2 Announce Type: replace
Abstract: Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain...
By Nafiseh Nikeghbal, Amir Hossein Kargaran, Shaghayegh Kolli, Jana Diesner
The paper introduces Drex, a family of four open-source decision models that output probability distributions over user-supplied options without autoregressive decoding. These models—ranging from 8B to 35B parameters—are trained with supervised fine-tuning, reinforcement learning from adjusted feedback, and vision tuning for the Max variant. Drex achieves state‑of‑the‑art performance on Decision Index and competitive results on Decision Bench, while delivering faster, more consistent responses than larger GPT‑style models.
Small language models can grade open‑ended exam answers as reliably as much larger models when they use an explicit rubric. In experiments with six cost‑efficient model configurations, the rubric decouples grading from judge intelligence, with answer identity explaining 95.6% of score variance and judge identity only 0.2%. Removing rubric criteria or the official answer collapses reliability and inflates scores, showing the rubric’s essential role.
By Jhen-Ke Lin
The paper introduces the pooled selection-rate standard deviation (pooled SD) as a way to quantify how unevenly multiple-choice option letters are chosen across different orderings. It shows that pooled SD decomposes into two label‑free factors: the flip rate, which measures how often a model’s chosen option changes with ordering, and the positional skew, which captures the variability in option‑letter selection rates. Experiments on 41 checkpoints and 12 benchmarks reveal that prior correction mainly reduces positional skew, while aggregating each ordering with its reversal mainly reduces flip rate; applying both mitigations together further narrows the accuracy range across answer letters.
By Hyunjong Ok, Minjae Park, Jaeho Lee
The paper investigates how the scoring rules used in multimodal large‑language‑model (LLM) benchmarks can dramatically alter reported accuracy. By reproducing official scorers for eight benchmarks and comparing them to a blinded human‑plus‑LLM adjudication, the authors show that the same model outputs can yield accuracy differences of up to 22 points depending on the rule. They argue that no single scoring rule is reliable and recommend that benchmarks report non‑compliance rates and use at least two distinct scoring rules.
By Ahmad Jarjis Hasan, Jonathan Lwowski
arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.
By Suyash Maniyar, Armaan Sandhu, Abhishek Mishra