arXiv Machine Learning
Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Oct 21
DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge
DeclineCraft is a dynamic abstention benchmark for retrieval‑augmented generation (RAG) that tests whether models can recognize when neither retrieved documents nor parametric knowledge can answer a question. The benchmark uses unanswerable Minecraft feature‑addition queries that appear after a corpus cut‑off, allowing evaluation of future models. Across 27 large language models, the study measures Correct Abstention Rate (CAR), Answerable Attempt Rate, and Answer Accuracy, finding that single‑shot retrieval boosts CAR, iterative RAG reduces CAR with minimal impact on accuracy, and larger models improve accuracy but not CAR.
arXiv AI
Oct 7
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
The paper introduces a new evaluation setting called penalty‑framed no‑valid‑option MCQA, where multiple‑choice questions may contain no correct answer. By removing the correct option from the MMLU‑Pro mathematics subset and allowing models to either pick an option or abstain, the authors penalize forced‑choice responses that are invalid. Experiments reveal that even models with high standard MCQA accuracy can still produce invalid forced‑choice answers, indicating that traditional accuracy metrics miss an important aspect of model reliability.
arXiv Machine Learning
Oct 23
Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
The paper investigates how compact an encoder can be for closed‑set decision tasks in language‑model pipelines, comparing joint versus per‑option scoring. In a 32‑million‑parameter study, per‑option scoring outperforms joint scoring by 5.8 chance‑corrected skill points on a 37‑benchmark suite, and two external benchmarks corroborate this advantage. Across six released joint encoders ranging from 17 million to 1 billion parameters, the 68‑ and 150‑million‑parameter models surpass a 435‑million‑parameter DeBERTa‑v3 zero‑shot classifier on 209 overlapping tasks and match it on 145 unseen tasks, while the smallest model processes a median request in 14 ms versus 728 ms for the classifier.
arXiv Computation and Language
Nov 2
EDU-Jev Training System-One Decision Models for Education
EDU-Jev is a family of single‑pass decision models (0.8B, 2B, and 4B parameters) designed for educational tasks that require selecting from a fixed set of options. The models output a probability distribution over predefined choices without generating text, using a pointer head to score each option. In twelve education benchmarks, EDU‑Jev 4B achieved 67.6% accuracy, outperforming five commercial decision APIs by 9.4 percentage points and the best open 4B model by 18.7 points, while the 2B version can process about 150 decisions per second on a single H100 GPU.
arXiv AI
Oct 16
LEV: Thinking Fast and Deep with Looped Decision Models
LEV is a decision model that performs reasoning by looping its layers instead of generating text, allowing it to deepen its computation on a per-question basis. Trained jointly across all loop steps, a single LEV checkpoint can answer at any depth, improving accuracy—especially on hard questions—while keeping inference fast. On benchmark tests, LEV outperforms dense and cascade models by 5.4 to 8.8 accuracy points with sub‑50 ms latency, and on‑demand depth selection saves 15–24% of computation for minimal accuracy loss.
arXiv AI
Aug 28
Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries
The paper introduces a penalty‑aware evaluation framework for Retrieval‑Augmented Generation (RAG) systems that uses asymmetric scoring, knowledge‑gap canaries, and a failure‑attribution pipeline. Applying this framework to three commercial RAG products and a baseline on SimpleQA‑Verified, the authors find that while overall accuracy is similar across systems, canary violation rates vary dramatically, showing that systems differ more in when they answer than in what they answer. The study demonstrates that penalty‑aware scoring can reorder system rankings and is robust across different penalty settings.