Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
Read the original on arXiv Computation and Language →The paper introduces Jev, a specialized decision model that selects among given options rather than generating text, and benchmarks it against 19 large language models (LLMs) across 13 multiple-choice tasks in knowledge, reasoning, and multilingual domains. Jev performs competitively with frontier LLMs on knowledge and commonsense benchmarks, achieving the top score on MMLU-Redux and ARC-Challenge, and surpasses most representative and all small LLMs except in mathematical word problems where it lags behind. The study demonstrates that a specialized decision model can match general-purpose LLMs on knowledge-based decisions but struggles with multi-step calculations.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.