← Back to all news
arXiv Machine Learning By Tardelli Ronan Coelho Stekel

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

Read the original on arXiv Machine Learning →

The paper investigates how compact an encoder can be for closed‑set decision tasks in language‑model pipelines, comparing joint versus per‑option scoring. In a 32‑million‑parameter study, per‑option scoring outperforms joint scoring by 5.8 chance‑corrected skill points on a 37‑benchmark suite, and two external benchmarks corroborate this advantage. Across six released joint encoders ranging from 17 million to 1 billion parameters, the 68‑ and 150‑million‑parameter models surpass a 435‑million‑parameter DeBERTa‑v3 zero‑shot classifier on 209 overlapping tasks and match it on 145 unseen tasks, while the smallest model processes a median request in 14 ms versus 728 ms for the classifier.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 16

Verdict: Encode Once, Decide Many

Verdict-MQ is an open encoder that processes a single piece of state once and then answers multiple typed questions about that state, avoiding separate passes for each question. By using a structured attention mask, it ensures the state encoding is independent of question order and prevents questions from influencing one another. On the typed‑decisions benchmark, Verdict‑MQ outperforms a larger specialist model, achieving higher accuracy and better calibration, while on other benchmarks it shows mixed results compared to fine‑tuned baselines.

By Avdhesh Singh Chouhan, Jack Perry
fine-tuningbenchmarks
More like this →
arXiv Computation and Language
Oct 1

Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...

By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
llmsragnlpbenchmarkssafety
More like this →
arXiv AI
Jul 29

How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model

arXiv:2607. 25583v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) and low-bit quantization are now standard tools for adapting language models under tight compute budgets, yet their interaction is most often studied on billion-parameter models where the design space is expensive to explore.

By Mahendra Singh Rathor, Anagheem Azzam
llmsfine-tuningefficiencybenchmarksdata-engineering
More like this →