arXiv AI

Where in the Input Does a One-Pass Decision Encoder Look?

The paper evaluates Laya, a small model that processes a multiple‑choice question, its options, and a text in a single 512‑token sequence. Experiments show that when the relevant evidence lies beyond the window, accuracy drops to about 26–27%, and confidence metrics fail to warn of truncation. Fine‑tuning on longer documents improves performance on visible evidence but not on cut evidence, and a larger window or a 312‑token scan can yield modest gains.

arXiv Machine Learning
Oct 22

Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models

arXiv:2610. 29052v1 Announce Type: cross Abstract: A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text.

By Yuchuan Weng, Chi Liu, Xin Chen, Srinivasan Manoharan
arXiv AI
3d ago

How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?

The study investigates how many large‑language‑model (LLM)‑generated training rows a decision encoder, specifically Laya (ModernBERT‑large), requires to achieve high accuracy on a choice‑making task. Using 19 fine‑tunes on nested subsets of a 2,818‑row pool, accuracy improves from 69.4 % before fine‑tuning to 86.9 % with all rows, with diminishing returns as more data is added. The experiment also examines the impact of label noise, finding that 10 % symmetric noise has negligible effect, while 20 % and 40 % noise degrade performance by 3.4 and 8.7 points respectively, and that a single run with 10 % confusable‑pair noise reduces confidence. whyItMatters":"The results quantify the data efficiency and robustness of Laya to noisy LLM‑generated labels, informing how much synthetic training data is needed for reliable decision‑encoding models."

By Mohammed Nayeem Khan, Aliyas Ahmad Misri
arXiv AI
Sep 7

When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models

A 0.6B language model consistently answers YES to 1,200 logical tests, yet its behavior shows no discrimination. Linear probes reveal the correct verdict with high AUC (0.96) and transfer to unseen structures, but a single scalar readout fails due to a saturated decision threshold offset by +4.6 σ. Adjusting this threshold restores behavior accuracy from 50 % to 81 % and improves higher‑scale models, demonstrating that miscalibrated readouts, not hidden knowledge loss, drive performance gaps.

By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
arXiv AI
Oct 5

Sentence-Level Context Sensitivity as a Training-Free Detector of Unsupported Content, Evaluated Against Trained Verifiers

The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.

By Mohamed Aly Bouke
arXiv AI
Sep 15

Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA

The paper presents a system for the MedReason 2026 challenge that tackles both multiple‑choice and open‑ended medical visual question answering using offline, containerized inference. Key findings include that comparing answer semantics rather than labels boosts retrieval‑only accuracy from 20.0 % to 57.5 % on a 200‑case holdout, and that varying the number of in‑prompt retrieved examples has minimal impact on final accuracy (93.5 %–94.0 %). The final system achieves 94.0 % MCQ accuracy on the development set and 93.20 % on the official pre‑evaluation, far surpassing the off‑the‑shelf baseline. "whyItMatters":"The results demonstrate that semantic‑aware retrieval and careful adapter tuning can dramatically improve medical VQA performance, offering a practical approach for high‑accuracy, offline inference in clinical settings."

By Tristan Kirscher (ICube, Institut Strauss), Niklas C. Koser (CAU), Soren Pirk (CAU)
arXiv AI
Oct 16

Verdict: Encode Once, Decide Many

Verdict-MQ is an open encoder that processes a single piece of state once and then answers multiple typed questions about that state, avoiding separate passes for each question. By using a structured attention mask, it ensures the state encoding is independent of question order and prevents questions from influencing one another. On the typed‑decisions benchmark, Verdict‑MQ outperforms a larger specialist model, achieving higher accuracy and better calibration, while on other benchmarks it shows mixed results compared to fine‑tuned baselines.

By Avdhesh Singh Chouhan, Jack Perry
arXiv AI
Sep 25

Sequential knowledge editing breaks a model's ability to tell good evidence from bad, without costing it accuracy

The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.

By Atul Anand
arXiv Machine Learning
1d ago

How Much Is Left Matters: False Positives and False Negatives in Partial-Prompt Contamination Tests

The paper evaluates partial‑prompt completion tests—where a model is given the first part of a benchmark problem and checked for reproducing the rest—on seven open‑weight models using MATH‑500 and recent AIME 2025/2026 problems as true negatives. It finds that memorization flags are highly sensitive to the amount of text left: problems with fewer than eight remaining words are flagged far more often than those with 25 or more, and most ROUGE‑based flags arise from short leftovers. On true negatives, even a high ROUGE‑L threshold flags up to 6.7% of problems at a 60% prefix, and exact‑match flags 8.6% at an 80% prefix, while models distilled from Qwen2.5‑Math show almost no recall of contamination. The authors recommend length‑stratified reporting, limiting prefixes to 60%, using style‑matched post‑cutoff controls, and cross‑family calibration to reduce false positives.

By Abhishek Anand