← Back to all news
arXiv Machine Learning

Calibrated Evidence Inference With a Small, Permissively Licensed Open Model

The paper presents a calibrated evidence inference model fine‑tuned from Qwen3‑4B‑Base using low‑rank adapters and a classification head to classify trial outcomes as significant increase, significant decrease, or no significant difference. Trained on 1,431 open‑access PubMed Central articles, the released model (Apache 2.0) achieves an expected calibration error of 0.017, a log loss lower than Gemma 4 31B‑it in zero‑shot settings, and a macro‑F1 of 0.925 on the test split. Accepting its top‑80 % most confident predictions yields only a 2.6 % error rate, and ablation studies show that article context influences answers while the title alone still provides useful signal.

Related stories

arXiv Machine Learning
Oct 7

Calibrated Answers About Randomized Trials From a 4-Billion-Parameter Open Model: A Registered Test and a License-Clean Release

Fiorillo v0.5 is an open 4‑billion‑parameter model that reads a randomized trial article (cut to 6,144 tokens) and predicts whether an intervention significantly increased, decreased, or had no significant effect on an outcome. Fine‑tuned on 1,431 license‑clean training articles, it achieved an expected calibration error of 0.0168, log loss 0.8603 lower than the prior, and macro‑F1 of 0.9248 on a public test split of 1,218 prompts. The model’s release, licensed under Apache 2.0, met all four pre‑registered criteria on the Open Science Framework and was validated against the evaluated predictions.

By Johann Emmanuel Li
fine-tuning
More like this →
arXiv Computation and Language
Sep 24

What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs

The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.

By Han Chen, Yingrui Li
llms
More like this →
arXiv AI
Oct 2

Jev in Medicine: A Benchmark Evaluation

The study evaluates Jev 1.13, a non‑generative model that selects from predefined answer options, on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena‑MCQ, and the NEJM Case Challenges. Jev’s top‑1 accuracy matches GPT‑6 Sol with medium reasoning on PubMedQA but falls behind on MetaMedQA, DiagnosisArena‑MCQ, and NEJM cases. While Jev shows strong calibration on MetaMedQA and is fast and inexpensive, its performance on examination and complex diagnostic tasks is substantially lower, indicating the need for task‑specific validation before clinical deployment.

By Alfredo Madrid-Garc\'ia, Beatriz Merino-Barbancho
llmsbenchmarks
More like this →
arXiv AI
Sep 7

When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models

A 0.6B language model consistently answers YES to 1,200 logical tests, yet its behavior shows no discrimination. Linear probes reveal the correct verdict with high AUC (0.96) and transfer to unseen structures, but a single scalar readout fails due to a saturated decision threshold offset by +4.6 σ. Adjusting this threshold restores behavior accuracy from 50 % to 81 % and improves higher‑scale models, demonstrating that miscalibrated readouts, not hidden knowledge loss, drive performance gaps.

By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
llms
More like this →