← Back to all news
arXiv Machine Learning By Xiangwei Wang, Peng Wang, Saman Halgamuge

CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels

Read the original on arXiv Machine Learning →

CORDIAL is a calibration method for large language models that treats ordinal outputs as noisy measurements of true labels and corrects them using a small, interpretable channel of five parameters. The method can be calibrated from only a few labels and preserves first‑order stochastic order. Experiments on Amazon reviews and CMU‑MOSEI transcripts with four LLMs show that CORDIAL achieves the lowest log loss in most settings, outperforming other calibrators when the calibration set is small, and can also learn priors from other tasks or fuse multiple LLMs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Oct 22

S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

S-EDL is a new evidential method that calibrates large language models for open‑ended generation by using the model’s own sequence likelihoods to build a differentiable Dirichlet prior over prompt‑specific semantic classes. This approach optimizes an evidential loss that provides a calibration‑aware training signal without relying on fixed multiple‑choice options. Experiments on four open‑ended benchmarks and three models show that S-EDL reduces semantic calibration error and improves accuracy, offering a framework for trustworthy LLM deployment in complex applications.

By Yawei Li, Jiazheng Li, David R\"ugamer, Bernd Bischl, Yulan He, Mina Rezaei
llmsbenchmarks
More like this →
arXiv Machine Learning
Jun 24

LLMs are Bayesian, In Expectation, Not in Realization

arXiv:2507. 11768v3 Announce Type: replace-cross Abstract: Bayesian accounts of in-context learning face a direct objection: exact posterior predictives for exchangeable data are invariant to task-preserving order, yet transformers change next-token probabilities when the same examples are serialized differently.

By Leon Chlon, Fatima Sheaib, Zein Khamis, Maggie Chlon, Mahdi El Zein, MarcAntonio M. Awada
llmsbenchmarks
More like this →
arXiv Computation and Language
Sep 1

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

The paper argues that traditional global calibration metrics, such as Expected Calibration Error and Brier Score, are confounded by differences in model accuracy when comparing large language models. It introduces ACE, an accuracy‑controlled evaluation framework that offers Instance‑Aligned, Distribution‑Aligned, and Candidate‑Aligned views to provide fairer cross‑model comparisons. Experiments across various benchmarks reveal that many reported calibration advantages disappear after accuracy control and that model rankings often reverse, indicating that raw global metrics are unreliable for cross‑model calibration assessment.

By Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang
llmsbenchmarks
More like this →