The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions

The paper investigates the "score granularity gap" in black-box large language model (LLM) classifiers, asking how finely a confidence score can be thresholded for deployment. By comparing seven confidence construction methods across 25 model-dataset pairs, the authors find that single-shot verbalized confidence, when properly converted to a probability, ranks well but offers only a few distinct threshold values, limiting operational flexibility. The study also shows that multi-query aggregation can improve weak models but may harm strong ones, and provides concrete guidance for deployment trade-offs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• benchmarks