arXiv Machine Learning October 23, 2026 By Sachin Gupta

Cost-Aware Abstention for LLM-Based IT Ticket Classification

Read the original on arXiv Machine Learning →

The paper evaluates cost‑aware abstention for routing IT support tickets, comparing four non‑LLM classifiers and four large language models on a 47,837‑ticket, eight‑class dataset. Using a cost model (c_ok=0, c_triage=1, c_wrong=5), a TF‑IDF logistic regression model with a 0.60 threshold reduced test cost from 0.763 to 0.472 and lowered wrong routes from 15.3% to 3.5%. The study also shows that calibration‑selected policies outperform always‑route and always‑defer strategies in most settings and introduces new metrics such as correctness AUROC and tie‑aware risk‑coverage area.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 30

Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds

arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.

By Mengdie Flora Wang, Haochen Xie, Guanghui Wang, Devin Zhang, Jae Oh Woo

llms agents benchmarks safety

arXiv Machine Learning

Sep 22

When Calibration Depends on the Scoring Rule: Quantized Biomedical LLM Classification

arXiv:2608.03854v4 Announce Type: replace Abstract: Quantized large language models can run on consumer hardware, which motivates interest in on-premises processing of sensitive data. The reliability...

By Anton Rasmussen, Hong Qin

llms efficiency

arXiv AI

Oct 21

When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders

The paper investigates when post‑training quantization (PTQ) calibration helps or hurts text embedders, focusing on the amount of calibration (dose) and whether to calibrate at all. Across three models and 161 calibration runs at 4‑bit weight quantization, the effect ranges from significant gains to catastrophic degradation, driven mainly by the functional displacement of the quantized model. A validation‑gated abstention rule is proposed that avoids harmful calibrations while capturing most benefits, highlighting the trade‑off between safety and opportunity.

By Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon

efficiency safety

arXiv AI

Aug 18

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

arXiv:2608. 15565v1 Announce Type: new Abstract: Experience-learning agents for optimization modeling improve by storing verified skills, but existing learners admit knowledge by checking against known answers, which real ticket streams do not provide.

By Junbo Jacob Lian, Huiling Chen, Hanzhang Qin, Chung-Piaw Teo

llms agents benchmarks

arXiv Machine Learning

Aug 18

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

arXiv:2608. 14551v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty.

By Arya Rahgozar, Pouria Mortezaagha

llms benchmarks

arXiv Machine Learning

Jul 1

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

llms benchmarks

One email a morning, machine-written

Related stories

Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds

When Calibration Depends on the Scoring Rule: Quantized Biomedical LLM Classification

When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models