The paper introduces a signed lexical gate that combines a sentence classifier’s logit margin with a sparse lexical model’s support for the predicted intent, assigning positive evidence to lexical agreement and negative evidence to a lexically favored competing intent. This gate retains more information than unsigned lexical confidence or a hard agreement rule and is calibrated via an independent binomial procedure to meet specified risk targets. Experiments on BANKING77, CLINC150, and HWU64 show that the proposed score reduces the area under the risk‑coverage curve by up to 15.8% and increases accepted coverage at low error rates, offering a compact, interpretable confidence enhancement for risk‑calibrated intent routing.
By Yezhou Cheng, Zehua Yang, Bojun Lin
The paper evaluates whether zero‑shot large language models (LLMs) can replace fine‑tuned natural language understanding (NLU) classifiers for intent detection in conversational systems. Experiments on ATIS and CLINC150 show that fine‑tuned RoBERTa outperforms Claude Haiku zero‑shot when ample in‑domain labels are available, but the LLM matches the supervised model on the broader 150‑intent CLINC150 schema. The authors identify three production‑relevant scenarios where LLMs excel—out‑of‑scope detection, robustness to ASR noise, and dynamic per‑deployment schemas—and present a decision framework to guide practitioners.
whyItMatters:"The study provides concrete evidence and a practical framework for deciding when to deploy LLMs versus fine‑tuned models in real‑world conversational applications."
By Carson Rodrigues, Oysturn Vas
The paper presents the AI Appeals Processor, a deep‑learning system deployed in a CPU‑only government environment to classify and route citizen appeals. On a dataset of 10,000 Russian‑language appeals, multilingual BERT achieved 82% accuracy on a three‑way appeal‑type task, outperforming other models such as fastText, Word2Vec+LSTM, and SVM. After deployment, the system reduced end‑to‑end handling time by 53‑56% and, following several retraining cycles, achieved over 95% correct classifications in production, though the high accuracy is not directly comparable to the test‑set result due to taxonomy issues.
By Vladimir Beskorovainyi
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.
By Amir Rafe, Subasish Das
arXiv:2610.17205v1 Announce Type: new
Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a ty...
By Jonathan Geuter, Andrew Cheng, David Alvarez-Melis