arXiv Computation and Language October 16, 2026 By Vaibhav Prajapati, Pushp Garg

Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation

Read the original on arXiv Computation and Language →

The paper investigates how few-shot on-device personalization of virtual assistants, specifically using LoRA fine‑tuning, affects the system’s ability to reject out‑of‑scope (OOS) inputs. It finds that naive personalization consistently lowers OOS‑rejection AUROC across five intent‑classification benchmarks while preserving in‑scope accuracy, partly due to miscalibration of the confidence scorer and partly due to a genuine shrinkage of embedding margins. The authors propose a low‑cost fix—scoring rejection with Mahalanobis distance instead of softmax confidence—that restores most of the lost OOS performance on most datasets without harming accuracy, and they release all code for reproducibility.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jun 30

The Calibrated Deepfake Trust Score (CDTS): Competence-Coupled Trust Degradation Across Deepfake Detectors

arXiv:2606. 29484v1 Announce Type: cross Abstract: Modern deepfake detectors are rarely consumed as bare classifiers.

By Md Anas Biswas

llms computer-vision

arXiv Machine Learning

Jul 21

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.

By Max Fomin

llms agents benchmarks safety

arXiv Computer Vision

Sep 10

How (Mis)calibrated is your Federated CLIP and what to do about it?

arXiv:2512.04305v3 Announce Type: replace Abstract: Vision-language models (VLMs) such as CLIP are increasingly adapted across decentralized data silos, yet the reliability of their predictions under...

By Mainak Singha, Masih Aminbeidokhti, Paolo Casari, Gianni Franchi, Elisa Ricci, Subhankar Roy

llms fine-tuning multimodal

arXiv AI

4d ago

PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

arXiv:2605.30803v2 Announce Type: replace Abstract: LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rub...

By Swastik Roy, Rajkumar Pujari, Tharindu Kumarage, Charith Peris, Rahul Gupta, Anna Rumshisky, Pradeep Natarajan, Venkatesh Saligrama

llms safety

arXiv Computation and Language

Aug 25

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

arXiv:2608.21606v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whet...

By Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi, Abhishek Lalwani, Somya Chatterjee, Sadid Hasan

llms fine-tuning multimodal benchmarks safety

arXiv AI

Oct 2

On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence

The paper evaluates nine on‑device named‑entity recognition models ranging from classical taggers to large language models, measuring not only accuracy but also latency and output validity. Using a silver‑gold benchmark derived from an LLM judge panel and a human‑validated corpus, the study shows that encoder‑based models achieve comparable accuracy to a 4 B instruct LLM while being much smaller, faster, and producing no malformed output. Confidence calibration of GLiNER is analyzed, revealing over‑confidence but improved reliability after temperature scaling and thresholding.

By Vinay Kumar Chaganti

llms benchmarks