The paper investigates how few-shot on-device personalization of virtual assistants, specifically using LoRA fine‑tuning, affects the system’s ability to reject out‑of‑scope (OOS) inputs. It finds that naive personalization consistently lowers OOS‑rejection AUROC across five intent‑classification benchmarks while preserving in‑scope accuracy, partly due to miscalibration of the confidence scorer and partly due to a genuine shrinkage of embedding margins. The authors propose a low‑cost fix—scoring rejection with Mahalanobis distance instead of softmax confidence—that restores most of the lost OOS performance on most datasets without harming accuracy, and they release all code for reproducibility.
arXiv:2606. 29484v1 Announce Type: cross Abstract: Modern deepfake detectors are rarely consumed as bare classifiers.
arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.
arXiv:2512.04305v3 Announce Type: replace Abstract: Vision-language models (VLMs) such as CLIP are increasingly adapted across decentralized data silos, yet the reliability of their predictions under...
arXiv:2605.30803v2 Announce Type: replace Abstract: LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rub...
arXiv:2608.21606v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whet...
The paper evaluates nine on‑device named‑entity recognition models ranging from classical taggers to large language models, measuring not only accuracy but also latency and output validity. Using a silver‑gold benchmark derived from an LLM judge panel and a human‑validated corpus, the study shows that encoder‑based models achieve comparable accuracy to a 4 B instruct LLM while being much smaller, faster, and producing no malformed output. Confidence calibration of GLiNER is analyzed, revealing over‑confidence but improved reliability after temperature scaling and thresholding.
The paper introduces the concept of perfect aliasing, where a truth probe that aligns truthful reporting with a task’s prescribed action cannot differentiate between the two based solely on its labels. In a binary reporting game, probes fitted on compliant contexts yield identical optimizations, while on rival contexts their labels are complementary, causing their AUROCs to sum to one across 751 cell-layer pairs. By employing randomized codebooks and mixed-context fitting, the authors demonstrate that separating prescribed output symbols from semantic action enables perfect recovery of truth, achieving an AUROC of 1.000 on rival trials for a reward-trained Gemma-2-9B policy, whereas conventional probes perform near chance.
arXiv:2606. 29280v1 Announce Type: cross Abstract: We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction.
arXiv:2610.20417v1 Announce Type: new Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
The paper evaluates whether zero‑shot large language models (LLMs) can replace fine‑tuned natural language understanding (NLU) classifiers for intent detection in conversational systems. Experiments on ATIS and CLINC150 show that fine‑tuned RoBERTa outperforms Claude Haiku zero‑shot when ample in‑domain labels are available, but the LLM matches the supervised model on the broader 150‑intent CLINC150 schema. The authors identify three production‑relevant scenarios where LLMs excel—out‑of‑scope detection, robustness to ASR noise, and dynamic per‑deployment schemas—and present a decision framework to guide practitioners. whyItMatters:"The study provides concrete evidence and a practical framework for deciding when to deploy LLMs versus fine‑tuned models in real‑world conversational applications."
arXiv:2610.17747v1 Announce Type: cross Abstract: Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step agai...
arXiv:2602.13576v2 Announce Type: replace-cross Abstract: Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-langu...