arXiv AI
To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 2
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
The paper introduces an action‑class diagnostic framework for multi‑turn tool‑calling in large language model agents, breaking failures into action‑class miscalibration and action‑execution failure across a four‑class action space (TOOL_CALL, ASK, REFUSE, CONFIRM). It defines a self‑revealing upper bound (Acc GAR) to expose state‑grader masking of miscalibration and shows that miscalibration is a significant, previously hidden failure mode, especially for heavily tool‑trained families. The study demonstrates that calibration can be reshaped by context‑only perturbations, but the effects vary widely across models and perturbation mechanisms, underscoring the need for diagnostics beyond aggregate accuracy.
arXiv AI
Aug 25
CallScreenBench: Benchmarking Small Language Models as Phone Secretaries
CallScreenBench is a benchmark for evaluating small, on-device language models that act as phone secretaries, focusing on their ability to handle unknown inbound calls without a cooperative task. The benchmark measures owner endorsement through five call-and-note metrics, each paired with counter-metrics and uncertainty estimates, and includes guardedness diagnostics to identify safe, tool‑free proxies. Results across 4‑bit checkpoints of 0.6‑4 B parameter models show varying performance on service, recall, plausibility, and triage discrimination, highlighting trade‑offs between quality and guardedness.
arXiv AI
Jun 16
Rational Sparse Autoencoder
arXiv:2606. 14990v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK.
arXiv Machine Learning
Sep 30
RAISE: Diagnosing Acquisition Collapse in Costly LLM Signals
The paper introduces RAISE, a diagnostic framework that tests whether a costly large language model (LLM) signal provides enough pre-call information to justify selective use. It identifies the failure mode of acquisition collapse, where an LLM appears useful overall but lacks actionable evidence for individual decisions. The authors demonstrate RAISE with Structured Hypothesis Embeddings (SHE) and evaluate it across multiple study designs, showing that predictable incremental benefit, rather than average lift, indicates recoverable selective value.
arXiv Machine Learning
Jul 1
Probing Memorization of Tabular In-Context Learning
arXiv:2606. 31208v1 Announce Type: new Abstract: Large tabular models (LTMs), i.
arXiv AI
Jun 15
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
arXiv:2606. 14211v1 Announce Type: new Abstract: LLMs are increasingly deployed as agents that interact with external environments and observe feedback such as execution results, error messages, and tool outputs.