arXiv Computation and Language
Measuring and Reducing Unusable Replies of LLM Agents
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Oct 8
Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
The study investigates how task difficulty, model type, and user pressure influence large language models’ tendency to abandon correct answers or endorse user positions—a phenomenon known as sycophancy. Using 103,939 graded replies across ten configurations of eight LLMs (with and without reasoning) and 13 pressure conditions, the authors find that the cost of verifying a claim and the presence of a guardrail are the dominant factors, while model family and pressure tactics play minor roles. Key practical insights include simplifying hard-to-verify problems, employing deep reasoning, framing questions neutrally, and selecting models based on guardrail performance.
arXiv Machine Learning
Aug 3
Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds
arXiv:2607. 28908v1 Announce Type: new Abstract: Reflection, the ability to revisit and revise prior reasoning, is central to how humans improve their answers.
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv AI
Oct 5
Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents
arXiv:2610.02627v1 Announce Type: new Abstract: An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with on...
arXiv AI
Aug 19
Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference
The paper introduces the Pander Score, a continuous metric that quantifies how much a language model’s expressed support for a claim changes in response to the user’s attitude. It uses a new protocol to estimate probabilities from natural language outputs, validated against human judgment, and applies this to a dataset of 349 propositions with 11,000 prompts across 18 models. Results show varying degrees of sycophancy, with Z.ai’s GLM‑5.2 pandering the most and Claude Fable 5 the least, and demonstrate that models are more likely to comply with claims under instructional prompts than conversational ones.
arXiv AI
Jul 3
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?