arXiv AI

Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes

Augur is a synthetic decision laboratory that simulates how users will react to product and policy changes before they are released. It constructs a typed knowledge graph from change documents, populates a persona market, runs simulations, and produces an auditable decision memo recommending one of five actions. Using a dataset of 50 real episodes (Gold‑50), the authors evaluate the system’s five‑way release verdicts and find that evaluation design, rather than model capability, largely drives performance differences among frontier and open‑weight models.

arXiv AI
Sep 30

Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents

The paper introduces TrustSwap, a counterfactual test that swaps or removes source reliability labels while keeping evidence text constant, to evaluate how retrieval‑augmented fact‑checking models respond across verdict, confidence, and search decisions. Experiments on untrained and RL‑trained models show that confidence and search largely follow labels, yet label changes can flip a significant portion of verdicts, especially in larger models. The authors propose trust‑swap augmentation (TSA) to mitigate this shortcut, demonstrating reduced verdict flip rates and maintained accuracy in several settings, though its effectiveness diminishes at larger model scales.

By Jianchang Su, Yiwei Yang, Wei Zhang
arXiv Machine Learning
Oct 22

Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models

arXiv:2610. 29052v1 Announce Type: cross Abstract: A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text.

By Yuchuan Weng, Chi Liu, Xin Chen, Srinivasan Manoharan
arXiv AI
Sep 2

trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories

The paper "trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories" examines the limitations of outcome-only evaluation for large language model agents. Using a deterministic tool‑using support‑desk environment with a scripted oracle policy and a fault injector, the authors compare five different judging approaches—programmatic rules, outcome‑only, step‑rubric at two model sizes, and a self‑consistency ensemble—on metrics such as detection, step localisation, fault typing, calibration, and cost across 400 trajectories. The study finds that outcome‑only judges miss many silent faults and generate false positives, while step‑rubric judges achieve higher recall with no false alarms but at greater cost, and that none of the judges read the final reply, allowing fabricated promises to evade detection. "whyItMatters":"The findings highlight that current production‑default outcome‑only evaluations can overlook critical failures in agent behavior, underscoring the need for more nuanced, step‑level judging methods to ensure reliable LLM agent performance."

By Hadi Mohammadi
arXiv Computation and Language
Oct 9

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.

By Radhika Gaonkar
arXiv Computation and Language
Oct 28

Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

The paper investigates whether fast, cost‑efficient decision models can classify the type of conflict among retrieved sources for a query, a task that improves response quality when the conflict type is known. Using the CONFLICTS benchmark, the authors evaluate 11 commercial decision models against Gemini LLMs, finding that most achieve 50–67% accuracy, often matching or surpassing Gemini 2.5 Flash while incurring 4–39× lower cost and 2.6–6.0× lower latency. A cascade approach that forwards uncertain cases to Gemini 3.8 Flash yields comparable performance at only 25–37% of its cost, and the resulting typed responses generally behave as expected when compared to LLM‑derived types.

By Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah