← Back to all news
arXiv AI By Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Read the original on arXiv AI →

JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents

StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.

By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
llmsagentsreinforcement-learningbenchmarks
More like this →
arXiv AI
Sep 24

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.

By Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
llmsagentsbenchmarks
More like this →