HarvestBench is a new benchmark that evaluates how large language model agents decide whether to pay a fuel cost to avoid killing animals in a farm simulation. In the gridworld, agents control tractors that must choose between driving through or swerving around animals, with the cost of swerving being a variable price. Across nine models and 7,201 decisions, kill rates varied widely, and models were sensitive to price and briefing instructions, showing differing moral behavior without relying on an LLM grader.
Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow...
arXiv:2605. 16301v2 Announce Type: replace-cross Abstract: Evaluating animal welfare reasoning in LLMs remains an open challenge despite rapid deployment in consumer and professional contexts where welfare considerations appear implicitly in everyday queries.
JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.
CrashGym is a deterministic, replayable limit‑order‑book simulator that lets language‑model agents decide between a deleveraging intervention and a liquidity backstop before a market crash. Agents can query intervention experiments in a laboratory version of the market, and their choices are scored by simulating every feasible intervention on an independently seeded target realization, allowing direct regret calculation. In a study of 24 new worlds, laboratory access reduced mean regret by 57.4% for one deployment, and exploratory results suggest lower regret across all evaluated deployments, while an evolving strategy mix offers no clear advantage over a fixed mix.
Language-model agents are increasingly asked to run experiments, yet benchmarks rarely measure whether those experiments improve the decisions they are meant to inform. We introduce CrashGym, a determ...
The paper introduces a continued‑pretraining corpus called "coupled welfare" that embeds ecological reasoning into large language models. After a single QLoRA run, the corpus was applied to five open models across three architectures, resulting in a dramatic reduction of harmful decisions under high‑pressure scenarios while leaving general performance unchanged. The study also demonstrates that the improvement is content‑specific, as a neutral control corpus did not yield similar gains, and that the models remain robust against adversarial attempts to justify environmentally destructive actions.
arXiv:2608. 14642v1 Announce Type: new Abstract: Reinforcement Learning (RL) agents trained on a single reward signal exploit the gap between the designed reward and the intended behavior.
arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.
arXiv:2608. 04719v1 Announce Type: new Abstract: Agent evaluations tell us that a model picked the wrong tool, but rarely why.
StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.
Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness.
arXiv:2606. 00914v1 Announce Type: new Abstract: LLM agents increasingly act after consuming ranked external information streams such as social feeds, search results, retrieval contexts, and email queues, yet safety evaluations almost always test the model or the user prompt in isolation, never the upstream ranker that decides what the agent reads just before it acts.