Hugging Face Trending Papers

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Chronicle introduces a method called cut‑point replay to make regression testing of large language model (LLM) agents reproducible. It records an agent run at non‑deterministic boundaries as immutable envelopes and replays selected boundaries while executing the rest live, turning recorded failures into continuous‑integration tests. Benchmarks show minimal overhead, bit‑stable full replay, and effective detection of faulty code changes in a mutation study.

arXiv AI
Sep 18

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Chronicle introduces a method called cut‑point replay to make regression testing of large language model (LLM) agents reproducible. It records an agent’s run at non‑deterministic boundaries as immutable envelopes and then replays selected boundaries while executing the rest live, enabling continuous‑integration tests that detect faulty code changes. Benchmarks show minimal overhead, perfect bit‑stability, and effective detection of unsafe actions in a mutation study.

By Tisha Chawla, Susheem Koul
arXiv AI
Oct 9

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.

By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
arXiv Computation and Language
Nov 2

Agentic World Modeling for Agent Test-time Scaling without Environment Replica

Agentic World Modeling (AWM) redefines next-observation prediction as a multi-turn evidence-acquisition task, allowing agents to query a real environment safely and construct observations without mutating shared state. Integrated with trajectory-level test-time scaling, AWM enables parallel exploration over a single shared environment, outperforming single-shot simulation by 40.07% and trained world-model baselines by 12.36% on benchmarks such as τ-bench, AppWorld, and OfficeBench. This approach provides a practical foundation for safe and scalable agentic test-time scaling in shared, non-resettable environments.

By Wendi Li, Huseyin A. Inan, Wei-Ning Chen, Yanzhi Chen, Chetan Bansal, Saravan Rajmohan, Sharon Li
arXiv AI
Jul 24

DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers

arXiv:2607. 20531v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful.

By Jerzy Kami\'nski, Ilya Galyukshev, Artem Kuznetsov, Sergey Chuprin, Kirill Redko, Aidar Shumbalov, Anna Kalyuzhnaya
arXiv AI
Sep 30

Boundary-State Control for Tool-Using Language-Model Agents: Commit-Time Consistency under State Drift

The paper introduces BSC‑R, a deterministic effect‑boundary mechanism that ties a single‑use commit authorization to the specific action and the semantic state that justified it, aiming to close the proposal‑to‑commit gap in tool‑using language‑model agents. Experiments on 2,847 AgentDojo episodes and 10,302 frozen proposals show that BSC‑R preserves the agent’s original behavior while rejecting unauthorized changes, and further tests on a boundary‑drift experiment and the CONTINUITY suite demonstrate high success rates in valid contexts and robust handling of replay and ambiguous cases. However, broader testing reveals that BSC‑R still allows a 25% invalid‑effect commit rate in a larger attack set, indicating that it provides scoped, not universal, safety.

By Wesley Shu
arXiv AI
Oct 15

SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

SolBehave is a benchmark that evaluates the behavioural correctness of smart contracts generated by large language models, going beyond simple compilation checks. It applies three tests—differential fuzzing, historical‑transaction replay, and DCR‑guided synthetic‑transaction replay—on 250 real Ethereum contracts to assess whether generated code matches the original’s behaviour. The study finds that while compilation success varies widely, only a minority of compiled contracts are behaviourally equivalent, highlighting that compilation alone is insufficient for correctness.

By Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni, Gianluca Moro, Lorenzo Bacchiani, Mario Bravetti, Thomas Troels Hildebrandt
arXiv AI
Aug 26

Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal

The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.

By Parker Fawcett