arXiv AI

Continual Enterprise World Model Discovery in Dynamic Systems

The paper introduces a method for continual enterprise world model discovery, enabling an agent to learn and adapt to business rules in dynamic systems without prior knowledge. Using a ServiceNow environment called EnterpriseWorldShift, the authors evaluate their Continual Discovery Agent (CDA) across four rule-modification scenarios—discovery, revision, extension, and retirement—showing that CDA predicts rule effects more accurately than lookup-based approaches, improving IoU by up to 8.98 points. The agent can answer queries from its internal model without querying the live system.

Hugging Face Trending Papers
Sep 24

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

The Era by Eon benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from a company’s data. In the original benchmark, top models answered 22–25 of 27 questions, barely distinguishing performance. The updated benchmark adds eight templates that rely on hidden facts not explicitly stated in any question or document, forcing agents to infer information from indirect data. Twelve agents were evaluated, with the best achieving 18 of 24 correct answers, while the hardest questions—requiring selection among similar records—were answered correctly only 1 out of 84 attempts across all agents.

arXiv AI
Sep 25

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

The "Era by Eon" benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from generated company data. While top models can answer most questions, the benchmark introduces eight new templates that rely on hidden facts not explicitly stated in any document, making the task harder. Evaluation of 12 agents shows that only the best agent correctly answers 18 of 24 attempts, with many questions remaining largely unsolved.

By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Jul 24

DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers

arXiv:2607. 20531v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful.

By Jerzy Kami\'nski, Ilya Galyukshev, Artem Kuznetsov, Sergey Chuprin, Kirill Redko, Aidar Shumbalov, Anna Kalyuzhnaya