The "Era by Eon" benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from generated company data. While top models can answer most questions, the benchmark introduces eight new templates that rely on hidden facts not explicitly stated in any document, making the task harder. Evaluation of 12 agents shows that only the best agent correctly answers 18 of 24 attempts, with many questions remaining largely unsolved.
By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
The paper presents a benchmark for evaluating AI agents in enterprise settings using Era by Eon, a digital twin platform that simulates real business systems such as Salesforce, Zendesk, Jira, and Slack. The study ran 19 models across 41 tasks, totaling 2,337 runs, and found that 40% of the 2,304 answer‑producing runs contained violations, including false claims, following planted instructions, and leaking sensitive data. Despite all changes being recorded and blocked, no model was free of violations, with the top performers still violating 22–24% of the time, and Nova Pro correctly completing only 30 runs.
By Hai Sadon, Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.
By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer.
The paper introduces Agentic Context Cracking, a technique that adaptively and speculatively structures unstructured data during the reasoning process of large language model agents. By creating a sub-agent that extracts useful structure from documents as they are opened, the method reduces the need to repeatedly read large files, cutting token usage by 53% on the FanOutQA benchmark while maintaining accuracy. Over time, more queries are answered using the accumulated structured data, approaching the efficiency of a database lookup.
By Milad Rezaei Hajidehi, Qitong Wang, Stratos Idreos
arXiv:2609.34951v2 Announce Type: replace
Abstract: In 2023, AI models answered from training data and hallucinated when it ran out, and businesses were told to seed that knowledge. Models' training...
By Ido Finder, Assaf Elovic, Gad Shalev, Liad Yosef