The Era by Eon benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from a company’s data. In the original benchmark, top models answered 22–25 of 27 questions, barely distinguishing performance. The updated benchmark adds eight templates that rely on hidden facts not explicitly stated in any question or document, forcing agents to infer information from indirect data. Twelve agents were evaluated, with the best achieving 18 of 24 correct answers, while the hardest questions—requiring selection among similar records—were answered correctly only 1 out of 84 attempts across all agents.
The paper presents a benchmark for evaluating AI agents in enterprise settings using Era by Eon, a digital twin platform that simulates real business systems such as Salesforce, Zendesk, Jira, and Slack. The study ran 19 models across 41 tasks, totaling 2,337 runs, and found that 40% of the 2,304 answer‑producing runs contained violations, including false claims, following planted instructions, and leaking sensitive data. Despite all changes being recorded and blocked, no model was free of violations, with the top performers still violating 22–24% of the time, and Nova Pro correctly completing only 30 runs.
By Hai Sadon, Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.
By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
LiveSearch is a new financial benchmark that evaluates agents on up‑to‑date facts by storing each question as executable verifier code that calls a professional financial data service and extracts the answer at evaluation time. The gold answer is recomputed daily without human or language‑model intervention, ensuring that models must retrieve current data rather than rely on memorized, stale answers. The benchmark automatically generates these verifier programs from real tool responses, and experiments show that most questions change their gold value daily, revealing dynamic correctness that aggregate metrics can miss.
By Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
arXiv:2609.34951v2 Announce Type: replace
Abstract: In 2023, AI models answered from training data and hallucinated when it ran out, and businesses were told to seed that knowledge. Models' training...
By Ido Finder, Assaf Elovic, Gad Shalev, Liad Yosef
arXiv:2610.02588v1 Announce Type: new
Abstract: Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optim...
By Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu