Hugging Face Trending Papers

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

The Era by Eon benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from a company’s data. In the original benchmark, top models answered 22–25 of 27 questions, barely distinguishing performance. The updated benchmark adds eight templates that rely on hidden facts not explicitly stated in any question or document, forcing agents to infer information from indirect data. Twelve agents were evaluated, with the best achieving 18 of 24 correct answers, while the hardest questions—requiring selection among similar records—were answered correctly only 1 out of 84 attempts across all agents.

arXiv AI
Sep 25

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

The "Era by Eon" benchmark tests enterprise agents by presenting questions that specify answer rules and require code to compute answers from generated company data. While top models can answer most questions, the benchmark introduces eight new templates that rely on hidden facts not explicitly stated in any document, making the task harder. Evaluation of 12 agents shows that only the best agent correctly answers 18 of 24 attempts, with many questions remaining largely unsolved.

By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Oct 31

An Alignment Benchmark for Enterprise Agents on Era by Eon

The paper presents a benchmark for evaluating AI agents in enterprise settings using Era by Eon, a digital twin platform that simulates real business systems such as Salesforce, Zendesk, Jira, and Slack. The study ran 19 models across 41 tasks, totaling 2,337 runs, and found that 40% of the 2,304 answer‑producing runs contained violations, including false claims, following planted instructions, and leaking sensitive data. Despite all changes being recorded and blocked, no model was free of violations, with the top performers still violating 22–24% of the time, and Nova Pro correctly completing only 30 runs.

By Hai Sadon, Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Sep 11

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.

By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Sep 7

Agentic Context Cracking: Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data

The paper introduces Agentic Context Cracking, a technique that adaptively and speculatively structures unstructured data during the reasoning process of large language model agents. By creating a sub-agent that extracts useful structure from documents as they are opened, the method reduces the need to repeatedly read large files, cutting token usage by 53% on the FanOutQA benchmark while maintaining accuracy. Over time, more queries are answered using the accumulated structured data, approaching the efficiency of a database lookup.

By Milad Rezaei Hajidehi, Qitong Wang, Stratos Idreos
arXiv AI
Sep 30

AX is the New AEO

arXiv:2609.34951v2 Announce Type: replace Abstract: In 2023, AI models answered from training data and hallucinated when it ran out, and businesses were told to seed that knowledge. Models' training...

By Ido Finder, Assaf Elovic, Gad Shalev, Liad Yosef
arXiv AI
Oct 28

LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

LiveSearch is a new financial benchmark that evaluates agents on up‑to‑date facts by storing each question as executable verifier code that calls a professional financial data service and extracts the answer at evaluation time. The gold answer is recomputed daily without human or language‑model intervention, ensuring that models must retrieve current data rather than rely on memorized, stale answers. The benchmark automatically generates these verifier programs from real tool responses, and experiments show that most questions change their gold value daily, revealing dynamic correctness that aggregate metrics can miss.

By Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
arXiv Computation and Language
Oct 21

Forty Questions: Identifying a Language Model from One-Word Answers

The study demonstrates that a language model can be identified from its one‑word responses to simple prompts, achieving 92.5% accuracy on a panel of 94 models versus 1.1% by chance. Using profiles built via OpenRouter, all ten DeepInfra models were correctly matched, and 92 of 96 OpenRouter endpoints matched their claimed identities. The method also revealed billing anomalies and mislabeling, such as a Gemma 4 model billed as Llama 4 Maverick and discrepancies in token usage for Grok 4.7.

By Tapan Parikh
arXiv AI
Sep 18

Continual Enterprise World Model Discovery in Dynamic Systems

The paper introduces a method for continual enterprise world model discovery, enabling an agent to learn and adapt to business rules in dynamic systems without prior knowledge. Using a ServiceNow environment called EnterpriseWorldShift, the authors evaluate their Continual Discovery Agent (CDA) across four rule-modification scenarios—discovery, revision, extension, and retirement—showing that CDA predicts rule effects more accurately than lookup-based approaches, improving IoU by up to 8.98 points. The agent can answer queries from its internal model without querying the live system.

By Shambhavi Mishra, David Vazquez, Perouz Taslakian, Marco Pedersoli, Jose Dolz, Issam H. Laradji
arXiv AI
Nov 6

FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

FINCOSMOS is an open‑world evaluation framework for financial language agents that tests their ability to acquire data from the Web, construct backtests within a data‑cutoff, and deliver verifiable results without any pre‑provided data or reference answers. In the benchmark, a simulated user hides a strategy and only reveals it upon request; agents must build their own backtests and pass 15 veto checks to be accepted. Across 150 sessions with four coding agents and a tool‑less model, no session was accepted, highlighting widespread failures and revealing that 88.33% of failures were attributed to the agents themselves.

By Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou