arXiv Machine Learning
Jul 30

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

arXiv:2604. 18584v2 Announce Type: replace-cross Abstract: Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity.

By Shaden Alshammari, Kevin Wen, Abrar Zainal, Mark Hamilton, Navid Safaei, Sultan Albarakati, William T. Freeman, Antonio Torralba
arXiv Machine Learning
Oct 16

Lean Graph Search for Formal Theorem Proving

LeanGraphSearch is a graph‑based retrieval method for the Lean 4 theorem prover that builds a dependency graph from Mathlib’s forward dependencies and traverses it to find supporting lemmas via multi‑hop chains. By linking each declaration to its premises, propagating relevance through a graph‑ranking algorithm, and re‑ranking candidates against the original query, it improves recall by up to 12% over LeanSearch v2 on MathlibQR and increases coverage of complete annotated proof routes on MathlibMPR by up to 3.7%. When integrated into an iterative proving loop, LeanGraphSearch raises pass rates at 8, 16, and 32 reflection turns by as much as 40% compared to a vanilla prover and up to 10% compared to LeanSearch v2, demonstrating higher accuracy‑cost efficiency.

By Yihan Xu, Jia-Chen Gu, Junyi Zhang, Di Wu, Haikang Deng, Po-Nien Kung, Kai-Wei Chang, Amit Sahai, Wei Wang, Nanyun Peng
arXiv AI
Aug 24

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

Clarify-Then-Search is a benchmark that tests whether large language models can ask clarification questions to improve the usefulness of deep search results. It uses 518 real-world query pairs from Baidu, where each intent query is paired with an underspecified version. The evaluation involves a clarifier asking up to three questions, a user answerer providing only explicit information, and a rewriter generating a new query that is then searched; performance is measured by a weighted nugget-recall score.

By Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen
arXiv Computation and Language
Sep 10

Benchmarking Hybrid Deep Research Across Database Querying and Web Search

arXiv:2609.09410v1 Announce Type: new Abstract: While autonomous agents have made significant strides in "deep research" by iteratively navigating the open web to synthesize information, real-world p...

By Ruofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon, Yite Wang, Xiaodong Yu, Boyi Liu, Feng Yan, Debiao Li, Yuxiong He, Zhewei Yao