arXiv AI

PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites

PatentAgentBench is a new benchmark that tests large language model agents on 510 real-world patent tasks across eight suites, including novelty search, freedom‑to‑operate, design FTO, invalidity search, claim charting, patent drafting, office‑action response, and patent translation. The benchmark uses curated patent records and expert‑reviewed annotations to evaluate task‑specific metrics, requiring agents to search patent collections without pre‑supplied candidate pools and to produce high‑quality drafting and response work. Seven LLMs were evaluated on four agent harnesses, revealing that current systems still struggle with novelty identification, risk assessment, and understanding practitioner intent throughout patent workflows.

arXiv AI
Oct 27

Novel or Not? PatEx-Gym: Agentic Prior-Art Search for Novelty Assessment in Patent Examination

PatEx-Gym is a new framework for developing and evaluating agents that perform patent examination. It includes PatEx-Bench, a benchmark of 1,000 realistic tasks with large retrieval corpora, and PASS, an annotation‑free method for synthesizing search trajectories. A 4B‑parameter PatEx‑Agent trained on these synthetic trajectories provides a lightweight baseline that competes with much larger general‑purpose models, yet prior‑art discovery remains challenging.

By Qiyao Wang, Xinyi Chen, Hongbo Wang, Jiayuan Lu, Jianhang Li, Minghuan Tan, Yuan Lin, Min Yang
arXiv Computation and Language
Oct 26

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...

By Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou, Weiqi Zhai, Qile Jin, Jiajia Li, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Shiwei Ye, Bing Zhao, Hu Wei, Lin Qu, Xianpei Han
Hugging Face Trending Papers
Aug 18

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

StartupBench is a benchmark that evaluates general‑purpose agents on end‑to‑end workflows derived from real AI startup products that have proven market adoption. It translates these product workflows into deliverable‑oriented tasks and assesses them with detailed rubrics that capture complex requirements. Even the best current models complete only about 30% of the tasks, highlighting failures in instruction following and domain expertise.

arXiv AI
Oct 19

Optimist Deep Research: An Iterative, State-Centric Agent for Autonomous Literature Research

Optimist Deep Research is an open‑source agent that conducts literature research as an iterative, state‑centric process. It breaks a research question into a tree of subquestions, uses search intents to query multiple scholarly databases, and merges results into a provenance‑preserving document registry. The agent iteratively expands the registry via citation snowballing, evaluates evidence, and refines the research plan through a critic that can add, split, rephrase, deprioritize, or close subquestions, ultimately producing a report with every citation linked to a retrieved document.

By Gihan Panapitiya
arXiv AI
Aug 19

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

StartupBench is a new benchmark that evaluates general-purpose agents on end-to-end workflows derived from AI startup products that have proven market adoption. It translates real-world product workflows into deliverable-oriented tasks and assesses them with detailed rubrics. The study finds that even the best models complete only about 30% of these tasks, highlighting challenges such as complex instruction following and domain expertise.

By Liya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang, Jingzhe Ding, Qingshui Gu, Yongjie Zhong, Jinxiang Meng, Yuan Gao, Yunqiu Zhou, Hao Zhu, Jifeng He, Yongzhi Liao, Xinyi Zhang, Chaoxin Li, Yi Zhu, Xi Lin, Duju Zeng, Xiang Gao, Wen Zhang, Yunyang Wang, Duo Wang, Huan Zhou, Zuo Wang, Jin Chen, Kaiyuan Zhang, Chuqian Yu, Tianhao Yu, Longxiang Liu, Jianbo Xue, Huimin Che, Jiahao Wang, Yujia Qin, Jiaheng Liu, Shen Yan, Xiaolong Chang, Wenhao Huang
Hugging Face Trending Papers
Jul 11

When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation

Patent claim drafting is a challenging legal drafting task that requires technical expertise, precise linguistic control, strict adherence to formal conventions, and the preservation of complex logical relationships among claim elements. While Chain-of-Thought (CoT) prompting has been widely used to improve the reasoning capabilities of large language models (LLMs), recent evidence suggests that its benefits may be limited, or even negative, in highly structured or pattern-sensitive tasks.