← Back to all news
arXiv AI By Ritiz Tambi, Sahar Shayegan, Joshua Meyer, Sun Kim, Ali Khan, Victor Shih, Mehdi Jamei, Andi Partovi

TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?

Read the original on arXiv AI →

TreasuryQA Bench is a new benchmark that tests AI agents on real-world tasks involving searching, reading, and extracting exact figures from scanned U.S. Treasury PDFs stored on a shared drive. The benchmark presents 70 questions over 1,435 scanned documents, requiring agents to locate the relevant file via Dropbox’s Model Context Protocol, OCR the necessary pages, and return a precise numeric answer. Results from ten agents show a wide performance gap, with the best agent solving 64.8% of tasks and the worst only 16.7%, and scan quality is shown to significantly impact success rates.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 12

Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA

KnowForge is a modular pipeline for open‑domain multimodal document question answering that separates retrieval, judging, and reading into distinct agents. It uses a hybrid‑retrieval knowledge‑point index, a Locator agent for searching, a Bridge Judge to enforce a second hop, a Deep Reader that handles text, images, and tables, and a synthesizer with format checking. On the M3DocVQA benchmark it achieves an F1 of 47.5, outperforming prior systems by 11 points, and shows strong gains on MultiHop‑RAG and MMLongBench‑Doc as well.

By Wenlin Lin, Fanzhe Wei
llmsragagentsnlpmultimodalbenchmarks
More like this →
arXiv AI
Oct 23

CompLit: Scientific Literature Search Benchmarks Must Cover Implicit, Cumulative, and Unmet Needs

CompLit is a new literature search benchmark covering all eight arXiv domains, designed to evaluate agents on three diagnostic settings: implicit, cumulative, and unmet needs. The benchmark shows that top agents like Codex and Claude Code perform well on standard queries (86‑87%) but drop to 66‑68% on implicit queries, lose additional accuracy when requirements accumulate, and abstain on up to 53% of unmet queries. The study finds that agents often locate the correct or near‑correct paper but misjudge it, indicating a need for more reliable search capabilities.

By Dayoon Ko, Jihyuk Kim, Soyeong Jeong, Young-jun Lee, Dahyun Lee, Juyeon Kim, Gunhee Kim, Moontae Lee, Kyungjae Lee
llmsagentsbenchmarks
More like this →