arXiv Computation and Language

A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

arXiv AI
Sep 3

Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English

The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.

By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
arXiv AI
Jul 22

Benchmarking Generalization in Financial Statement Fraud Detection: robust evaluation and novel tasks

arXiv:2607. 19259v1 Announce Type: cross Abstract: Financial statement fraud detection (FSFD) is crucial for market integrity but faces challenges from increasingly sophisticated schemes and under-utilized textual data in financial reports.

By Guy Stephane Waffo Dzuyo (Forvis Mazars, LORIA CNRS Universit\'e de Lorraine), Ga\"el Guibon (LORIA CNRS Universit\'e de Lorraine, LIPN CNRS Universit\'e Sorbonne Paris Nord), Christophe Cerisara (LORIA CNRS Universit\'e de Lorraine), Luis Belmar-Letelier (Forvis Mazars)
arXiv AI
Jun 9

Summarization is Not Dead Yet

arXiv:2606. 08000v1 Announce Type: cross Abstract: The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem.

By Dongqi Liu, Chenxi Whitehouse, Zheng Zhao, Zhuchen Cao, Jian Li, Yabiao Wang
arXiv AI
Sep 4

Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements

FinRAG-QA is a new benchmark dataset for financial question answering, featuring 999 practitioner-curated questions on 10 standardised indicators drawn from 209 annual and Pillar 3 reports of 24 major European and U.S. banks between 2019 and 2023. The dataset focuses on cross‑institutional retrieval over documents averaging 198k words, making it longer than any existing financial QA resource. Experiments on a multi‑stage Retrieval‑Augmented Generation pipeline show that contextual chunk enrichment and a retrieval‑optimised embedding model significantly improve NDCG@10, while a reasoning‑optimised generator boosts answer accuracy from 44.6% to 79.0% when the correct document is retrieved.

By Arianna Miola, Bruno Spaccavento, Lorenzo Silotto, Marco Bianchetti, Luca Cagliero