arXiv Computation and Language

IFRS Compliance Verification: An Open Multi-LLM Benchmark and Content-Conditioned Ensemble Classifier

The paper introduces a new benchmark of 19,998 IFRS regulation‑disclosure pairs and a multi‑stage framework that combines domain‑adapted sentence embeddings with a learned ensemble classifier to verify corporate financial disclosures against IFRS. The ensemble, trained on GPT‑4.1, Claude Sonnet 4, and Gemini 2.5 Pro, achieves 81.7% weighted F1 on the disagreement set—outperforming the best single LLM by 10.4 points and majority voting by 14.6 points—and the retrieval pipeline reaches 45.3% Recall@15 on unseen regulations. All data, models, and code will be publicly released.

Hugging Face Trending Papers
Jul 29

Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text

Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways. Detecting a conflict is only the first step: review workflows may also need to determine its type, since numerical, temporal, referential, factual, and normative inconsistencies require different evidence and downstream checks.

arXiv AI
Sep 25

Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation

The paper presents a retrieval‑augmented generation pipeline for answering regulatory compliance questions in finance. It builds a three‑stage retriever on LegalBERT and a compact 2B–12B generator served with 4‑bit quantization, achieving a Recall@10 of 0.774 on the ObliQA benchmark and improving answer quality via RAFT‑LoRA fine‑tuning. However, the adapted models fail to transfer to Australian case‑law questions, and a closed‑book model performs almost as well while lacking verifiable grounding.

By Tobias Deu{\ss}er, Abhishek Pillai, Aurelio F. Bariviera, Dhananjay Bhardwaj, Lorenz Sparrenberg, David Berghaus, Christian Bauckhage, Rafet Sifa
arXiv Machine Learning
Aug 11

UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers

arXiv:2608. 09209v1 Announce Type: cross Abstract: Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on adversarial or out-of-distribution inputs.

By Chidaksh Ravuru, Shashank Srivastava
arXiv AI
Jun 12

Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

arXiv:2602. 07294v4 Announce Type: replace-cross Abstract: With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures.

By Yidong Jiang, Junrong Chen, Eftychia Makri, Jialin Chen, Peiwen Li, Ali Maatouk, Leandros Tassiulas, Eliot Brenner, Bing Xiang, Rex Ying
Hugging Face Trending Papers
Oct 29

ScienceJudge: Pattern Perturbation Augmentation for Sentence-Level Scientific Claim Verification

ScienceJudge introduces Pattern Perturbation Augmentation (PPA) to generate hard negative examples for sentence‑level scientific claim verification by modeling reporting errors as faithfulness‑breaking transformations. PPA applies these patterns to supported claims, creating controlled hard negatives and expanding coverage through pattern‑level, operation‑level, and cross‑domain augmentation. The resulting ScienceJudge system, which ensembles PPA‑trained vision‑language predictors with instruction‑guided LLM predictors, achieves state‑of‑the‑art performance, improving the average score from 0.8124 to 0.8692 and ranking first on the NLPCC 2026 Task 10 Track 1 leaderboard.

arXiv AI
Jun 3

Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.

By Andrea Brunello, Cristian Curaba, Luca Geatti, Michele Mignani, Angelo Montanari, Nicola Saccomanno
arXiv AI
Sep 2

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.

By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate