Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries

The paper investigates how large language models (LLMs) can produce incorrect answers to financial queries that are underspecified, a phenomenon termed "frame collapse." By testing models on Indian mutual fund data—where product variants and tax rules are formally defined—the authors show that LLMs often rely on ambiguous labels, default assumptions, or outdated rules, leading to answers that match the data but answer a different question. The study demonstrates that even a judge that checks for mislabelled figures can miss these errors if the model silently chooses an alternate interpretation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents