When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions.
arXiv:2607. 22841v1 Announce Type: cross Abstract: We present DS@GT's submission to FinMMEval 2026 Task 1, a multilingual financial exam question answering benchmark spanning English, Spanish, Greek, Chinese, and Hindi.
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv:2608. 15964v1 Announce Type: cross Abstract: Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt.
arXiv:2609.15079v1 Announce Type: cross Abstract: Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, eve...
The IGT system tackles PolyFiQA Task 2 of the FinMMEval Lab, a multilingual financial QA challenge involving English SEC filings and news in five languages. It distinguishes two question families: numeric‑structured queries are answered via keyword extraction from filings, while synthesis queries use rule‑based passage selection from news. The approach yields a development ROUGE‑1 of ~0.395, a 60% boost over a generic RAG baseline, and places third among twelve teams on the official test set.
arXiv:2605.29637v2 Announce Type: replace Abstract: Large language models often exhibit a substantial gap between their performance in English and in lower-resourced languages on equivalent knowledge...
arXiv:2609.37104v1 Announce Type: cross Abstract: Open-source reasoning models provide unequal access to reasoning capability across languages. When a model can solve a problem but cannot deliver a c...
The paper investigates how multilingual large language models can be guided to reason more reliably in low- to mid-resource languages by selecting appropriate language modes during inference. Experiments with LLaMA and Qwen models show that using English context can correct errors from non‑English comprehension, but adding redundant bilingual context can cause interference. To balance this trade‑off, the authors propose Reliability‑Aware Adaptive Inference (RAAI), a training‑free test‑time framework that routes prompts based on Expected Calibration Error and gates reasoning with a mid‑layer Risk Index, achieving up to 37.7% accuracy gains and reduced calibration error on low‑resource languages.
arXiv:2609.39001v1 Announce Type: cross Abstract: LLM services are billed per token and context windows are measured in tokens, yet the number of tokens needed for the same content varies across lang...
The paper introduces Multilingual GSM‑Symbolic, a large multilingual math dataset with 30,000 question‑answer pairs across 15 languages, designed to evaluate cross‑lingual capability transfer using symbolic templates that generate millions of variations. It quantifies key determinants of transfer—model size, language resource level, reasoning ability, and typological distance—showing that larger models and better reasoning reduce performance gaps between low‑ and high‑resource languages, while typological distance has limited impact. The authors present a predictive framework that explains 92 % of language‑to‑language variation and can estimate unseen language performance within 6 pp, improving to 4.2 pp with only ten target‑language templates.
arXiv:2607. 19856v1 Announce Type: cross Abstract: FinMMEval 2026 Task 1 evaluates multilingual financial multiple-choice question answering in English, Chinese, Arabic, and Hindi.