arXiv AI

MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks

arXiv:2507. 03162v2 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) has transformed various domains, particularly computer science (CS) education.

arXiv Computation and Language
Aug 31

Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts

The paper investigates why large reasoning language models struggle to transfer parametric knowledge across different scripts. Through observational data and regression analysis on ECLeKTic and MultiLoKo datasets, the authors find that script mismatch—not language family—is the main predictor of transfer failure when controlling for model capability and question difficulty. By providing key entities in the source language and training models to reason about transliteration ambiguities, they demonstrate a reduction in the cross‑script transfer gap, suggesting that post‑training improvements can enhance cross‑lingual knowledge transfer.

By Lucas Bandarkar, Alan Ansell, Trevor Cohn
arXiv AI
Jul 17

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.

By Saima Afrin, Alessandro Midolo, Camilo Escobar-Vel\'asquez, Mario Linares-V\'asquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo
Hugging Face Trending Papers
Jul 16

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored.

arXiv Machine Learning
Jun 5

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models

arXiv:2606. 05531v1 Announce Type: cross Abstract: Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and chart meaningful progress toward human-like multimodal intelligence.

By Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Mohamed Hefeeda, Ehsaneddin Asgari