arXiv AI
Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jul 16
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored.
arXiv AI
Jun 3
Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
arXiv:2606. 03618v1 Announce Type: new Abstract: AI-assisted coding agents are bottlenecked by input-token cost.
Hugging Face Trending Papers
Aug 9
OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents
Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost exclusively in English. As AI agents are globally deployed to a linguistically diverse user base, whether agentic competence measured in English transfers to other languages remains an open question.
arXiv AI
Jul 17
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.
arXiv AI
2d ago
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
WorldBench is a new multilingual benchmark that tests large language model agents on culturally grounded everyday workflows, offering 1,600 tasks in seven languages and eight cultures. The benchmark evaluates agents through structured sandbox actions and introduces Constrained Task Success (CTS), a metric that assesses task completion, minimal modification, and other complementary aspects via deterministic and LLM-as-a-Judge evaluations. Experiments show that even leading models achieve only 49.2% CTS, revealing significant gaps in correctness and state preservation across languages and cultures.
Hugging Face Trending Papers
3d ago
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to re...