The paper introduces ${ au}$-bench-LILT, a multilingual extension of the ${ au^3}$-bench benchmark that evaluates conversational agents on realistic tool‑use tasks in German, Korean, Turkish, and Farsi across airline, retail, telecom, and banking domains. It localizes every layer of the benchmark, including culturally appropriate customer databases, native‑language user‑simulator prompts refined by contrastive annotation, and tasks tailored to realistic regional scenarios. Evaluation of eight agent models reveals the most significant performance drops in the Korean and Farsi retail domains, with reductions of up to 61 points in pass^4.
By Kaden Uhlig, Kiarash Mokhtari, An{\i}l Arslan, Yunsu Kim, Joern Wuebker
The paper introduces SalesLLM, a bilingual (Chinese/English) benchmark for evaluating large language models (LLMs) in realistic sales dialogues. It comprises 30,074 scripted configurations and 1,805 curated multi‑turn scenarios from Financial Services and Consumer Goods, with controllable difficulty and personas. An automatic evaluation pipeline uses an LLM judge for sales‑process progress and fine‑tuned BERT classifiers for end‑of‑dialogue buying intent, while a user model, CustomerLM, is trained to improve simulation fidelity. SalesLLM scores correlate strongly with human ratings (Pearson r = 0.86) and reveal that top Chinese LLMs match junior‑to‑intermediate human salespeople but not experts, with cross‑lingual consistency remaining poor.
By Xuanbo Su, Wenhao Hu, Le Zhan, Yuting Xie, Kailin Lyu, Kaijie Chen, Ziwei Li, Yeqiang Wang, Haibo Su, Yunzhang Chen, Ling Huang
arXiv:2609.15079v1 Announce Type: cross
Abstract: Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, eve...
By Kushagra Agrawal, Yuming Feng, Man-Fai Leung
arXiv:2610.27205v1 Announce Type: cross
Abstract: Web agents are typically evaluated on English-language interfaces, leaving their ability to operate across the world's multilingual websites largely...
By Ocean Monjur, Daud Ahmad Nisar, Shahriar Kabir Nahin, Tadesse Destaw Belay, Sushruti Mishra, Israel Abebe Azime, Anshuman Chhabra
arXiv:2606. 15862v1 Announce Type: new Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.
By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang
Compass‑v3 is a 245B‑parameter Mixture‑of‑Experts language model tailored for Southeast Asian e‑commerce, featuring 71B active parameters per token and hardware‑efficient expert parallelism. It is trained on 12 trillion multilingual tokens and synthetic e‑commerce instructions, and incorporates Optimal‑Transport Direct Preference Optimization to improve instruction adherence. Benchmarks show it outperforms GPT‑4, DeepSeek‑V3.1, and Qwen3‑235B, and it is already deployed at scale on Shopee, handling over 70% of the platform’s LLM traffic.
By Sophia Maria
arXiv:2606. 28715v1 Announce Type: cross Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI.
By My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya
TauIndianBankBench is a new benchmark that evaluates conversational agents on 1,000 Indian retail‑banking tasks by checking not only the final records but also whether every required procedural step was performed and no unrequested changes were made. It highlights that models can appear successful when judged only on final records, yet may skip necessary steps or alter records incorrectly. The benchmark provides a more accurate assessment of language model performance in the Indian banking domain.
By Krishanu Adhikary, Prashant Devadiga
arXiv:2610.19638v1 Announce Type: new
Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While...
By Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa, Ryota Hirobuchi
arXiv:2603. 16453v3 Announce Type: replace Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.
By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang
arXiv:2604. 04532v2 Announce Type: replace-cross Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings.
By Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban
arXiv:2606.28715v2 Announce Type: replace-cross
Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly und...
By My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya