The study evaluates whether aggregated brand recommendation profiles can identify the language model that generated them. Using 6,475 responses from five deployed endpoints, a character‑n‑gram classifier accurately attributes single responses to the correct system (97.84% accuracy). However, when responses are aggregated into domain‑condition units, the classifier’s performance drops to 66.53%, and a forest model misclassifies all gift‑domain units, indicating that aggregated brand behaviour does not reliably reveal the underlying system.
By Dmitrij \.Zatuchin
The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.
By Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen
arXiv:2610.32144v1 Announce Type: new
Abstract: When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, an...
By Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
The study investigates how different prompting strategies affect large language model (LLM) judges across multiple languages and tasks. Using a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks, the authors collected over 1 million pointwise scores validated against human judgments. Results show that prompting effectiveness is highly dependent on the judge model, can reverse direction across judges, and becomes less stable in low‑resource languages, indicating that prompt choice must be tailored to the specific evaluation setting.
By Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
arXiv:2606. 23701v1 Announce Type: cross Abstract: Qualitative product feedback can reveal nuanced user experiences, but its implicit sentiment is difficult to measure.
By Sherri Weitl-Harms, John Hastings
The paper introduces the latent space item response model (LSIRM) as a tool for uncovering how language models differ on specific benchmark items. By mapping 1,348 models across 5,761 BBH items and comparing scoring methods across BBH, GPQA, MMLU-Pro, and MuSR, the authors show that items with similar correct answers cluster together, especially when scored by option labels. LSIRM separates a model’s tendency to perform better on certain answer categories from other performance differences, revealing that small changes in scoring can dramatically alter model rankings.
By Jinwen Luo, Han Jiang, Ziang Xiao, Susu Zhang