arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that co...
The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
arXiv:2606. 01252v1 Announce Type: cross Abstract: Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored.
arXiv:2601.16390v2 Announce Type: replace-cross Abstract: Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant la...
arXiv:2604. 18347v2 Announce Type: replace-cross Abstract: Vision Language Models (VLMs) achieved rapid progress in the recent years.
arXiv:2410. 07809v2 Announce Type: replace-cross Abstract: Multilingual instruction tuning (MIT) is challenged by the curse of multilinguality, data scarcity, and high computational cost.
arXiv:2605.28190v2 Announce Type: replace Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...
arXiv:2607. 19243v1 Announce Type: cross Abstract: Although Large Language Models (LLMs) demonstrate remarkable multilingual fluency, their internal knowledge representations remain disproportionately biased toward high-resource languages.
arXiv:2608.03446v2 Announce Type: replace Abstract: Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the giv...
The study introduces a zero‑shot benchmark for eight large language models on English and Bengali abstractive summarization, using both lexical/semantic metrics and reference‑free factuality checks. Results show significant cross‑lingual differences: GPT‑5.6 Luna excels in English but not in Bengali, where LLaMA 3.3 70B tops ROUGE and BLEU, while Gemini 2.5 Flash achieves the best Bengali factuality score. The evaluation also reveals that different metrics rank models differently, underscoring the need for multi‑dimensional assessment in low‑resource languages.
The paper proposes using multilingual sentiment lexicons to steer language models for sentiment control, especially in low-resource languages. It finds that aggregated multilingual lexicons outperform English-derived steering vectors for mid- and low-resource languages, while monolingual lexicons are still best for languages the model was pre-trained on. The study also explains how representational density, pre-training priors, and architectural biases affect steering effectiveness, noting that large English vectors can cause linguistic collapse in low-resource settings but this is mitigated by norm-dampening in multilingual aggregation.