Neo-Classic is a new benchmark designed to evaluate linguistic‑aesthetic reasoning in Classical Chinese poetry. It uses an out‑of‑sample dataset of strictly metrical poems written by contemporary experts and a set of reverse‑understanding probes, avoiding reliance on historical corpora. Experiments with leading LLMs show a 20–50% performance drop on contemporary texts and low accuracy (0–13%) on discourse‑level ordering, indicating that current models excel at local patterns but struggle with global hierarchical planning.
arXiv:2606. 12392v1 Announce Type: cross Abstract: Recently, large language models (LLMs) have achieved promising progress in the fields of classical Chinese translation and the generation of classical poetry.
The paper introduces Peony, a benchmark designed to evaluate large language models’ ability to comprehend the ‘poetic logic’ of modern Chinese poetry. It defines this logic through four tasks across stanza, line, and imagery levels and tests six mainstream LLMs under both non‑thinking and thinking configurations. Results show current LLMs struggle with this literary reasoning, highlighting Peony’s role in revealing these limitations.
Recently, large language models (LLMs) have achieved promising progress in the fields of classical Chinese translation and the generation of classical poetry. However, domain-specific research on precise translation and affective-semantic understanding of classical poetry remains limited.
arXiv:2608. 11452v1 Announce Type: cross Abstract: Text-to-image (T2I) models are increasingly asked to illustrate literary and cultural content, yet we cannot measure how well an image renders the meaning of a poem.
CNeo-Bench is a new benchmark comprising 4,759 Chinese neologisms, each with reference definitions and categorized by linguistic mechanisms such as phonetic substitution and visual character decomposition. The benchmark includes a two-tier evaluation framework that tests whether models can describe a neologism and whether they can manipulate its underlying mechanism. Evaluation of 18 large language models shows that most perform poorly on definition generation (below 40%) and exhibit a recognition‑manipulation gap, often paraphrasing rather than restoring the original form; few‑shot prompting helps but does not fully resolve the errors.
arXiv:2608. 03095v1 Announce Type: cross Abstract: We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese.
arXiv:2608.23098v1 Announce Type: new Abstract: Classical Chinese poetry composition has long valued Tuiqiao, the iterative refinement of words, imagery, and prosody. However, many current AI poetry...
arXiv:2607. 04147v1 Announce Type: cross Abstract: Automated fine-grained perception of calligraphy styles--a task vital to cultural heritage preservation--remains a critical challenge for Large Vision-Language Models (LVLMs), largely constrained by existing datasets that suffer from modal mixture and flattened labels.
arXiv:2608. 10444v1 Announce Type: cross Abstract: Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains.
The paper introduces the first large‑scale benchmark dataset of Bangla idioms, along with a synthetic multiple‑choice question set for idiom meaning identification. It evaluates recent large language models on three idiom‑related tasks—paraphrasing, idiom span detection, and meaning identification—using zero‑shot and few‑shot prompting. Results show significant variability across models, with Phi‑4‑mini‑instruct best at paraphrasing, Kimi‑K2‑32b‑instruct excelling at span detection, and Gemini‑2.5‑flash leading in meaning identification.
The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.
The study compares four approaches for Chinese sentence-level metaphor identification: BERT fine‑tuning, QLoRA-based large language model fine‑tuning, zero‑shot LLM prompting, and zero‑shot prompting with an expert‑informed procedural Skill. Results show that fine‑tuning yields the highest accuracy on the native test set, while the Skill‑based zero‑shot method provides the most stable performance across three datasets, achieving the highest external floor and the smallest performance range. Adding the Skill reduces false positives on one dataset but increases false negatives on others, indicating a trade‑off between precision and recall.