arXiv Computation and Language

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

The paper introduces Peony, a benchmark designed to evaluate large language models’ ability to comprehend the ‘poetic logic’ of modern Chinese poetry. It defines this logic through four tasks across stanza, line, and imagery levels and tests six mainstream LLMs under both non‑thinking and thinking configurations. Results show current LLMs struggle with this literary reasoning, highlighting Peony’s role in revealing these limitations.

arXiv Computation and Language

Sep 18

Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

Neo-Classic is a new benchmark designed to evaluate linguistic‑aesthetic reasoning in Classical Chinese poetry. It uses an out‑of‑sample dataset of strictly metrical poems written by contemporary experts and a set of reverse‑understanding probes, avoiding reliance on historical corpora. Experiments with leading LLMs show a 20–50% performance drop on contemporary texts and low accuracy (0–13%) on discourse‑level ordering, indicating that current models excel at local patterns but struggle with global hierarchical planning.

By Han Zhang, Zihan Gu, Zhiyuan Wang, Tianyi Ma, Jiacheng Lu, Xinyan Zhang, Yuhao Wei, Cheng Hua

llms benchmarks

arXiv AI

Jun 11

System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5

arXiv:2606. 12392v1 Announce Type: cross Abstract: Recently, large language models (LLMs) have achieved promising progress in the fields of classical Chinese translation and the generation of classical poetry.

By Haotao Xie

llms fine-tuning benchmarks safety

Hugging Face Trending Papers

Jun 10

System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5

Recently, large language models (LLMs) have achieved promising progress in the fields of classical Chinese translation and the generation of classical poetry. However, domain-specific research on precise translation and affective-semantic understanding of classical poetry remains limited.

llms fine-tuning benchmarks safety

arXiv AI

Aug 13

TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation

arXiv:2608. 11452v1 Announce Type: cross Abstract: Text-to-image (T2I) models are increasingly asked to illustrate literary and cultural content, yet we cannot measure how well an image renders the meaning of a poem.

By Haoqi Hu, Tongji Luo, Li Zhang, Boning Zhou

llms diffusion benchmarks

arXiv AI

Aug 25

Jiuge-Tuiqiao: An Interpretable Human-AI System for Classical Chinese Poetry Refinement

arXiv:2608.23098v1 Announce Type: new Abstract: Classical Chinese poetry composition has long valued Tuiqiao, the iterative refinement of words, imagery, and prosody. However, many current AI poetry...

By Yufeng Han, Lifan Deng, Cunliang Kong, Wenhao Li, Xin Cong, Yuzhuo Bai, Kangyang Luo, Maosong Sun

agents safety

arXiv AI

Jun 24

Predicting Poets' Origins from Verse: A Computational Analysis of Regional Linguistic Fingerprints in the Complete Tang Poems

arXiv:2606. 24093v1 Announce Type: cross Abstract: We ask whether the geographic origin of Tang-dynasty poets leaves a detectable linguistic trace in their work.

By Chi-Sheng Chen, Hung-Yun Liu

llms

arXiv Computation and Language

Sep 1

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

arXiv:2608.28986v1 Announce Type: new Abstract: LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a K...

By Keunhyeung Park, Seunguk Yu, YoungBin Kim

llms

arXiv Computation and Language

1d ago

MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors

arXiv:2607.00848v4 Announce Type: replace Abstract: In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present ch...

By Jiahui Liang, Lifeng Han

llms rag nlp benchmarks

arXiv Computation and Language

3d ago

HaikuS2S: A Cascaded System For Responding In Verse

arXiv:2609.23951v1 Announce Type: new Abstract: Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior w...

By Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe

llms fine-tuning multimodal safety

arXiv AI

Sep 15

Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku

arXiv:2609.15511v1 Announce Type: cross Abstract: This paper investigates the generation and human evaluation of Japanese haiku by contemporary Large Language Models (LLMs), focusing on authorship pe...

By Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese

llms safety

arXiv Machine Learning

Aug 5

VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

arXiv:2608. 03095v1 Announce Type: cross Abstract: We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese.

By Tu Tran Do, Nhat Ngoc Nguyen, Khanh-Tung Tran, Hoang D. Nguyen, Tu Minh Phuong, Long Hoang Dang

llms nlp benchmarks

arXiv AI

Aug 18

A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs

arXiv:2510. 06039v2 Announce Type: replace-cross Abstract: Reliable evaluation of knowledge-grounded Large Language Models (LLMs) in Chinese requires resources that explicitly align Chinese-language text with verifiable Knowledge Graph (KG) facts.

By Chengwei Wu, Xingrui Zhuo, Mingyang Gao, Xinghe Cheng, Zhichao Yan, Jiapu Wang

llms computer-vision nlp fine-tuning benchmarks safety

Related stories