The paper "Evaluating Style-Personalized Text Generation: Challenges and Directions" examines the difficulties of assessing text that is tailored to individual users’ styles. It critiques common metrics such as BLEU, embeddings, and LLM-as-judges, and introduces a style discrimination benchmark covering domain discrimination, authorship attribution, and LLM-generated personalized versus non-personalized discrimination across eight writing tasks. The study finds that ensembles of diverse evaluation metrics outperform single-evaluator approaches and offers guidance for reliable assessment of style-personalized generation.
arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that co...
arXiv:2609.22112v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated the ability to generate user-specific text with high stylistic fidelity. However, the personal data that...
arXiv:2608.23124v1 Announce Type: cross Abstract: Personalized text generation for authors and literary writing is essential for applications such as adaptive writing assistants, creative support too...
Personalized text generation for authors and literary writing is essential for applications such as adaptive writing assistants, creative support tools, and computational literary analysis. However, e...
arXiv:2608. 19746v1 Announce Type: new Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing.
The paper examines whether existing automatic methods can reliably assess creativity in text produced by large language models (LLMs). By collecting human ratings on 11 creativity dimensions for both human and AI short stories, the authors compare these judgments with automated metrics and LLM-as-a-Judge evaluations. The results show a significant misalignment: automated metrics and LLM judges favor AI-generated stories and show near-zero correlation with human assessments, revealing fundamental limitations in current computational approaches to evaluating creative text.
The paper introduces PerFine, a training‑free critique‑refine framework that improves large language model personalization. In each iteration, a generator drafts text conditioned on a retrieved user profile, while a critic—also conditioned on the same profile—provides structured feedback on tone, vocabulary, sentence structure, and topicality. The generator revises the draft, and a knockout strategy selects the best version across iterations; additional inference‑time techniques such as Best‑of‑N and Topic Extraction further balance quality and efficiency. Experiments on Yelp, Goodreads, and Amazon datasets show PerFine consistently outperforms existing retrieval‑augmented methods like PGraphRAG, achieving G‑Eval gains of +8.5‑12% and steady improvements over 3‑5 refinement iterations.
The paper introduces VOLM, a framework that quantifies how much original value a human adds to a document beyond what a language model could generate from a task description alone. Unlike existing tools that focus on stylistic detection, VOLM extracts content at varying granularities, reconstructs it with an LLM, and compares these reconstructions to those derived from the task description. Evaluations across news articles, ICLR peer reviews, and argumentative essays show that VOLM can distinguish human-authored texts from LLM-generated ones while remaining robust to content-preserving transformations.
The study investigates bias in large language model (LLM) judges by having ten LLMs evaluate narrative constraint selections rather than generated text. Results show that self-preference largely disappears under blind evaluation when quality and evaluator severity are controlled, but self- and other-labels alone shift scores bidirectionally when quality is matched. The authors conclude that authorship attribution drives evaluation bias and that open-ended, ground‑truth‑free tasks can effectively study LLM judge behavior.
arXiv:2310.00436v2 Announce Type: replace Abstract: Authorship identification uses patterns in writing to infer who wrote a text, but those patterns also reflect topic, genre, and register. This surv...
arXiv:2510.13302v4 Announce Type: replace-cross Abstract: Computational stylometry studies writing style through quantitative textual patterns, enabling applications such as authorship attribution, i...
The paper introduces FAVoR, a method for federated personalized generation that mitigates author‑style homogenization caused by standard aggregation in parameter‑efficient fine‑tuning. Using the BlogText benchmark and ASCE diagnostics, the authors show that common federated PEFT baselines preserve semantic utility but blur author‑specific style. FAVoR employs a shared‑private adapter design, where clients upload shared updates while keeping author‑specific residual corrections locally, leading to improved style retention with minimal utility loss.