arXiv Computation and Language November 9, 2026 By Ruichen Zheng, Yihe Wang, Fabrice Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng

Does Reasoning Improve Psychological Depth in Large Language Models?

Read the original on arXiv Computation and Language →

The study examined whether reasoning-enhanced large language models produce psychologically deeper short stories. Using 60 blinded pairwise evaluations, human annotators found no consistent advantage for reasoning-oriented models: GPT‑5 slightly outperformed GPT‑4o, while DeepSeek‑R1 lagged behind DeepSeek‑V3. In contrast, an LLM-as-a-Judge consistently favored reasoning models, revealing a strong pro‑reasoning bias that may mask reader subjectivity in creative tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Sep 15

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

arXiv:2609.13773v1 Announce Type: new Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may...

By Ruichen Zheng, Yihe Wang, Fabrice Y Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng

llms

arXiv AI

Jul 17

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.

By Inder Preet, Shuxin Lin, Dhaval Patel

llms nlp benchmarks

arXiv AI

Oct 7

Universe of Thoughts: A Computational Framework for Creative Reasoning in Large Language Models

arXiv:2511.20471v3 Announce Type: replace Abstract: Recent advances in Large Language Model (LLM) reasoning have improved conventional problem solving, but creative reasoning remains comparatively un...

By Yuto Suzuki, Farnoush Banaei-Kashani

llms benchmarks

arXiv Computation and Language

Oct 1

Evaluating Whether LLMs Can Reliably Connect the DOTs?

arXiv:2609.38406v1 Announce Type: new Abstract: Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct mi...

By Eftekhar Hossain, John Salvador, Santu Karmaker

llms benchmarks

arXiv AI

Jul 10

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

arXiv:2601. 22588v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design.

By Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He

llms benchmarks

arXiv AI

Aug 17

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.

By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen

llms benchmarks

One email a morning, machine-written

Related stories

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

Universe of Thoughts: A Computational Framework for Creative Reasoning in Large Language Models

Evaluating Whether LLMs Can Reliably Connect the DOTs?

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI