In LLM Reasoning, there is Irrationality on top of Value Misalignment

The paper introduces the concept of rational value risk, a gap between a large language model’s deployed reasoning strategy and an ideal rational strategy that maximises utility. It shows that even well‑aligned models can fail to achieve the aligned value during reasoning, and decomposes the estimation error into bounded prompts, bounded responses, and imperfect verifiers. Experiments across several LLMs and benchmarks confirm that rational value risk is common, can be mitigated but not eliminated by value alignment, and that techniques like self‑consistency and longer chains of thought can improve rationality with diminishing returns.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• benchmarks
• safety