arXiv AI

Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training

arXiv:2606. 26102v1 Announce Type: cross Abstract: Standard post-training pipelines apply supervised fine-tuning (SFT) and reinforcement learning (RL) to make language models helpful, but these processes may inadvertently degrade values instilled during pre-training.

arXiv AI

Aug 25

Alignment midtraining for animals

The paper studies how midtraining with synthetic documents can align AI models to the value of animal compassion. It introduces ANIMA, a 26‑question benchmark covering 13 ethical dimensions, and shows that training on 3,000 documents yields 77% accuracy versus 40% for instruction‑tuning. The improvement disappears after 5,000 samples, indicating that document‑based interventions may need explicit preservation strategies.

By Jasmine Brazilek, Miles Tidmarsh

benchmarks safety

arXiv AI

Jun 4

Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning

arXiv:2605. 16301v2 Announce Type: replace-cross Abstract: Evaluating animal welfare reasoning in LLMs remains an open challenge despite rapid deployment in consumer and professional contexts where welfare considerations appear implicitly in everyday queries.

By Isabella Luong, Joyee Chen, Arturs Kanepajs, Jasmine Brazilek, Sankalpa Ghose, David Williams-King, Linh Le, Allen Lu

llms benchmarks safety

arXiv Machine Learning

Jun 5

Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability

arXiv:2605. 03217v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning, yet existing bias evaluations treat model outputs as simply "biased" or "unbiased.

By Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur

llms efficiency safety

arXiv Computation and Language

Sep 17

Does Moral Reasoning Training Help or Hurt? Red-Teaming RL-Trained Ethical Agents with Persona Attacks

arXiv:2609.17552v1 Announce Type: new Abstract: Moral-reward RL can make language-model agents more cooperative, but whether that alignment survives adversarial persona pressure is unknown. Such atta...

By Arth Singh

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jul 15

Rethinking Reward Models for Multi-Domain Test-Time Scaling

arXiv:2510. 00492v3 Announce Type: replace Abstract: The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic.

By Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song

llms reinforcement-learning

arXiv AI

Sep 1

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

arXiv:2605.11467v2 Announce Type: replace-cross Abstract: Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberativ...

By Swapnil Parekh, Naman Goyal

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 14

Length Penalties Make Chain-of-Thought Less Monitorable

arXiv:2607. 09786v1 Announce Type: new Abstract: Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer.

By Bryce Little

reinforcement-learning benchmarks

arXiv AI

Aug 11

CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models

arXiv:2608. 08061v1 Announce Type: new Abstract: The key question in moral judgement is not simply whether someone chooses the "right" answer, but how they decide what matters most when moral principles conflict.

By Siddarth Singh, Victoria Williams, Simon Rosen, Ebenezer Gelo, Helen Sarah Robertson, Ibrahim Suder, Benjamin Rosman, Geraud Nangue Tasse, Steven James

llms agents robotics benchmarks

arXiv AI

4d ago

What Does Post-Training Change in Multilingual Reasoning?

arXiv:2609.37104v1 Announce Type: cross Abstract: Open-source reasoning models provide unequal access to reasoning capability across languages. When a model can solve a problem but cannot deliver a c...

By Hongyang Li, Xiao Li, Caesar Wu, Gr\'egoire Danoy, Pascal Bouvry

reinforcement-learning fine-tuning

arXiv AI

Jul 24

Training Large Language Models for Self-Explanation Faithfulness

arXiv:2607. 21090v1 Announce Type: cross Abstract: We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations - the extent to which a model's generated reasoning accurately reflects its internal decision-making process.

By Yeoktatt Cheah, Mar\'ia P\'erez-Ortiz, Noah Y. Siegel, Oana-Maria Camburu

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jun 9

From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning

arXiv:2606. 09092v1 Announce Type: new Abstract: Theory of Mind (ToM) is a must-acquire skill for modern foundation model systems to operate effectively and safely in the real world.

By Jike Zhong, Yuxiang Lai, Ming Li, Yuheng Li, Wuao Liu, Behzad Dariush, Konstantinos Psounis, Shao-Yuan Lo

reinforcement-learning fine-tuning multimodal

arXiv AI

Jul 20

Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

arXiv:2510. 12229v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear.

By Bianca Raimondi, Daniela Dalbagno, Maurizio Gabbrielli

llms fine-tuning safety

Related stories