← Back to all news
arXiv Machine Learning By Wenxi Dai, Jingyu Hu, Yanting Yang, Mengnan Du

From Representation to Readout: Mechanistic Analysis of Membership Signals in Language Models

Read the original on arXiv Machine Learning →

The paper investigates how membership inference signals arise in large language models by distinguishing between information encoded in internal representations and the extent to which it is accessed during readout. It introduces three metrics—Decodable Information Score, Used Information Score, and Readout Accessibility Gap (DUG)—to quantify these aspects. Experiments show that supervised fine‑tuning affects both decodable information and readout, whereas reinforcement‑learning‑based inference mainly influences readout, with intermediate layers identified as key sites of membership effects.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 27

Demystifying Reinforcement Learning Post-Training of Language Models

The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers. "whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."

By Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
llmsnlpreinforcement-learningfine-tuning
More like this →
arXiv AI
6d ago

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

The paper investigates whether reinforcement learning (RL) truly endows large language models (LLMs) with reasoning skills or merely tunes them to exploit prompt information. By training small LLMs on the finite logical domain of Tic‑Tac‑Toe and applying Group Relative Policy Optimization (GRPO), the authors show that RL improves accuracy and robustness. However, mechanistic interpretability using sparse autoencoders reveals that the models mainly rely on explicit prompt cues—such as turn order, game progression, and board occupancy—while higher‑level concepts like board symmetries and strategic forks remain weakly represented, suggesting that perceived reasoning may be an artifact of task‑specific adaptations.

By Aishik Nagar, Cheston Tan
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv AI
Jul 16

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

arXiv:2510. 11686v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.

By Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash
llmsreinforcement-learning
More like this →