arXiv Computation and Language
Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
Hugging Face Trending Papers
3d ago
Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training
Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different...
Hugging Face Trending Papers
2d ago
Verbalizing Activation Differences with Natural Language Autoencoders
Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing...
arXiv Computation and Language
1d ago
Verbalizing Activation Differences with Natural Language Autoencoders
The paper investigates natural language autoencoders (NLAs) as tools for verbalizing activation differences in large language models. It examines whether NLAs capture compositional structure, generalize to fine‑tuned models, recover fine‑tuning objectives, and can be specialized for model diffing. Experiments across four models show that NLAs largely succeed in these tasks, though they do not yet match state‑of‑the‑art diffing methods on emergent misalignment detection.
arXiv Computation and Language
Sep 17
Modelling Adjectival Modification Effects on Semantic Plausibility
The paper investigates how adjectival modifiers affect the semantic plausibility of events, using the Adept benchmark of 16,000 English sentence pairs that differ by a single adjective. Experiments show that sentence transformers, despite being conceptually suited to the task, underperform compared to models like RoBERTa. The authors provide an error analysis and discuss the implications of their findings for future work on balancing training and test data.
arXiv Computation and Language
Sep 4
The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification
The study examines how the geometric placement of synthetic data affects discourse‑pragmatic function classification. Using 410 annotated instances of the word "look" from the British National Corpus, synthetic examples were generated with Llama 3.1 and grouped by cosine distance from real data in RoBERTa space. Six training conditions were compared, showing that examples close to real data (NEAR) yield the largest macro‑F gain, while a distance‑balanced mix gives the highest accuracy, yet none improve AUC.
arXiv Machine Learning
Jun 2
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
arXiv:2606. 00230v1 Announce Type: new Abstract: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs.