arXiv Computation and Language September 3, 2026 By Momose Oyama, Yusuke Takase, Hidetoshi Shimodaira

Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors

Read the original on arXiv Computation and Language →

The paper introduces a method that represents language models as log‑likelihood vectors over prompt‑response pairs, enabling the construction of model maps that compare conditional distributions. Squared Euclidean distances in this vector space approximate KL divergence, and experiments show that these maps reveal global structure related to model attributes and task performance. The approach also captures systematic shifts from prompt changes, supports additive compositionality for predicting downstream scores, and offers PMI vectors to mitigate unconditional distribution effects, thereby aiding analysis and prediction of input‑dependent behavior.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

llms

arXiv Machine Learning

Jun 25

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.

By Wa\"iss Azizian, Ali Hasan

llms

arXiv Machine Learning

Jun 5

Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution

arXiv:2606. 05486v1 Announce Type: cross Abstract: Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens.

By Govind Ramesh, Yao Dou, Wei Xu

llms benchmarks

arXiv Machine Learning

Jul 10

Prompt Compression via Activation Aggregation

arXiv:2607. 08399v1 Announce Type: cross Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response.

By Thibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder

llms

arXiv AI

Sep 1

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.

By Injin Kong, Hyoungjoon Lee, Yohan Jo

llms diffusion safety

arXiv AI

Aug 18

When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness

arXiv:2608. 16627v1 Announce Type: cross Abstract: Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL).

By Mahdi Dhaini, Adam Dejl, Juraj Vladika, Volkan \"Ozer, Barbara Plank, Gjergji Kasneci

llms benchmarks safety

arXiv Machine Learning

Jun 2

Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization

arXiv:2606. 00544v1 Announce Type: new Abstract: Modern language-model fine-tuning typically pairs each prompt with a single response, even though many prompts admit multiple valid completions.

By Hasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution

Prompt Compression via Activation Aggregation

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness

Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization