From Copying to Composition: How Language Models Develop In-Context Learning
Read the original on arXiv Machine Learning →The study investigates how language models acquire in-context learning (ICL) by distinguishing between copying a nearby demonstration and integrating information across a prompt. Across 17 models, roughly one-third default to copied answers, a few are undecided, and the remainder favor rule-consistent answers that yield higher accuracy and are guided by the requested value rather than the nearest conflicting example. Analyses show that models prioritize task-relevant prompt tokens before accuracy surpasses chance, and that synthetic or fine-tuned training can shift models toward compositional responses or improve accuracy when prompts reassign label meanings.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.