The paper introduces Adaptive Local Relational Alignment (ALRA), a logit‑based knowledge distillation method for autoregressive language models that combines student‑generated token proposals with teacher guidance at each prediction position. ALRA dynamically selects the number of candidate tokens based on the teacher’s probability spread, uses Adaptive Local Divergence to match both mass and relative token distributions, and applies Student‑Weighted Pairwise Relational Alignment to focus on high‑probability token pairs. Experiments on The Pile show that 200M‑ and 500M‑parameter students trained with ALRA outperform the best baseline by roughly 1 percentage point and surpass pre‑training without distillation by over 2 percentage points on nine zero‑shot benchmarks.
By Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc, Vo Nguyen Le Duy
The paper examines the intermediate‑entropy regime of natural language generation, where a single input can have multiple valid outputs such as in peer‑review generation and image captioning. It introduces a target‑side consolidation method that merges all references into a single target preserving every distinct information unit, and presents the MERC‑67K dataset of 36,537 papers and 31,014 images with both original and consolidated references. Human evaluations show that models trained on consolidated references achieve higher coverage and overall quality scores, outperforming multi‑reference baselines across several large language models while using fewer individual references during training.
By Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary, Manish Shrivastava, Vasudeva Varma
arXiv:2610.16182v1 Announce Type: new
Abstract: Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit t...
By Daoping Wu, Dong Wang, Siyu Liu, Chen Liang, Jian Geng, Ziqi Li, Shuguang Xiong, Ying Zhang
arXiv:2606. 24259v1 Announce Type: cross Abstract: Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge.
By Noor Islam S. Mohammad, Ulug Bayazit
Large language models (LLMs) achieve strong relation extraction (RE), but their computational demands and reliance on proprietary APIs limit deployment in resource-constrained or privacy-sensitive settings. We investigate how far small language models (SLMs) can close this gap across general-domain and literary text.
arXiv:2607. 04733v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities.
By Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang
Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.
By Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto
arXiv:2410.02343v2 Announce Type: replace
Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...
By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining.
arXiv:2610.27652v1 Announce Type: cross
Abstract: Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it...
By Wonsup Shin, Jingu Kim
arXiv:2608. 08989v1 Announce Type: cross Abstract: Public infant cry corpora are small, label-incompatible, and almost always evaluated one corpus at a time.
By Wu Hangyu
arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.
By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin