← Back to all news
arXiv Computation and Language By Martin Kuo, Jianyi Zhang, Dongting Li, Yiran Chen

DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration

Read the original on arXiv Computation and Language →

The paper introduces DA-Cramming, a cost‑effective pretraining method that incorporates dependency agreement information into BERT‑style language models. It builds on the Cramming technique to enable training with a single GPU in a day, using a dual‑stage workflow and four submodels to embed chunk‑level dependency agreements. Experiments show that this approach outperforms prior methods on a range of tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
3d ago

Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts

Ready2Blend is a method that blends natural-language instructions with learned alignment prompts to enable continual alignment of large language models without retraining the backbone. It uses AlignFormer to map each requirement to a fixed-length prompt stored in a modular bank, while keeping the backbone and prior prompts frozen. The approach achieves 93.1–98.5% of joint‑training performance, retains prior knowledge, and reduces training time by up to 4.3×, also allowing weighted personalization and order‑free composition.

By Jeesu Jung, Hwan Chang, Juseon Do, Jeonghwan Choi, Jinho Choo, Sungwoo Nam, S. K. Hong, Hwanjun Song
llmssafety
More like this →
arXiv Computation and Language
Sep 11

A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation

The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.

By Miguel Moura Ramos, Duarte M. Alves, Andr\'e F. T. Martins
llmsreinforcement-learningfine-tuningefficiencysafety
More like this →