arXiv AI
Aug 18
Automating the Detection of Requirement Dependencies Using Large Language Models
arXiv:2602. 22456v2 Announce Type: replace-cross Abstract: Requirements are inherently interconnected through various types of dependencies.
DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration
The paper introduces DA-Cramming, a cost‑effective pretraining method that incorporates dependency agreement information into BERT‑style language models. It builds on the Cramming technique to enable training with a single GPU in a day, using a dual‑stage workflow and four submodels to embed chunk‑level dependency agreements. Experiments show that this approach outperforms prior methods on a range of tasks.
Related stories
arXiv AI
Aug 11
One Adapter Pair per Model: A Universal Activation Interface for Language Models
arXiv:2608. 09521v1 Announce Type: new Abstract: Activation-based tools are usually tied to one model's native hidden space, requiring probes, sparse autoencoders, and natural-language interpreters to be rebuilt or rediscovered for each new language model.
arXiv AI
Jun 24
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.
arXiv AI
Aug 21
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
arXiv AI
3d ago
Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts
Ready2Blend is a method that blends natural-language instructions with learned alignment prompts to enable continual alignment of large language models without retraining the backbone. It uses AlignFormer to map each requirement to a fixed-length prompt stored in a modular bank, while keeping the backbone and prior prompts frozen. The approach achieves 93.1–98.5% of joint‑training performance, retains prior knowledge, and reduces training time by up to 4.3×, also allowing weighted personalization and order‑free composition.
arXiv Computation and Language
Sep 11
A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation
The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.
arXiv AI
Jun 15
Implicit Reasoning for Large Language Model-based Generative Recommendation
arXiv:2606. 14142v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly adopted as backbones for Generative Recommendation (GR), promising access to pretrained world knowledge.
arXiv AI
Sep 18
QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training
QVAC Genesis III is a 191.43 B‑token synthetic STEM corpus covering 19 domains and multiple difficulty levels, created through a dual generation strategy that uses a weak edge‑scale student model to generate corrective explanations and contrastive reasoning. The authors evaluate the corpus with an LLM‑as‑a‑parser protocol and demonstrate that 1.7 B‑parameter models trained on QVAC Genesis III outperform those trained on Cosmopedia‑v2 and the Cosmo‑1B model on ARC, GPQA Diamond, and MMLU STEM benchmarks, achieving up to +28.57% improvement on ARC‑E and a 99.45% valid answer rate.
arXiv AI
Jun 12
WildIFEval: Instruction Following in the Wild
arXiv:2503. 06573v3 Announce Type: replace-cross Abstract: Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge.
arXiv Machine Learning
Jun 30
MOSAIC: Orchestrating Collaborative Knowledge Tracing with Hierarchical Semantic Alignment
arXiv:2606. 29049v1 Announce Type: new Abstract: Knowledge Tracing (KT) is important for personalized education but traditionally suffers from two key limitations: a reliance on shallow ID-based representations that neglect semantic depth and a restriction to single-granularity mastery estimation that overlooks hierarchical knowledge dependencies.
arXiv Computation and Language
Aug 28
SPT: Skills as Pre-Training Data for Agentic Language Models
The paper introduces Skill Pre-Training (SPT), a mid‑training approach that uses public skill packages as data for agentic language models. By applying causal language modeling to a collection called SkillCorpus and employing a Reference Insert strategy to keep file relationships intact, SPT improves agentic performance across various model sizes while largely preserving general capabilities. Experiments also show that mixing skill data with general corpora yields further benefits.
arXiv AI
Sep 4
RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.