arXiv AI
Jul 3
Introduction to Transformers: an NLP Perspective
arXiv:2311. 17633v2 Announce Type: replace-cross Abstract: Transformers have dominated empirical machine learning models of natural language processing.
A Survey of Transformer-based Language Models with Focus on Efficiency
The paper surveys Transformer-based large language models (LLMs) with a focus on efficiency, reviewing 312 articles that cover data curation, model design, downsizing, and dynamic inference. It also examines efficiency in adaptation strategies such as pre‑training, fine‑tuning, prompt‑engineering, and Retrieval‑Augmented Generation (RAG). A statistical analysis and evaluation of over 30 prominent NLP models on 13 benchmarks provide insights into both efficiency and efficacy, highlighting trends toward sustainable NLP practices.
Related stories
arXiv AI
Jun 29
POTracker: Optimizing Large Language Models for Standard-Compliant Power Outage Report Generation
arXiv:2606. 23533v2 Announce Type: replace Abstract: Recent large language models (LLMs) are good at general text generation, but it is still hard to use them for domain-specific data generation because the output must follow strict formatting and structural rules.
arXiv Machine Learning
Sep 10
RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
RePro is a web‑recycling technique that trains a small language model (as little as 1 B parameters) with reinforcement learning to produce high‑quality, faithful rephrasings of pretraining data. The method uses one quality reward and three faithfulness rewards to preserve core semantics and structure while converting organic data into better training examples. Experiments show that RePro boosts downstream accuracy by 3.7–14.5 % over organic‑only baselines and improves data efficiency 2–3×, outperforming prior prompting‑based recycling approaches.
arXiv AI
Jul 7
Polarity Detection of Sustainable Development Goals in News Text
arXiv:2509. 19833v4 Announce Type: replace-cross Abstract: The United Nations' Sustainable Development Goals (SDGs) provide a globally recognised framework for addressing major societal, environmental, and economic challenges.
arXiv Machine Learning
Jun 5
A Survey on Diffusion Language Models
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
arXiv AI
Sep 1
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
arXiv:2601.19926v3 Announce Type: replace-cross Abstract: We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on o...
arXiv Machine Learning
Jul 31
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.
arXiv AI
Aug 21
Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa
arXiv:2608. 19200v1 Announce Type: cross Abstract: Text summarization refers to the task of condensing a document into a shorter version while preserving its key information.
arXiv Machine Learning
Aug 28
Squeezing More from Limited Data with Recursive Transformers
The paper investigates how to effectively pre‑train language models when the data budget is limited but compute is plentiful. It shows that increasing model size only improves performance up to an optimal point, after which overfitting degrades generalization, and that this optimal size varies with both the data budget and downstream tasks. To overcome the inefficiencies of standard Transformers in this regime, the authors propose recursive Transformers that reuse a shared block across depth and employ factorized embeddings, achieving better results than standard models on 10M–100M word pre‑training budgets and competitive performance with BabyLM Challenge 2025 winners.
arXiv AI
Sep 10
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
arXiv:2609.07798v1 Announce Type: cross Abstract: Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, polic...
arXiv AI
Aug 10
Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages
arXiv:2605. 02608v2 Announce Type: replace-cross Abstract: Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood.
arXiv Computation and Language
Aug 31
Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects
The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.