arXiv AI
Jul 3
Introduction to Transformers: an NLP Perspective
arXiv:2311. 17633v2 Announce Type: replace-cross Abstract: Transformers have dominated empirical machine learning models of natural language processing.
Improving language understanding with unsupervised learning
We’ve obtained state-of-the-art results on a suite of diverse language tasks with a scalable, task-agnostic system, which we’re also releasing. Our approach is a combination of two existing ideas: transformers and unsupervised pre-training.
Related stories
OpenAI Blog
Feb 14, 2019
Better language models and their implications
We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization—all without task-specific training.
arXiv AI
Jun 2
EuroBERT: Scaling Multilingual Encoders for European Languages
arXiv:2503. 05500v3 Announce Type: replace-cross Abstract: General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models.
arXiv AI
Jul 28
LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish
arXiv:2510. 07074v2 Announce Type: replace-cross Abstract: Instruction tuning has become a key technique for enhancing the performance of large language models, enabling them to better follow human prompts.
arXiv AI
Jun 24
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.
arXiv AI
Jul 14
Polarization Detection: A Hybrid Approach with AfroXLMR-Social and DeBERTa for Low- and High-Resource Settings
arXiv:2607. 10312v1 Announce Type: cross Abstract: The rapid proliferation of online polarization threatens social cohesion, necessitating robust automated detection systems that operate effectively across diverse linguistic contexts.
arXiv Machine Learning
Jul 22
Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty
arXiv:2607. 19101v1 Announce Type: cross Abstract: Reliable Text Difficulty Assessment is a prerequisite for valid text simplification workflows and personalized learning applications.
arXiv Machine Learning
Aug 3
Imbalanced Data Clustering via Targeted Data Augmentation Using GMM and LLM
arXiv:2607. 28635v1 Announce Type: cross Abstract: In Natural Language Processing (NLP), dealing with underrepresented topics is challenging, especially in unsupervised tasks where clustering might not adequately capture minority topics.
arXiv AI
Jul 28
Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languages
arXiv:2506. 10292v2 Announce Type: replace-cross Abstract: Training deep learning networks with minimal supervision has gained significant research attention due to its potential to reduce reliance on extensive labelled data.
OpenAI Blog
Jun 10, 2021
Improving language model behavior by training on a curated dataset
Our latest research finds we can improve language model behavior with respect to specific behavioral values by fine-tuning on a small, curated dataset.
arXiv AI
Jul 21
Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances
arXiv:2405. 12775v2 Announce Type: replace-cross Abstract: Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions.
arXiv Machine Learning
Aug 4
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.