← Back to all news
arXiv Machine Learning By Thomas Gebhart, Russell J. Funk

SciTBERT: A family of chronologically consistent language models for scientific and technological language processing

Read the original on arXiv Machine Learning →

SciTBERT is a family of BERT-derived language models trained on chronologically consistent corpora of scientific papers, patents, and educational web text, with training cutoffs spanning 2013 to 2025. The models are further post‑trained using paper and patent citations to create the SciTBERT‑CI family, which better captures temporal dynamics. Experiments show that these chronologically consistent encoders outperform earlier domain‑specific models, especially when training data is limited by early year restrictions, and they perform well on the new PatRepEval benchmark of patent‑related tasks at the science‑technology interface.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 22

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

arXiv:2609.23231v1 Announce Type: new Abstract: Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a dif...

By Mahdi Astaraki, Mohammad Khodadad, Reza Namazi, Mohammad Arshi Saloot, Amir Reza Behzad Moghadam, Hamidreza Mahyar, Soheila Samiee
ragbenchmarks
More like this →
arXiv AI
5d ago

Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings

The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.

By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
llmsragfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Sep 30

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...

By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
llmsbenchmarks
More like this →