arXiv AI
Jul 3
PreScience: A Dataset and Benchmark for Scientific Forecasting
arXiv:2602. 20459v2 Announce Type: replace Abstract: Can AI systems trained on the existing scientific record forecast the advances that will follow?
SciTBERT: A family of chronologically consistent language models for scientific and technological language processing
SciTBERT is a family of BERT-derived language models trained on chronologically consistent corpora of scientific papers, patents, and educational web text, with training cutoffs spanning 2013 to 2025. The models are further post‑trained using paper and patent citations to create the SciTBERT‑CI family, which better captures temporal dynamics. Experiments show that these chronologically consistent encoders outperform earlier domain‑specific models, especially when training data is limited by early year restrictions, and they perform well on the new PatRepEval benchmark of patent‑related tasks at the science‑technology interface.
Related stories
arXiv Computation and Language
Sep 22
ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents
arXiv:2609.23231v1 Announce Type: new Abstract: Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a dif...
arXiv AI
5d ago
Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings
The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.
arXiv Machine Learning
Sep 30
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...
arXiv AI
Aug 11
Enhancing Scientific Named Entity Recognition via Large Language Models: A Type-driven Multi-task Learning Approach
arXiv:2608. 08636v1 Announce Type: cross Abstract: Scientific named entity recognition (SciNER) plays a crucial role in information extraction and knowledge discovery from scientific texts.
arXiv Machine Learning
Aug 5
Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
arXiv:2608. 03855v1 Announce Type: new Abstract: Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry.
arXiv Computation and Language
Sep 4
Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer
The paper introduces Distilled Rapid Embedding Transfer (DRET), a parameter‑efficient method that injects biomedical domain knowledge from large specialized models into a smaller general‑purpose model without retraining on the original specialized corpora. DRET evolves through iterative strategies—tokenizer‑merge (DRET 1.x), hybrid embedding averaging (DRET 2.0), priority‑based embedding transfer (DRET 3.x), and further refinements (DRET 4.x)—and demonstrates that a 66‑million‑parameter DistilBERT can achieve competitive or superior performance on token‑level PICO classification compared to much larger models, while remaining lightweight. The authors validate the embedding‑level transfer with cosine similarity, semantic‑shift, and t‑SNE analyses, highlighting DRET’s potential for scalable, resource‑efficient biomedical text mining.
arXiv Computation and Language
Sep 18
Improving Cross-Lingual Transfer for Sequential Sentence Classification in Research Papers via Structural Similarity
The paper investigates cross‑lingual transfer for sequential sentence classification (SSC) in research papers, focusing on 13 non‑English languages. Experiments show that linguistic proximity does not reliably predict transfer success, whereas structural similarity in rhetorical organization—particularly label distribution similarity—correlates positively with performance. The authors introduce three generative‑model methods that exploit structural cues, achieving parity with strong encoder baselines on‑domain and outperforming them when transferring to unseen languages.
Hugging Face Trending Papers
Aug 12
Task- and dataset-specific information in protein language models
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs).
arXiv Computation and Language
Aug 25
What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization
arXiv:2601.17609v3 Announce Type: replace Abstract: In domains like medicine and finance, large-scale labeled data is costly and often unavailable, leading to models trained on small datasets that st...
arXiv Computation and Language
Aug 25
How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV Through Claim-Grounded Typed Citations
arXiv:2606.22342v2 Announce Type: replace Abstract: How does research evolve, and can we trace it at the level of individual claims? Scientific progress is not simply a uniform accumulation of facts....
arXiv AI
Jul 15
Scaling Point-in-Time Language Models
arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.