← Back to all news
arXiv Machine Learning By Francisco Galuppo Azevedo, Clarissa Lima Loures

Can LLMs Use Relational Transformer Embeddings?

Read the original on arXiv Machine Learning →

The paper investigates whether large language models (LLMs) can leverage frozen relational‑transformer embeddings by injecting them as soft tokens. Using a learned MLP projection and LoRA adaptation, the authors fine‑tune Qwen3.5‑4B on chain‑of‑thought reasoning traces and group‑based reinforcement learning, then evaluate on ten binary classification tasks across six RelBench databases. The hybrid approach consistently underperforms the standalone relational transformer, showing sensitivity to serialization format, token budget, and RL stability, leading the authors to conclude that stronger alignment objectives and schema‑aware design are needed for reliable relational prediction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 19

Path to Prior-Data Fitted Relational Models via Learned Latent Multi-table Representations

The paper investigates whether prior‑data‑fitted networks (PFNs) can be extended to graph‑based relational models. The authors pretrain a message‑passing foundation model, Griffin, on large synthetic relational databases and evaluate its frozen encoder on 16 real‑world relational classification tasks. A curriculum that starts with schema‑guided synthetic data and then uses schema‑agnostic data yields the best in‑context performance, outperforming real‑data‑only pretraining and other mixtures.

By Mohammad Sadeq Abolhasani, Viswanath Ganapathy
llmsfine-tuning
More like this →
Hugging Face Trending Papers
Jun 29

FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks

We introduce FlexTab, a flexible encoder-decoder architecture for in-context learning on tabular data that pairs a single, task-agnostic encoder with a suite of task-specific decoders. Unlike existing tabular in-context learners, which entangle feature representations with a specific prediction target, our design produces \textit{target-agnostic} row embeddings that can be leveraged across a wide range of downstream tasks within a table-native in-context learning setup.

llmsragbenchmarks
More like this →
arXiv Computation and Language
Sep 25

CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models

CONSISTRE is a consistency‑aware framework for document‑level relation extraction that tackles contradictions in large language model predictions. It offers two tracks: an inference‑time track that refines black‑box LLM outputs through constraint‑aware prompting, verification, and self‑reflection, and a training‑time track that distills consistency knowledge into smaller open‑source models via supervised fine‑tuning and reinforcement learning. Experiments on DocRED show both tracks outperform baselines, with the inference‑time track matching competitive F1 scores and the training‑time track narrowing the performance gap to proprietary LLMs while reducing inference cost.

By Mingxuan Sun
llmsreinforcement-learningfine-tuningefficiencybenchmarkssafety
More like this →