We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedi...
KinyaEmbed is the first sentence‑embedding model specifically designed for Kinyarwanda, built on KinyaBERT‑large and trained through a four‑stage curriculum that incorporates paraphrase pairs, translated MNLI triplets, OPUS‑100 translation pairs, and high‑quality KinyaCOMET pairs. It outperforms existing multilingual embeddings on the SemRel2024‑rw benchmark, achieving a Spearman ψ of 0.7298, and introduces the Wiki‑RW‑STS benchmark of 300 contamination‑free Kinyarwanda sentence pairs. All model checkpoints, filtered pairs, and the new benchmark are publicly released.
By Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre
The paper introduces a reproducible benchmark for evaluating attention mechanisms in tabular foundation models, focusing on the distinct row and column attention patterns that differ from language model attention. It compares several backends—Torch SDPA, FlashAttention variants, vLLM, and SageAttention—across realistic tabular shapes on A100, H100, and B200 GPUs, revealing that optimal backend choice varies by attention type, hardware, and model specifics. The study finds FlashAttention generally performs best, but CuDNN can outperform it for column attention on longer sequences, while SageAttention excels for large row sequences beyond 16k rows.
By Maximilian Schambach, Clemens Biehl, Sam Thelin
Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering (TabVQA). While recent approaches predominantly rel...
arXiv:2608. 01400v1 Announce Type: new Abstract: Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity.
By Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han, Valentin Thomas, Anthony L. Caterini
Causilo is a new tabular foundation model that delivers state‑of‑the‑art predictive performance while achieving exceptionally fast inference. On the TabArena benchmark it scores 1785.4 Elo with a median inference time of 0.10 seconds per 1 K test samples, outperforming TabPFN‑3.5‑Fast by 31.6% in speed and reaching the performance–efficiency Pareto frontier. The architecture builds on TabICL’s column‑then‑row design, adding a row‑refinement module that exchanges information among cell representations before a final column stage, and uses cross‑attention with a fixed number of summary tokens to keep attention cost linear in the number of features.
By Minyong Cho, Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo