arXiv:2609.22227v1 Announce Type: cross Abstract: Generative retrieval represents each item by a short Semantic ID and casts recommendation as autoregressive generation of that sequence. Because the...
arXiv:2606. 24259v1 Announce Type: cross Abstract: Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge.
The paper introduces SAE‑IDF, a data‑driven approach that uses sparse autoencoders to encode symbols into interpretable sparse codes, re‑weights dimensions by rarity, and matches units across heterogeneous symbolic systems. On 12 tasks spanning six symbolic system types, SAE‑IDF outperforms dense LLM baselines (Gemma‑2‑9B and Llama‑3.1‑8B) by up to 45 points, consistently across layers and configurations. Each sparse dimension carries a human‑readable meaning, allowing matches to decompose into named concepts, thereby enabling a scalable, universal symbolic backbone.
AlignSAE is a method that aligns Sparse Autoencoder (SAE) features with a predefined ontology using a two‑stage curriculum: an initial unsupervised pre‑train followed by supervised post‑train that binds specific concepts to dedicated latent slots. This approach preserves general reconstruction capacity while creating an interpretable interface for inspecting and controlling individual concepts. Experiments on GPT‑2 show that AlignSAE enables precise causal interventions—such as reliable concept swaps with 94.8% success in a 1‑hop QA task—and achieves 9–∞× better disentanglement than standard SAEs, as measured by Mutual Information Gap, while supporting multi‑hop reasoning and probing generalization dynamics.
arXiv:2609.01564v1 Announce Type: cross Abstract: Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific...
The paper introduces SAFE‑QE, a training‑free, retriever‑agnostic method for query expansion that mitigates hallucinated terms from large language models. It uses a subsequence‑association filter to score candidate terms based on their reproducibility across randomised query contexts, down‑weighting terms that lack corpus support. SAFE‑QE then selects a broad yet consistent set of expansions via constrained coverage maximisation, and defines a new metric, the Hallucinated Expansion Term Rate, to quantify faithfulness. The approach matches state‑of‑the‑art ranking performance while injecting far fewer unsupported terms on TREC and BEIR benchmarks.
The paper introduces Reward‑Informed Sparse Autoencoders (RI‑SAEs), which use reinforcement‑learning rewards to curate data for training sparse autoencoders on language‑model activations. On Llama‑3.1‑8B, a sparse subset of features separates high‑reward from low‑reward reasoning continuations, but control experiments show this separation largely reflects solution completeness rather than true reasoning quality. The authors conclude that reward filtering can cheaply reuse RL signals for interpretability, though most of the discovered features capture completion form rather than deep reasoning.
arXiv:2608. 10970v1 Announce Type: cross Abstract: Recent advances in Large Language Models (LLMs) have demonstrated strong capabilities in generating semantically relevant concepts and relations, making them promising tools for taxonomy enrichment.
The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%. "whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."
The paper reports that in English all‑words word sense disambiguation (WSD), the scarcity of high‑quality labels—not the models—has become the limiting factor. The authors introduce lexEN, a human‑adjudicated correction layer over the Maru2022 ALL_NEW benchmark, and SenseBench, a living leaderboard for LLM WSD evaluation. They show that frontier large language models reach about 95 % accuracy on lexEN‑v1, that relabeling corpora with these models improves downstream systems, and that fine‑grained WordNet senses are often ill‑posed, with coarsening improving both annotator agreement and model performance. "whyItMatters":"The study highlights that improving label quality and managing annotation costs are now the critical challenges for advancing WSD performance, as model accuracy is already near its theoretical ceiling."
The paper presents a fully self‑supervised contrastive learning framework that learns lexical representations from raw IPA‑transcribed wordlists without any cognacy annotations or expert input. Using a dual contrastive objective—word‑level and language‑level losses—the model produces word representations that enable fast computation of pairwise language distances and the inference of a global phylogenetic tree for 3,399 language varieties. The resulting tree achieves a generalized quartet distance competitive with baselines while requiring only minutes of computation on a standard notebook GPU, and the representations also capture diachronic concept stability.
arXiv:2610.27580v1 Announce Type: cross Abstract: Factual recall is a lookup: some representation inside the network serves as the key, and what that key is made of decides which queries can retrieve...