The paper investigates whether latent representations of text preserve the ordered information present in the input. Using an information‑theoretic decomposition, the authors show that even perfectly isotropic Gaussian representations can contain no target information, and that rank alone does not predict prediction risk. They introduce CANOPE, a non‑autoregressive framework with ordered latent canvases and geometric regularization, and demonstrate that while latent‑agreement and token‑grounded models achieve similar ranks, only the token‑grounded variant yields high positional recall and low word error rates on corrupted data.
arXiv:2607. 00852v1 Announce Type: cross Abstract: This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states.
The paper investigates when compressed vector representations can provide exact linear or affine readouts for a finite lexicon’s truth conditions, establishing a necessary and sufficient row‑space condition. It shows that the augmented truth matrix’s rank determines the minimal dimension needed for exact linear (rank r) and affine (rank r − 1) readouts, and that exact readouts preserve Boolean connectives. Experiments on GloVe and word2vec embeddings reveal that while many predicates are linearly separable, none achieves exact affine recovery from pretrained embeddings, yet supervised transductive training can attain exact affine recovery at dimensions meeting the theoretical bound, preserving most of the original variance. "whyItMatters":"The results provide a precise mathematical criterion for when vector embeddings can faithfully encode logical truth conditions, informing both theoretical understanding and practical training of language models."
arXiv:2603.13911v2 Announce Type: replace Abstract: A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decoda...
The paper reports that scene text recognition models, while achieving 89–97% accuracy on standard benchmarks, perform significantly worse on rare word–trigram combinations, with a 10–18 point drop in accuracy at the rare‑word/rare‑trigram corner across multiple languages and models. Scaling the vision backbone improves overall accuracy but does not alleviate this corner‑specific deficit. The authors identify the autoregressive decoder’s lexical prior as the root cause and show that architectural changes—specifically moving from autoregressive to CTC decoding—yield the largest improvement for these rare compositions.
arXiv:2608. 09093v1 Announce Type: cross Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.
arXiv:2609. 31181v1 Announce Type: new Abstract: Black-box model identification works by scoring a model's response to natural-language prompts.
arXiv:2603. 18482v2 Announce Type: replace-cross Abstract: Standard decoding strategies for text generation, including top-$k$, nucleus sampling, and contrastive search, select tokens based on likelihood, restricting outputs to high-probability regions.
arXiv:2607. 20433v1 Announce Type: cross Abstract: While language models remain frozen at their training state, the world evolves continuously.
arXiv:2607. 01018v1 Announce Type: cross Abstract: Reading order inference remains a critical bottleneck in the digitization of complex historical manuscripts, where pages contain multiple spatially interleaved reading streams, the canonical example being the Glossa Ordinaria layout, in which a central text is surrounded by commentaries that wrap around it in non-rectangular, non-convex regions.
The paper argues that meaning identity—whether two sentences convey the same idea after wording changes—is not encoded in the geometry of independently produced sentence embeddings. Experiments on frozen off‑the‑shelf encoders and language models show that identity can only be reliably computed when both sentences are processed together in a single forward pass, yielding high accuracy (0.90–0.96) on PAWS‑X, whereas independent embeddings or simple fusion methods perform near chance. Even advanced bi‑encoder fine‑tuning improves performance on PAWS but fails to generalize to other similarity tasks, underscoring that identity is a cheap computed operator rather than a property of individual sentence vectors.
SimLens is a training‑free decoder that improves early‑layer predictions in large language models by keeping only the start token and a candidate answer token and performing a lightweight continuation through the remaining layers. It outperforms direct linear readouts, yielding higher accuracy on tasks such as ARC, BoolQ, and HeadQA with LLaMA‑7B and Vicuna‑7B. The method is extended to Linear SimLens for confidence estimation and combined into SimExit, a hybrid early‑exit mechanism that achieves significant speedups while maintaining accuracy.
arXiv:2609.08372v2 Announce Type: replace Abstract: A dense retriever encodes a question as one vector, but the question arrives one word at a time. We read 2,144 held-out headlines from Thu Vien Pha...