Hugging Face Trending Papers July 5, 2026

Geometry of Ordinal Representations in Language Models

Read the original on Hugging Face Trending Papers →

Recent work showed that language models represent character counts on curved 1D manifolds, with attention heads performing geometric transformations to enable computation. We test whether this generalizes across four ordinal tasks (bracket depth, indentation, table position, numeric magnitude) in Gemma-2-2B, Gemma-2-9B, and Qwen3-4B.

Summary generated by The Flow from the publisher's feed. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning

Jul 7

Geometry of Ordinal Representations in Language Models

arXiv:2607. 04167v1 Announce Type: new Abstract: Recent work showed that language models represent character counts on curved 1D manifolds, with attention heads performing geometric transformations to enable computation.

By Saksham Bassi, Sharvi Tomar

llms benchmarks

arXiv AI

Jun 9

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

arXiv:2509. 10534v3 Announce Type: replace-cross Abstract: The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where.

By Anand Gopalakrishnan, Robert Csord\'as, J\"urgen Schmidhuber, Michael C. Mozer

llms rag fine-tuning

arXiv Machine Learning

Jul 20

Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization

arXiv:2601. 22402v2 Announce Type: replace-cross Abstract: Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation.

By Kanishk Awadhiya

llms rag robotics

arXiv Machine Learning

Jun 11

Visualizing LLM Latent Space Geometry Through Dimensionality Reduction

arXiv:2511. 21594v3 Announce Type: replace Abstract: Large language models (LLMs) achieve state-of-the-art results across many natural language tasks, but their internal mechanisms remain difficult to interpret.

By Alex Ning, Vainateya Rangaraju, Yen-Ling Kuo

llms rag diffusion benchmarks

arXiv AI

Jul 15

Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task

arXiv:2510. 18315v2 Announce Type: replace-cross Abstract: We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps.

By Brady Bhalla, Honglu Fan, Nancy Chen, Tony Yue YU

llms rag reinforcement-learning safety

arXiv AI

Jun 2

Emergent Ordinal Geometry in Transformers Trained on Local Comparisons

arXiv:2606. 01269v1 Announce Type: new Abstract: Transitive inference is the challenge of inferring that A < C from knowing only adjacent relations (A < B, B < C).

By Nishit Singh

llms rag

Related stories

Geometry of Ordinal Representations in Language Models

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization

Visualizing LLM Latent Space Geometry Through Dimensionality Reduction

Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task

Emergent Ordinal Geometry in Transformers Trained on Local Comparisons