arXiv AI August 28, 2026 By Mantas Lukauskas

Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors

The study evaluates how extractive prompt compressors affect token costs across ten languages, finding that compressors trained on English data widen the token premium gap for non‑English languages, while a multilingual compressor does not. The gap is tied to the supervision data rather than model architecture, and aggressive compression can reduce non‑English contexts to near‑zero utility. A translate‑then‑compress approach can match or outperform native compression at roughly half the token cost in several languages.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 1

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

arXiv:2606. 31796v1 Announce Type: cross Abstract: We study three complementary techniques for training compute-efficient language models.

By Dohyeon Kwon, Youngjin Park

llms

arXiv AI

Jul 17

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.

By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

llms rag nlp

arXiv AI

5d ago

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

arXiv:2608.21384v1 Announce Type: cross Abstract: Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disp...

By Ivan Dobrovolskyi

rag nlp benchmarks

arXiv Machine Learning

Jul 7

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).

By G\"ul Sena Alt{\i}nta\c{s}, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel

llms nlp benchmarks

arXiv AI

Jun 3

Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing

arXiv:2606. 03618v1 Announce Type: new Abstract: AI-assisted coding agents are bottlenecked by input-token cost.

By Mehmet Utku Colak

llms agents nlp benchmarks

arXiv Machine Learning

Aug 19

TokEval: A Tokenizer Evaluation Suite

TokEval is a tokenizer evaluation suite that introduces metrics beyond traditional fertility and compression rate, capturing linguistically and structurally meaningful properties such as UTF‑8 character boundary integrity and digit place‑value alignment for mathematics. The authors validate these metrics by pretraining language models with varied tokenizers and measuring downstream performance on bits‑per‑byte and benchmarks covering linguistic understanding, mathematical reasoning, and code generation. Their results show that information‑theoretic metrics predict language modeling performance, while structure‑sensitive metrics correlate with task accuracy, suggesting TokEval can guide tokenizer selection more principledly.

By Clara Meister

llms nlp benchmarks safety

One email a morning, machine-written

Related stories

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

In-Place Tokenizer Expansion for Pre-trained LLMs

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing

TokEval: A Tokenizer Evaluation Suite