← Back to all news
arXiv Computation and Language By Tanja Baeumel, Josef van Genabith, Simon Ostermann

When Tokenization is Secretly Output Supervision

Read the original on arXiv Computation and Language →

The paper argues that tokenization in language models should be viewed as an output supervision decision rather than merely input preprocessing. In autoregressive models, the granularity of the tokenizer determines the supervision signal the model receives, influencing learning difficulty, internal representations, and task performance. Experiments on numeric reasoning show that output tokenization, rather than input tokenization, drives differences in performance and training dynamics, and a survey of recent CL papers reveals that tokenization choices are rarely reported or acknowledged.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

Hugging Face Trending Papers
Aug 18

TokEval: A Tokenizer Evaluation Suite

TokEval is a tokenizer evaluation suite that introduces metrics beyond traditional fertility and compression rate, capturing linguistically and structurally meaningful properties such as UTF-8 character boundary integrity and digit place-value alignment for mathematics. The authors validate these metrics by conducting controlled language model pretraining experiments that vary tokenizer training data, pretokenization strategy, and training algorithm, then evaluate the models on bits-per-byte and benchmarks covering linguistic understanding, mathematical reasoning, and code generation. Results show that information-theoretic metrics predict language modeling performance, while structure-sensitive metrics correlate with task accuracy, suggesting TokEval can guide tokenizer selection more principledly.

llmsnlpbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 19

TokEval: A Tokenizer Evaluation Suite

TokEval is a tokenizer evaluation suite that introduces metrics beyond traditional fertility and compression rate, capturing linguistically and structurally meaningful properties such as UTF‑8 character boundary integrity and digit place‑value alignment for mathematics. The authors validate these metrics by pretraining language models with varied tokenizers and measuring downstream performance on bits‑per‑byte and benchmarks covering linguistic understanding, mathematical reasoning, and code generation. Their results show that information‑theoretic metrics predict language modeling performance, while structure‑sensitive metrics correlate with task accuracy, suggesting TokEval can guide tokenizer selection more principledly.

By Clara Meister
llmsnlpbenchmarkssafety
More like this →