Hugging Face Trending Papers

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

arXiv Computation and Language
Oct 12

zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

The paper introduces zip2zip++, a method for inference‑time token compression that reduces token counts using a dynamic Lempel‑Ziv‑Welch (LZW) codec while preserving language model quality. It identifies four problems with prior LZW‑based compression—embedding conflict, improper RoPE application, harmful digit merges, and segmentation ambiguity—and proposes solutions: separate input/output embeddings, size‑aware RoPE, and a selective‑merge LZW codec. Experiments on four models show that these changes halve the quality gap in perplexity and reasoning accuracy compared to the uncompressed baseline, achieving similar compression ratios.

By Saibo Geng, Andrea Gentilini, Xinxian Ma, Nathan Ranchin, Yunzhen Yao, Robert West
arXiv Computation and Language
Sep 3

Debias-SparseGPT: Bias-Aware Pruning for Large Language Models

Debias‑SparseGPT is a post‑training pruning technique that adds a representational debiasing term based on demographically contrasting inputs to mitigate bias amplification caused by weight sparsification. The method is validated across various generative LLMs and sparsity levels (25%, 50%, and structured 2:4), consistently reducing pruning‑induced bias while maintaining perplexity and zero‑shot accuracy. In the most aggressive 2:4 sparsity regime, enriching the calibration set with long‑context, content‑rich examples further improves both downstream performance and fairness.

By Irina Proskurina, Guillaume Metzler, Antoine Gourru, Julien Velcin
arXiv Computation and Language
Sep 1

Manac\'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.

By Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto
arXiv AI
Aug 28

Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors

The study evaluates how extractive prompt compressors affect token costs across ten languages, finding that compressors trained on English data widen the token premium gap for non‑English languages, while a multilingual compressor does not. The gap is tied to the supervision data rather than model architecture, and aggressive compression can reduce non‑English contexts to near‑zero utility. A translate‑then‑compress approach can match or outperform native compression at roughly half the token cost in several languages.

By Mantas Lukauskas
Hugging Face Trending Papers
Jul 14

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.

arXiv Machine Learning
Oct 23

Rethinking Sequence Compression with Test-time Regression

The paper introduces TTR‑Compressor, a new method for sequence compression that treats token aggregation as a test‑time regression problem. Unlike traditional pooling or learned projections, the regression coefficients are learned end‑to‑end and computed in closed form during inference. Experiments show that TTR‑Compressor outperforms existing compressors for block‑wise sparse attention and visual token merging, achieving better performance with fewer tokens.

By Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
arXiv Computation and Language
Sep 7

Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG

The paper introduces DEX-Comp, a two‑stage training method for soft context compression in Retrieval‑Augmented Generation (RAG). First, a pure distillation warm‑start trains the compression model on correct responses from an uncompressed RAG. Then, hard exploration uses reinforcement learning on queries where the uncompressed RAG fails, encouraging better computation patterns for compressed representations. Experiments on five open‑domain QA benchmarks show that DEX‑Comp compresses retrieved contexts 16×, speeds inference 4×–24×, and matches or surpasses the uncompressed RAG baseline across various retrieval depths.

By Shuyu Guo, Shuo Zhang, Zhaochun Ren