Hugging Face Trending Papers
Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
Sep 24
Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and...
arXiv Computation and Language
Sep 3
Debias-SparseGPT: Bias-Aware Pruning for Large Language Models
Debias‑SparseGPT is a post‑training pruning technique that adds a representational debiasing term based on demographically contrasting inputs to mitigate bias amplification caused by weight sparsification. The method is validated across various generative LLMs and sparsity levels (25%, 50%, and structured 2:4), consistently reducing pruning‑induced bias while maintaining perplexity and zero‑shot accuracy. In the most aggressive 2:4 sparsity regime, enriching the calibration set with long‑context, content‑rich examples further improves both downstream performance and fairness.
arXiv Computation and Language
Sep 1
Manac\'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.
arXiv Computation and Language
Sep 15
Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
arXiv:2609.13154v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have made prompts increasingly large and complex. Techniques such as chain-of-thought reasoning (Wei et...
Hugging Face Trending Papers
Aug 31
Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
Brazilian Portuguese remains under-served by open language models, and the few that exist are difficult to reproduce and are often compared without measures of uncertainty. We release Manacá-1B, an op...
arXiv AI
Aug 28
Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors
The study evaluates how extractive prompt compressors affect token costs across ten languages, finding that compressors trained on English data widen the token premium gap for non‑English languages, while a multilingual compressor does not. The gap is tied to the supervision data rather than model architecture, and aggressive compression can reduce non‑English contexts to near‑zero utility. A translate‑then‑compress approach can match or outperform native compression at roughly half the token cost in several languages.