The study evaluates whether prompt‑token counts can reliably identify the lineage of large language models served via APIs. Using a frozen‑threshold approach on 24 labeled endpoint pairs, the authors find that token‑count consistency perfectly separates development pairs but only half of the holdout pairs meet the strict repeatability criteria, yielding moderate accuracy and perfect specificity. The results confirm token‑count consistency as a fingerprint of shared tokenization stacks but reject it as a standalone test for model‑family attribution.
By Bo Chen
Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.
By Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
By Netanel Eliav
arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.
By Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon
The paper evaluates CASCADE, a fully local layered defense for Model Context Protocol (MCP)-based systems, by conducting a component ablation and corpus audit on a fixed 5,000-sample dataset. It demonstrates that the choice of aggregation convention heavily influences reported metrics, that detection performance varies with provenance, and that the released configuration does not fully disclose the operating point. The study also shows that a local review model invoked for a third of requests does not alter classification outcomes, highlighting the importance of reproducibility and transparency in defense evaluations.
By \.Ipek Abas{\i}kele\c{s} Turgut, Edip G\"um\"u\c{s}
arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.
By Yongxi Zhou, Junwei Yao, Yuanzhe Liu, Zihan Dong, Wenbo Ye, Jiaxi Wen, Lai Yun Choi
arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
By Renuka Oladri, Mohan Vamsi Varadaraju Priya, Jerry Wu
The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.
By Nicol\'as Vera Z\'u\~niga
The paper introduces a benchmark of 3,600 exact‑rational word problems and 8,600 prompts that test whether language models give the same canonical answer when the same quantity is expressed in different numeric forms (decimal, fraction, percentage, number word, scientific notation, or unit‑converted). After normalizing answer syntax, canonical accuracy is high (0.969–0.996), but correctness across equivalent representations drops to 0.848–0.981, revealing that many errors stem from the evaluator’s number grammar rather than the models’ reasoning. The study also finds that representation consensus does not outperform paraphrase consensus on low‑error subsets and that certain models (e.g., Mistral Small 4) exhibit systematic unit‑conversion errors.
By Ephraim Atta-Duncan
The paper audits the widely used ISOT/Kaggle Fake and Real News corpus and finds that extremely high reported accuracies (≈0.98) are largely due to shortcut signals rather than genuine veracity detection. A simple TF‑IDF linear classifier achieves perfect F1 when using only subject metadata, and even after removing metadata, newswire tags, and duplicate documents, the F1 drops only modestly, indicating that editorial style rather than specific tokens drives performance. Under topic‑disjoint and temporal transfer tests, performance collapses, and models transfer poorly to the independent LIAR benchmark, showing that within‑corpus scores reflect source and topic separability, not truth verification.
whyItMatters:"The study demonstrates that current high accuracy metrics on this fake‑news dataset are misleading, highlighting the need for more robust evaluation protocols that guard against shortcut learning."
By Yuvraj Verma
arXiv:2607. 18310v1 Announce Type: cross Abstract: Synthetic-population tools increasingly run every individual as an independent large language model (LLM) agent.
By Gurkan Ozkan
The paper demonstrates that a byte‑level BPE tokenizer can be sliced to create multiple vocabulary sizes from a single trained model, preserving exact logits while reducing deployed weights by 66%. Experiments on 30 models show that while sliced models match the full model numerically, they underperform fixed‑cap specialists by a few percentage points in bits‑per‑byte. Multi‑cap training improves robustness to typographical noise, suggesting benefits from training across multiple granularities rather than from control tokens alone.
By Christos Koutsiaris