arXiv Computation and Language October 12, 2026 By Khoi-Nguyen Tran

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Read the original on arXiv Computation and Language →

The paper introduces perturber, a stateless service and Python library designed to apply single, semantics‑preserving text perturbations. It offers a catalog of 68 perturbations across 11 families, enabling reproducible, isolated application of each perturbation without coupling to a specific benchmarking stack. The authors evaluate the tool’s correctness, reproducibility, and cost, and provide a live playground and introspectable catalog for benchmark selection.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Oct 23

Brittlebench: Quantifying LLM robustness via prompt sensitivity

The paper introduces Brittlebench, a framework and evaluation pipeline that measures how sensitive large language models (LLMs) are to prompt variations. By applying semantics‑preserving perturbations to popular benchmarks, the authors show that model performance can drop by up to 12% and that such perturbations can change model rankings in 63% of cases. The study finds that these input variations can explain up to half of the performance variance for a given model, underscoring the importance of robust evaluation methods.

By Angelika Romanou, Mark Ibrahim, Candace Ross, Kerem Oktar, Chantal Shaib, Samuel J. Bell, Anaelia Ovalle, Antoine Bosselut, Jesse Dodge, Koustuv Sinha, Adina Williams

llms benchmarks

arXiv AI

Aug 5

Can Large Language Models Recover Semantic Optimization Opportunities That Compilers Miss?

arXiv:2608. 03983v1 Announce Type: cross Abstract: Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation.

By Hailong Jiang, Feng Yu, Emran Hossain, Jianfeng Zhu, Mengfei Ren, Qiang Guan, Chunwei Xia

llms benchmarks

arXiv AI

Aug 11

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.

By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

llms agents benchmarks safety

arXiv Machine Learning

Jul 22

Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

arXiv:2607. 18642v1 Announce Type: new Abstract: Mined code corpora are abundant but uncontrolled: a snippet's semantics, surface "messiness," and difficulty are whatever the wild contained; there is no known-optimal reference to grade against; and any public sample may already sit in a model's training set.

By Yuxiang Ji

Hugging Face Trending Papers

Aug 18

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repa...

llms agents

arXiv Computation and Language

Oct 7

Turnslide: Scalable Multi-Turn Data Synthesis by Walking a Finite-State Machine

arXiv:2610.07070v1 Announce Type: new Abstract: Small language models are inexpensive to serve and can run on private infrastructure, but base models are often not good enough at multi-turn tool call...

By Aaron Fainman, Gabriela Kadlecov\'a, Maciej Gryka, Bartosz Kruszczy\'nski, Usman Zafar, C\'edric Archambeau, Aaron Klein, David Salinas, Selim Nowicki, Jacek Golebiowski

llms agents fine-tuning data-engineering