arXiv:2609.37076v1 Announce Type: new Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.
arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.
Large Language Models: A Plain-Language Overview for Engineers explains that LLMs are neural networks trained on vast text collections to predict subsequent words, and outlines key concepts such as Transformers, pre‑training, fine‑tuning, reinforcement learning from human feedback, retrieval‑augmented generation, chain‑of‑thought prompting, and tool‑using agents. It also discusses efficiency techniques like FlashAttention and QLoRA, provides a life‑cycle figure and comparative tables, and offers a checklist for evaluating LLM claims, including checks for data contamination, model‑based grading, long‑context handling, and hallucination. The article aims to give engineers a clear picture of how training changes and prompts affect generative AI tools like ChatGPT.
arXiv:2610.19185v1 Announce Type: new Abstract: Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them....
The paper introduces Prompt-agnostic Language Models, focusing on train‑time techniques to embed prompt robustness during instruction tuning. It shows that mixing paraphrases in training causes significant gradient interference—57% of per‑paraphrasing gradients conflict—leading to the development of two gradient‑merging methods, Prompt‑TIES and Prompt‑DARE, which resolve this interference. Experiments across four models demonstrate that these methods substantially reduce per‑example variance and improve local robustness with minimal impact on overall accuracy.
The paper introduces a token‑level log‑likelihood ratio to pinpoint where large language models memorize training sequences. It proposes a probe‑steer framework, instantiated as Gated Subspace Steering (GSS), that selectively corrects memorization‑relevant activations using the leading singular vectors of a gradient‑weighted activation matrix. Experiments on four benchmarks show GSS matches or surpasses state‑of‑the‑art memorization reduction methods while using 10–100× less compute.
LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.
Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured...
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
arXiv:2609.38929v1 Announce Type: new Abstract: Machine learning systems increasingly face the need to remove the influence of entire data domains, such as toxic language, harmful behavior, or topica...
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.