arXiv AI
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 28
Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.
arXiv Computation and Language
Sep 10
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
arXiv:2602.02219v3 Announce Type: replace Abstract: Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examin...
arXiv AI
Aug 24
Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems
The paper introduces Trustworthy RAG, an evaluation agent designed to detect misinformation and knowledge poisoning in Retrieval-Augmented Generation systems. It combines natural language inference verification, a five-signal poison detector, and a weighted Trust Index to assess the reliability of retrieved content. Experiments on multiple LLMs show high accuracy and precision, with the agent effectively blocking unsafe advice in a secure-coding assistant scenario.
arXiv AI
Jun 16
CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
arXiv:2606. 15396v1 Announce Type: cross Abstract: Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.
arXiv AI
Oct 15
StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge
arXiv:2610.16814v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as p...
arXiv AI
5d ago
Follow the Gradient: Tracing Safety Degradation Back to Training Data
The paper introduces SimpleLoRIF, a scalable gradient‑embedding technique that pinpoints fine‑tuning examples whose gradients align with unsafe behavior in large language models. By applying this method, the authors create SafeTulu, a version of Tulu 3 with the 20 % of most unsafe‑aligned examples removed, which reduces GCG attack success by up to 22 % across several LLMs while largely preserving or improving capability. The study also highlights previously under‑studied safety‑degrading data sources such as tool‑execution failures and narrative arithmetic.