First-Order In-Context Shapley: Scalable Data Auditing for LLM Alignment and Evaluation

The paper introduces First-Order In-Context Shapley, an inference‑only method that evaluates data utility for large language model alignment by computing the first‑order term of an in‑context cooperative game over semantic k‑NN neighborhoods in O(k) forward passes. It projects log‑likelihood shifts onto a directed influence graph, yielding three signals—Exemplifying Advantage, Exemplified Advantage, and One‑of‑a‑Kind Score—to identify high‑utility demonstrations and rare prompts. The approach reduces compute by 19× compared to Monte‑Carlo Shapley, filters 1–5% of records, and improves error detection and reward accuracy in DPO training and benchmark auditing. whyItMatters":"By enabling scalable, inference‑only data auditing, the method addresses label noise and safety inversions that hinder post‑training alignment of large language models, leading to measurable improvements in model reliability and reward accuracy."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• rag
• reinforcement-learning
• fine-tuning
• benchmarks
• safety
• data-engineering