arXiv Machine Learning

CSC: Calibrated Simplicity for Conflict-Aware Social Bot Detection in the LLM Era

The paper introduces CSC, a calibrated-simplicity framework for detecting social bots in the era of large language models. CSC combines a simplified prototype-guided graph expert, calibrated simplex-constrained fusion, and a lightweight inconsistency expert to address modality conflict between semantic and structural signals. Experiments on TwiBot-22, TwiBot-20, and MGStBot-large demonstrate that CSC improves calibrated decision quality while maintaining competitive performance across benchmarks.

arXiv AI
Aug 24

Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems

The paper introduces Trustworthy RAG, an evaluation agent designed to detect misinformation and knowledge poisoning in Retrieval-Augmented Generation systems. It combines natural language inference verification, a five-signal poison detector, and a weighted Trust Index to assess the reliability of retrieved content. Experiments on multiple LLMs show high accuracy and precision, with the agent effectively blocking unsafe advice in a secure-coding assistant scenario.

By Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson
arXiv Computation and Language
2d ago

Social bot detection in the age of ChatGPT: Challenges and opportunities

The article reviews the challenges and opportunities of detecting social bots amid the rise of advanced AI chatbots. It highlights gaps in current detection methods, especially regarding AI-generated conversations, and identifies emerging trends such as synthetic data generation, multimodal cross‑platform detection, low‑resource language support, and federated learning approaches. The authors propose these directions as promising avenues for future research.

By Emilio Ferrara
arXiv AI
Sep 4

KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents

KC-Bench is a dynamic interactive benchmark designed to evaluate how large language model agents reconcile user instructions, internal knowledge, and real‑time environmental observations. It contains 238 manually curated multi‑turn tasks that test world‑knowledge conflicts, input inconsistencies, and multi‑source temporal conflicts, using a user simulator, stateful tools, deterministic environment assertions, an open‑source natural‑language evaluator, and human trajectory verification. Evaluation of nine models—including DeepSeek‑V4‑Flash, GLM‑5.2, and MiniMax‑M3—reveals significant cross‑domain variation, with no model reliably handling factual correction, identity consistency checking, and temporal conflict resolution across all settings, and shows that missed conflicts can propagate to tool calls or synthetic protected‑data flows.

By Yaxing Lyu, Shengjie Zhou, Binbin Toh, Pengyu Zhu, Lijun Li
Hugging Face Trending Papers
Sep 3

KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents

KC-Bench is a dynamic interactive benchmark designed to evaluate how large language model agents reconcile user instructions, internal knowledge, and real‑time environmental observations. It consists of 238 manually curated multi‑turn tasks that test world‑knowledge conflicts, input inconsistencies, and multi‑source temporal conflicts, using a user simulator, stateful tools, deterministic environment assertions, an open‑source natural‑language evaluator, and human trajectory verification. Evaluation of nine models—including DeepSeek‑V4‑Flash, GLM‑5.2, and MiniMax‑M3—reveals significant cross‑domain variation, with no model reliably handling factual correction, identity consistency, and temporal conflict resolution across all settings, and shows that missed conflicts can propagate to tool calls or synthetic protected‑data flows.