Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

The paper argues that current fallacy-detection benchmarks are flawed because they pair fallacy classes with a broad "valid" or "none" class that includes many correct arguments, leading to misleadingly low false‑positive rates. By constructing scheme‑matched negatives—correct arguments that use the same argumentation scheme as the fallacy—the authors show that false‑positive rates rise dramatically, indicating that classifiers are learning to recognize schemes rather than detecting fallacies. The study releases these scheme‑matched examples as Scheme Foils and cautions that reported false‑positive rates should not be trusted until the valid class is audited for scheme coverage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• benchmarks