← Back to all news
arXiv AI By Sander Lanbo Shi, Miranda Anna Christ, Eric Modesitt, Andrew Louis, Konrad Cheng, Brandon Thai Tran, Sewoong Lee, Ratnakar Pawar, Gaia Molinaro, Juliana Bain, Nandika Donthi, Maja Pavlovic, Annya Dahmani, Francesca Bianco, Abhishek Shah, Tyler Crosse, Arth Singh, Karissa Huang, Sukanya Krishna, Konstantinos Voudouris, David Demitri Africa, Joshua Jacob, Rishub Jain

Towards Human-AI Complementarity for AI Safety

Read the original on arXiv AI →

The paper investigates human‑AI complementarity as a strategy for safer AI oversight, arguing that humans and models making different mistakes can combine judgments to produce stronger oversight signals. The authors present the largest known suite of safety‑relevant tasks (3,957 instances across 8 datasets and 6 domains) and a platform for measuring performance of humans, AI, and hybrid systems. Their study, based on 35,613 judgments from 416 raters, shows a notable complementarity effect but also finds that the tested assistance methods rarely improve over the best single source and that human raters corrected AI errors less often even as overall accuracy rose.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 8

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

The paper introduces CARE, a calibrated adaptive rectification and escalation framework that integrates AI models with human reviewers to ensure safe, human-aligned decisions. CARE continuously learns from human feedback, improving automation while reducing the need for human queries. Experiments on driving, language, and robotics datasets show a 25–81% reduction in human queries compared to baseline methods.

By Chenyu Zhang, Rachel Luo, Boyi Li, Anjali Parashar, Marco Pavone, Apoorva Sharma
roboticssafety
More like this →
arXiv AI
Oct 16

Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

The paper introduces ImpactBench, an open, expert-guided platform designed to measure the multi-dimensional impact of generative AI on users. It evaluates AI models across psychological, physical, and social wellbeing metrics derived from domain experts and existing benchmarks, using simulated multi-turn conversations with teen and adult scenarios. Ten leading LLMs were tested on 48,540 conversations, showing 68.7% promotion of desirable behaviors but only 53.1% avoidance of harms, with an 83.3% agreement rate between automated judges and expert reviewers.

By Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn
llmsbenchmarks
More like this →
arXiv AI
Aug 13

On Benchmarking Human-Like Intelligence in Machines

arXiv:2502. 20502v2 Announce Type: replace Abstract: Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition.

By Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum
benchmarks
More like this →