NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging

NIBench is a neuroimaging benchmark that evaluates autonomous AI agents on three fronts: execution and workflow reconstruction across 56 tasks in seven imaging modalities; skill evolution through distillation of execution trajectories; and scientific judgment via analysis of 36 Alzheimer’s disease claims. The study tests 19 language‑model agents, finding that top performers execute protocols nearly flawlessly, while distilled skills boost performance on known tasks but yield mixed results on unseen ones. Agents can reproduce reported results yet sometimes draw conclusions that diverge from expert answers, indicating that data access alone does not guarantee accurate scientific judgment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• diffusion
• nlp
• efficiency
• multimodal
• benchmarks