LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

LongDS-Bench is a new benchmark for evaluating long-horizon, multi-turn data analysis by agents, featuring 68 tasks derived from real-world Kaggle notebooks that span 2,225 turns across six domains such as Geoscience, Business, and Education. The benchmark focuses on agents’ ability to maintain, update, restore, and compose evolving analytical states, with tasks designed around state-evolution patterns like counterfactual perturbation, rollback, and multi-state composition, and an average dependency span of 11.3 turns. Evaluation of five state-of-the-art models shows that the best model achieves only 48.45% average accuracy, with performance dropping nearly 47 points from early to late turns and long-horizon errors accounting for 52%–69% of failures, indicating that maintaining a correct analytical state is the key bottleneck.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• agents
• benchmarks