arXiv:2607. 24991v1 Announce Type: cross Abstract: Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs).
By Barbara Kitchenham, Sebasti\'an Pizard, Lech Madeyski, Ronnie de Souza Santos, Martin Shepperd, David Budgen
Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require.
arXiv:2606. 17588v1 Announce Type: cross Abstract: Several studies have examined the use of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), reporting mixed accuracy.
By Mika M\"antyl\"a, Patricia Matsubara, Katia Romero Felizardo, Miikka Kuutila, Marco Gerosa, Savio de Sousa Sampaio, Tayana Conte, Igor Steinmacher
The paper introduces PRISMA-LLM, a reporting framework for AI-assisted systematic reviews. It is based on an analysis of 888 review-automation papers, showing a shift toward LLM- and software-driven workflows and inconsistent reporting of evaluation and limitations. The framework separates implementation details from consequence-sensitive evaluation and limitation reporting.
By Miguel Zabaleta, Baihan Lin
The study investigates whether large language models (LLMs) have improved in screening systematic reviews (SRs) within software engineering. Using the SESR‑Eval benchmark and a new smaller dataset, the authors evaluated eight recent LLMs and found only marginal performance gains over older models (average MCC rose from 0.347 to 0.365). The research also examined prompt variations, inter‑model agreement, and the impact of refining inclusion/exclusion criteria, concluding that LLMs still fall short of replacing human reviewers and that future work should explore agent‑based methods, prompt engineering, and further criteria refinement.
By Aleksi Huotala, Miikka Kuutila, Mika M\"antyl\"a
arXiv:2606. 28363v1 Announce Type: cross Abstract: Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points.
By Hsieh-Ting Lin, Jiunn-Tyng Yeh