Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation
Read the original on arXiv AI →The study reports a mixed‑methods pilot evaluation of an AI‑orchestrated research workflow in which a human researcher coordinated multiple LLM‑based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple‑choice questions (MCQs) as a testbed, the team collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24‑criterion quality framework. Across all evaluations, average MCQ quality was high, but criterion‑level analysis and equivalence testing show that generated MCQs are not fully comparable to expert‑vetted baseline questions, with persistent gaps in skill depth, cognitive engagement, difficulty calibration, and metadata alignment. "whyItMatters":"The findings highlight how AI‑orchestrated workflows shift researcher labor toward specification, orchestration, verification, and governance, underscoring emerging AI research operations skills needed for AI‑empowered research pipelines."
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.