The paper investigates how large language models (LLMs) can perform multi-coder qualitative coding by independently coding, debating, and reconciling disagreements. It quantifies the effectiveness of this approach across diverse datasets, identifying key factors—such as codebook length, data similarity, and agent disagreement—that influence coding accuracy. The study finds that intense, unresolved debates improve accuracy but that LLMs still lack adaptive responsiveness to context, leading to design recommendations for automated coding systems.
arXiv:2607. 28889v1 Announce Type: cross Abstract: Qualitative researchers increasingly encounter interaction corpora whose scale exceeds what manual coding alone can address, and large language models (LLMs) are frequently proposed as analytic assistants.
The paper introduces Agent-as-Peer-Debriefing, a multi‑agent framework that incorporates peer debriefing into qualitative data analysis with large language models. A Hierarchical Coding Agent generates codes and reflections, which are then refined by three Peer‑Debriefing Agents applying Theory‑Driven, Data‑Driven, or Applied perspectives. Experiments on three datasets show that perspective‑based refinement aligns more closely with human codes than a single‑LLM baseline, and that the choice of perspective offers meaningful trade‑offs.
The paper examines the mandatory Limitations sections that have appeared in many top-tier NLP conferences since late 2022. By applying a novel human‑AI hybrid qualitative coding framework, the authors analyze Limitations sections from ACL and EMNLP papers published between 2020 and 2025 to uncover trends over time, correlations with paper attributes, and recurring writing patterns. The study provides a critical reflection on the diverse challenges researchers report and on their self‑reporting practices within the NLP community.
arXiv:2411.08881v3 Announce Type: replace-cross Abstract: AI-based systems, including Large Language Models (LLMs), impact millions by supporting diverse tasks but face issues like misinformation, bi...
The paper titled "Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025" examines how experienced developers employ AI agents in software development. Through field observations and surveys, it finds that developers value agents for productivity but maintain control over design and implementation to ensure quality. They use agents as collaborative tools rather than full delegation, selecting tasks based on suitability and leveraging their expertise to guide agent behavior.
arXiv:2507.21831v2 Announce Type: replace-cross Abstract: LLMs are seeing widespread use for task automation, including automated coding in the social sciences. However, even though researchers have...
QuAD is a framework that quantitatively assesses the quality of qualitative response datasets before analysis. It compares an intrinsic cluster distribution derived from the responses with an extrinsic, researcher-defined analytical lens based on metadata, using a four‑quadrant assessment that considers lens‑level relations and within‑cluster variation. The method employs a content‑aware multi‑agent system with two‑stage instruction generation and adaptive anchors to improve scoring consistency, and is validated through synthetic experiments and a real‑world social survey case study.
arXiv:2608. 12355v1 Announce Type: cross Abstract: Recent progress in AI coding agent research has led to rapid improvements in agents' ability to autonomously perform complex software engineering tasks, from editing large codebases to executing long-horizon development workflows.
arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.
arXiv:2607. 21482v1 Announce Type: new Abstract: Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models.
arXiv:2606. 06462v1 Announce Type: new Abstract: Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance.
arXiv:2610.33205v1 Announce Type: new Abstract: Current uses of large language models (LLMs) for qualitative coding often rely on ad hoc prompting with limited guidance for reliability and human over...