arXiv AI

Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

arXiv:2607. 00048v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed.

arXiv AI

Jul 2

Prompting GPT-5 on Scrum Certification Questions: An Empirical Accuracy Study

arXiv:2607. 00049v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in Agile Software Development for documentation, coaching, and training.

By Mirko Perkusich, Danyllo Albuquerque, Jo\~ao Paiva, Robson Vilar, Emanuel Dantas, Ademar Fran\c{c}a de Sousa Neto, Rohit Gheyi, Kyller Gorg\^onio, Angelo Perkusich

llms

arXiv AI

Jul 17

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.

By Inder Preet, Shuxin Lin, Dhaval Patel

llms nlp benchmarks

arXiv AI

Jul 3

Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

arXiv:2607. 02432v1 Announce Type: new Abstract: Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation.

By Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez, Lorena Otero-Cerdeira

llms robotics safety

arXiv AI

Jun 17

Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

arXiv:2606. 17588v1 Announce Type: cross Abstract: Several studies have examined the use of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), reporting mixed accuracy.

By Mika M\"antyl\"a, Patricia Matsubara, Katia Romero Felizardo, Miikka Kuutila, Marco Gerosa, Savio de Sousa Sampaio, Tayana Conte, Igor Steinmacher

llms

arXiv Machine Learning

Jun 8

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.

By Tao Liu, Ye Lu, Ruohua Zhang, Siyu Song, Wentao Liu, Aimin Zhou, Hao Hao

llms agents benchmarks safety

arXiv AI

Jul 3

LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability

arXiv:2607. 01247v1 Announce Type: cross Abstract: Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps.

By Aastha Sapkota, M. G. Sarwar Murshed

llms

arXiv AI

Aug 11

WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management

arXiv:2608. 07529v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as technical assistants, but their competence in solid waste management (SWM) remains difficult to assess because existing benchmarks emphasize general knowledge rather than professional decisions under engineering, environmental, and policy constraints.

By Yi Zhang, Hongyang Wang, Zheng Hao Leong, Zihao Wu, Kaijun Lin, Zhixing Pan, Qixun Huangfu, Wei Ren, Wenyan Wu, Fangyun Wang, Wenting Yu, Hengyu Lin, Muling Yang, Zongguo Wen

llms benchmarks

arXiv AI

Jul 10

InvestPhilBench: A Multi-Layer Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

arXiv:2606. 25984v2 Announce Type: replace Abstract: Large language models are increasingly deployed as investment research assistants, yet no benchmark tests whether they can accurately reconstruct and apply the specific procedural decision frameworks of expert investors.

By Mingguang Chen, Bo Qu

llms benchmarks

arXiv AI

Jul 17

Harnessing LLMs for Reliable Academic Supervision: A Comparative Study

arXiv:2607. 14707v1 Announce Type: cross Abstract: Large language models routinely produce fluent answers to single-shot prompts, yet deploying them as reliable components of a domain decision system is substantially harder.

By Akash Raj

llms benchmarks

arXiv AI

Jul 21

When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering

arXiv:2607. 17063v1 Announce Type: new Abstract: The rapid advancement of large language models (LLMs) has led practitioners to increasingly rely on them for answering questions about hardware description languages (HDLs).

By Ziteng Hu, Jiachi Chen, Wenhao Lv, Huan Zhang, Yingjie Xia

llms agents nlp

arXiv AI

Jul 22

Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

arXiv:2607. 18725v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult.

By Shaswata Mitra, Subash Neupane, Trisha Chakraborty, Himanshu Tripathi, Sudip Mittal, Aritran Piplai, Shahram Rahimi

llms fine-tuning safety

arXiv AI

Jul 20

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

arXiv:2607. 16122v1 Announce Type: new Abstract: Evaluations should do more than measure a models current performance.

By Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

llms fine-tuning benchmarks

Related stories