The study investigates whether large language models (LLMs) have improved in screening systematic reviews (SRs) within software engineering. Using the SESR‑Eval benchmark and a new smaller dataset, the authors evaluated eight recent LLMs and found only marginal performance gains over older models (average MCC rose from 0.347 to 0.365). The research also examined prompt variations, inter‑model agreement, and the impact of refining inclusion/exclusion criteria, concluding that LLMs still fall short of replacing human reviewers and that future work should explore agent‑based methods, prompt engineering, and further criteria refinement.
By Aleksi Huotala, Miikka Kuutila, Mika M\"antyl\"a
The paper introduces a benchmark of 400 synthetic healthcare workflow episodes to evaluate large language model (LLM) agents’ reliability in escalating care when evidence is missing or conflicting. It finds that while clean decision accuracy is high (0.97), models miss a large proportion of necessary escalations (32–97%) under uncertainty, and even the largest 9.7B model’s performance drops to 0.78 in conflict scenarios. A workflow-aware controller improves utility and reduces missed escalations, but its effectiveness is limited by lexical generalization failures.
By Zihao Zheng (Washington University in St. Louis), Junyi Yao (Washington University in St. Louis), Baichuan Li (Southern Methodist University), Jiayu Long (Washington University in St. Louis)
The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.
By Mohamed Abouzahra
The paper investigates how personalized agents decide to use, ignore, update, or query retrieved user memory before acting on a task. An empirical audit protocol is developed to test structured intermediate outputs, revealing that while exposing state definitions improves accuracy, an explicit state-output field does not significantly enhance policy accuracy for large language models. The study also shows that example-level accuracy overstates consistency, with full four‑way family success being rare, and that providing benchmark‑associated state labels merely conditions predictions rather than proving internal fidelity.
By Yihang Chen, Pin Qian, Su Wang, Chong Peng, Huan Xu, Shuaiting Li, Yiqi Sun
The study evaluates how to reduce fabricated claims in multi‑stage large language model (LLM) hiring pipelines. Prompt guardrails alone cut fabrication density by 86 % but still left half of outputs containing false claims, while adding a human‑in‑the‑loop checkpoint after resume improvement eliminated all identity fabrications and significantly lowered overall fabrication rates. The results show that a layered approach—combining prompt guardrails with human checkpoints—provides stronger protection against severe failures without harming the quality of the final outputs.
By Hiroko Takano
TWIST is a new benchmark suite designed to evaluate the quality of interventions in conversational memory systems, focusing on how well these systems act correctly at belief change points. It includes four tracks—tension detection, draft vetting, belief-consistent answering, and sensitive recall governance—each paired with hard-negative controls to prevent gaming. The benchmark is rigorously validated through double annotation, adjudication, and a separability audit, revealing that current models face trade-offs between contradiction recall and false-positive rates.
By Subrat Panda