arXiv:2608. 12324v1 Announce Type: cross Abstract: People increasingly ask large language models (LLMs) for counsel on questions of faith, doctrine, and pastoral care.
arXiv:2609.13634v1 Announce Type: cross Abstract: Do AI assistants help believers reason about moral dilemmas consistently with their faith? We present FaithfulBench, the first benchmark to score AI...
arXiv:2608. 07508v1 Announce Type: cross Abstract: Large language models are already advisors to millions of people of faith who bring them real decisions.
arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faste...
PetQA is a Korean long‑form question‑answering benchmark designed to assess veterinary knowledge and clinical reasoning in large language and vision‑language models. It comprises 10,076 text‑only and 8,751 multimodal QA pairs about dogs and cats, with expert veterinarian answers, and a test split called PetQA‑Bench that includes question type and clinical condition annotations. The study evaluates 18 models across zero‑shot, retrieval‑augmented generation, and supervised fine‑tuning settings using ROUGE, BERTScore, and LLM‑as‑a‑judge metrics, revealing current models’ strengths and limitations and underscoring the need for better adaptation methods for clinically reliable veterinary AI; translated versions in five languages are also provided.
arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.
Large language model assistants are being deployed to clinicians, but their readiness is judged mainly by benchmark scores that are based on examination questions or curated cases. This study analyzed 127,833 queries from 6,342 healthcare professionals across 35 specialties, finding that most requests were for documentation, administration, and knowledge retrieval, with a significant portion of queries unanswerable. When compared to 58 public benchmarks, the median benchmark lacked documentation requests and only shared 31% of the real task mix, indicating that current benchmarks poorly reflect actual clinical use.
ZenGen is a comprehensive framework designed to develop socially intelligent large language models (LLMs). It introduces SoMBench, a psychology‑grounded benchmark with 3 primary and 17 secondary dimensions, and shows that current LLMs perform poorly on social cognition tasks. ZenGen combines supervised fine‑tuning, on‑policy distillation, and rubric‑based reinforcement learning to improve internalization, while Actio provides deployment‑time grounding with typed supports such as PRISM, Starling, SAGE, and gated RAG, yielding significant performance gains across benchmarks.
Ansari is a retrieval‑grounded Islamic AI assistant that has handled over 140,000 conversations in more than 25 languages since June 2023. It uses an agentic retrieval loop where a language model searches authenticated Islamic corpora—including the Qur’an, hadith collections, fiqh encyclopedias, and tafsir sources—and answers only based on retrieved content, providing citations for verification. The paper details Ansari’s architecture, multi‑platform deployment, evaluation results (including top performance on the IslamicMMLU leaderboard and strong resistance to false premises), and lessons for faith‑sensitive LLM deployments.
arXiv:2601. 02854v2 Announce Type: replace Abstract: As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning.
arXiv:2609.37788v1 Announce Type: cross Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, draw...
arXiv:2607. 03953v1 Announce Type: cross Abstract: This study independently replicates and extends the Natural Language Tools (NLT) framework of Johnson et al.
The paper introduces MentorQA, a multilingual dataset and evaluation framework for mentorship-oriented question answering derived from long‑form videos. It contains nearly 9,000 QA pairs across four languages and defines evaluation dimensions such as clarity, alignment, and learning value that extend beyond factual accuracy. Experiments show that Multi‑Agent QA pipelines outperform other architectures, especially on complex topics and low‑resource languages, while automated LLM‑based evaluation shows variable alignment with human judgments.