arXiv AI
Aug 11
JaleesBench: Are AI Assistants Good Spiritual Company?
arXiv:2608. 07508v1 Announce Type: cross Abstract: Large language models are already advisors to millions of people of faith who bring them real decisions.
When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models
arXiv:2608. 12324v1 Announce Type: cross Abstract: People increasingly ask large language models (LLMs) for counsel on questions of faith, doctrine, and pastoral care.
Related stories
arXiv AI
Jun 9
SLMJury: Can Small Language Models Judge as Well as Large Ones?
arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.
arXiv AI
Aug 3
M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities
arXiv:2601. 02854v2 Announce Type: replace Abstract: As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning.
arXiv AI
23h ago
Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models
arXiv:2608. 17183v1 Announce Type: new Abstract: Small Language Models (SLMs) are increasingly deployed in resource-constrained, privacy-sensitive settings, where safety and bias failures can cause security and societal risks.
arXiv AI
Jul 7
The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Models
arXiv:2607. 03953v1 Announce Type: cross Abstract: This study independently replicates and extends the Natural Language Tools (NLT) framework of Johnson et al.
arXiv Machine Learning
Jun 8
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.
arXiv AI
Jul 2
Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns
arXiv:2607. 00048v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed.
arXiv Machine Learning
Jun 10
The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring
arXiv:2606. 10327v1 Announce Type: cross Abstract: Automated Essay Scoring (AES) systems must judge interdependent discourse elements (e.
arXiv Machine Learning
Aug 5
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv AI
23h ago
Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference
arXiv:2606. 07897v2 Announce Type: replace Abstract: Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user.
arXiv AI
Jul 16
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
arXiv:2603. 00546v2 Announce Type: replace Abstract: Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains.
arXiv AI
Jun 9
The AI Epistemic Deference Index: A Continuous Measure of Sycophancy
arXiv:2606. 07897v1 Announce Type: new Abstract: Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user.