The paper argues that large language models (LLMs) are evaluated too narrowly, focusing on isolated technical metrics rather than holistic, developmental, and societal aspects. It proposes a diagnostic ontology that links evaluation dimensions to the LLM training pipeline, turning evaluation into a root‑cause analysis tool. The authors introduce an anthropomorphic framework—IQ, PQ, EQ, and VQ—to assess LLM capabilities, operationalize it with a modular architecture, and validate it through meta‑analysis of over 200 benchmarks, outlining key challenges and future directions.
arXiv:2607. 10871v1 Announce Type: new Abstract: Contemplative traditions have long guided ethical behavior and prosocial interaction, and recent work suggests that contemplative principles (e.
MAVEN is a hierarchical framework for evaluating whether multimodal content aligns with macro‑societal values such as peace, justice, and freedom. It organizes values into six primary dimensions and 72 secondary indicators, enabling multi‑level quantitative scoring. The authors build a human‑verified multimodal benchmark, a soft‑match metric, and propose efficient evaluator optimization techniques, demonstrating that a compact 2B evaluator performs comparably to larger models and approaches state‑of‑the‑art closed‑source VLMs.
arXiv:2607. 16122v1 Announce Type: new Abstract: Evaluations should do more than measure a models current performance.
arXiv:2607. 25891v1 Announce Type: new Abstract: Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules.
arXiv:2607. 14673v1 Announce Type: new Abstract: Evaluations (Evals) are a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale.
arXiv:2602. 07840v3 Announce Type: replace-cross Abstract: Evaluating relevance in large-scale search systems is fundamentally constrained by the governance gap between nuanced, resource-constrained human oversight and the high-throughput requirements of production systems.
arXiv:2603. 23522v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context.
ZenGen is a comprehensive framework designed to develop socially intelligent large language models (LLMs). It introduces SoMBench, a psychology‑grounded benchmark with 3 primary and 17 secondary dimensions, and shows that current LLMs perform poorly on social cognition tasks. ZenGen combines supervised fine‑tuning, on‑policy distillation, and rubric‑based reinforcement learning to improve internalization, while Actio provides deployment‑time grounding with typed supports such as PRISM, Starling, SAGE, and gated RAG, yielding significant performance gains across benchmarks.
arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.
arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...
arXiv:2607. 05638v1 Announce Type: cross Abstract: Teams deploying large language models in business contexts need evaluation systems, yet most treat evaluation as static model selection: run benchmarks, rank models, deploy the winner.
arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.