arXiv AI
Jun 26
Life After Benchmark Saturation: A Case Study of CORE-Bench
arXiv:2606. 26158v1 Announce Type: new Abstract: When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.
From Checklists to Clusters: A Homeostatic Account of AGI Evaluation
arXiv:2510. 15236v2 Announce Type: replace Abstract: Contemporary AGI evaluations report multidomain capability profiles, yet they typically assign symmetric weights and rely on snapshot scores.
Related stories
arXiv AI
Jul 15
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
arXiv:2607. 12790v1 Announce Type: new Abstract: Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists.
arXiv AI
Jul 1
Toward AI-Resilient Assessment in Computer Science Courses in an AI-Native World
arXiv:2606. 30655v1 Announce Type: cross Abstract: AI-native course assessments in senior computer science courses and related fields should grade students by \emph{AI-resilient skill}: the ability to achieve outcomes beyond a strong AI baseline.
arXiv Machine Learning
Jul 15
Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection
arXiv:2607. 11969v1 Announce Type: cross Abstract: Point-adjustment (PA), long the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al.
arXiv AI
Aug 11
Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
arXiv:2603. 00077v3 Announce Type: replace-cross Abstract: Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks.
arXiv AI
Jun 2
SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems
arXiv:2606. 01314v1 Announce Type: new Abstract: Recent self-evolving agents have shown that skills can be discovered, refined, and accumulated through execution.
arXiv Machine Learning
Jun 10
FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses
arXiv:2606. 09878v1 Announce Type: new Abstract: Standard benchmarks report aggregate accuracy, but practitioners need to know which specific capabilities a model lacks.
Hugging Face Trending Papers
Jul 14
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not.
arXiv AI
Jun 16
Not All Skills Help: Measuring and Repairing Agent Knowledge
arXiv:2606. 15390v1 Announce Type: cross Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone.
arXiv Machine Learning
Jul 23
Building Fast, Evaluating Slow: Pipeline Choices Dominate Autointerpretability Score Variance
arXiv:2607. 19386v1 Announce Type: new Abstract: Cross-paper comparison of sparse autoencoder (SAE) interpretability often relies on autointerpretability scores.
arXiv AI
Jun 10
Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm
arXiv:2605. 27914v2 Announce Type: replace-cross Abstract: Benchmarking is mature where answers are verifiable -- math, code, reasoning -- but the fastest-growing uses of LLMs are subjective and human-facing: companionship, emotional support, counseling.
arXiv AI
3d ago
Demystifying Agent Skills: Why They Work-Until They Don't
arXiv:2608. 14036v1 Announce Type: new Abstract: Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge.