arXiv:2606. 10287v1 Announce Type: new Abstract: Evaluating Knowledge Graph Completion (KGC) models remains challenging because standard assessment relies on isolated rank-based metrics such as MRR, Hits$@$k, and Mean Rank, which often produce conflicting model orderings across datasets.
arXiv:2606. 08921v1 Announce Type: new Abstract: Knowledge graph completion (KGC) aims to predict missing facts from an observed knowledge graph (KG), playing a crucial role in a wide range of real-world applications such as drug discovery, recommender systems, and retrieval-augmented generation (RAG).
arXiv:2606. 08926v1 Announce Type: new Abstract: Knowledge graph completion (KGC) models are commonly evaluated using rank-based metrics such as MRR and Hits@K, despite different users often requiring different evaluation perspectives.
The paper examines whether economic benchmarks used in frontier AI leaderboards measure a distinct capability or merely reflect general test-taking ability. Using a structural factor analysis and a predictive leave-one-benchmark-out test on a snapshot of 421 model configurations, the authors find that economic benchmarks do not form a separate factor but are better predicted by a multi‑factor representation than by a single general index, especially for linear learners. They argue that construct validity should be evaluated with both structural and predictive tests and provide a two‑test protocol along with data and code.
arXiv:2604. 24827v2 Announce Type: replace-cross Abstract: Closed-source frontier labs do not disclose parameter counts.
arXiv:2607. 09739v1 Announce Type: new Abstract: We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite.
arXiv:2606. 10358v1 Announce Type: cross Abstract: Learning Bayesian network (BN) structure from sparse discrete data is hard: when each instance records only a few variables, most variable pairs lack the joint observations needed for reliable scoring, and data-only methods recover little structure.
The paper audits the reproducibility of knowledge‑graph extraction from threat reports by re‑implementing matching rules for only five of twelve systems and re‑scoring ten system outputs under eight protocols. The audit shows that different matching protocols can reverse most pairwise system rankings and that a fixed prediction set can vary from 0.16 to 0.70 F1. The authors also build CTIForge to isolate validation effects, finding that validation changes precision across backbones and increases entity‑type disputes, and they release the full pipeline, protocol suite, and audit records.
arXiv:2607. 10212v1 Announce Type: new Abstract: Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance.
The paper argues that typical tabular machine learning benchmarks, which aggregate results by averaging scores or ranks, can hide which models are essential for achieving the best performance on specific datasets. It proposes evaluating models against a data‑centric peak performance frontier, classifying them as irreplaceable, sufficient, redundant, or fallible based on their position relative to other models. Applying this to the TabArena benchmark shows that common aggregation metrics mainly capture consistency and failure avoidance, but fail to reflect dataset‑specific strengths, leading to a misalignment between aggregate rewards and true model utility.
arXiv:2606. 03365v1 Announce Type: new Abstract: Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs.
Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs.
arXiv:2608. 08069v1 Announce Type: cross Abstract: Foundation models are increasingly reused as software components, making model selection a critical software-engineering decision.