arXiv AI

Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs

arXiv:2606. 01400v1 Announce Type: cross Abstract: Evaluating large language models (LLMs) across comprehensive benchmarks is expensive and time-consuming.

arXiv AI

Jul 14

Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

arXiv:2607. 09739v1 Announce Type: new Abstract: We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite.

By Jihan Yao, Gantavya Bhatt, Arnav Das, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhardwaj, Chang Su, Jialei Wang, Yikai Zhu, Sugam Devare, Damon Mosk-Aoyama, Zhen Dong, Venkat Krishna Srinivasan, Yineng Zhang, Oleksii Kuchaiev, Jiantao Jiao, Banghua Zhu, Jeff Bilmes

llms rag benchmarks

arXiv Machine Learning

Aug 7

Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings

arXiv:2608. 05724v1 Announce Type: cross Abstract: Sparse word embedding pipelines can avoid dense co-occurrence matrix materialization, dense factorization, and gradient training while still relying on sparse global corpus statistics.

By Sriram Loganathan, Gokul Anand, Aung Bo Bo, Yourui Shao, William B. Andreopoulos

rag nlp efficiency

arXiv Machine Learning

Jun 5

The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models

arXiv:2606. 05169v1 Announce Type: new Abstract: We give a stereological theory of LLM benchmark coverage.

By Jason Z Wang

llms benchmarks

arXiv Machine Learning

Jun 30

Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation

arXiv:2606. 29947v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes.

By Zhe Dong (University of Maine at Presque Isle), Fang Qin (Stanford University), Manish Shah (Independent Researcher), Yicheng Wang (Independent Researcher)

llms rag benchmarks

Hugging Face Trending Papers

Aug 6

Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings

Sparse word embedding pipelines can avoid dense co-occurrence matrix materialization, dense factorization, and gradient training while still relying on sparse global corpus statistics. This paper studies Random Indexing (RI) vectors refined by weighted averaging on a sparse Positive Pointwise Mutual Information (PPMI) graph.

rag nlp efficiency

arXiv AI

Jun 26

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.

By Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fatimah Oozeer, Ant\'ia Garc\'ia, Philip Quirke, Amirali Abdullah, Fazl Barez, Shriyash Kaustubh Upadhyay

llms agents benchmarks

arXiv Machine Learning

Jun 3

Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings

arXiv:2606. 03365v1 Announce Type: new Abstract: Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs.

By Guillaume M\'erou\'e, Fabien Gandon, Pierre Monnin

rag benchmarks

arXiv Machine Learning

Jun 29

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.

By Rostislav Gusev, Alexey Zaytsev

nlp benchmarks

arXiv Machine Learning

Aug 11

On the Effect of Sampling Diversity in Scaling LLM Inference

arXiv:2502. 11027v5 Announce Type: replace Abstract: Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhance it.

By Tianchun Wang, Zichuan Liu, Yuanzhou Chen, Jonathan Light, Weiyang Liu, Haifeng Chen, Xiang Zhang, Wei Cheng

llms

arXiv Machine Learning

Jun 11

GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs

arXiv:2606. 11562v1 Announce Type: new Abstract: Graph analysis underlies many applications whose answers cannot be looked up in a single record or retrieved along a path: laundering rings, drug repurposing, user preference, and scientific theme are all inferred from a node together with its neighbourhood.

By Zhuoyi Peng, Jingzhou Jiang, Hanlin Gu, Lixin Fan, Yi Yang

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

Covering the Unseen: Information Demand Coverage Optimization for Retrieval-Augmented Generation

arXiv:2606. 29328v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) typically treats context selection as ranking chunks against a single query embedding.

By Bingxue Zhang, Jianying Jia, Feida Zhu

rag multimodal benchmarks

arXiv Machine Learning

Jul 20

VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.

By Shahrzad Esmat, Dhawal Shah, Ali Jannesari

llms efficiency benchmarks

Related stories