Evaluating Multiple LLM Generations with Validated Task Coverage

The paper introduces VTC-Bench, a five‑domain benchmark designed to evaluate multiple outputs from large language models (LLMs) by measuring Validated Task Coverage (VTC). VTC quantifies how many distinct, useful results are produced within a set number of attempts, using real‑data tasks that allow automatic, reproducible checks of output quality and task‑relevant distinctness without relying on model‑based judges. Experiments show that models which perform best on single‑draw quality do not always achieve the highest coverage, and simple output‑variation metrics fail to capture task‑relevant diversity, highlighting the importance of evaluating finite candidate sets directly.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• benchmarks