arXiv Machine Learning
Identifying Intelligent Processes via Online Sequential Testing
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 26
Evaluating Multiple LLM Generations with Validated Task Coverage
The paper introduces VTC-Bench, a five‑domain benchmark designed to evaluate multiple outputs from large language models (LLMs) by measuring Validated Task Coverage (VTC). VTC quantifies how many distinct, useful results are produced within a set number of attempts, using real‑data tasks that allow automatic, reproducible checks of output quality and task‑relevant distinctness without relying on model‑based judges. Experiments show that models which perform best on single‑draw quality do not always achieve the highest coverage, and simple output‑variation metrics fail to capture task‑relevant diversity, highlighting the importance of evaluating finite candidate sets directly.
arXiv Machine Learning
Jul 2
Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
arXiv:2509. 17314v4 Announce Type: replace-cross Abstract: Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation.
arXiv AI
Sep 28
Cost-Aware Best-LLM Identification using Dueling Feedback
The paper introduces a new variant of the multi‑armed bandit problem that incorporates dueling feedback—pairwise comparisons of model responses—and heterogeneous sampling costs to identify the best large language model (LLM) from a set with varying query costs. Assuming a Condorcet winner, the authors propose a Track‑and‑Stop style algorithm that guarantees asymptotically optimal cost as the error probability approaches zero. Extensive experiments on synthetic and real‑world data show that this cost‑aware approach consistently outperforms both classical cost‑unaware algorithms and other cost‑aware extensions.
arXiv Machine Learning
Sep 17
Long-Context Demonstration Selection Using State Space Models
The paper addresses the challenge of selecting demonstrations for long-context language model queries, where transformer inference costs grow quadratically with sequence length. It proposes two algorithms that distill transformer behavior into state space models (SSMs) with linear inference time, partitioning transformer layers into groups and estimating separate SSMs for each. The distilled SSMs achieve less than 0.7% approximation error, and in downstream tasks they reduce FLOPs by 14.2× while improving accuracy by 6.48% compared to baseline methods.
arXiv Machine Learning
Oct 9
Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs
The study investigates how many synthetic conversation samples are required to train activation probes that monitor language models for high‑stakes, harmful, and instruction‑following behaviors. By tracking learning curves across 10–590 samples on various probe models, generators, and prompts, the authors find that the number of samples needed is largely determined by coverage of the monitored concept rather than the difficulty of individual sample types. They show that high‑stakes and harmful probes reach near‑plateau performance with as few as 80 samples, while instruction probes require several times more, and that breadth of sample kinds is more critical for instruction monitoring than for high‑stakes scenarios. The paper also releases evaluation suites, dev sets, and generated data for further research.
arXiv AI
Jun 9
More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
arXiv:2601. 21522v2 Announce Type: replace-cross Abstract: The performance of large language models (LLMs) on verifiable tasks is usually measured by pass@k, the probability of answering a question correctly at least once in k trials.