arXiv AI
Sep 25

Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.

By Tapan Parikh
arXiv Computation and Language
Nov 9

Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation

The paper introduces DriftBench, a benchmark that tests whether large language models (LLMs) maintain hard constraints while refining scientific ideas over multiple turns. Using 38 research briefs across 24 domains, the study evaluates seven models from five providers on 2,116 runs, finding that iterative pressure increases proposal complexity and reduces constraint adherence, with a high rate of ‘knows‑but‑violates’ instances. The authors also compare judge stability, re‑elicitation effects, and human versus LLM judgments, and release all benchmark materials.

By Garvin Kruthof