The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.
By Tapan Parikh
arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.
By Yongxi Zhou, Junwei Yao, Yuanzhe Liu, Zihan Dong, Wenbo Ye, Jiaxi Wen, Lai Yun Choi
arXiv:2607. 10202v2 Announce Type: replace Abstract: Forced-choice probes with counterbalanced orientations are a standard tool for measuring language-model "value dispositions," and a concentration/extremity index over repeated draws is read as how sharply a model commits.
By Hong-In Won, Jinseok Jang, Hyoseop Kim
arXiv:2609.13936v1 Announce Type: new
Abstract: Datasets that ship automatically generated feature annotations invite a question rarely asked of them: would a human agree with those labels? This repo...
By Levent Bulut
The paper introduces DriftBench, a benchmark that tests whether large language models (LLMs) maintain hard constraints while refining scientific ideas over multiple turns. Using 38 research briefs across 24 domains, the study evaluates seven models from five providers on 2,116 runs, finding that iterative pressure increases proposal complexity and reduces constraint adherence, with a high rate of ‘knows‑but‑violates’ instances. The authors also compare judge stability, re‑elicitation effects, and human versus LLM judgments, and release all benchmark materials.
By Garvin Kruthof
arXiv:2607. 18310v1 Announce Type: cross Abstract: Synthetic-population tools increasingly run every individual as an independent large language model (LLM) agent.
By Gurkan Ozkan
arXiv:2608. 06953v1 Announce Type: cross Abstract: Agent memory systems compress what they store, and compression is built to drop qualifiers, so a claim's epistemic standing tends not to survive being written to memory.
By Alex Kwon
The paper investigates how large language model (LLM) agents decide whether to act on user requests when their tool outputs may be ambiguous. Using presupposition theory, the authors show that agents often act without confirmation when a lookup cannot distinguish between one or two records, and that the way search results are presented (listing, counting, or omitting) significantly affects this behavior. Experiments with Claude and GPT models reveal that counting a second record prevents action, while listing both records or user-provided claims can reduce erroneous actions.
By Yue Zhao, Angela Huyue Zhang, S. Alex Yang, Hua Wei, Xiyang Hu
The study investigates how task difficulty, model type, and user pressure influence large language models’ tendency to abandon correct answers or endorse user positions—a phenomenon known as sycophancy. Using 103,939 graded replies across ten configurations of eight LLMs (with and without reasoning) and 13 pressure conditions, the authors find that the cost of verifying a claim and the presence of a guardrail are the dominant factors, while model family and pressure tactics play minor roles. Key practical insights include simplifying hard-to-verify problems, employing deep reasoning, framing questions neutrally, and selecting models based on guardrail performance.
By Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
arXiv:2610.15938v1 Announce Type: new
Abstract: Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown re...
By Nitish Kumar, Prateik Pothuneedi, Greg Bandak
The paper reports on a deployed multi‑agent tender‑response system that uses an open‑weights language model under sovereignty constraints. In a blind comparison, the system’s answers were judged at least as good as human‑written bids in 40 of 55 sections, with only a few gaps attributable to missing knowledge rather than writing quality. The study also demonstrates an asymmetry in conditioning: while structural markup improves reading tasks, converting instruction material from prose to nested XML degrades answer quality, and naming forbidden constructions concentrates defects.
By Cheng Yu, Nikhil Mathew, Zhengjie Wang
arXiv:2607. 10202v1 Announce Type: new Abstract: Cross-model comparisons read divergence in value dispositions as evidence that language models hold individuated values.
By Hong-In Won, Jinseok Jang, Hyoseop Kim