arXiv Computation and Language

Forty Questions: Identifying a Language Model from One-Word Answers

The study demonstrates that a language model can be identified from its one‑word responses to simple prompts, achieving 92.5% accuracy on a panel of 94 models versus 1.1% by chance. Using profiles built via OpenRouter, all ten DeepInfra models were correctly matched, and 92 of 96 OpenRouter endpoints matched their claimed identities. The method also revealed billing anomalies and mislabeling, such as a Gemma 4 model billed as Llama 4 Maverick and discrepancies in token usage for Grok 4.7.

arXiv AI
Oct 16

System1-Arena: Which Decisions Can Be Left to System 1?

System1-Arena is a benchmark that evaluates which decisions can be handled by System‑1 models—those that output probabilities in a single forward pass—versus System‑2 models that generate reasoning steps. It contains 92,000 questions across 105 tasks from 268 public sources, grouped into text classification, applied decisions, evaluation and control, and agent/action, and also includes 138 closed‑loop environments such as web shopping, coding control, robotics, games, and driving. The study tests 30 models, showing that large System‑1 models can match or exceed fast closed LLMs in non‑agentic accuracy with better‑calibrated probabilities at lower latency, while reasoning LLMs are most accurate but far more expensive, and large open LLMs perform best in closed‑loop settings.

By Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou
arXiv AI
Oct 12

The One-Word Census: Answer-Choice Conformity Across 105 Language Models

The study examines how 105 language models choose a single word from a large set of valid options, finding that 46% of the time they all pick the same word, "serendipity." Using 96 prompts that ask for one-word answers across various categories, the researchers measure convergence and each model’s share of the consensus. They observe that post‑training and persona‑tuned models diverge more, while heavily post‑trained assistants are more conformist, and that the field’s answer choices are more concentrated than human norms in most categories.

By Tapan Parikh
arXiv Machine Learning
Nov 2

From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings

The paper introduces BULLPEN, a framework that builds compact embeddings of large language models (LLMs) by jointly analyzing model answers, correctness, and question text. BULLPEN expands on prior work by adding 20 new benchmarks focused on behavior, style, and provenance, and demonstrates that embeddings derived from only 400 questions retain most of the predictive power of full‑pool methods. Across 260 open‑weight models, BULLPEN’s answer‑text encoder achieves higher accuracy on new tasks and preserves over 91% of full‑pool skill with minimal inference cost.

By Russell Ro, Qiming Cui, Reza Abbasi-Asl
arXiv Machine Learning
Jul 14

One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions

arXiv:2607. 10252v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly consumed through opaque serving chains - API aggregators, resellers, and inference providers - in which the client has no technical means to confirm that the model answering is the model advertised, and recent audits show that a substantial fraction of commercial endpoints deviate from the vendor's reference weights.

By Tomas Bruckner
arXiv AI
Sep 25

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

The paper shows that chain‑of‑thought (CoT) instructions can distort evaluation of vision‑language models (VLMs) when a scorer reads answer‑label logits before the model generates a rationale. On ScienceQA, Qwen2.5‑VL‑7B’s accuracy falls from 80.76% to 45.48% under this CoT‑prefix scoring, and most predictions incorrectly pick the first option. Linear probes and free generation recover most of the lost accuracy, indicating that the answer information remains in the hidden states but is missed by the early readout. The authors explain the mismatch with vocabulary and layer diagnostics, noting that probability mass shifts toward continuation tokens while answer information stays linearly accessible in later layers. The effect varies across datasets and models, but the study demonstrates that CoT‑prefix scoring can misrepresent model knowledge unless the requested and scored outputs are aligned.

By Zeyan Li, Siyuan Qiu, Jianfeng Xu
arXiv AI
Sep 25

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.

By Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Alexander Evseev, Sergey Bolovtsov
arXiv AI
Sep 25

Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.

By Tapan Parikh