arXiv Computation and Language By Tapan Parikh

Forty Questions: Identifying a Language Model from One-Word Answers

Read the original on arXiv Computation and Language →

The study demonstrates that a language model can be identified from its one‑word responses to simple prompts, achieving 92.5% accuracy on a panel of 94 models versus 1.1% by chance. Using profiles built via OpenRouter, all ten DeepInfra models were correctly matched, and 92 of 96 OpenRouter endpoints matched their claimed identities. The method also revealed billing anomalies and mislabeling, such as a Gemma 4 model billed as Llama 4 Maverick and discrepancies in token usage for Grok 4.7.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Oct 16

System1-Arena: Which Decisions Can Be Left to System 1?

System1-Arena is a benchmark that evaluates which decisions can be handled by System‑1 models—those that output probabilities in a single forward pass—versus System‑2 models that generate reasoning steps. It contains 92,000 questions across 105 tasks from 268 public sources, grouped into text classification, applied decisions, evaluation and control, and agent/action, and also includes 138 closed‑loop environments such as web shopping, coding control, robotics, games, and driving. The study tests 30 models, showing that large System‑1 models can match or exceed fast closed LLMs in non‑agentic accuracy with better‑calibrated probabilities at lower latency, while reasoning LLMs are most accurate but far more expensive, and large open LLMs perform best in closed‑loop settings.

By Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou
arXiv AI
Oct 12

The One-Word Census: Answer-Choice Conformity Across 105 Language Models

The study examines how 105 language models choose a single word from a large set of valid options, finding that 46% of the time they all pick the same word, "serendipity." Using 96 prompts that ask for one-word answers across various categories, the researchers measure convergence and each model’s share of the consensus. They observe that post‑training and persona‑tuned models diverge more, while heavily post‑trained assistants are more conformist, and that the field’s answer choices are more concentrated than human norms in most categories.

By Tapan Parikh
arXiv Machine Learning
Nov 2

From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings

The paper introduces BULLPEN, a framework that builds compact embeddings of large language models (LLMs) by jointly analyzing model answers, correctness, and question text. BULLPEN expands on prior work by adding 20 new benchmarks focused on behavior, style, and provenance, and demonstrates that embeddings derived from only 400 questions retain most of the predictive power of full‑pool methods. Across 260 open‑weight models, BULLPEN’s answer‑text encoder achieves higher accuracy on new tasks and preserves over 91% of full‑pool skill with minimal inference cost.

By Russell Ro, Qiming Cui, Reza Abbasi-Asl