arXiv Machine Learning
Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 12
The One-Word Census: Answer-Choice Conformity Across 105 Language Models
The study examines how 105 language models choose a single word from a large set of valid options, finding that 46% of the time they all pick the same word, "serendipity." Using 96 prompts that ask for one-word answers across various categories, the researchers measure convergence and each model’s share of the consensus. They observe that post‑training and persona‑tuned models diverge more, while heavily post‑trained assistants are more conformist, and that the field’s answer choices are more concentrated than human norms in most categories.
arXiv AI
Oct 16
Prism-1: Position- and Rival-Independent Scoring for System One Models
Prism-1 introduces Position- and Rival-Independent Scoring for System One models, ensuring that option ordering and set changes do not affect decision probabilities. It achieves this by reading each option in its own block at shared positions, preserving softmax independence across options. Experiments on Qwen3-8B and Gemma 4 12B show Prism-1 maintains baseline accuracy while eliminating sensitivity to shuffling and irrelevant options.
arXiv AI
Aug 18
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
arXiv:2608. 15428v1 Announce Type: cross Abstract: Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question.
Hugging Face Trending Papers
Oct 15
Prism-1: Position- and Rival-Independent Scoring for System One Models
Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in one forward pass, without generated reasoning. We...
arXiv Computation and Language
1d ago
Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals
The paper investigates the reliability of LLM judges in the Arena-Hard benchmark, showing that aggregate scores can mask significant variability in individual verdicts. By running GPT‑4.1 across 500 prompts under eight different judge configurations and repeating each setup, the authors find that small changes—whether in temperature, criteria, or instruction framing—can shift win/tie/loss outcomes on up to 28% of prompts and alter aggregate scores by several points. The study concludes that stable aggregate scores do not guarantee repeatable individual judgments, and recommends that evaluations report both aggregate results and prompt‑level agreement.
arXiv Machine Learning
Jul 14
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
arXiv:2607. 10202v1 Announce Type: new Abstract: Cross-model comparisons read divergence in value dispositions as evidence that language models hold individuated values.