arXiv Computation and Language
2d ago
Where a Model Sends Its Own Repeated Token
arXiv:2609. 31181v1 Announce Type: new Abstract: Black-box model identification works by scoring a model's response to natural-language prompts.
Same Quantity, Different Answer: Numerical Representation Invariance in Language Models
The paper introduces a benchmark of 3,600 exact‑rational word problems and 8,600 prompts that test whether language models give the same canonical answer when the same quantity is expressed in different numeric forms (decimal, fraction, percentage, number word, scientific notation, or unit‑converted). After normalizing answer syntax, canonical accuracy is high (0.969–0.996), but correctness across equivalent representations drops to 0.848–0.981, revealing that many errors stem from the evaluator’s number grammar rather than the models’ reasoning. The study also finds that representation consensus does not outperform paraphrase consensus on low‑error subsets and that certain models (e.g., Mistral Small 4) exhibit systematic unit‑conversion errors.
Related stories
arXiv Computation and Language
Sep 22
Euston: Training Away Mathematical Sycophancy Without Losing the Mathematics
Euston is an 8‑B parameter mathematical claim‑verification model that resists producing false derivations when presented with corrupted theorems. It was trained on 3,026 matched true/corrupted statement pairs generated by GraphSynth, a probabilistic factor‑graph generator, and fine‑tuned from DeepSeek‑R1‑8B using GRPO. On a balanced held‑out split, Euston’s balanced accuracy rose from 29.50 % to 63.75 %, and its discrimination gap improved from –0.5 % to +27.5 %, while maintaining comparable general mathematical ability and reducing response length and truncation rates.
arXiv Machine Learning
Sep 2
Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR
arXiv:2609.01354v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text ans...
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv AI
Aug 13
Moxia: A Trust-First Neuro-Symbolic Execution Architecture for Self-Explaining Mathematical Reasoning
arXiv:2606. 00671v3 Announce Type: replace Abstract: We present Moxia (formerly AXIOM), a trust-first neuro-symbolic architecture for self-explaining mathematical reasoning over natural-language input.
arXiv Machine Learning
Jul 14
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
arXiv AI
Aug 12
AXIOM: A Trust-First Neuro-Symbolic Execution Architecture for Self-Explaining Mathematical Reasoning
arXiv:2606. 00671v2 Announce Type: replace Abstract: We present AXIOM, a trust-first neuro-symbolic architecture for natural-language mathematical reasoning.
arXiv Machine Learning
Aug 19
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
The paper evaluates how three large mixture‑of‑experts models (Alibaba, OpenAI, NVIDIA) can be fine‑tuned to reason in a low‑resource language, specifically Greek. Accuracy metrics show little change, but the authors uncover significant qualitative improvements: after supervised fine‑tuning, models reason in Greek on ~98% of items, with better grammaticality and retained general ability. Reinforcement learning with pre‑registered rewards further eliminates reasoning‑channel leaks and format skips, while the Greek‑reasoning habit remains robust to an accuracy‑only gradient.
arXiv AI
Aug 5
Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity
arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.
arXiv AI
Sep 1
SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization
arXiv:2608.29270v1 Announce Type: cross Abstract: Autoformalization translates informal mathematical theorems into code for proof assistants such as Lean. A central challenge is that current evaluati...
arXiv AI
Jun 3
Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks
arXiv:2606. 03606v1 Announce Type: cross Abstract: Large language models achieve strong performance on arithmetic reasoning benchmarks, and one common response to arithmetic brittleness is to delegate computation to code.
arXiv AI
Sep 1
Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification
arXiv:2608.28725v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final...