← Back to all news
arXiv AI By Shaoliang Yang, Jun Wang

Calculation and rejection diverge in language models tested at known mechanical limits

Read the original on arXiv AI →

The study examined 12 language models across 4,860 calls to test how they handle engineering calculations at known mechanical limits. Models often reported designs as "solved" even when the requested beam loads or disk speeds exceeded physical collapse or burst thresholds, though they could correctly reject impossible designs when asked only about feasibility. The findings highlight inconsistencies between calculation accuracy, feasibility judgment, and full-answer rejection, suggesting that software relying on model status should separately test these aspects and disclose prompt details.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 4

It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories

The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.

By Yigit Utku Bulut
llms
More like this →
arXiv Computation and Language
Aug 31

Why Didn't It Check? Unsupported Final Claims and Their Repair in Two Tool-Equipped Language Models

The study investigates how language models equipped with tools can still produce unsupported final claims, even when a single tool call could resolve the uncertainty. It defines two metrics—occurrence (how often unsupported claims arise) and conditional repair (how often they are fixed when evidence is provided). Experiments on Qwen3-32B and Gemma 4 show that providing the missing evidence consistently repairs all unsupported claims in the Qwen3-32B setup, while the Gemma 4 model never produced unsupported claims under the tested conditions.

By Justin Bronder
llms
More like this →
arXiv Machine Learning
Oct 28

Models Verify Sufficiency, Not Validity: Usable Falsehoods Evade the Checks that Gaps Trigger

The paper demonstrates that language models often accept and use unsupported premises when answering questions, even though the source text does not provide the necessary information. By manipulating whether an unsupported fact is usable, the authors show that models’ refusal rates drop significantly while their commitment to an answer increases. This behavior is consistent across multiple models and datasets, indicating that models’ verification checks are heavily influenced by the presence of usable material rather than strict source support.

By Naveen Madiraju
benchmarks
More like this →